Using unix domain socket with nginx is not reliable. better not using it.
I think it is related to the kernel, somehow, your Linux kernel scheduler assign most process to one CPU when persistent connection is used. Please check the CPU idle time during the test.
You can try increase "max...