I have developed a PHP based Multi-Threading crawler for cache warmup that uses cURL. This crawler also supports multiple cache varies. It works like it should and crawls up to 100,000 URLs within 1 hour, but cache varies drives my crazy. All cache varies work if I request URLs in browser with different cookies and they also work if I warmup the cache with cURL requests in shell script. For cache varies I use different and multiple cookies and also different user agents. Everything works fine, but not if I request URLs in PHP cURL requests with this crawler. For debugging I added cache status and cookies in access_log, but everything looked okay. After hours of testing I found out that cookie needs also cookie value in PHP cURL request.
So the question is, is this behaviour caused by PHP cURL or LSWS?
CURLOPT_COOKIE => "cookie_1=;cookie_2=" // doesn't work CURLOPT_COOKIE => "cookie_1=abc;cookie_2=def" // that works