How to set Connection Timeout in .htaccess or php.ini?

[serpent_driver]

great. I think this is the right place/method for your crawler to run.

Sorry, this is not great. Offering a CLI variant of crawler is an option, but not the one and only option. This option is intended to run in cronjob, but is only a small part of complex application with a lot of features and functions.

in general, web server is not suitable for long time running task/script. better to use command line if possible in such case.

This comment is like a knockout for LiteSpeed Enterprise webserver and stands completely against all what LSWS makes valuable. If this is a official statement you better should remove a lot of descriptions in LSWS website and documentations!

If web server is not suitable for long time running task/script what is noconntimeout directive good for?!

for($i=1 to 100){ curl_exec(...); print "finished url[$i]"; flush(); };

My crawler is a multi threading crawler and doesn't work serial. It makes 100 or whatever number of requests parallel. Up to 1 million requests within 1 hour.

 

[serpent_driver]

I just did some further tests and you won't believe it... noconntimeout directive works. It was my fault. A typo in rewrite rule to limit noconnection timeout to the crawler has caused it, so I am very sorry! Ashes on my head!