Search results

Why to wait if sales go up? This crawler improves your page speed and with this crawler your sales go up! I swear! ;)

Selfmade Professional :) I work since 10 years with LiteSpeed and already have developed a couple of cache plugins.

This question is answered in chapter above LogRotationSize -> https://docs.litespeedtech.com/lsws/configuration/#apache-style-directives

Do you have any control panel for your server in use?

Good idea, but I already had a lot of discussions with leaders of LiteSpeed, but it seems they have not much interests in business. If they think an alternative crawler might be good, they would have released one already...... I'll send you PM to give you the link to control panel.

@Germont Last weekend I found the time to generate a standalone version of "HyperSpeed Multithreading Cache Warmup Crawler." :) I am just doing some fine tuning, but it is almost done. Do you want to have a look on it?

NO, never ever! But to understand why it doesn't matter you have to know how crawling with cUrl works.

Memcached, Object cache or any other caching doesn't matter for warmup crawling. I noted no difference if it is used or not, because I am not shure if these kind of cache is available in command line interface (CLI). On this shared hosting I have PHP 7.4 and MariaDB, but it seems also not...

@Germont Have a look on attached log file and check the amount of crawled URLs and the total time to crawl 1000 URLs. Result: 1000 URLs within ~1 minute, but it can go faster depending on server ressources. This crawl session ran on a shared hosting. Server load: Low

Which feature do you need? Free Netflix access while crawling, free entry to all soccer (football) games in premier league in your country, a dozen virgins? :) I have integrated all features that are possible with cURL. Only cURL limits what is possible and impossible. Witout cURL no HighSpeed...

@Germont I invested a lot of time to develop this script. This crawler is x-time faster than everything that LiteSpeed has ever published. Before I invest time again to make a standalone version of this crawler, are you ready to pay for it? I will publish it, but not for free!!

In 2013 curl already had commands I use to make requests much faster. Possible, but this script (no plugin) is part of an application and uses sources from this application. I have to extract this sources from application to make a standalone version. It is already on my to do list, but with...

@Germont Preload ByCron crawler script is good, but he overlooks something in curl requests, that causes too much load and traffic and as result of this, requests take too much time. There are also some other deficits....

Why not? I do it and it works as fast as with my dedicated server. :)

I've developed similar working multithreading crawler, but mine is extended and has some more features for user agent, cookies, request headers for webp image, cache-control support and much more. This allows me to crawl up to 100k URLs within 1 hour on a shared hosting.

Check this: https://docs.litespeedtech.com/lsws/configuration/#logrotationsize and this: https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:config:real-time-stats and basically all information about Logging...

There are a lot of ways to warmup the cache, but each method does the same. It does request URLs of your page like LiteSpeed crawler does it. The question is if your provider will check this and disallows each method that crawls your page. The best way would be to kick this provider off. He...

PHP 5! WOW! You know we have year 2022?

https://www.litespeedtech.com/support/forum/threads/please-use-the-official-wordpress-forum.16028/

Is anybody interested in how to configure LSWS to use different cache directory for each (sub) domain?