How to setup LiteMage warmup for big site?

A Magento store is a very large site with 17k pages. You may have setup your LiteMage TTL to 86400, which is 24 hours, and your warmup internal set to 57600, which is 16hours. You may have an issue whereby the entire website is being re-cached every 24 hours. This is resulting in the website running very slowly until the cache warmup is complete. Crawling all 17k pages takes around 6-8 hours to complete. How to solve this problem?

For such as big site, you can adjust your settings based on your situation. How often do you products change? For example, if it does not change much, you can set your public TTL = 604800, which is 1 week, and set up your warm-up interval to 345600, which is 4 days. So every 3 days it will try to recrawl again. You can also enable delta crawler, so when something change, the related URLs are automatically recrawled in 5 minutes.

When you flush all LiteMage cache files, the whole site warm-up will be automatically started. So it's safe to set a long TTL when you have delta crawler enabled.