Need help for install Litemage Crawler for Magento 2

[Toni Moya]

I followed the instructions to be able install in my store Litemage Crawler for Magento 2: https://blog.litespeedtech.com/2018/08/28/cache-crawler-for-prestashop-and-magento-2/ but for some reason it has not been installed correctly.

 

[Pong]

Can you provide more details what you did and what errors you experienced?
https://www.litespeedtech.com/support/wiki/doku.php/litespeed_wiki:cache:litemage2:crawler
Anything not clear from the wiki?

 

[Toni Moya]

I´ve followed all the steps for the installation and is currently active, but there is no increase in the cached pages, the vast majority of pages respond with a: x-litespeed-cache: miss

 

[Pong]

Please double check if the page URLs are in https://yourmain.com/sitemap.xml ?

 

[Toni Moya]

Yes, you can check: https://www.salelsol.eu/sitemap.xml

 

[Pong]

what exact command did you run? Can you give an example URL which doesn't warm up?
seem you site map url should be https://www.salelsol.eu/sitemap_001.xml
sh M2-crawler.sh https://www.salelsol.eu/sitemap_001.xml

 

[Unique_Eric]

Hi Toni,
The script seems not crawl your sitemap, I will test it on my local site and update here shortly
Best,

 

[Unique_Eric]

Hi Toni,
I found out why your sitemap fail to crawl, because your sitemap has no line change. Not sure if it being minified by some optimize function?
Could you tell us how your get this sitemap? and try not optimize this sitemap see if it works.
Best,
Eric

 

[Toni Moya]

- Hi Eric, many tanks for your help. Sitemap is created by an Mageworx app named: SEO Suite Ultimate Extension for Magento 2 what includes XML Sitemap.
- I'm investigating what the problem may be.
Best Regards
Toni

 

[Unique_Eric]

Hi @Tony

No problem.

Method 1: Generate sitemap by Magento2 build-in tool instead of seo (https://www.litespeedtech.com/suppo...e:litemage2:crawler#how_to_generate_a_sitemap)
Method 2: Create line change on each url result from script for seo sitemap

May I ask this is a popular plugin for Magento 2 user https://www.mageworx.com/magento-2-seo-extension.html?
Best,

 

[Toni Moya]

- Hi Eirc, just created all new sitemap.xml by Magento2 build-in tool, please check now.

Best Regards
Toni

 

[Unique_Eric]

Checked, it's working.
./M2-crawler.sh [URL]https://www.salelsol.eu/sitemap.xml
Prepare to crawl [URL]https://www.salelsol.eu/sitemap.xml XML file
There are 313 urls in this site map
Starting to run First view on desktop agent...
https://www.salelsol.eu/acerca-salelsol.html -> Caching
https://www.salelsol.eu/bolsademicrofibra.html -> Caching
https://www.salelsol.eu/catalog/category/view/id/1796 -> Caching
https://www.salelsol.eu/catalog/product/view/id/5845 -> No Need To Cache
https://www.salelsol.eu/catalog/product/view/id/6116 -> No Need To Cache
https://www.salelsol.eu/catalog/product/view/id/6117 -> No Need To Cache
https://www.salelsol.eu/catalog/product/view/id/6125 -> No Need To Cache
https://www.salelsol.eu/catalog/product/view/id/6136 -> No Need To Cache
https://www.salelsol.eu/compras-seguras.html -> Caching
https://www.salelsol.eu/d1-coal.html -> Caching
https://www.salelsol.eu/d1-dense.html -> Caching
https://www.salelsol.eu/d1-focus.html -> Caching

You can try it and add it to cronjob

Best,
Eric

 

[Toni Moya]

- Indeed Crawler is working but when I verify it using the browser developer tool, many pages already cached show up: cache miss.

Best Regards
Toni

 

[Unique_Eric]

Hi Toni,
May I ask how you view the page? Did you view with guest view(non logged in)?
You can try an incognito page and see if it works to you.
Best,
Eric