Do Not Cache User Agents - for Googleboot

[bobby47]

my sites pages are not being indexed by google, after waiting a while compared to other sites there is something not quite right & after going through a process of elimination i think it might be googlebot being served up the cached pages


so under "Do Not Cache User Agents" under "excludes" under "LiteSpeed Cache Settings" what do you put in the box for Googlebot ?

so that googlebot is not served up the cached pages & gets to cache all the new pages.

 

[bobby47]

Hello,

any news on this ?


thank you

 

[Unique_Eric]

Hi,

Try adding

Google|Googlebot

to the field.

Best,
Eric

 

[bobby47]

Thanks very much for that, its started indesxing the pages, but much slower than normal, & that was after i purged the cache, i would have thouht this would be one of the most important & sought after answers because if google is'nt caching the new pages then everything else is irrelevant .

i was adding loads of new pages, and eventually it has started caching & indexing all of them, but only after i purged the cache, so i dont know if its that that done it.

 

[Unique_Eric]

Hi,

Basically page cache will be purged if you add/update/delete anything on it. So I am not sure why index not working until you purge it or make google bot agent as no cache.

You can test it by:
1. check page "A" is a cache miss/hit
2. edit page "A" and update
3. Revisit page "A" and it should be a miss

Best

 

[Behinesazy]

Hello, I'm having the same problem while excluding these agents in the LiteSpeed plugin. They are indeed in the .htaccess file, but Googlebot reports a hit.

### marker NOCACHE USER AGENTS start ###
RewriteCond %{HTTP_USER_AGENT} APIs\-Google|Googlebot\ smartphone|GoogleOther\-Video/1\.0|Google\-Extended|Googlebot|GoogleOther|AdsBot\-Google\-Mobile|GoogleOther\-Image|Google\-InspectionTool|Storebot\-Google|Googlebot\-News|Googlebot\-Image|AdsBot\-Google|Mediapartners\-Google|Google\-Read\-Aloud|google\-speakr|Google\ Inspection\ Tool\ smartphone|Googlebot\ News|Googlebot\ Video|Googlebot\ Desktop|Google\ Favicon|Storebot\-Google/1\.0|Googlebot/2\.1|Googlebot\-Image/1\.0|Google\-InspectionTool/1\.0|Google\ ADT\-2|bingbot|Google\-Safety|FeedFetcher\-Google|GoogleOther\-Video|Google\-CloudVertexBot|http\://www\.google\.com/bot\.html|Google\|Googlebot [NC]
RewriteRule .* - [E=Cache-Control:no-cache]
### marker NOCACHE USER AGENTS end ###

is there any fix for this? ty

 

[Behinesazy]

Now I do a live URL test in Google Search Console or using Google Rich Results:

As you see its working good, but I don't know why for Googlebot smartphone it doesn't work.

 

[Behinesazy]

Plugin Settings:

 

[serpent_driver]

It works.

 

[Behinesazy]

It won't work for Google Smartphone or Desktop bot. You can test it here:

https://httpstatus.io/

Also, I sent a screenshot from Google Search Console itself (your screenshot is probably from Chrome Developer Tools or Firefox). Search Console is more trusted and shows what Google sees.

 

[Behinesazy]

Google Search Console and https://httpstatus.io/ both show hit!


I will disable the plugin until a fix comes.

 

[serpent_driver]

The inspection tool doesn't use Googlebot User-Agent. If you want to check if it works use any browser extension or plugin to change the User-Agent of your browser. There is no need to use any external services.

User-Agent for Googlebot (Desktop):

Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

User-Agent for Googlebot (Mobile):

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.6778.204 Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

 

[Behinesazy]

Sir, the inspection tool in Google Search Console is indeed Googlebot. I don't change anything with extensions or from the developer; I only care about what Google says officially in their Search Console.

 

[serpent_driver]

User-Agent for Google Inspection Tool:

Mozilla/5.0 (compatible; Google-InspectionTool/1.0;)

https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers

 

[Behinesazy]

here my settings :

### marker NOCACHE USER AGENTS start ###
RewriteCond %{HTTP_USER_AGENT} APIs\-Google|Googlebot\ smartphone|GoogleOther\-Video/1\.0|Google\-Extended|Googlebot|GoogleOther|AdsBot\-Google\-Mobile|GoogleOther\-Image|Google\-InspectionTool|Storebot\-Google|Googlebot\-News|Googlebot\-Image|AdsBot\-Google|Mediapartners\-Google|Google\-Read\-Aloud|google\-speakr|Google\ Inspection\ Tool\ smartphone|Googlebot\ News|Googlebot\ Video|Googlebot\ Desktop|Google\ Favicon|Storebot\-Google/1\.0|Googlebot/2\.1|Googlebot\-Image/1\.0|Google\-InspectionTool/1\.0|Google\ ADT\-2|bingbot|Google\-Safety|FeedFetcher\-Google|GoogleOther\-Video|Google\-CloudVertexBot|http\://www\.google\.com/bot\.html|Google\|Googlebot [NC]
RewriteRule .* - [E=Cache-Control:no-cache]
### marker NOCACHE USER AGENTS end ###

And as you see, I have no Mozilla/5.0 (compatible; Google-InspectionTool/1.0, just Google-InspectionTool and Google-InspectionTool/1.0. I did the same for the smartphone, but it's not working.

LiteSpeed needs to fix it ASAP. It prevents pages from being indexed. I don't know why no one ever noticed, only this article from 2019.

 

[serpent_driver]

There is nothing to fix. Settings to exclude certain User-Agents are correct and the x-litespeed-cache-control header is correct as well: "no-cache" and this is what you want to achieve.

 

[Behinesazy]

Chrome/131.0.6778.204

The problem is this version changes with every release, and one can't go to plugin settings to change these numbers. It's painful even if it works.

 

[Behinesazy]

Settings to exclude certain User-Agents are correct and the x-litespeed-cache-control header is correct as well: "no-cache" and this is what you want to achieve.

Don't you see the pictures I sent from Google Search Console? its a hit for samrtphone bot

 

[serpent_driver]

Therefore, you do not have to use the browser version, but rather clear and unchangeable designations: "Googlebot" and nothing else.

 

[Behinesazy]

i set this too alone but same result.