I've developed similar working multithreading crawler, but mine is extended and has some more features for user agent, cookies, request headers for webp image, cache-control support and much more. This allows me to crawl up to 100k URLs within 1 hour on a shared hosting.