23 Aug 2022 - 22:01 — https://memex.marginalia.nu/log/63-marginalia-crawler.gmi
There are more search engines on the internet than just one. Read a blogpost about how Marginalia - a growing new search engine - does the crawling.