robots.txt Crawl-delay
Crawl-delay is not in the standard; Google ignores it. Bing may honor it.
Common causes
- Expecting Google to honor Crawl-delay.
- Wrong syntax.
How to fix
- Do not rely on Crawl-delay for Google.
- Use Search Console crawl rate if needed.
FAQ
- Crawl-delay Google?
- Ignored.
- Bing?
- May honor.
Fix it now
Try in validator (prefill this example)