robots.txt Disallow All
Disallow: / with User-agent: * blocks all crawlers. Often a mistake.
Common causes
- Disallow: / for *.
- Intended for one bot only.
How to fix
- Remove or narrow Disallow.
- Use specific User-agent if needed.
FAQ
- Disallow: / blocks?
- All crawlers for *.
- How allow again?
- Remove the rule or use Allow.
Fix it now
Try in validator (prefill this example)