robots.txt Missing User-agent
Disallow and Allow apply to a User-agent. Rules before any User-agent are ignored.
Common causes
- Disallow without User-agent above.
- Wrong order.
How to fix
- Start with User-agent: * (or specific).
- Then Disallow/Allow.
FAQ
- User-agent required?
- Yes for rules to apply.
- Order?
- User-agent then rules.
Fix it now
Try in validator (prefill this example)