robots.txt Directive Order
Each group starts with User-agent. Rules apply until the next User-agent.
Common causes
- Rules in wrong group.
- User-agent after rules.
How to fix
- Put User-agent first, then rules.
- New User-agent starts new group.
FAQ
- Order of directives?
- User-agent then Allow/Disallow.
- Multiple User-agent?
- Each starts a group.
Fix it now
Try in validator (prefill this example)