Quick answer
robots.txt should be UTF-8.
robots.txt Encoding
robots.txt should be UTF-8. Wrong encoding can break parsing.
Common causes
- Saving as Latin-1.
- BOM or wrong charset.
How to fix
- Save as UTF-8.
- No BOM recommended.
FAQ
- Encoding?
- UTF-8.
- BOM?
- Avoid.
Fix it now
Try in validator (prefill this example)