seo - Why Google robots.txt Tester has error and it's not valid -


as can see in below image google webmaster tools robots.txt tester tell me 9 error don't know how fix , problem? enter image description here

please me figure out

that valid robots.txt - you've got utf-8 bom (\xef\xbb\xbf) @ beginning of text file. that's why there's red dot next 'user' in first line. mark tells browsers , text editors interpret file utf-8 whereas robots.txt expected use ascii characters.

convert text file ascii , errors go away. or copy after red dot , try pasting in again.

i tested on live version, here's result translated byte form:

\xef\xbb\xbfuser-agent: *\r\ndisallow: /en/news/iranology/\r\ndisallow:  /en/tours-services/tour-the-soul-of-iran\r\ndisallow:  /en/tours-services/tour-a-whistle-stop-tour\r\ndisallow: /en/to 

you can see bom @ beginning. browsers , text editors ignore may mess crawlers ability parse robots.txt. can test live version using python script:

import urllib.request  text = urllib.request.urlopen('http://www.best-iran-trip.com/robots.txt')  print(repr(text.read())) 

if you're able install notepad++, has encoding menu lets save in format.


Comments

Popular posts from this blog

mysql - Dreamhost PyCharm Django Python 3 Launching a Site -

java - Sending SMS with SMSLib and Web Services -

java - How to resolve The method toString() in the type Object is not applicable for the arguments (InputStream) -