Robots.txt is the first page which is crawled by a search
engine while it starts crawling a website. If robots.txt does not exist, then
server return to the search engine crawler a page not found error (404).
Page not found (404 or broken links) is not a good
indication for the search engines and our website should not have any page not
found issue. You can monitor all broken links
in a website by our website monitoring tool.
No comments:
Post a Comment