Robots.Txt refers to a text file which is stored in a website’s server. This text contains various rules which should be followed by indexing robots when crawling the site. You can use this text file to decide what files should be viewed by the crawler bots and what shouldn’t.