10. Gzip Compression: Enable Gzip compression on your server to reduce the dimension of data files transferred among the server as well as the person’s browser. txt file is then parsed and can instruct the robot regarding which web pages are usually not to generally be crawled. Like a search https://www.youtube.com/watch?v=cZppEXx1zPU