Understanding the Robots.txt File for Non-Developers

Fear not the techy-sounding robots.txt file, my friends. I’ve created a comprehensive, easy-to-understand guide to the robots.txt file for the average inbound marketer – no developer knowledge required. What is robots.txt? The robots.txt file is a document stored in the root directory of a website that tells search engine crawlers and other robots how to and how not to crawl …