Understanding the Robots.txt File for Non-Developers

Fear not the techy-sounding robots.txt file, my friends. I’ve created a comprehensive, easy-to-understand guide to the robots.txt file for the average inbound marketer – no developer knowledge required. What is robots.txt? The robots.txt file is a document stored in the root directory of a website that tells search engine crawlers and other robots how to and how not to crawl …

How do I allow search engines to crawl and index restricted content?

So you have white papers, articles or other information reserved for registered users or paying customers only on your site. (Otherwise known as “restricted content.”) The question is – how do you allow search engine crawlers to index these restricted content pages, without giving away all your restricted content for free? What are the SEO best practices for restricted content? You …