How do I allow search engines to crawl and index restricted content?

So you have white papers, articles or other information reserved for registered users or paying customers only on your site. (Otherwise known as “restricted content.”) The question is – how do you allow search engine crawlers to index these restricted content pages, without giving away all your restricted content for free? What are the SEO best practices for restricted content? You certainly don’t want to be penalized for showing the search engines one set of content, yet showing the visitors to your site something entirely different.

In order for this content to be visible to potential customers or new members, you probably want these pages showing up in the search results index, right? But you also don’t want to give away the content for free or to users without registering. News organizations (like for example) do this exact thing all the time!

The two best solutions I know of to date are the Preview method and the First Click Free method, but I welcome your alternative suggestions.

The “Preview Restricted Content” Method Restricted Content LoginYou can show a preview of the content up to a certain number of words, like does in this screen shot. You have optimized meta data and a short bit of the optimized content, but the full view isn’t available until a user signs in, registers or pays. Once a user signs in, registers or pays, you grant access to the full content on a separate unique page, which you could theoretically specify in the robots.txt to nocrawl/noindex (though be sure not to nocrawl/noindex the content preview page – that totally defeats the purpose). So show a preview, allow it to be indexed, but nocrawl/noindex the full content.

It's certainly not a perfect solution - you don't have the full content for the bots to crawl and rank. Which is no doubt why Google created a different solution back in 2008 called “Google First Click Free for Web Search.”

The “First Click Free” Method

The basic premise is that the first page a user clicks in the search engine results will be displayed in full without requiring registration or payment. From that point, you block the user with a requirement to log in or make a payment to view additional content when he or she tries to navigate to another restricted resource. The content that visitors see and Googlebot sees is identical, and the first-click user must be able to see the full article even if it has multiple pages (which you can specify to display all on one page for Googlebot and visitors, apparently, or you can use a cookie instead). You need to ensure that Googlebot can access all the restricted content in full using the robots.txt file, and the referring URL in the HTTP request-header field will be a Google domain.

Although it's a useful resource for sites with lots of restricted content to be able to be indexed fully while protecting most of the content from users initially, First Click Free isn’t a perfect solution either. It has two inherent problems. First, I imagine that many users who searched very specific and/or long-tail key terms to arrive at your content could get everything they need with that one click, and have no intention to pay or register for your restricted content. Or second, if you only have a few restricted resources, or even just one, a “first click free” solution could give away your entire revenue or registration stream. In this case, I suggest the preview solution I mentioned above instead.

Heather Physioc
Heather Physioc is a Kansas City inbound marketer, and President of Tentacle Inbound, LLC. Tentacle Inbound offers services in digital marketing, website development and design, and more. Connect with Heather on Twitter.

Comments 2

  1. Interesting topic and a common problem I don’t see covered much. I also like the preview method the best. More straightforward. Seems like a canonical tag would help with this approach.

    I’ve never tried it, but I wonder if sites could take advantage of the fact that Google “reads” a page much faster then people do – as long as the user/bot experience is consistent.

  2. Great post Heather. I was researching this topic for a client and came across your post (second time that’s happened lately, so kudos).

    You sum it up nicely and that’s pretty much what I’ve found. My client is a bit different that a news publisher, so the “get what you need on the first fix” isn’t an issue for them.

Leave a Reply

Your email address will not be published. Required fields are marked *