Creating great content is hard work. It requires hours of researching, rewriting, and editing. And if you’re like me, you spend just as much time agonizing over the perfect picture to include in your post. Before you hit publish, you’re certain that you’ve done everything you could to create a massively successful piece of content that will drive tons of traffic.

But what you might not know is that there are people waiting to take (and get credit for) your content the moment you hit the publish button. It happens all the time: scrapers and content aggregators take other people’s content, publish it, and outrank the truly original source in Google.

One of the most common reasons this can happen is that aggregators take your newly published content and get their copy indexed before you do. Sometimes, Google will assume that the aggregator is the original author, and that you republished their content. This can lead to a loss of traffic, or worse, being caught in the Panda filter.

Now, there’s nothing we can do to change the way that Google works. But the good news is that there are some things we can do on our own sites to prevent this type of thing from happening.

Below, I’m going to show you a few tactics that you can use on any WordPress site to make sure that Google always indexes your content first. Follow these steps, and you will always get authorship credit for the content that you create.

1. Delay your RSS feed

RSS feeds can be great traffic drivers if you have a lot of subscribers. But what you might not know is that RSS feeds make it really easy for people to take and distribute your content however they see fit.

I don’t have a problem with people distributing my content in general, but I do have a problem with someone else getting authorship credit (and outranking me) for something that I wrote. To make sure that my content gets indexed before anybody else can take it from my feed, I like to delay the RSS build.

On this particular site, my new content is normally in Google’s index within a minute or two after I publish it. Sometimes it can take a little longer, so to be safe, I delay my RSS build so it doesn’t show new content for 10 minutes after it’s published. This way, Google has plenty of time to index the post before anybody else is able to consume it through RSS.

Changing the RSS wait time is simple. Just copy and paste the following code into functions.php. To change the amount of time you want to wait before building the feed, change the value of the $wait variable.

function publish_later($where) {

	global $wpdb;

	if ( is_feed() ) {
		// get the WordPress timestamp
		$now = gmdate('Y-m-d H:i:s');

		// set the amount of time that you want to wait before building the feed
		$wait = '5'; // integer

                // now define the unit of time that you would like to wait
		$device = 'MINUTE'; //MINUTE, HOUR, DAY, WEEK, MONTH, YEAR

		// now we turn the timestamp into a format that MYSQL can understand
		$where .= " AND TIMESTAMPDIFF($device, $wpdb->posts.post_date_gmt, '$now') > $wait ";
	}
	return $where;
}

add_filter('posts_where', 'publish_later');

2. Turn off auto cache clearing

If you’re not using a caching plugin on your site, you really should be. These plugins take a lot of strain off your database, and generally result in a website that loads more quickly than one without a caching plugin (WP Super Cache and W3 Total Cache are good options).

Most (if not all) of the available WordPress caching plugins have a setting that clears the cache each time you publish a new post. This is a great idea in theory (and it’s just fine for authority sites), but it also gives scrapers a chance to grab your content before Google indexes it.

To combat this, turn off the function that automatically clears the cache each time you publish a new post (found within the settings of your caching plugin). Once your post is in the index, manually clear the cache through your plugin’s admin panel. This way, any scrapers who hit your site before your new content is indexed will not see the new content.

3. Give your sitemap a funky name

One of the easiest ways to scrape the entirety of someone’s site is by using the sitemap as a starting point and then following all the links. Since most sitemaps include every piece of content on a particular site, this is by far the easiest method of content discovery for scrapers.

We don’t want to make it any easier for scrapers than it has to be, but we also need our sitemap to be updated right away so Google can quickly find and index the new post.

Most people name their sitemap sitemap.xml, or something very similar. Almost every WordPress site uses this convention, as it’s the default setting for just about every sitemap plugin. I would encourage you to name your sitemap something else. Maybe your dog’s name, or even a random string of characters. This will make it very difficult to randomly guess the location of the sitemap.

Additionally, you will want to turn off the setting that shows your sitemap’s location in robots.txt. If you’re using the popular Google XML Sitemap Generator plugin, you can do that like this:

Removing the location from robots.txt will not hurt you at all, as long as you submit your sitemap in Google and Bing webmaster tools. Aside from those guys, nobody needs to see it anyway.

4. Submit your new post through Google Webmaster Tools

This is one of my favorite features of Google Webmaster tools, and a lot of people don’t know that it exists. If you have a new website, it take some time for Google to index your new content after you publish a post. This tip will speed up the indexing process significantly.

As soon as you publish a new post, login to Webmaster Tools and go to Health->Fetch as Googlebot. Enter the URL you want to submit into the field:

After a moment, a ‘submit to index’ button will show like this:

Click it, and your content will be submitted to the index. In my experience, it normally takes between 12-14 hours for Google to index a URL submitted to them in this manner. While that sounds like a long time, it’s often much faster than Google would index a page posted on a new site.

Closing

If you follow the above steps, it becomes just about impossible for anyone else to consume and publish your content before Google is able to discover and index your version.

If you have any alternate methods of protecting your content, please leave a note in the comments and I’ll be sure to add it to the post.

Best WordPress Themes for Business Sites

1. Setup a 301 redirect (ie: redirect yoursite.com/affiliate to affiliatesite.com)
2. Block the redirected page through robots.txt
3. Add the nofollow attribute to all affiliate links

The idea is to make it look like you’re linking to internal product pages, when you’re really linking to an external site. Then you’re adding nofollow and /disallow directives to keep Google from discovering that you’re linking offsite.

It sounds like a good idea in theory, right? I mean, how could Google know that a link redirects if they can’t follow it or crawl it?

The truth is, there are some advantages to running your affiliate links through redirects (more on that later). But if you’re trying to fool Google by masking your affiliate links in this way, the fact of the matter is that you’re wasting your time.

Google can see right through you

Before we dive in here, I need to clear up some common misconceptions about how Google works.

Nofollow attributes on links tell search engines not to give editorial credit to the links. But the spiders will still follow the link and discover the page on the other side. So you’re not stopping the crawl by adding a nofollow attribute to your affiliate link.

Robots.txt disallow commands tell Google not to crawl a page. That’s it. They will still index the page, and they can still tell if the page is redirected without crawling it. This is how:

HTTP headers are sent from your server each time a file is requested. These headers return status codes that communicate whether or not the document exists, or if it has been moved (redirected).

Now, Google can check HTTP headers without crawling a page. In fact, you can do the same thing yourself with just two lines of PHP (try it out for yourself if you’re curious — just replace the value of the $url variable with your own affiliate url):

$url = 'http://google.com'; 
var_dump(get_headers($url,0)); 

When executed, the above code will return a response that looks something like this:

/*array(18) { 
  [0]=>  string(30) "HTTP/1.0 301 Moved Permanently" 
  [1]=>  string(32) "Location: http://www.google.com/" 
  [2]=>  string(38) "Content-Type: text/html; charset=UTF-8" 
  [3]=>  string(35) "Date: Mon, 30 Jul 2012 00:59:50 GMT" 
  [4]=>  string(38) "Expires: Wed, 29 Aug 2012 00:59:50 GMT" 
  [5]=>  string(38) "Cache-Control: public, max-age=2592000" 
.... 
  string(15) "HTTP/1.0 200 OK" 
  [10]=>  string(35) "Date: Mon, 30 Jul 2012 00:59:51 GMT" 
  [11]=>  string(11) "Expires: -1" 
  [12]=>  string(33) "Cache-Control: private, max-age=0" 
..... 
}*/ 

See how the header communicates that google.com 301 redirects to www.google.com? With two lines of code, even a well-behaved bot that honors robots.txt could discover that your ‘sneaky’ redirect is a 301 affiliate link.

In short, your masked links aren’t fooling anyone. But they can still be worthwhile to implement.

Good reasons to redirect affiliate links

Even though your redirects aren’t fooling Google, there are still a few good reasons why you might want to implement them:

1. Click tracking. By running your affiliate links through an internal redirect, you can track the number of times that visitors click on your links. This is necessary information if you want to calculate your own conversion rates (GoCodes is a popular WordPress plugin that will do this for you).

2. Pretty URLs. Some affiliate programs (I’m looking you, CJ), provide incredibly ugly URLs like fewrxrk.com/32535346342. Your users don’t want to click on that link, because they have no idea what’s going to be on the other side. An internal redirect can help entice clicks.

3. Privacy. Many affiliate programs include your affiliate ID in the links that they give you. If you use the same ID on multiple sites, it can make it very easy for people to discover all of the affiliate sites that you’re running through a simple Google search. By not including the ID on your page, you can make it a little bit harder for them.

While bloggers like myself appreciate all that WordPress can do, the truth is, most websites don’t need a lot of  the out of the box functionality to be enabled. For example, when I build a business card website for a small business, one of the first things I do is disable the RSS feed. They just won’t get any benefit out of it on their site.

Why get rid of the feed?

RSS is great for syndicating content, but if you have a business card website or don’t regularly publish content that people would want to subscribe to, all it’s really doing for you is making your content easier for other people to take and re-purpose on their own sites (there are even WordPress plugins that make it crazy easy for people to do this). This can result in a lot of crappy links, and even some duplicate content issues in some cases.

When you have a blog, it’s well worth the risk to keep the RSS feed running. But in some cases, there’s just no benefit.

How to get rid of the feed

Disabling the RSS feed is simple, though you will need to be comfortable editing functions.php (remember to make a backup before you start).

Step 1: Open your functions.php file in your favorite text editor. Make a backup, and then scroll to the bottom of the file. Copy and paste this code into functions.php. This code will replace the contents of your RSS feed with a simple message that directs visitors to your homepage:

function disable_feed() {
wp_die( __('Our RSS feed is disabled. Please <a href="/">visit our homepage</a>.') );
}

add_action('do_feed', 'disable_feed', 1);
add_action('do_feed_rdf', 'disable_feed', 1);
add_action('do_feed_rss', 'disable_feed', 1);
add_action('do_feed_rss2', 'disable_feed', 1);
add_action('do_feed_atom', 'disable_feed', 1);

Step 2: Save functions.php. Load yoursite.com/feed in your browser, and you should see the new error message.

Step 3: Your feed is now disabled, which is a good first step. But there is one more thing that we still need to do. If you view the source of your website, you will see two references to yoursite.com/feed in the head section of your site. Since your feed isn’t functional anymore, let’s go ahead and remove those references.

With functions.php still open, copy and paste the following code:

function removeHeadLinks() {
      remove_action( ‘wp_head’, ‘feed_links_extra’, 3 );
      remove_action( ‘wp_head’, ‘feed_links’, 2 );
      remove_action( ‘wp_head’, ‘rsd_link’ );
}
add_action('init', 'removeHeadLinks');

Now view your page source again, and the references to your RSS feed should be gone.

Congratulations! You’ve successfully disabled your site’s RSS feed.

In cases where Google feels extremely confident that you are looking for one particular item, they will show it to you — and nothing else. This is pretty revolutionary for a search engine that’s been pretty consistent with its 10 result set.

This morning I searched ‘occupational outlook handbook’ and got the following results page (try it out for yourself and see if you get it too). As you can see in the screenshot below, my results consisted only of a knowledge graph entry and links to the handbook itself. That’s it. There are no other domains in the results.

I’ve never seen this happen before, but I have a feeling that we will be seeing more and more of this before too long (article continues after the screenshot).

‘What happens when I click to page 2,’ I thought. Well, that’s just as interesting. Page 2 is essentially just a site:bls.gov command. Here’s a screenshot:

It wasn’t until the bottom of page 3 that I came across listings for any other domains.

I don’t mean to imply that this is a bad thing. In this particular case, it’s perfect — Google knew exactly what I was looking for and gave it to me.

For those affiliates out there who are piggybacking off of brand names, it might be time to reassess your business model. How long until brands and products get a similar treatment?

If you’ve seen similar results, please leave a note in the comments. I’d like to see some other examples of queries that trigger this behavior.

Dear Google,

Something has happened to you.

Since you released Panda into the wild, you have become a clingy high school girlfriend. You know, the one whose world view has yet to develop to the point where they understand that most of the things you do and say have nothing at all to do with them.

With recent updates, you have demonstrated a belief that everything we do is an attempt to manipulate you. At worst, this is the type of deep-seated paranoia that usually precedes imminent self destruction (see Nixon, Richard and Stalin, Joseph). At best, it’s childish, manipulative, and egomaniacal.

See, somewhere along the line, you forgot something really important:

We can add value to our sites without adding value to the internet.

I’m going to say that one more time:

We can add value to our sites without adding value to the internet.

Sometimes, when we’re trying to make our sites better for our users, we add value to our sites without adding value to the internet. There *should* be nothing wrong with that.

As an example, let’s say I’m a cat expert and I have an informational site about breeds and cat care. In addition to my regular expert content, maybe I like to post a funny cat picture each day, and each picture is a new post. My users love the cat pictures, and they keep coming back each day to see them. I’m happy, because the pictures increase my site’s engagement. My users are happy because the pictures are hilarious. Everyone is happy.

Except you.

You’re not happy, because a year from now, half of the pages on my site have nothing on them except a photograph. You interpret this to mean that my content is not adding value to the internet.

From your newfound perspective, if content doesn’t add value to the internet,  it’s thin. And if it’s thin, it’s spam. And if it’s spam, the entire site must burn.

Here’s the thing, Google. And you have to understand this: As site owners,  just because we create a piece of content doesn’t mean we’re expecting you to send us traffic for it. 

Some things, we do just for us. Some things, we do just for our users. Not everything is about you. In fact, most things aren’t.

You used to know this.

You used to be good at identifying sections of a site that aren’t adding value to the internet, while highlighting the sections that are.

You used realize that everything we do isn’t an attempt to get traffic from you.

But not now.

Now, you assume that every page that we create is an attempt to squeeze out one extra visit from the Google machine.

Now, you see every word, every photo, every comment as yet another attempt to manipulate your results.

This paranoia has got to stop.

It isn’t all about you, Google. And it never was.

For a site like mine, a high bounce rate from search isn’t necessarily a bad thing. My content is geared towards helping people solve one specific problem. They read one page to help them solve it, and then they bounce. So long as they find what they’re looking for, I’m a happy camper.

But I’m not happy about my analytics reports. Each time I log in, I’m presented with pages that look something like this:

These numbers are so bad that it makes me want to hole myself up in my bedroom and spend entire days staring at the ceiling fan like Martin Sheen at the beginning Apocalypse Now.

You can see that the bounce rate is 100% for all keywords, and the average visit duration is zero seconds.

Zero seconds on site. So not only is my design repulsive, but my readers are equipped with cheetah-like reflexes. How can that be?!

Thankfully, there’s a simple explanation — and my bounce rate is to blame.

How Google Analytics measures time on site

When you install Google Analytics (or any other analytics software) on your site, it’s as simple as copying a neat little JavaScript function they provide you and pasting it onto the pages of your site. This script is then executed each time one of your pages loads.

This is how time on site measurements work:

1. When a visitor enters your site, the time of the initial pageview is logged.
2. As a visitor navigates through your site, the time of each subsequent pageview is logged.
3. Time on site is calculated by subtracting the time of the initial pageview from the time of the last pageview.

See the problem?

When a visitor views only one page, there’s nothing to subtract the timestamp from. Your analytics software has no way to calculate time on site, so they give you a big fat zero.

Your bounce rate is absolutely killing your analytics.

What to do about it

This is a frustrating problem. My bounce rate is making my analytics next to worthless when it comes to measuring engagement.

I have accepted that the high bounce rate is just a fact of life (until I do something to try and reduce it), but it sure would be nice to know how users are interacting with my pages. Are they reading the entire articles? Are they just reading a sentence and then bouncing?

With a little bit of creativity, you can setup Google Analytics to measure scroll depth. This allows you to see how far people scroll down on your content pages (in terms of percentages).

Scroll depth is not a perfect metric by any means, but it at least gives you an idea of how people are interacting with your site (some information is always better than no information).

With scroll depth events enabled, I can see reports that look something like this:

If you’d like to implement scroll depth events within Analytics, check out this jQuery plugin by Rob Flaherty.

A few weeks ago, I was contacted by a site owner who had a very serious problem: He was noticing what he believed to be a lot of false positive soft 404 errors in his Google Webmaster Tools account. Alarmed, he started checking his pages in Google, and noticed that a lot of his content had been completely de-indexed.

Curious, I took a look at his site, and my initial impression was positive. It was well architected, and home to a lot of original and high quality content that demonstrated a solid understanding of its audience. A quick check of his backlinks removed any suspicion that he was engaged in anything he shouldn’t have been. Everything appeared to be in top shape.

Stranger still, the pages that Google was reporting as soft 404s appeared to be unique and valuable. Aside from the header and footer, they had nothing at all in common with his real 404 pages (which sent the proper response code).

Why was Google throwing the soft 404 flag and removing his content from their index?

I was interested in helping him solve this problem and agreed to help.

Where things got nasty

With the initial research out of the way, I fired up Firefox and used this user agent switcher plugin to see the site as Google sees it.

The news wasn’t good. I’m pretty sure I reacted the same way Samuel L Jackson reacted in Jurassic Park when the electricity in the raptor fences went down. That is, mumbling obscenities to myself and warning everyone to hold onto their butts.

In short, I saw that the site had been hacked to show pharmacy spam links to Google, and the regular site to everyone else. With the Googlebot user agent activated, every page on the site looked like this:

Suddenly, the problem made sense. In Google’s eyes, the content of every page on his site was identical to his 404 pages (which were also hacked). Hence the soft 404 errors.

And they were de-indexing the site because every page was a duplicate of every other page on the site (not to mention the content was nothing but spam links).

Can we fix it? Yes we can!

I requested the login credentials for his site and cpanel admins.

The site used Joomla as its CMS, which I’m not terribly familiar with, and it took some time to get acquainted. As I was starting to get a handle on how all the pieces worked together, I stumbled across a php file that was buried in a very deep folder.

In it, I found what must be the simplest malicious code in the history of malicious code. It was a just a couple lines of php  that looked like this:

*note* I’ve removed the actual URLs from the screenshot because I promised anonymity — I don’t want you super sleuths digging through backlinks and identifying the hacked site. 

In case you can’t read the code, basically what it’s doing is checking to see if the user agent of the visitor is Googlebot. If it is, then it shows the pharmacy spam links. If it isn’t, then it shows the site as normal.

Fixing the site was as simple as removing that conditional statement.

The recovery

Within one week, many of the de-indexed pages started making their way back into the index.

Within two and a half weeks, the site made a full recovery. All of its pages were back in the index, and traffic had returned to normal.

In this case, the site benefited greatly by having someone in charge who was monitoring Google Webmaster Tools, saw the soft 404 errors, and noticed that pages were dropping from the index. I suspect that if the site had continued to show those spam links for an extended period of time, recovery could have been much slower.

In the end, the relatively quick recovery was great news for them. Unfortunately, not everyone is so lucky.

Check your Webmaster Tools each day, ladies and gentlemen.

I immediately went on a record store rampage, spending all the money I could earn on CDs by bands I’d never heard of, trying to rediscover that feeling of *it* I experienced when I heard that song for the first time.

I remember starting with the Pixies (the guy at the record store told me they were a big influence on Kurt Cobain. He was right). Then I started moving backwards through time to Talking Heads, David Bowie, The Beatles, and on and on until I finally made it all the way back to Bo Diddley and his little square guitar.

I ended up falling in love with the psychedelic music of the 60s and 70s, because to me it felt less like a product and more like an adventure. And after my first viewing of The Wall, I promised myself that no matter how old I got, I would never become a cog in the machine.

My definition of a cog: when you sacrifice your freedom or individuality for the good of someone else.

Our lives take strange paths sometimes. Though I never would have guessed it (nor would it have even been possible to at the time), my love of music directly led to my career in affiliate marketing.

This is how.

I don’t need no stinking diploma

I had no interest in going to college when I graduated high school. Not because I didn’t like to learn (I loved it and still do), but because I recognized that I had no idea what I wanted to do with myself. At that point, I only knew what I didn’t want to be:  a cog in the machine.

I was certain that going to college without a clear direction would just accelerate my transition into being a cog. Within 10 years I’d wind up like Patrick Bateman, applying a wax face each morning and debating the artistic merits of business cards.

Meanwhile, I’d run into a musical rut. I wasn’t feeling *it* anymore. All the music I could find in the record store was feeling old and tired. It was around this point that a guy I worked with gave me an audience recording of a Phish show (3/12/93, for those of you who you care), and turned me on to live music.

So while my friends were all packing up their stuff for college, I took the money I’d saved from working after school as a steakhouse cook and hit the road in search of *it*.

Ladies and gentlemen, the Phish from Vermont!

I spent the better part of 3 years following Phish around the country, and it was one of the defining experiences of my life.

I got to see the country, experienced *it* on a near nightly basis, achieved a great perspective of what’s actually important in life (hint: not business cards),  met a lot of great people (including my future wife), and I learned how to make money on the road.

While following Phish, I learned these key lessons about life and business:

1. Look like you’re supposed to be there and nobody will question you. Sometimes you might have to carry around different colored wrist bands in your backpack.

2. Value is relative. I used to sell bottled water in the parking lots for gas and food money. I did ok. But it was much more valuable to people when I stood right by the entry gates (I sold more and for higher prices).

3. Provide value and people will love you — even on your off nights. I learned this from the band.

4. Authority is an illusion.

5. If you’re good to people, you will always be welcome. I quit so many jobs to go on tour, but always tried hard to leave gracefully. I was always welcomed back when I got home.

6. Freedom is worth a whole lot more than money. I didn’t get any value out of money beyond what I needed for food and gas. Once those basic needs were met, money wasn’t good for much.

It was an adventure. But it couldn’t last forever.

Get a job, hippie

When my daughter was about a year old, we moved to Charlottesville, VA. I needed a job, but I wasn’t willing to sell my happiness for money. As luck would have it, I managed to get what was a dream job for me at the time: working in merchandising for Dave Matthews Band.

Now, when I say that I worked in merchandising, what I really mean is this: when people ordered Dave Matthews merchandise online, I would grab the order sheet off the printer, take the merch off a shelf, put it in a mailer, and drop it in a bin that the mailman would pick up at the end of the day.

It was a simple job, but it was my first exposure to how online business worked. It wasn’t long before my wheels started spinning. I wanted to make money online for myself. But how?

I didn’t have the answer to that question right away, but I knew right then that I wanted to learn how to build websites.

With a clear direction in mind, it was time to go to college.

On second thought, I’ll take that diploma

After a long educational hiatus I applied to The College of William and Mary, got in, and enrolled in the computer science program. It was *tough* going to school full-time with a family, but the experience taught me a strong work ethic and I tip my hat to everyone who makes it through it. When I left college, I had a good foundation of the technical skills I needed to make a living online, but I didn’t have the business sense.

I took an entry-level job working in the online advertising department at Media General (a pretty big news corporation in the southeast). I made considerably less money in that position than I would have as a programmer, but it was a trade-off I was willing to make for more exposure to the business side of running a website.

I ended up learning a ton about how the advertising business works on the internet, and it wasn’t long before I started building my own affiliate sites in my spare time and applying what I learned at work (for the following five years, I would spend 2-3 hours a night working on my sites after the kids went to bed. That’s what it takes when you have a job, wife and kids. There is no golden bullet.).

I learned SEO as I went along, and with a lot of practice I started getting pretty good at it. I eventually took over SEO for Media General, and it wasn’t too long before I quit for an in-house SEO position at Snagajob. And again, I applied everything I learned to my own sites.

I enjoyed my job, and I worked with great people. But I’d started noticing something about myself. I was taking things too seriously. I was losing my perspective on what’s really important (not business cards). And I’d started to hit that point on the professional development scale where I was sacrificing freedom for money.

When I looked into my future, I could see only one path remaining: a cog.

It was nobody’s fault but own. And I didn’t like it.

I quit

On February 3rd, 2012 I quit my job. I’m hopeful that it’s the last job I’ll ever have to quit.

After five years of steady work, my affiliate sites were successful enough that I believed I could match my salary in six months with full-time dedication.

I hope I’m right, but only time will tell. Either way, I’m happier now than I’ve been in years. I have the freedom that I’ve always needed to be happy. And I owe it all to rock and roll.