Getting There With Search

Six steps you can take to help search engines find and index your content

You’re trying to get your website noticed and search engines are an excellent channel for visibility. In this article, we’ll look at some basic things you can do to make sure your content is being indexed by search engines. The more of the items you can implement below, the more noticeable your site will be to the search engines. 

1. Check your “Robots.TXT” file
One little line of code in a simple txt file can be very costly if it’s blocking your site from search engines. A robots.txt file allows you to tell search engines to crawl or not crawl certain directories or files in your site.

For example, this simple line of code disallows all crawlers (a.k.a. robots):

User-agent: *
Disallow: /
To find your robots.txt file simply type in your domain followed by /robots.txt. For example, here is the robots.txt file for the W3C: www.w3.org/robots.txt. For more details on what can be in this file and how search engines treat it visit robotstxt.org.

2. Make sure your content is indexable
Although search engines have come a long way over the years in terms of indexing all kinds of content on the web, there are still some types of content that may not be fully indexed or not indexed at all. If you are seeing missing content when you look at the search engine’s cache of your page, you may want to check if the content is presented in one of the tough-to-index ways below.

This is also true for people using screen readers due to a disability. A screen reader “sees” the page much in the same way a search engine crawler does – by crawling content and deciphering the elements.

JavaScript
Some JavaScript is crawled by search engines today and more will likely be in the future, but JavaScript (including its related scriping technique AJAX) can still present an issue.  Because most content in JavaScript is usually not indexed, things like navigation, on-page apps, and any other content presented by using JavaScript may not be seen and therefore cannot contribute to the context of the page (or sometimes cannot be followed, in the case of links) for search engines.

Flash
With better indexing capabilities coming about recently, Flash sites are becoming more prevalent in SERPS (Search Engine Result Pages), although a site built entirely in Flash is still probably not the best idea if you care about search engine traffic.

Today search engines primarily attempt to index links and text from Flash files. While this is better than it used to be, 100 percent of content still may not be indexed depending on how your Flash site is created. Navigation through “pages” in a Flash file is all contained within a single swf file that lives on one URL, eliminating separate topical content for separate pages.  This can be problematic when you’re up against competitors with much more targeted topical and sub-topical content living on distinct URLs (with links to each of those specific URLs providing even more context).

To minimize indexing difficulties, try to use Flash in smaller pieces. Make sure each topical page of your site has its own unique URL first, then put Flash elements on each page if you like.  Beware though—the more of your content you put in Flash, the less content and context you may be providing to the search engines. 

Image Text
It’s rare to see the entire content of a page posted as a .jpg or other image these days, but it still happens. And when it does, a crawler goes through the code and just sees an image instead of seeing all of the pictures, content, and link text on the page. Search engines simply cannot read any textual content you present in an image, whether it is the entire content of the page or just titles or headers.  You will see the images displayed when you check the cached version of a page.  This is because it is displaying the actual image that is cached, not reading the text content within it.

3. Strenghten your link structure
Links to and from your pages are very important for the “findability” of your pages.  If a page has no links connecting it with any other indexed pages on the web, it may not be found by search engines, since they follow links to discover new content. 

Internal links
Make sure you have a sensible linking structure in place on your site that is crawlable, links to top level as well as deeper level pages, and links to content relevant to the page the links are on.

Crawlable links are links that can be seen by search engines, meaning they’re not in JavaScript or in unindexable links within a Flash file. Also link to different pages within your site, not just from the home page, but all pages. Deeper pages in a site tend to be tougher to find and index, since they are linked to less often, or from more obscure pages in a site. Try to include links to pages most relevant to the content of each page, to give the search engines better context, and to provide a good mix of deeper links. 

You can also include a sitemap page on the site (similarly named xml sitemap files are discussed later).  Provide the sitemap link from your home page and/or from a header or footer on all pages.

External links
If you provide worthwhile content, your site and the pages within it will attract links naturally.  These links from external sites help search engines find and classify your site, especially if your site is newly published. To kick-start your visibility,you can add your site to  trusted directories like The Open Directory Project and Yahoo! Directory.  If it is relevant, you can also add your site to online local listings pages like Yelp, Yahoo! Local or CitySearch. 

Promote your website in your advertising campaigns, add it to your business card, and provide any other means for visibility that you can. If people find your site interesting and useful they will link to it.

To see what your inlinks looks like, go to https://siteexplorer.search.yahoo.com/ and type in your URL. Click on the Inlinks button.  Use the dropdowns to look at links to one page or the entire site, or to look at links from all pages, all pages except that subdomain, or all pages except that domain.

4. Create a  sitemap XML file
The major search engines we’re addressing here all support xml sitemap files. These sitemap files are different from the onsite sitemap pages previously described. They are xml files that contain a list of the URLs on your site along with a small amount of information about the URLs that is placed on your server and crawled by search engines. This allows you to tell search engines about your URLs, even if they haven’t crawled them naturally by following links on the Web.

Visit sitemaps.org for more information, or see Yahoo!, Google, and Bing’s support of sitemaps.

5. Verify your “nofollow” and “noindex” tags
Noindex and nofollow tags can be used to block search engines from crawling specific links or content. 

Noindex
The noindex meta tag tells search engines not to index a page.  It looks like this:

<meta name=”robots” content=”noindex” />

To check for noindex tags on any of your pages, right click on the page in the browser and choose “View Source”.  Search for noindex on the page.

For more information on the search engines’ support of noindex, see these Yahoo!, Google and Bing pages mentioning it.

Nofollow
Nofollow tags can be found in a robots meta tag at the page level, or within the <a> tag at the link level.

Nofollow at the page level tells search engine robots not to follow any of the links in the body of the page that the nofollow meta tag is on.  It looks like this:

<meta name=”robots” content=”nofollow” />

Nofollow at the link level tells search engine robots not to follow that particular link that the nofollow attribute is applied to.  It looks like this:

<a href=”http://www.example.com/” rel=”nofollow”>link text</a>

To check for nofollows on any page, you can look at the source code of the page by right clicking on the page and choosing “View Source.” Then do a search for the word nofollow in the source code.

For more information on nofollows, see this Wikipedia article, or see Yahoo!,  Google and Bing’s  support of nofollow.

6. Specify your site’s language
You can also help search engines by specifying what language your site is written in. This is a simple meta tag that looks like this:

<meta http-equiv=”content-language” content=”en”>

See all ISO codes at the Library of Congress site for more information.

To check for language meta tags on any page, you can look at the source code of the page by right clicking on the page and choosing “View Source.” Then do a search for the word language (or content-language) in the source code.

Still having problems?
If you’ve tried everything above and believe you still have indexing issues, browse the webmaster guidelines below for more information, troubleshooting, and contact information for the search engines. 

Search Engine Guidelines for webmasters

• Yahoo! Search Content Quality Guidelines
• Google Webmaster Guidelines
• Bing guidelines for successful indexing

For more on getting your site noticed, refer to Laura Lippay’s previous post “Is Your Site Invisible?”

— Laura Lippay, Director of Technical Marketing

(Image by Kapungo via Flicker, CC 2.0)

More Video Marketing Case Studies from Leading Brands

Yesterday, I posted Video Marketing Case Studies with David Meerman Scott and Me. At the end of the post, I said, "If you have any video marketing case studies that you'd like to share with us, just mention them." Silly me.

Click to read the rest of this post...

Google Goes Live With “In-Stock, Nearby” Mobile Shopping

In the range of thigs discussed at Google’s Search Evolution event in December Engineering VP Vic Gundotra demonstrated mobile shopping with real-time inventory information.  Now Google has announced that product is live. Google is working with a limited number of retailers for the time being but is inviting others to apply to become part of the [...]

*** Read the full post by clicking on the headline above or, in Facebook, by clicking on the "View Original Post" link below. ***

Google Is Bing’s 4th Largest Referring Source

What sites send Bing the most traffic? Here’s a surprise. According to Hitwise, Google is Bing’s fourth largest referring source. Below are the top five sites that drive traffic to Bing, or “upstream” to Bing, as Hitwise calls it: MSN is by far the top source, the last site visited by 42.7% of Bing users before they [...]

*** Read the full post by clicking on the headline above or, in Facebook, by clicking on the "View Original Post" link below. ***

Get the Best Results from Yahoo’s Partner Network

Efficient Frontier offers insights

Recently, we introduced some new features and functionalities that make it easier and more efficient for you to manage your account and better target your campaigns. These were pretty much a hit with the search marketing cogniscenti.

So how are these updates performing? Pretty well, according to Efficeint Frontier’s Shay O’Reilly. He offers several tips on how to make them perform even better.

— The Team

The Problem Of Inconsistent SMB Contact Details – Part I

Have you ever thought twice about how to find the correct contact details for a local business – i.e., their name, address, phone number? The answer should be so simple:  just Google it. Well, the reality is not that simple. In fact, few questions in the local online search world are as complicated as this [...]

*** Read the full post by clicking on the headline above or, in Facebook, by clicking on the "View Original Post" link below. ***

Google Shows Inventory on Mobile Shopping Search

Find out what's in stock nearby.

Click to read the rest of this post...

MapQuest Adds Facebook Share Feature

Easily enable stalking by anyone your network. Just kidding. (Sort of)

Click to read the rest of this post...

Yahoo!’s March 2010 Search Update Underway

The announcement was as short as usual.

Click to read the rest of this post...

Questions about the Yahoo! and Microsoft Search Alliance

Advertisers and agencies have questions; we have answers

A few weeks ago, we announced U.S. and E.U clearance for the Yahoo! and Microsoft Search Alliance. This was, naturally, big news in the search marketing world. That post garnered some 60 comments from our engaged advertising and agency customers. Most of these comments were positive.

Understandably, some of you had questions and concerns about your accounts. But the most important fact for you to know regarding the search alliance is that, as noted in our FAQ’s:

We will begin with the algorithmic search transition, with a goal of completing transition of at least the U.S. market by the end of 2010. We also hope to make significant progress transitioning U.S. advertisers and publishers in 2010 prior to the crucial holiday season, but may wait until 2011 if we determine that the transition will be more effective after the holiday season. All global customers and partners are expected to be transitioned by early 2012.

What do I do now?
This naturally invokes the question, “So what do I, as an advertiser, need to do in the meantime?” The short answer is “nothing.” Keep on doing what you’re doing. During this time, we’ll continue innovating and we’ll ping you when routine updates and enhancements are made available—just as we do now. When the time comes to transition your account to Microsoft adCenter, you’ll be informed about what you’ll need to do, with instructions on how to do it.

What about all of Yahoo’s bells and whistles?
Several of you asked, in short, if all the bells and whistles that you have come to expect from Yahoo! will be ported over to your future Microsoft adCenter account—things like negative keywords, character length of ads, and so forth. Our answer:

As you might expect, Yahoo!’s Sponsored Search platform and Microsoft’s adCenter platform are not identical, but they offer many similar features, tools and reports. Before any advertiser transitions begin, Yahoo! and Microsoft will work together so that our advertisers have a high-quality product that is as familiar as possible and easy to use.

Finally, here are answers to some of your very specific questions:

Nick Gowdy asks: Are we still going to be able to choose the networks on which we advertise? E.g. Yahoo! Search, Bing Search, Yahoo! Search Partners, Bing Search Partners, etc.

Answer: Once Yahoo! Sponsored Search advertisers are transitioned to Microsoft’s search platforms, advertisers will not be permitted to select traffic solely from Bing or solely from Yahoo!

Avromie asks: Anybody know what will happen with SearchMonkey[http://developer.yahoo.com/searchmonkey/] results?

Answer: Yahoo! and Microsoft are sharing ideas for how to advance the SearchMonkey vision of building an ecosystem for developers, publishers, and the semantic web. The landscape is complex, so we’re working hard to determine which path provides the best value for site owners and end users.

SEM All Stars asks: I use YWA [Yahoo! Web Analytics[http://web.analytics.yahoo.com/]]. Is that still going to work?

Answer: Yahoo! and Microsoft plan to provide high-quality, competitive search analytics, but the details are still being determined. As transition plans are established, we will communicate more detail about what you can expect in terms of analytics tools.

Jo asks: Why not give people the choice—those who want to switch can, but those who want to stay with YSM should be able to have that option as well. Is this possible?

Answer: Sorry, no: All Sponsored Search advertisers will use the adCenter platform for their paid search campaigns following their transition, and the “Panama” system will no longer be available. The post-transition unified search marketplace will provide numerous benefits to advertisers, including increased volume, greater ease of doing business, and increased innovation.

Brett West asks: When the alliance takes effect, will it be necessary to maintain advertising accounts with both Yahoo! and Bing/Microsoft or will they be merged somehow?

Answer: Approximately three months in advance, advertisers will receive multiple communications from Yahoo! and Microsoft that will provide details on what to expect during the transition process. When the transition date arrives, Yahoo! and Microsoft will work together to transition your account to the adCenter platform. You will have flexibility to keep your existing adCenter accounts or create new accounts for the unified search marketplace.

Was your question not answered?
As the agreement between Yahoo! and Microsoft only recently received regulatory clearance, we have not had the opportunity to fully discuss every one of your questions with Microsoft and make a determination on each. But we will certainly provide more details to both agencies and advertisers before the transition occurs. Stay tuned.

For more, visit the Yahoo! Transition Center.

— The Team