Search Engine Optimization Demystified
Part 3 - Road Blocks to Search Engines
In the first part of this series Search Engine Optimization Demystified - Part One, we examined the players in the search engine optimization game. Understanding the industry helps us to see through the hype and separate the actual workings of the search engines from the mythology popularized by those looking to make a quick buck off of SEO services.
In the second part of the series Search Engine Optimization Demystified - Part Two, we looked at how the search engines see your web site. We went through ranking, indexing and the rules to follow to get the best results for excellent natural search engine results.
In this, the third article in this series I will explain how search engines can be prevented from ever getting to your site and how they can be restricted from parts of your web site through bad web development practices. This results in poor search engine ranking and low natural search traffic levels.
There are ways that the informed webmaster can intentionally instruct search engine robots not to index a web site. There are some good reasons to do this. For example, we have a client who only serves the government on construction projects and only uses his web site as an addendum to his bids. He does not want the general public finding his site since answering calls from the public would be a nuisance.
We added this Meta tag to the code of his site:
It does what it looks like it does; it instructs the search engine "robots" not to index this page or follow the links from it.
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Now before you get paranoid - I have never seen this done maliciously, but if you have a robots meta tag on your site it should say:
Since search engines assume this to be the default setting, it is not really necessary to have this tag on your site.
<META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">
There are also meta tags that can tell the search engines what kind of content to expect on this web site. This may reduce traffic intentionally, for instance:
This tag is used by search engines to filter search results for people who set their search preferences to exclude explicit content.
< META NAME ="AUDIENCE" CONTENT="ADULT">
You can see the META tags that have been used on your web site by looking at the source code of the site through your browser. For example, with Internet Explorer choose "Source" from the "View" menu. The meta tags will be in the top part of the file between the
tags - if you hit the
tag you've gone too far.
A frame set is a set of web pages coded so that parts of your screen show content from different files. This used to be an easy way for an amateur to place a navigation bar on all of the pages of a site. Frame sets are not used by professional webmasters anymore due to the negative impact they have with search engines. Since frame sets first came out in the mid-1990's, coding has evolved to the point where webmasters can develop web sites faster and more easily without the need for frame sets.
The first page of the site is the most heavily weighted page by search engines. Weighting is done according to various factors, including the amount of relevant text on the page. Since the first page of a frame set is just a set of code instructions to the browser on how to find and use the files for the next page, what a frame set does is to "demote" all of the pages in a site to one level down in the page hierarchy. Search engines use the page hierarchy to determine the importance of the page within your web site. If your first page has no content on it, it will rank poorly, and if all other pages are secondary to the first page they will be ranked even lower.
A typical frame set page looks something like this in the code view:
<TITLE>A simple frameset document</TITLE>
<FRAMESET cols="20%, 80%">
<FRAMESET rows="100, 200">
As you can see there is absolutely nothing in the way of actual words in the page that may be indexed by the search engines. You can understand now why utilizing a frame set would depress your search engine rankings.
In addition, search engines like to follow a nice set of hierarchical links in order to index the content of the site. While your menu with all of your links may be contained in frame1.html in the code view above, you are making the search engine go that extra step to find that menu. As we already mentioned, this is effectively how the frame set knocks everything down one level and makes the search engine jump through an extra hoop to get to your content. Search engines expect us as site owners to jump through extra hoops to appease them - they don't like it at all when the tables are turned and will rank your site accordingly.
Media-Rich Home Pages
Media-rich home pages contain Flash, images, video and other multimedia files as the primary content. If you only remember one thing about search engines, remember this: search engines can only read text.
Search engines cannot read images or text written on images. They cannot read text inside of Flash movies or understand voiceover commentary from a video. Pdfs are often encoded as jpeg images and cannot be read by search engines when saved in this format. Search engines can only read text contained in HTML and in meta tags (which are specifically written for search engines).
One common misuse of Flash is the "Splash" or "Landing" page. This is a page that is just a picture or movie with only an "Enter Here" link on it. Since search engines cannot read the content of an image or Flash movie, this page looks blank to them. If the "Enter Here" link is also coded inside the Flash movie the search engines cannot see the link and will not be able to get to any of the pages inside your site regardless of whether they employ Flash or HTML.
Some web sites are written entirely in Flash and are not accessible to search engines as is. There are a few ways around the Flash barrier, like creating an HTML version of the site with meta tags and using additional navigation links in HTML. But why place a barrier to search engines on your site in the first place? We recommend using Flash for decorative purposes. We recommend that it only be used in such a way that if it were removed, search engines would still have everything necessary to index the site correctly.
To test your site's accessibility to search engines check if you can copy and paste the text from the web browser to your text editor. If the text is written in HTML you will be able to do this. If you can't, it is likely that search engines will not be able to read this text. A site done entirely in Flash or with images is usually the result of a company web development effort driven primarily by graphic design personnel with little or no input from web development professionals. If you are developing a web site for business rather than for a movie, an event, or an art exhibit, you are better off listening to the advice of web development professionals who are trained in SEO concepts for maximum ROI through better search engine placement.
Other Landing Pages
A landing page may not be media rich, but may have very little content on it in order to "direct" users to a specific location on the site. In Canada we often see landing pages directing users to click on links for either French or English versions of the site. This can be avoided by using scripts that detect the default language of the browser and direct the user to their preferred language without the need for them to click a link. Using language specific web site addresses is another good practice. Alternatively put the link to the choice of language on every page of the site in the navigation. This allows users to switch languages from any page on the site. This is important once you realize that search engines often display internal pages rather than the home page of a site on their search result pages. Well optimized web sites avoid low content landing pages.
Dynamic Content and Menus
Search engines cannot read text that is dynamically created when a visitor asks for it. A search engine will follow all of the links it can see on a web site. But a search engine will not type in search terms in a "search box" to see what other content you have in your database. If you have a database driven web site you must have "hard coded" links to the data that the search engine can follow, or much of your web site will be not be indexed.
As an aside, java script drop down menus are usually also less human user-friendly due to the fact that they difficult to manually operate, and they provide no navigational reference point since the drop down 'snaps back up' disappearing as soon as it is clicked. While there are again various methods to "get around" this obstacle, these are stop-gap measures that can and should be avoided.
Remove the road blocks!
As we have seen, there are many web site design practices commonly in use today that put up barriers to search engines. The more experience your webmaster has in SEO the less likely they are to engage in practices that can confuse or mislead search engine spiders. But you, as a web site owner, must make it clear to the developer of your site that SEO is important to you. SEO is time consuming and requires training. Unless you request a search engine friendly web site - and pay for one - you are not likely to get one. If you have any of the above barriers in place on your site, we encourage you to give us a call today to find out how we can get you out of the search engine quagmire that you are likely in.
By Candace Carter, Back2Front - The Web Site People. December 2008