Keyword Basics Part 1: How Search Engines Work by Ken McGaffin, 24 November 2006
One of the most frequently asked questions from website owners is, "Why can't my site be found on Google?" They know it's important to appear in search engine results but they just don't know why it doesn't happen to them.
Key points
- The Spider software crawls the web looking for new pages to collect and add to the search engine indices.
- The Index software catches everything the Spider can throw at it.
They may well be in awe of the 'black arts' of search engine optimization or puzzled by the complexity of it all. If they're unlucky they will have paid out money to some snake oil salesman guaranteeing to get them to the top in 48 hours - and been sorely disappointed with the lack of traffic that results.
Most search engine optimizers are highly ethical, professional people but they do tend to keep their cards close to their chest.
The big secret is there is no big secret
It's true: the 'big secret' of search engine optimization is that there is no big secret.
It is all about understanding what is going on, followed by the hard work and attention to detail that are common to many business activities.
This article is about understanding what is going on inside a search engine. It is just under 800 words in length but that is more than enough to give you what you need to know, albeit in simple terms.
Inside the guts of a search engine
For simplicity's sake, let's say there are three pieces of software that together make up a search engine - the Spider software, the Index software and the Query software.
If you understand what these three do, then you have the foundation for getting your website to the top of the search engines.
Here's what the three types of software do:
The Spider software 'crawls the web looking for new pages to collect and add to the search engine indices'.
This is a metaphor. In reality, the spider doesn't do any 'crawling' and doesn't 'visit' any web pages. It requests pages from a website in the same way as Microsoft Explorer, or Firefox or whatever browser you use requests pages to display on your screen.
The difference is that the spider doesn't collect images or formatting - it is only interested in text and links AND the URL, (for example, http://www.Unique-Resource-Locator.html) from which they come: it doesn't display anything and it gets as much information as it can is the shortest time possible.
A spider loves links because they lead it to other web pages that have the things that it loves, guess what? Text, links and URLs!
The Index software catches everything the Spider can throw at it (yes, that's another metaphor). The index makes sense of the mass of text, links and URLs using what is called an algorithm - a complex mathematical formula that indexes the words, the pairs of words and so on.
Essentially, an algorithm analyses the pages and links for word combinations and assigns scores that allow the search engine to judge how important the page (and URL) might be to the person that is searching.
And of course it stores all of this information and makes it available.
The Query software is what you see when you go to a search engine - it is the front end that everybody thinks of as a search engine - familiar ground at last. It may look simple but it presents the results of all the quite remarkable search engine software that works away invisibly on our behalf.
The main feature of the query software is the box into which people type their search terms.
Type in your words, hit search and the search engine will try to match your words with the best web pages in can find through searching the web.
But this too is a metaphor and perhaps the most important one.
The query software doesn't search the web - it checks the records that have been created by its own index software. And those records have been made possible by the raw material the spider software collects.
What you need to understand about search engines
That is it. What you need to understand is that the search engine has done all the hard work of collecting and analysing web pages, BUT it only makes that information available when someone does a search by entering words in the search box and hitting return.
The words people use when they search therefore determine the results the search engine presents. So search engine optimizers want to know the words people use when they search - we call them keywords - that might sound fancy but keywords are only 'the words people use when they search'.
And that's what Wordtracker provides. Use keywords in your website copy and you will prosper: ignore them and your online business will surely perish.
About Ken McGaffin
Ken McGaffin is an experienced internet marketing consultant. He has worked for major pharmaceutical companies, advertising agencies, government bodies and non-profit organizations.
148 comments
Excellent information, thank you
Well done! Written in plain language so I could understand. Thank you.
It's always good to be reminded that sometimes you think things are more complex than they actually are. Beyond generating inbound links, I'm looking forward to finding out more about building a friendly site that can grow our search rankings.
Congrats on the new publication, gents! Good look, great content.
Great article. I needed a refresher and there you are. Looking forward to the next instalment.
It now makes sense. Thankyou
Great article! Knowing the basics of how search engines work explains a lot of other things about the Internet that I did not understand. Thank You Much!
What do you mean by "website copy"
You made sense out of what has been a scarey topic. I can't wait for part two and three! I am so glad I signed up for your newsletter. Thank you, thank you.
You made sense out of what has been a scarey topic. I can't wait for part two and three! I am so glad I signed up for your newsletter. Thank you, thank you.
Good article. Thanks Ken. I can see the articles in the new Word Tracker newsletter will definately be worth reading.
I do SEO writing and I have found that yahoo is more responsive than google i.e. discovers incoming links more quickly, ranks sites higher for targeted keywords more quickly. I have a theory that google is more skewed toward incoming links (rather than keywords distributed through text) but crawls less often. Do you have an opinion?
Dear Ken McGaffin
My name is Jyoth Mahalingam 56 years old and 36 years experienced telecom man. I am into internet since 1995 when it came to india. I have worked as web site marketer and wrote for ebooks and web sites. I am keenly watching this changes in SEO and the dance by the search engines. Your article is simple and very good to follow. I would be too happy if you can make this full article available to me as a ebook. I intend to share what you have written with my own friends to make them understand the basics of internet and SEO. You are doing a good service please enjoy doing it.
Thanking you. Yours sincerely A.Jyothi mahalingam Web writer & Telecom consultant ajm0750@gmail.com
Good information. I needed this. Thank you.
Thank You I enjoyed the article about how search engines work
Thank you for the information, every day i am fighting with this site to get it out there, you see my webmaster is out of town and dont know when he will be back. I want to learn how to work my website copy.
Great article for beginners. Very easy to understand.
Love the new newsletter, keep 'em comming.
i should search for the keyword search option very easy way
Very interesting and useful. Thanks very much.
This is great! Finally an explaination that even I can understand! Thank You
Very informative. Looking forward to reading future articles
realy its great information provided. we are thankfull to you for this infoormation privilages
Warm Regadrs
shekar
Great article...Can't wait for part 2 and 3!!!
Keep 'em coming! others make it sound difficult. your's hit the spot.
I found this information extremely helpful and stated very clearly for a novice like myself.
My son built my webpage and made a brilliant job of it but he is a little impatient with people who don't have an obvous knowledge of basic computor skills? I,m the dummy in the books computors for Dummies.
Thank you for this concise article. Crystal clear - NOW I get it!
I am trying to build a website, and any good information I can gather really helps. Thanks for the really good stuff.
Very informative. Thanks.
How soon is...coming soon?
This was a good piece of info. short and simple. Thanks
I'm having trouble getting good high rating links. I'm finding I'm not sure who to link to and how to obtain the links.
My thanks. This article was concise and informative, clearly explaining the essentials of search engine functionality and how we can use it to advantage with the inclusion of proper keywords in our sites.
Excellent mini-tutorial! Clear and simple. Thanks for your great tools and very useful service.
Ken,
Thank you for that very simple and concise explanation of how search engines work. I'll definitely be looking forward to parts 2 & 3 of your article series.
I am sure they will help me break my addiction to purchasing mysterious software programs that claim to be a panacea for SEO. Thanks.
Ariel Keis
Couldn't have put it better myself.
very helpful!
Wow -- I have been using the Internet extensively since the early 90s, and that's the first time I've ever heard anyone explain these things in this way! Makes things seem a lot simpler. Looking forward to the next articles in this series. Thanks! Rose.
Hi Ken,
Thank you for the explanation, you put it across very well, I'll use this when I explain search engines
Eddie Ewan
This was a great value 800 word document. You did a great job at explaining how things work without getting technical. Thanks
Plain and simple. great job! Look forward to part 2.
I am trying to start a business on the internet but I am completely ignorant of what to do. I don't even know what you want on this page. My email address is tome1231@sbcglobal.net. If anyone needs help I do. Please learn me what to do.
Thanks! Keep sending easy to read valuable search info.
Never seen it written so simply. I am going to refer all my clients to this page. Thanks.
This is very good information. I am looking forward to parts 2 and 3. Thank you.
Well written, straight to the point and on target with what people are really asking. I am particularly interested in long tail keyword phrases and duplicate content vs. original. I'll bet many others are also.
Superb! Step by step, jargon busting wisdom. Love it. Thank you.
Wow, what an eye opener... I wish I know about this a lot earlier. I just put my site up on the 1st of this month. I was just going to look for a way to do some SEO on this site. If you look at it add home.html after the link and you will go to the landing page. I used some key words I got from your site. If you get a chance and do look at it, do you think you can give me some advice on what I could do? Or what to buy or who to buy... lol, to get it up there fast? Great information. MIKE
I've been testing the trial version, one thing annoys me. Not being able to enter my own keywords and go directly to step 4, comparing how many ask for them and how much competition is. I have to go through steps 1, 2 and 3 and in the end the phrase I like to research is not allways on offer in step 3 and thus I can't test it in step 4. Is this different in the pay version?
Also I would like to enter the phrase "travel to" and then get the results what most people search for, be it "travel to venice" or "travel to paris" is that possible?
Wonderful, easy to understand the basics.
Fantastic information easy to understand and follow. I need and want more of this kind of instuction. Thanks for the vine !
Very good article...I cannot wait until Part 2 and 3. It seems as if almost everything I do is all about the fundamentals and basics of life.
Love the metaphores - thanks for the info!
very clearly. i've got all the idea. thank you for sharing your skill with us!
Could you recommend some on-line SEO and SEM courses? I have been self taught doing what you indicated in the artilce above: reading, researhing, submitting to emails and trying out the skills that I learned on a company web site where I was working and also created new SEM programs that were successful but would like to investigate classes - I just need to be pointed in the right direction. Thank you.
Now That makes sense Thank You Let me in one Part 2 Keyword Matrix
Straight forward and informative. Thanks.
please can you tell me what i need to do to get higher positions in the search engines... i will pay for a report telling me exactly whats needed as different people have been trying to juggle it and i dont think its working .. help!!!!!
Does anyone have suggestions about the best way to get this business listed on the search engines? I have paid and that didn't work. I have listed my own info and have yet to get a single customer from a search engine. It's very frustrating....
I found this to be an excellent article in visualizing the different aspects of a search engine - I had not quite looked at it in this way before.
Look forward to the next article! Thanks!
This article makes things much clearer. I will work on these principles when designing my sites. This is great information
Great article for us beginners in SEO. When can we expect part 2 and part 3? I'm chomping at the bit here! Cheers.
Your article is too simple and helped me understand about search engines in just 2 minutes...
Great job,
Keep going
Thanks to some great professional help and guidance from an English man, I am now successfully getting fantastic results from using better “keyword” written copy….wish I had read your great article a year or so ago :)
I have been reading reports & SEO newsletters and trying to improve my existing Realty Web Site Position in the search engines…. I sure would like to find a resource or study that would guide you thru the step by step implementation (development) of a fundamental web site and how the key words could best be placed for maximum effectiveness & best site positioning.
Really nice article. Kudos to writer and wordtracker team.
Cool. Makes a lot more sense now. Thanks!
Thank you.
Guys, for further reading I suggest reading about Google Pagerank formula. For example, this wikipedia article: http://en.wikipedia.org/wiki/PageRank Long story short, having links to your website is very important.
The problem with the plethora of books and articles like this is that they cause more harm than good.
They partialy educate the website owner into believing they are now an IT marketing genius.
If this were possible and true, fortune 500 companies such as HP and microsoft would’nt spend millions on hiring consultants and employees to achieve the great results they have.
I am a IT marketing consultant myself and am shocked when a potential client has an extremly poor ranking, web pressence and design and yet I’m told they know exactly what they are doing because they have read this book or that article.
It’s sad to be living in an age where patients are telling their doctors what meds they need and website owners without a marketing degree insist they can compete with the likes of HP, microsoft and other companies that hire PROFESSIONALS. :)
What a joke.
WOW! What an eyeopener. LOVE wordtracker academy.
Excellent. Nice work and thanks for being so simple and descriptive. Cheers
it’s nice information, thank’s
Thank you
For hepling me understand
keyword and how it is use. I had
given up on the idea until I found
wordtracker and you.
bernievon, 09/08/07
Thank you
For hepling me understand
keyword and how it is use. I had
given up on the idea until I found
wordtracker and you.
bernievon, 09/08/07
Ken:
From what I can read, it is well written.
I am partially blind.
If you would be willing to give me a few pointers for clarification, it would be most appreciated.
Thanks
Lonnie dokteur@gmail.com
Very interesting and informative info..Thanx a lot
Thank you Ken for this invaluable insight. Only when when is a true master of a subject can he explain it so clearly in words and pictures that even a little child can understand
thanks of lot ken for knwledgable information.i have just started SEO as a trainee but ithink there is a lot of information available on net but ur article is so valuable for me can u send me some more information as i can start basic work on seo on my Email-ID.
great article. Thanks for the information.
thank you for this information…
thank you for this information…
Thank You for another very interesting article. It’s really good written and I fully agree with You on main issue, btw. I must say that I really enjoyed reading all of Your posts. It’s interesting to read ideas, and observations from someone else’s point of view… it makes you think more. So please try to keep up the great work all the time. Greetings
I have been on a steep learning curve all year since I decided to become an internet marketeer. I have found that keyword optimization is an exact art and there are few short cuts, however there is help if you know where to find it.
This course has been a great help.
Many thanks
Dennis Nicholson
Keywords are greatly important but other things ought not to be forgotten. We changed to php, otherwise keeping the content the same including the meta tags, but lost half our visitors. So watch out!
This article has helped me to improve my website. Thanks a lot!
Crystal clear and extremely helpful....Thanks
Three metaphors are nice examples. Thanx
You guys reaffirmed everything I thought to be true about optimization for the best site. Thank you!
Extremely helpful information. I was referred to this article by one of my friends, it definitely is a great starter..
Humayun Sohel http://www.questivity.com
Great Tips! Thank you!!
Thanks for very interesting article. btw. I really enjoyed reading all of your posts. It’s interesting to read ideas, and observations from someone else’s point of view… makes you think more.
Yhank you vaery much for this article.
I’ve been thinking lately that being frugal is not a bad idea. My credit history is not that great mostly because of lat payments on credit cards. But I still feel like the only way to rebuild it is to get another credit card and quickly pay it back. Which one do you feel will work among those at
discover no annual fee credit cards no fees universal chase discover
This article makes things much clearer. I will work on these principles when designing my sites. This is great information
Very well explained. This article gave me an idea how the SE works. Thanks a lot.
Thank you for the insightful information into how search engines work!
Great info, I think we do forget from time to time that google for example is not searching the web but its own database.
I have tried and used nearly every KW tool that has been made available over the years but Word Tracker most certainly has survived the test of time and advancement.... This is still by far the most effective kw tool I have ever used for the United States.
Regards
Great article, very insightful and straight to the point. Allows all level of online users understand how the search engine works.
great info. Thanks for sharing.
Wow, great information about search engines working. This is first time I am knowing this type of information. This will be very useful to me and every online marketer.
A very useful explanation of the basic mechanics. Thanks - Ewan Kennedy.
I feel like I finally found the information I needed to help me run my site and get it out of the dark place to the light.
Extremely helpful information. I was referred to this article by one of my friends, it definitely is a great starter..great future
This article makes things much clearer. I will work on these principles when designing my sites. This is great information!
Have a nice Day, nice greetings from Nehemia
Some great tips thanks. We sometimes forget the importance of good quality content and relevant keywords on our sites :-)
Thanks for explaining it so simply most of the guides I've read just blind you with science. I will concentrate on my main keywords and see how i get on, thanks Wordtracker !
Such a simple way to explain a complex subject.Thanks
Hi, my name vipin i so seo, i know how important is this information, good post..................
I think, too many keywords on a site spread the power of the site. One special keyword means, that you have all the power pointing to this word. Many keywords mean, that only a little power points to each word, when searching in a search engine for one of those words. Anyway. Thanks for this description.
The search engine spiders have problems with java, and flash-only pages. They can't read its content. And be aware of javascript links! Spiders usuallay can't follow them.
Terrific article, Ken, thanks - finally the search engines made easy to understand. Very enlightening.
David Portney President, American Seminar Academy www.SeminarAcademy.com
Hi Ken, I appreciate that you write your articles in an understandable manner but don't make me feel like the newbie I am. I have bee enjoying the first 6 keyword articles and find the comments and questions very helpful. The one thing that I may be missing is the answers to some of the questions. As you know for everyone who takes the step to ask there are a ton of us who are chicken to ask. Since I am new to this stuff (nice word) I may need to look somewhere else for the answers. I will be watching for the answer. Thanks again, NEIL
Good information given here which will help people alot.
Thanks, this is useful information - it is nice to see a common sense, easy to understand explanation.
Hello,
I would like to know how come my Website doesn't show on a search?
Thank You Very Much for helping Me.
Sylvie
Hmmm, I was sceptical before...
After I have some time to digest the valuable information you have generously provided, I am sure I will be able to put it into practice. Wish me success! Thank you again for sharing your SEO wisdom. Daniel Schainen
I've been looking for a simple way to explain SEO to a technology challenged manager. This is it!
But what is about the PageRank? A half year ago I had a PageRank 5, then suddenly 3 and now my PageRank is 0. I haven't change anything. I only added products.....
great info. Thanks
merten: Hard to say without looking at your site in detail. You might not have changed anything but Google is always changing so that what is ok with Google one day gets you 'slapped' the next.
Hi Ken,
It's been so helpful to know this basic knowledge you've written. I honestly say that I've already doing some traffic method for my website but forgotten to know the basic. It really helps me realized now and know how keyword research really works.
More power to you and I would say this is the start for me to really be successful int generating traffic for my site! Thank you once again.
Hi Ken,
i think this is one of the best articles i have ever read and you convinced me to continue reading the rest of your articles, Great Job!
I follwed your fourmula and I got better rank for my site http://www.agranews.com
Thanks
Now I am happy in Google search etc for www.agranews.com
Really good advise.
Foundation, Basics, Basics and more basics is the key to everything in life. Nice article! Simple but very informative and very strong. Thanks.
Regards Crewlane Technologies dot us
I have recommended Keyword tracker to most of my friends, Although you people are too lazy to do most things but if they do, it will surely help. A good weapon does not make a good soldier. Thanks again.
Regards Prince Crewlane Technologies
Excellent tool! Just using the free version now but I am 99% sure i'll be keeping it on and purchasing. The pricing is a little crazy though, I will almost pay double as I can't afford to buy 1 full years subscription and will have to pay monthly.
Hello Ken, Thanks for the help and the explanation. The reason I've been reading your article on 'How Search Engines Work' is because my website on the face of it seems successful. All my website product front pages seem to appear on the 1st page of the major search engines using the relevent keywords. All except MSN and I can't find my website product front pages anywhere. I am able to find our website Home page on MSN for each product. That appears much further back, sometimes appearing twice on the page, sometimes on page 20 or so. Sometimes, even nowhere. Using your article will give me good ammunition to help sort out the problem. Once again thank you.
Dave Badcock of Gifts Are Us
Hi,Ken. Thanks for the imformation,i think i should to read the reamin Parts!Cool article.
Ken, are you married?!!!?? :) Wow, what a great communicator! Only my 'contact' pages are up right now for my sites as full development is underway. My fortune cookie the other night said that 'a Ken Mc will be your SEO angel'....soooo, where you be? Excellent. www.interiordesignersla.com
Hey Ken- Im a talent agent in Hollywood who's starting to really focus my clients on the web. Id love to do a great web site with John Sally (the ex basketball star)..
Anyone have any specific ideas how to monetize a site with him?. Your articles and so fascinating.. !...
Thx Steve
Thank you so much for this information. I am recently disabled, run over by a car, and looking to try and make a living from home. This information was very helpful. MiChal
Dear Sir a wonderful article which will not only help amatures but experienced people also. Venkatteshprasanna
There where good news from Matt Cuts, so Google can read flash files....
After a lot of time, google as the first search engine can read flash files. So i think, that this is a really really good thing. Now nobody must have a table or css (div) website, to get good positions in the search engine.
Thank you for explaining search engines in "lay mans" language. This helps me a great deal.
Very good artice. Good to understand. Thanks for the work.
My comment is more of a question:
1) If your title is: How to grow the perfect red rose.
2) Your description is: Learn the fast and easy way to grow the perfect red rose.
3) Your content is all about learning how to grow the perfect red rose the fast and easy way.
Then why does a person's website that is "NOT SEO" and talks about red toilet paper come up on the 1st page of a search engine (Google) and the fully SEO website comes up on the 14th page even though the query that was typed in was: red roses? By the way - in case you haven't guessed this is real and happens all the time.
Thank you Allen
I have found some helpful and great tips here that make a lot of things easier to understand. I will try it on my homepage and I'll see, if it works.
KEN: I am a novice marketeer just starting out in my own business. I am still "lost" on most of this stuff, but after reading articles like yours, it surely helps me understand the nuts and bolts better. Thanks for a wonderful ariticle that certainly opened my eyes. Mike
It is well explained in plain english. Very informative. Thank you.
Reading and researching before challenging myself to Affiliate Marketing! Thanks, after people trying to hoodwink you into purchases (extremely expensive) on how to learn. I think this is the jackpot. Thank you - fantastic to find people who still have morals and integrity. Website: Under construction - Actually, learning phase...
Goog infos and easy to follow the instuctions. Thanks for this!
Very interesting and useful. Thanks very much.
Baldur Bjarnason
What search engines does Wordtracker get it's results from and how often does it update it's database files? Is there a margin of error %?
Webtraffic: see here
Hi Ken
This article is an absolute goldmine! Takes the mystery out of what can seem to be a mysterious and impossible to crack world of keywords and search engines.
Thanks for sharing this great info.
Angela Baden
Great stuff! I signed up for the 7 day and after several minutes I am in love!!! My site www.whitebirchstudios.com ranks for words like exterior rendering, but I am going to use your app to find those keywords that people are actually typing in to find architectural renderings and illustrations.
Again, thanks.