WebMaster World: The Inner Workings of Robots, Spiders, and Web Crawlers

<body valign="top"><script type="text/javascript"> function setAttributeOnload(object, attribute, val) { if(window.addEventListener) { window.addEventListener('load', function(){ object[attribute] = val; }, false); } else { window.attachEvent('onload', function(){ object[attribute] = val; }); } } </script> <div id="navbar-iframe-container"></div> <script type="text/javascript" src="https://apis.google.com/js/platform.js"></script> <script type="text/javascript"> gapi.load("gapi.iframes:gapi.iframes.style.bubble", function() { if (gapi.iframes && gapi.iframes.getContext) { gapi.iframes.getContext().openChild({ url: 'https://www.blogger.com/navbar.g?targetBlogID\x3d9311008\x26blogName\x3dWebMaster+World\x26publishMode\x3dPUBLISH_MODE_BLOGSPOT\x26navbarType\x3dBLUE\x26layoutType\x3dCLASSIC\x26searchRoot\x3dhttps://webmasterworld.blogspot.com/search\x26blogLocale\x3den_US\x26v\x3d2\x26homepageUrl\x3dhttp://webmasterworld.blogspot.com/\x26vt\x3d-1617876758125134333', where: document.getElementById("navbar-iframe-container"), id: "navbar-iframe" }); } }); </script>

Wednesday, December 01, 2004

The Inner Workings of Robots, Spiders, and Web Crawlers

Robots, a.k.a. spiders or Web crawlers are used to feed data to a user, such as Google. Most robots are useful, but some, such as EmailSiphon and Cherry Picker are spambots. Here, you'll learn what robots do and how to gain control over them.

There are three basic types of search engines: crawler-based, human-powered, and a combination of both.