Robots In the Sandbox |
Programs which automatically surf the web are known as robots; a search engine crawler is a variety of robot. Crawlers are also sometimes called spiders. (What else would you find crawling around a web?)
Here is a list of recent visits by the crawlers of some well-known search engines to the Computer Science Department's web server. This page is generated from recent entries in the web server logs. Downloads are counted, and several of the most recent ones are listed.
For Google (29 hits):
3:10:25 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/ruby/code/ruby.css |
3:05:46 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/114_common/labs/python/p2.css |
3:02:18 am Mar 29, 2024 | http://sandbox.mc.edu/phpBB3/styles/prosilver/theme/links.css |
3:02:18 am Mar 29, 2024 | http://sandbox.mc.edu/phpBB3/styles/prosilver/theme/buttons.css |
3:02:17 am Mar 29, 2024 | http://sandbox.mc.edu/phpBB3/styles/prosilver/theme/colours.css |
3:02:16 am Mar 29, 2024 | http://sandbox.mc.edu/phpBB3/styles/prosilver/theme/responsive.css |
3:02:16 am Mar 29, 2024 | http://sandbox.mc.edu/phpBB3/styles/prosilver/theme/icons.css |
3:02:15 am Mar 29, 2024 | http://sandbox.mc.edu/phpBB3/styles/prosilver/theme/base.css |
3:02:15 am Mar 29, 2024 | http://sandbox.mc.edu/phpBB3/styles/prosilver/theme/forms.css |
3:02:14 am Mar 29, 2024 | http://sandbox.mc.edu/phpBB3/styles/prosilver/theme/content.css |
… |
For Bing (105 hits):
3:22:07 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/cs302/php/area.php |
3:21:41 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/cs302/php/area.php |
3:21:25 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/cs302/php/area.php |
3:21:16 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/cs302/php/area.php |
3:21:10 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/cs302/php/area.php |
3:21:06 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/cs302/php/area.php |
3:19:23 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/cs302/php/area.php |
3:19:06 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/cs302/php/area.php |
3:17:36 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/cs302/php/area.php |
3:17:24 am Mar 29, 2024 | http://sandbox.mc.edu/~bennet/cs302/php/area.php |
… |