factor

Author	SHA1	Message	Date
Doug Coleman	12b6fc17ea	robots: Rename	2013-03-24 10:11:18 -07:00
John Benediktsson	fd853f9958	Cleanup some lint warnings.	2011-10-15 19:19:44 -07:00
Doug Coleman	b2bf680959	allow robot-identifiers to be set for robots.txt	2009-06-19 14:41:48 -05:00
Doug Coleman	65802b6aaa	fix unit test in robots	2009-04-07 10:08:01 -05:00
Doug Coleman	b6064813c9	construct a glob-matching quote for robots.txt files	2009-04-06 17:39:44 -05:00
Doug Coleman	c626cab383	a robots.txt parser	2009-03-20 21:56:48 -05:00