John Benediktsson
|
712be93989
|
http.client: remove http-get* and its friends, change http-request* and with-http-request* to not check response codes.
|
2014-03-12 19:39:15 -07:00 |
|
John Benediktsson
|
435f54cf59
|
http.client: use new http-get* word.
|
2013-10-12 10:25:42 -07:00 |
Doug Coleman
|
b491b5aeb6
|
robots: Rename
|
2013-03-24 10:11:18 -07:00 |
|
John Benediktsson
|
6f1c44b46d
|
Cleanup some lint warnings.
|
2011-10-15 19:19:44 -07:00 |
|
Doug Coleman
|
1e14a83ee1
|
allow robot-identifiers to be set for robots.txt
|
2009-06-19 14:41:48 -05:00 |
Doug Coleman
|
65802b6aaa
|
fix unit test in robots
|
2009-04-07 10:08:01 -05:00 |
|
Doug Coleman
|
b6064813c9
|
construct a glob-matching quote for robots.txt files
|
2009-04-06 17:39:44 -05:00 |
|
Doug Coleman
|
c626cab383
|
a robots.txt parser
|
2009-03-20 21:56:48 -05:00 |