steve
/
factor
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Fix the max-depth variable in spider

db4
Doug Coleman 2010-08-29 13:21:50 -05:00
parent 720d634388
commit 74f3579644
1 changed files with 21 additions and 10 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

31
extra/spider/spider.factor
View File

@ -8,9 +8,23 @@ continuations calendar prettyprint dlists deques locals
spider.unique-deque combinators concurrency.semaphores ; spider.unique-deque combinators concurrency.semaphores ;
IN: spider IN: spider
TUPLE: spider base count max-count sleep max-depth initial-links TUPLE: spider
filters spidered todo nonmatching quiet currently-spidering base
#threads semaphore follow-robots? robots ; { count integer initial: 0 }
{ max-count number initial: 1/0. }
sleep
{ max-depth integer initial: 0 }
initial-links
filters
spidered
todo
nonmatching
quiet?
currently-spidering
{ #threads integer initial: 1 }
semaphore
follow-robots?
robots ;
TUPLE: spider-result url depth headers TUPLE: spider-result url depth headers
fetched-in parsed-html links processed-in fetched-at ; fetched-in parsed-html links processed-in fetched-at ;
@ -22,11 +36,8 @@ fetched-in parsed-html links processed-in fetched-at ;
over >>currently-spidering over >>currently-spidering
swap 0 <unique-deque> [ push-url ] keep >>todo swap 0 <unique-deque> [ push-url ] keep >>todo
<unique-deque> >>nonmatching <unique-deque> >>nonmatching
0 >>max-depth
0 >>count
1/0. >>max-count
H{ } clone >>spidered H{ } clone >>spidered
1 [ >>#threads ] [ <semaphore> >>semaphore ] bi ; 1 <semaphore> >>semaphore ;
: <spider-result> ( url depth -- spider-result ) : <spider-result> ( url depth -- spider-result )
spider-result new spider-result new
@ -91,9 +102,9 @@ fetched-in parsed-html links processed-in fetched-at ;
now >>fetched-at drop ; now >>fetched-at drop ;
:: spider-page ( spider spider-result -- ) :: spider-page ( spider spider-result -- )
spider quiet>> [ spider-result print-spidering ] unless spider quiet?>> [ spider-result print-spidering ] unless
spider spider-result fill-spidered-result spider spider-result fill-spidered-result
spider quiet>> [ spider-result describe ] unless spider quiet?>> [ spider-result describe ] unless
spider spider-result add-spidered ; spider spider-result add-spidered ;
\ spider-page ERROR add-error-logging \ spider-page ERROR add-error-logging
@ -108,7 +119,7 @@ fetched-in parsed-html links processed-in fetched-at ;
: spider-page? ( spider -- ? ) : spider-page? ( spider -- ? )
{ {
[ todo>> deque>> deque-empty? not ] [ todo>> deque>> deque-empty? not ]
[ [ todo>> peek-url depth>> ] [ max-depth>> ] bi < ] [ [ todo>> peek-url depth>> ] [ max-depth>> ] bi <= ]
[ [ count>> ] [ max-count>> ] bi < ] [ [ count>> ] [ max-count>> ] bi < ]
} 1&& ; } 1&& ;