steve
/
factor
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

modern: If we find whitespace, there was no delimiter, so it's just a token.

modern-harvey3
Doug Coleman 2019-10-17 00:48:52 -05:00
parent 2c6d958030
commit 4ef0a41d05
1 changed files with 12 additions and 18 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

30
extra/modern/modern.factor
View File

@ -358,18 +358,10 @@ DEFER: lex-factor-top*
over "\\" tail? [ drop (read-backslash) ] [ lex-factor-top* ] if over "\\" tail? [ drop (read-backslash) ] [ lex-factor-top* ] if
] if ; ] if ;
! If the slice is 0 width, we stopped on whitespace. ! If the slice is 0 width, we stopped on whitespace before any token.
! Advance the index and read again! ! Return it to the main loop as a ws form.
: read-token-or-whitespace ( string n slice -- string n' slice/f )
: read-token-or-whitespace-top ( string n slice -- string n' slice/f )
dup length 0 = [ dup length 0 = [
! [ 1 + ] 2dip drop lex-factor-top
merge-slice-til-not-whitespace <ws>
] when ;
: read-token-or-whitespace-nested ( string n slice -- string n' slice/f )
dup length 0 = [
! [ 1 + ] 2dip drop lex-factor-nested
merge-slice-til-not-whitespace <ws> merge-slice-til-not-whitespace <ws>
] when ; ] when ;
@ -418,9 +410,11 @@ DEFER: lex-factor-top*
[ slice-til-whitespace drop ] dip span-slices [ slice-til-whitespace drop ] dip span-slices
dup section-open? [ rewind-slice f ] when dup section-open? [ rewind-slice f ] when
] } ] }
{ char: \s [ read-token-or-whitespace-nested ] }
{ char: \r [ read-token-or-whitespace-nested ] } ! Two cases: zero width slice if we found whitespace token, otherwise text token
{ char: \n [ read-token-or-whitespace-nested ] } { char: \s [ read-token-or-whitespace ] }
{ char: \r [ read-token-or-whitespace ] }
{ char: \n [ read-token-or-whitespace ] }
[ lex-factor-fallthrough ] [ lex-factor-fallthrough ]
} case ; } case ;
@ -445,10 +439,10 @@ DEFER: lex-factor-top*
dup section-open? [ read-acute ] when dup section-open? [ read-acute ] when
] } ] }
! Two cases: zero width slice if we found whitespace, otherwise token ! Two cases: zero width slice if we found whitespace token, otherwise text token
{ char: \s [ read-token-or-whitespace-top ] } { char: \s [ read-token-or-whitespace ] }
{ char: \r [ read-token-or-whitespace-top ] } { char: \r [ read-token-or-whitespace ] }
{ char: \n [ read-token-or-whitespace-top ] } { char: \n [ read-token-or-whitespace ] }
[ lex-factor-fallthrough ] [ lex-factor-fallthrough ]
} case ; } case ;