steve
/
factor
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

simple-tokenizer: consider \t \n \r spaces also.

db4
John Benediktsson 2015-04-19 20:47:55 -07:00
parent 56bb141257
commit a30c0c0946
2 changed files with 7 additions and 2 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

5
basis/simple-tokenizer/simple-tokenizer-tests.factor
View File

@ -31,3 +31,8 @@ USING: simple-tokenizer tools.test ;
] [
"\"Hello world.app/Contents/MacOS/hello-ui\" -i=boot.macosx-ppc.image \"-include= math compiler ui\" -deploy-vocab=hello-ui \"-output-image=Hello world.app/Contents/Resources/hello-ui.image\" -no-stack-traces -no-user-init" tokenize
] unit-test
{ V{ "ls" "-l" } } [ "ls -l" tokenize ] unit-test
{ V{ "ls" "-l" } } [ "ls -l\n" tokenize ] unit-test
{ V{ "ls" "-l" } } [ "\nls -l" tokenize ] unit-test
{ V{ "ls" "-l" } } [ "\nls -l\n" tokenize ] unit-test

4
basis/simple-tokenizer/simple-tokenizer.factor
View File

@ -4,10 +4,10 @@ USING: peg.ebnf strings ;
IN: simple-tokenizer
EBNF: tokenize
space = " "
space = [ \t\n\r]
escaped-char = "\" .:ch => [[ ch ]]
quoted = '"' (escaped-char | [^"])*:a '"' => [[ a ]]
unquoted = (escaped-char | [^ "])+
unquoted = (escaped-char | [^ \t\n\r"])+
argument = (quoted | unquoted) => [[ >string ]]
command = space* (argument:a space* => [[ a ]])+:c !(.) => [[ c ]]
;EBNF