steve
/
factor
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

peg.javascript.tokenizer: allow escaped quotes in strings 

Previously, the tokenizer didn't understand
escaped quotes in string literals. Also added was
a test to ensure the escaping works.
locals-and-roots
catb0t 2016-06-11 17:36:14 -04:00 committed by John Benediktsson
parent 6984bae7ca
commit 385cf35e7f
2 changed files with 11 additions and 6 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
extra/peg/javascript/tokenizer/tokenizer-tests.factor
View File

@ -23,3 +23,7 @@ IN: peg.javascript.tokenizer.tests
{ V{ T{ ast-regexp f "<(w+)[^>]*?)/>" "g" } } } [ { V{ T{ ast-regexp f "<(w+)[^>]*?)/>" "g" } } } [
"/<(\\w+)[^>]*?)\\/>/g" tokenize-javascript "/<(\\w+)[^>]*?)\\/>/g" tokenize-javascript
] unit-test ] unit-test
{
V{ T{ ast-string { value "abc\"def\"" } } }
} [ "\"abc\\\"def\\\"\"" tokenize-javascript ] unit-test

7
extra/peg/javascript/tokenizer/tokenizer.factor
View File

@ -48,9 +48,10 @@ Name = !(Keyword) iName => [[ ast-name boa ]]
Number = Digits:ws '.' Digits:fs => [[ ws "." fs 3array "" concat-as string>number ast-number boa ]] Number = Digits:ws '.' Digits:fs => [[ ws "." fs 3array "" concat-as string>number ast-number boa ]]
| Digits => [[ >string string>number ast-number boa ]] | Digits => [[ >string string>number ast-number boa ]]
EscapeChar = "\\n" => [[ 10 ]] EscapeChar = "\\n" => [[ CHAR: \n ]]
| "\\r" => [[ 13 ]] | "\\r" => [[ CHAR: \r ]]
| "\\t" => [[ 9 ]] | "\\t" => [[ CHAR: \t ]]
| "\\\"" => [[ CHAR: \" ]]
StringChars1 = (EscapeChar | !('"""') .)* => [[ >string ]] StringChars1 = (EscapeChar | !('"""') .)* => [[ >string ]]
StringChars2 = (EscapeChar | !('"') .)* => [[ >string ]] StringChars2 = (EscapeChar | !('"') .)* => [[ >string ]]
StringChars3 = (EscapeChar | !("'") .)* => [[ >string ]] StringChars3 = (EscapeChar | !("'") .)* => [[ >string ]]