factor/extra/infix/tokenizer/tokenizer.factor

26 lines
1.0 KiB
Factor
Raw Normal View History

! Copyright (C) 2009 Philipp Brüschweiler
! See http://factorcode.org/license.txt for BSD license.
2009-02-07 19:03:35 -05:00
USING: infix.ast kernel peg peg.ebnf math.parser sequences
strings multiline ;
2009-02-07 19:03:35 -05:00
IN: infix.tokenizer
EBNF: tokenize-infix [=[
2009-02-07 19:03:35 -05:00
Letter = [a-zA-Z]
Digit = [0-9]
Digits = Digit+
Number = Digits '.' Digits => [[ "" concat-as string>number ast-value boa ]]
| Digits => [[ >string string>number ast-value boa ]]
String = '"' [^"]* '"' => [[ second >string ast-value boa ]]
2015-07-16 11:44:59 -04:00
Space = [ \t\n\r]
2009-02-07 19:03:35 -05:00
Spaces = Space* => [[ ignore ]]
NameFirst = Letter | "_" => [[ CHAR: _ ]]
NameRest = NameFirst | Digit
Name = NameFirst NameRest* => [[ first2 swap prefix >string ]]
Special = [+*/%(),] | "-" => [[ CHAR: - ]]
| "[" => [[ CHAR: \[ ]] | "]" => [[ CHAR: ] ]]
| ":" => [[ CHAR: \: ]]
Tok = Spaces (Name | Number | String | Special )
2009-02-07 19:03:35 -05:00
End = !(.)
Toks = Tok* Spaces End
]=]