steve
/
factor
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

io.encodings.utf8 fixed for bootstrap; add unit tests

db4
U-SLAVA-DFB8FF805\Slava 2009-03-04 21:33:34 -06:00
parent 039f44baa3
commit 83e8cdae1c
2 changed files with 5 additions and 3 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
core/io/encodings/utf8/utf8-tests.factor Normal file → Executable file
View File

@ -22,3 +22,6 @@ IN: io.encodings.utf8.tests
[ { BIN: 11110101 BIN: 10111111 BIN: 10000000 BIN: 10111111 BIN: 11101111 BIN: 10000000 BIN: 10111111 BIN: 11011111 BIN: 10000000 CHAR: x } ]
[ { BIN: 101111111000000111111 BIN: 1111000000111111 BIN: 11111000000 CHAR: x } encode-utf8-w/stream ] unit-test
[ 3 ] [ 1 "日本語" >utf8-index ] unit-test
[ 3 ] [ 9 "日本語" utf8-index> ] unit-test

5
core/io/encodings/utf8/utf8.factor Normal file → Executable file
View File

@ -1,8 +1,7 @@
! Copyright (C) 2006, 2008 Daniel Ehrenberg.
! See http://factorcode.org/license.txt for BSD license.
USING: math math.order kernel sequences sbufs vectors growable io
continuations namespaces io.encodings combinators strings
binary-search ;
continuations namespaces io.encodings combinators strings ;
IN: io.encodings.utf8
! Decoding UTF-8
@ -85,7 +84,7 @@ PRIVATE>
0 [ code-point-length + ] accumulate swap suffix ;
: utf8-index> ( n string -- n' )
code-point-offsets natural-search drop ;
code-point-offsets [ <= ] with find drop ;
: >utf8-index ( n string -- n' )
code-point-offsets nth ;