COMPLETED
0.1.1
-
Documentation and the whole project tree updated. No functional improvements.
-
Corrected typos.
0.1.0
-
Notion of binary tokenizer and a library for embedded tokenization.
-
Separation of punctuaction marks.
0.0.1
-
Simple tokenization is desired.