Token Tester
This little tool was created to assist in the development of Word Stats' text analysis algorithm. It works on individual tokens, not on sentences. Select a language, optionally adjust hyphenation settings, and enter a string of characters into the box. Hit Clean to see how this token is cleaned up in case it contains characters other than letters and digits, and/or hit Test to see how it parses. Note that text is case sensitive; e.g., I'm is recognized as a valid word but i'm is not.
Word ( Valid ) Number Hyperlink Empty Other
| Characters | -- | |
| Syllables | -- | |
| Hyphenation | -- |