 | TokenizerBase Properties |
[This is preliminary documentation and is subject to change.]
The TokenizerBase type exposes the following members.
Properties| Name | Description | |
|---|---|---|
 | InputReplacers |
Set of replacement rules to be applied on a text, before splitting the text into tokens
|
| LowerCase | |
| TokenSplitterChars |
Characters that are going to be used to split the text into tokens
|
See Also