 | TokenizerBasic Properties |
[This is preliminary documentation and is subject to change.]
The TokenizerBasic type exposes the following members.
Properties| Name | Description | |
|---|---|---|
 | InputReplacers |
Set of replacement rules to be applied on a text, before splitting the text into tokens
(Inherited from TokenizerBase.) |
| LowerCase | (Inherited from TokenizerBase.) |
| TokenSplitterChars |
Characters that are going to be used to split the text into tokens
(Inherited from TokenizerBase.) |
See Also