 | TokenizerBase Class |
Common base for different tokenizers.
Inheritance HierarchyNamespace: imbNLP.Toolkit.Processing
Assembly: imbNLP.Toolkit (in imbNLP.Toolkit.dll) Version: 0.2.30
SyntaxC#
public abstract class TokenizerBase : ITokenizer
The TokenizerBase type exposes the following members.
Constructors| Name | Description | |
|---|---|---|
 | TokenizerBase | Initializes a new instance of the TokenizerBase class |
Properties| Name | Description | |
|---|---|---|
 | InputReplacers |
Set of replacement rules to be applied on a text, before splitting the text into tokens
|
| LowerCase | |
| MinLength | |
| tokenSelector | |
| TokenSplitterChars |
Characters that are going to be used to split the text into tokens
|
Methods| Name | Description | |
|---|---|---|
 | Equals | Determines whether the specified object is equal to the current object. (Inherited from Object.) |
| ExecuteInputReplacers |
Executes all replacement rules from the InputReplacers collection
|
| Finalize | Allows an object to try to free resources and perform other cleanup operations before it is reclaimed by garbage collection. (Inherited from Object.) |
| GetHashCode | Serves as the default hash function. (Inherited from Object.) |
| GetType | Gets the Type of the current instance. (Inherited from Object.) |
| MemberwiseClone | Creates a shallow copy of the current Object. (Inherited from Object.) |
| Tokenize |
Tokenizes the specified text, according to the configuration of the tokenizer
|
| ToString | Returns a string that represents the current object. (Inherited from Object.) |
See Also