TokenizerBasic Class

Basic tokenizer for English - it removes all non-Latin characters during tokenization

Inheritance Hierarchy

SystemObject
imbNLP.Toolkit.ProcessingTokenizerBase
imbNLP.Toolkit.ProcessingTokenizerBasic

Namespace: imbNLP.Toolkit.Processing
Assembly: imbNLP.Toolkit (in imbNLP.Toolkit.dll) Version: 0.2.30

Syntax

public class TokenizerBasic : TokenizerBase

The TokenizerBasic type exposes the following members.

Constructors

	Name	Description
	TokenizerBasic	Initializes a new instance of the TokenizerBasic class.

Properties

	Name	Description
	InputReplacers	Set of replacement rules to be applied on a text, before splitting the text into tokens (Inherited from TokenizerBase.)
	LowerCase	(Inherited from TokenizerBase.)
	MinLength	(Inherited from TokenizerBase.)
	tokenSelector	(Inherited from TokenizerBase.)
	TokenSplitterChars	Characters that are going to be used to split the text into tokens (Inherited from TokenizerBase.)

Methods

	Name	Description
	Equals	Determines whether the specified object is equal to the current object. (Inherited from Object.)
	ExecuteInputReplacers	Executes all replacement rules from the InputReplacers collection (Inherited from TokenizerBase.)
	Finalize	Allows an object to try to free resources and perform other cleanup operations before it is reclaimed by garbage collection. (Inherited from Object.)
	GetHashCode	Serves as the default hash function. (Inherited from Object.)
	GetType	Gets the Type of the current instance. (Inherited from Object.)
	MemberwiseClone	Creates a shallow copy of the current Object. (Inherited from Object.)
	Tokenize	Tokenizes the specified text, according to the configuration of the tokenizer (Inherited from TokenizerBase.)
	ToString	Returns a string that represents the current object. (Inherited from Object.)