Click or drag to resize

TokenizerBasic Properties

Home | imbSCI | imbACE | imbNLP | imbWEM | imbWBI

The TokenizerBasic type exposes the following members.

Properties
  NameDescription
Public propertyInputReplacers
Set of replacement rules to be applied on a text, before splitting the text into tokens
(Inherited from TokenizerBase.)
Public propertyLowerCase (Inherited from TokenizerBase.)
Public propertyMinLength (Inherited from TokenizerBase.)
Public propertytokenSelector (Inherited from TokenizerBase.)
Public propertyTokenSplitterChars
Characters that are going to be used to split the text into tokens
(Inherited from TokenizerBase.)
Top
See Also