TokenizerBaseTokenSplitterChars Property

[This is preliminary documentation and is subject to change.]

Characters that are going to be used to split the text into tokens

Namespace: imbNLP.Toolkit.Processing
Assembly: imbNLP.Toolkit (in imbNLP.Toolkit.dll) Version: 0.2.16

Syntax

public char[] TokenSplitterChars { get; set; }

Type: Char
The token splitter chars.

Coding For Science - imbVeles Open Source Framework - Goran Grubić : 2017-2018 Project web site

Send comments on this topic to hardy@veles.rs

TokenizerBaseAddLanguageSpecificTextSet("LSTB5C7CFEB_0?cpp=::|nu=.");TokenSplitterChars Property