zorba::TokenizerProvider

#include <zorba/tokenizer.h>

A TokenizerProvider provides a Tokenizer for a given language.

Public Functions

bool	getTokenizer(locale::iso639_1::type lang, Tokenizer::State state=nullptr, Tokenizer::ptr t=nullptr) const =0 Creates a new Tokenizer.
	~TokenizerProvider()

bool getTokenizer(locale::iso639_1::type lang, Tokenizer::State *state=nullptr, Tokenizer::ptr *t=nullptr) const =0

Creates a new Tokenizer.

lang	The language of the text that the tokenizer will tokenize.
state	The State to use. If `null`, t is not set.
t	If not `null`, set to point to a Tokenizer for lang.

Returns true only if this provider can provide a tokenizer for lang.

~TokenizerProvider()