zorba::Tokenizer::State

#include <zorba/tokenizer.h>

A State contains inter-Tokenizer state, currently the current token, sentence, and paragraph numbers.

Public Types

Tokenizer::size_type

Public Attributes

value_type	para Paragraph number.
value_type	sent Sentence number.
value_type	token Token number.

Public Functions

Default constructor.

Public Types

value_type

Tokenizer::size_type value_type

Public Attributes

para

value_type para

Paragraph number.

sent

value_type sent

Sentence number.

token

value_type token

Token number.

Public Functions

State

State()

Default constructor.