TokenStream::GetNextToken (open)

Return the next token in the stream

Syntax

LOADLIB "wh::util/langspecific.whlib";

RECORD FUNCTION GetNextToken()

Return value

RECORD

The token that was read

endoffset

The (byte) position directly after the last character of the token within the stream

normalizedtext

For word tokens, the normalized text

startoffset

The (byte) position of the first character of the token within the stream

stemmedtext

For word tokens, the stemmed text

text

The token text

type

The token type

Description

Calling this function will read the next token from the internal buffer and return it. If the end of the stream was reached, a default record is returned.