nltk.test.unit.test_tokenize module

Unit tests for nltk.tokenize. See also nltk/test/tokenize.doctest

nltk.test.unit.test_tokenize.load_stanford_segmenter()

class nltk.test.unit.test_tokenize.TestTokenize

Bases: object

test_tweet_tokenizer()

Test TweetTokenizer using words with special and accented characters.

test_tweet_tokenizer_expanded(test_input: str, expecteds: Tuple[List[str], List[str]])

Test match_phone_numbers in TweetTokenizer.

Note that TweetTokenizer is also passed the following for these tests:

• strip_handles=True

• reduce_len=True

Parameters

• test_input (str) – The input string to tokenize using TweetTokenizer.

• expecteds (Tuple[List[str], List[str]]) – A 2-tuple of tokenized sentences. The first of the two tokenized is the expected output of tokenization with match_phone_numbers=True. The second of the two tokenized lists is the expected output of tokenization with match_phone_numbers=False.

test_sonority_sequencing_syllable_tokenizer()

Test SyllableTokenizer tokenizer.

test_syllable_tokenizer_numbers()

Test SyllableTokenizer tokenizer.

test_legality_principle_syllable_tokenizer()

Test LegalitySyllableTokenizer tokenizer.

test_stanford_segmenter_arabic()

Test the Stanford Word Segmenter for Arabic (default config)

test_stanford_segmenter_chinese()

Test the Stanford Word Segmenter for Chinese (default config)

test_phone_tokenizer()

Test a string that resembles a phone number but contains a newline

test_emoji_tokenizer()

Test a string that contains Emoji ZWJ Sequences and skin tone modifier

test_pad_asterisk()

Test padding of asterisk for word tokenization.

test_pad_dotdot()

Test padding of dotdot* for word tokenization.

test_remove_handle()

Test remove_handle() from casual.py with specially crafted edge cases

test_treebank_span_tokenizer()

Test TreebankWordTokenizer.span_tokenize function

test_word_tokenize()

Test word_tokenize function

test_punkt_pair_iter()

test_punkt_pair_iter_handles_stop_iteration_exception()

test_punkt_tokenize_words_handles_stop_iteration_exception()

test_punkt_tokenize_custom_lang_vars()

test_punkt_tokenize_no_custom_lang_vars()

punkt_debug_decisions(input_text, n_sents, n_splits, lang_vars=None)

test_punkt_debug_decisions_custom_end()

test_sent_tokenize(sentences: str, expected: List[str])

Parameters

• sentences (str) –

• expected (List[str]) –