nltk.translate.ibm_model module

Common methods and classes for all IBM models. See IBMModel1, IBMModel2, IBMModel3, IBMModel4, and IBMModel5 for specific implementations.

The IBM models are a series of generative models that learn lexical translation probabilities, p(target language word|source language word), given a sentence-aligned parallel corpus.

The models increase in sophistication from model 1 to 5. Typically, the output of lower models is used to seed the higher models. All models use the Expectation-Maximization (EM) algorithm to learn various probability tables.

Words in a sentence are one-indexed. The first word of a sentence has position 1, not 0. Index 0 is reserved in the source sentence for the NULL token. The concept of position does not apply to NULL, but it is indexed at 0 by convention.

Each target word is aligned to exactly one source word or the NULL token.

References: Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York.

Peter E Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19 (2), 263-311.

class nltk.translate.ibm_model.AlignmentInfo

Bases: object

Helper data object for training IBM Models 3 and up

Read-only. For a source sentence and its counterpart in the target language, this class holds information about the sentence pair’s alignment, cepts, and fertility.

Warning: Alignments are one-indexed here, in contrast to nltk.translate.Alignment and AlignedSent, which are zero-indexed This class is not meant to be used outside of IBM models.

__init__(alignment, src_sentence, trg_sentence, cepts)

alignment

tuple(int): Alignment function. alignment[j] is the position in the source sentence that is aligned to the position j in the target sentence.

center_of_cept(i)

Returns

The ceiling of the average positions of the words in the tablet of cept i, or 0 if i is None

cepts

list(list(int)): The positions of the target words, in ascending order, aligned to a source word position. For example, cepts[4] = (2, 3, 7) means that words in positions 2, 3 and 7 of the target sentence are aligned to the word in position 4 of the source sentence

fertility_of_i(i)

Fertility of word in position i of the source sentence

is_head_word(j)

Returns

Whether the word in position j of the target sentence is a head word

previous_cept(j)

Returns

The previous cept of j, or None if j belongs to the first cept

previous_in_tablet(j)

Returns

The position of the previous word that is in the same tablet as j, or None if j is the first word of the tablet

score

float: Optional. Probability of alignment, as defined by the IBM model that assesses this alignment

src_sentence

tuple(str): Source sentence referred to by this object. Should include NULL token (None) in index 0.

trg_sentence

tuple(str): Target sentence referred to by this object. Should have a dummy element in index 0 so that the first word starts from index 1.

zero_indexed_alignment()

Returns

Zero-indexed alignment, suitable for use in external nltk.translate modules like nltk.translate.Alignment

Return type

list(tuple)

class nltk.translate.ibm_model.Counts

Bases: object

Data object to store counts of various parameters during training

__init__()

update_fertility(count, alignment_info)

update_lexical_translation(count, alignment_info, j)

update_null_generation(count, alignment_info)

class nltk.translate.ibm_model.IBMModel

Bases: object

Abstract base class for all IBM models

MIN_PROB = 1e-12

__init__(sentence_aligned_corpus)

best_model2_alignment(sentence_pair, j_pegged=None, i_pegged=0)

Finds the best alignment according to IBM Model 2

Used as a starting point for hill climbing in Models 3 and above, because it is easier to compute than the best alignments in higher models

Parameters

• sentence_pair (AlignedSent) – Source and target language sentence pair to be word-aligned

• j_pegged (int) – If specified, the alignment point of j_pegged will be fixed to i_pegged

• i_pegged (int) – Alignment point to j_pegged

hillclimb(alignment_info, j_pegged=None)

Starting from the alignment in alignment_info, look at neighboring alignments iteratively for the best one

There is no guarantee that the best alignment in the alignment space will be found, because the algorithm might be stuck in a local maximum.

Parameters

j_pegged (int) – If specified, the search will be constrained to alignments where j_pegged remains unchanged

Returns

The best alignment found from hill climbing

Return type

AlignmentInfo

init_vocab(sentence_aligned_corpus)

maximize_fertility_probabilities(counts)

maximize_lexical_translation_probabilities(counts)

maximize_null_generation_probabilities(counts)

neighboring(alignment_info, j_pegged=None)

Determine the neighbors of alignment_info, obtained by moving or swapping one alignment point

Parameters

j_pegged (int) – If specified, neighbors that have a different alignment point from j_pegged will not be considered

Returns

A set neighboring alignments represented by their AlignmentInfo

Return type

set(AlignmentInfo)

prob_of_alignments(alignments)

prob_t_a_given_s(alignment_info)

Probability of target sentence and an alignment given the source sentence

All required information is assumed to be in alignment_info and self.

Derived classes should override this method

reset_probabilities()

sample(sentence_pair)

Sample the most probable alignments from the entire alignment space

First, determine the best alignment according to IBM Model 2. With this initial alignment, use hill climbing to determine the best alignment according to a higher IBM Model. Add this alignment and its neighbors to the sample set. Repeat this process with other initial alignments obtained by pegging an alignment point.

Hill climbing may be stuck in a local maxima, hence the pegging and trying out of different alignments.

Parameters

sentence_pair (AlignedSent) – Source and target language sentence pair to generate a sample of alignments from

Returns

A set of best alignments represented by their AlignmentInfo and the best alignment of the set for convenience

Return type

set(AlignmentInfo), AlignmentInfo

set_uniform_probabilities(sentence_aligned_corpus)

Initialize probability tables to a uniform distribution

Derived classes should implement this accordingly.

nltk.translate.ibm_model.longest_target_sentence_length(sentence_aligned_corpus)

Parameters

sentence_aligned_corpus (list(AlignedSent)) – Parallel corpus under consideration

Returns

Number of words in the longest target language sentence of sentence_aligned_corpus