nltk.tag.crf module

A module for POS tagging using CRFSuite

class nltk.tag.crf.CRFTagger

Bases: TaggerI

A module for POS tagging using CRFSuite https://pypi.python.org/pypi/python-crfsuite

>>> from nltk.tag import CRFTagger
>>> ct = CRFTagger()  

>>> train_data = [[('University','Noun'), ('is','Verb'), ('a','Det'), ('good','Adj'), ('place','Noun')],
... [('dog','Noun'),('eat','Verb'),('meat','Noun')]]

>>> ct.train(train_data,'model.crf.tagger')  
>>> ct.tag_sents([['dog','is','good'], ['Cat','eat','meat']])  
[[('dog', 'Noun'), ('is', 'Verb'), ('good', 'Adj')], [('Cat', 'Noun'), ('eat', 'Verb'), ('meat', 'Noun')]]

>>> gold_sentences = [[('dog','Noun'),('is','Verb'),('good','Adj')] , [('Cat','Noun'),('eat','Verb'), ('meat','Noun')]]
>>> ct.accuracy(gold_sentences)  
1.0

Setting learned model file >>> ct = CRFTagger() # doctest: +SKIP >>> ct.set_model_file(‘model.crf.tagger’) # doctest: +SKIP >>> ct.accuracy(gold_sentences) # doctest: +SKIP 1.0

__init__(feature_func=None, verbose=False, training_opt={})

Initialize the CRFSuite tagger

Parameters

• feature_func – The function that extracts features for each token of a sentence. This function should take 2 parameters: tokens and index which extract features at index position from tokens list. See the build in _get_features function for more detail.

• verbose (boolean) – output the debugging messages during training.

• training_opt (dictionary) – python-crfsuite training options

Set of possible training options (using LBFGS training algorithm).

‘feature.minfreq’

The minimum frequency of features.

‘feature.possible_states’

Force to generate possible state features.

‘feature.possible_transitions’

Force to generate possible transition features.

‘c1’

Coefficient for L1 regularization.

‘c2’

Coefficient for L2 regularization.

‘max_iterations’

The maximum number of iterations for L-BFGS optimization.

‘num_memories’

The number of limited memories for approximating the inverse hessian matrix.

‘epsilon’

Epsilon for testing the convergence of the objective.

‘period’

The duration of iterations to test the stopping criterion.

‘delta’

The threshold for the stopping criterion; an L-BFGS iteration stops when the improvement of the log likelihood over the last ${period} iterations is no greater than this threshold.

‘linesearch’

The line search algorithm used in L-BFGS updates:

• ‘MoreThuente’: More and Thuente’s method,

• ‘Backtracking’: Backtracking method with regular Wolfe condition,

• ‘StrongBacktracking’: Backtracking method with strong Wolfe condition

‘max_linesearch’

The maximum number of trials for the line search algorithm.

set_model_file(model_file)

tag(tokens)

Tag a sentence using Python CRFSuite Tagger. NB before using this function, user should specify the mode_file either by

• Train a new model using train function

• Use the pre-trained model which is set via set_model_file function

Params tokens

list of tokens needed to tag.

Returns

list of tagged tokens.

Return type

list(tuple(str,str))

tag_sents(sents)

Tag a list of sentences. NB before using this function, user should specify the mode_file either by

• Train a new model using train function

• Use the pre-trained model which is set via set_model_file function

Params sentences

list of sentences needed to tag.

Returns

list of tagged sentences.

Return type

list(list(tuple(str,str)))

train(train_data, model_file)

Train the CRF tagger using CRFSuite :params train_data : is the list of annotated sentences. :type train_data : list (list(tuple(str,str))) :params model_file : the model will be saved to this file.