nltk.tbl.template module

class nltk.tbl.template.BrillTemplateI[source]

Bases: object

An interface for generating lists of transformational rules that apply at given sentence positions. BrillTemplateI is used by Brill training algorithms to generate candidate rules.

abstract applicable_rules(tokens, i, correctTag)[source]

Return a list of the transformational rules that would correct the i-th subtoken’s tag in the given token. In particular, return a list of zero or more rules that would change tokens[i][1] to correctTag, if applied to token[i].

If the i-th token already has the correct tag (i.e., if tagged_tokens[i][1] == correctTag), then applicable_rules() should return the empty list.

Parameters

  • tokens (list(tuple)) – The tagged tokens being tagged.

  • i (int) – The index of the token whose tag should be corrected.

  • correctTag (any) – The correct tag for the i-th token.

Return type

list(BrillRule)

abstract get_neighborhood(token, index)[source]

Returns the set of indices i such that applicable_rules(token, i, ...) depends on the value of the index*th token of *token.

This method is used by the “fast” Brill tagger trainer.

Parameters

  • token (list(tuple)) – The tokens being tagged.

  • index (int) – The index whose neighborhood should be returned.

Return type

set

class nltk.tbl.template.Template[source]

Bases: BrillTemplateI

A tbl Template that generates a list of L{Rule}s that apply at a given sentence position. In particular, each C{Template} is parameterized by a list of independent features (a combination of a specific property to extract and a list C{L} of relative positions at which to extract it) and generates all Rules that:

  • use the given features, each at its own independent position; and

  • are applicable to the given token.

ALLTEMPLATES = []
__init__(*features)[source]

Construct a Template for generating Rules.

Takes a list of Features. A C{Feature} is a combination of a specific property and its relative positions and should be a subclass of L{nltk.tbl.feature.Feature}.

An alternative calling convention (kept for backwards compatibility, but less expressive as it only permits one feature type) is Template(Feature, (start1, end1), (start2, end2), …) In new code, that would be better written Template(Feature(start1, end1), Feature(start2, end2), …)

For instance, importing some features

>>> from nltk.tbl.template import Template
>>> from nltk.tag.brill import Word, Pos

Create some features

>>> wfeat1, wfeat2, pfeat = (Word([-1]), Word([1,2]), Pos([-2,-1]))

Create a single-feature template

>>> Template(wfeat1)
Template(Word([-1]))

Or a two-feature one

>>> Template(wfeat1, wfeat2)
Template(Word([-1]),Word([1, 2]))

Or a three-feature one with two different feature types

>>> Template(wfeat1, wfeat2, pfeat)
Template(Word([-1]),Word([1, 2]),Pos([-2, -1]))

deprecated api: Feature subclass, followed by list of (start,end) pairs (permits only a single Feature)

>>> Template(Word, (-2,-1), (0,0))
Template(Word([-2, -1]),Word([0]))

Incorrect specification raises TypeError

>>> Template(Word, (-2,-1), Pos, (0,0))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "nltk/tag/tbl/template.py", line 143, in __init__
    raise TypeError(
TypeError: expected either Feature1(args), Feature2(args), ... or Feature, (start1, end1), (start2, end2), ...
Parameters

features (list of Features) – the features to build this Template on

applicable_rules(tokens, index, correct_tag)[source]

Return a list of the transformational rules that would correct the i-th subtoken’s tag in the given token. In particular, return a list of zero or more rules that would change tokens[i][1] to correctTag, if applied to token[i].

If the i-th token already has the correct tag (i.e., if tagged_tokens[i][1] == correctTag), then applicable_rules() should return the empty list.

Parameters

  • tokens (list(tuple)) – The tagged tokens being tagged.

  • i (int) – The index of the token whose tag should be corrected.

  • correctTag (any) – The correct tag for the i-th token.

Return type

list(BrillRule)

classmethod expand(featurelists, combinations=None, skipintersecting=True)[source]

Factory method to mass generate Templates from a list L of lists of Features.

#With combinations=(k1, k2), the function will in all possible ways choose k1 … k2 #of the sublists in L; it will output all Templates formed by the Cartesian product #of this selection, with duplicates and other semantically equivalent #forms removed. Default for combinations is (1, len(L)).

The feature lists may have been specified manually, or generated from Feature.expand(). For instance,

>>> from nltk.tbl.template import Template
>>> from nltk.tag.brill import Word, Pos

#creating some features >>> (wd_0, wd_01) = (Word([0]), Word([0,1]))

>>> (pos_m2, pos_m33) = (Pos([-2]), Pos([3-2,-1,0,1,2,3]))

>>> list(Template.expand([[wd_0], [pos_m2]]))
[Template(Word([0])), Template(Pos([-2])), Template(Pos([-2]),Word([0]))]

>>> list(Template.expand([[wd_0, wd_01], [pos_m2]]))
[Template(Word([0])), Template(Word([0, 1])), Template(Pos([-2])), Template(Pos([-2]),Word([0])), Template(Pos([-2]),Word([0, 1]))]

#note: with Feature.expand(), it is very easy to generate more templates #than your system can handle – for instance, >>> wordtpls = Word.expand([-2,-1,0,1], [1,2], excludezero=False) >>> len(wordtpls) 7

>>> postpls = Pos.expand([-3,-2,-1,0,1,2], [1,2,3], excludezero=True)
>>> len(postpls)
9

#and now the Cartesian product of all non-empty combinations of two wordtpls and #two postpls, with semantic equivalents removed >>> templates = list(Template.expand([wordtpls, wordtpls, postpls, postpls])) >>> len(templates) 713

will return a list of eight templates

Template(Word([0])), Template(Word([0, 1])), Template(Pos([-2])), Template(Pos([-1])), Template(Pos([-2]),Word([0])), Template(Pos([-1]),Word([0])), Template(Pos([-2]),Word([0, 1])), Template(Pos([-1]),Word([0, 1]))]

#Templates where one feature is a subset of another, such as #Template(Word([0,1]), Word([1]), will not appear in the output. #By default, this non-subset constraint is tightened to disjointness: #Templates of type Template(Word([0,1]), Word([1,2]) will also be filtered out. #With skipintersecting=False, then such Templates are allowed

WARNING: this method makes it very easy to fill all your memory when training generated templates on any real-world corpus

Parameters

  • featurelists (list of (list of Features)) – lists of Features, whose Cartesian product will return a set of Templates

  • combinations (None, int, or (int, int)) – given n featurelists: if combinations=k, all generated Templates will have k features; if combinations=(k1,k2) they will have k1..k2 features; if None, defaults to 1..n

  • skipintersecting (bool) – if True, do not output intersecting Templates (non-disjoint positions for some feature)

Returns

generator of Templates

get_neighborhood(tokens, index)[source]

Returns the set of indices i such that applicable_rules(token, i, ...) depends on the value of the index*th token of *token.

This method is used by the “fast” Brill tagger trainer.

Parameters

  • token (list(tuple)) – The tokens being tagged.

  • index (int) – The index whose neighborhood should be returned.

Return type

set