A limitation on AttributeRuler (Mappings & Exceptions)?

[ricardojosehlima]

Hi,
I am trying some changes in the Mappings & Exceptions code (https://spacy.io/usage/linguistic-features#mappings-exceptions) that changes that tag and the POS of "The Who". The relevant part is:

patterns = [[{"LOWER": "the"}, {"TEXT": "Who"}]]
attrs = {"TAG": "NNP", "POS": "PROPN"}
ruler.add(patterns=patterns, attrs=attrs, index=0)  # "The" in "The Who"
ruler.add(patterns=patterns, attrs=attrs, index=1)  # "Who" in "The Who"

If I change it to:

patterns = [[{"TAG": "DT"}, {"TAG": "WP"}]]

The code still works. What I did was instead of capturing "The Who", capture any sequence of two tokens that have DT and WP as tags.

Now, when I change it to:

patterns = [[{"POS": "DET"}, {"POS": "PRON"}]]

Now capturing any sequence of tokens whose POS are DET and PRON... nothing happens after

attrs = {"TAG": "NNP", "POS": "PROPN"}
ruler.add(patterns=patterns, attrs=attrs, index=0)  
ruler.add(patterns=patterns, attrs=attrs, index=1)  

The POS classifications are correct and the sequence is captured.
I need to capture sequences of POS and modify them, any advice on why Spacy is behaving like this not allowing it and how to make it?

[polm]

I'm not entirely sure I understand what you're saying because you say "nothing happens" but you also say "the sequence is captured", but I think the issue is you can't seem to match a DET PRON sequence, right? Here's an example:

import spacy

nlp = spacy.load("en_core_web_sm")
text = "I saw The You perform. Who did you see?"
doc1 = nlp(text)
print(doc1[2].tag_, doc1[2].pos_)  # DT DET
print(doc1[3].tag_, doc1[3].pos_)  # WP PRON

# Add attribute ruler with exception for "The Who" as NNP/PROPN NNP/PROPN
ruler = nlp.get_pipe("attribute_ruler")
#print(ruler.patterns) # XXX use this to debug
# Pattern to match "The Who"
patterns = [[{"POS": "DET"}, {"POS": "PRON"}]]
# The attributes to assign to the matched token
attrs = {"TAG": "NNP", "TAG": "CHECKME"}
# Add rules to the attribute ruler
ruler.add(patterns=patterns, attrs=attrs, index=0)  # "The" in "The Who"
ruler.add(patterns=patterns, attrs=attrs, index=1)  # "Who" in "The Who"

doc2 = nlp(text)

for tok in doc2:
    print(tok, tok.tag_, tok.pos_, sep="\t")

The reason this doesn't work is because POS are set in the AttributeRuler based on tags - before it runs they are empty. The Ruler isn't recursive, so after it makes a change it doesn't run other rules. So you're trying to match empty fields, which won't work.

If you really want to match on POS values you can use a second AttributeRuler after the first, but I suspect it's usually easier to match on the TAG values or other attributes.

[ricardojosehlima]

Hi, thanks for the answer and sorry for misphrasing what was going on: "nothing happens" equals to the rules weren't applied and the tags and pos remained the same. I will try to focus on tags instead as you pointed.