How to Integrate NER and Textcat Multilabel Models in a Pipeline

[santhoshlinga]

Hi,

I am working on extracting information from raw text and have created an NER model with 6 entities. I want to pass the output of the NER model to textcat multilabel models. Specifically, I have created different textcat multilabel models for each entity to improve precision. Now that I have all the models ready, I need guidance on how to add them to the pipeline.

Here are the steps I am trying to implement:

1. NER Model: Outputs 6 entities in an array.
2. Textcat Multilabel Models: Use the output from the NER model to get the final output for each entity.

Could anyone provide advice or examples on how to integrate these models into a single pipeline?

Thank you!

[santhoshlinga]

Is there any possibility?

[bobbywilson0]

Not sure if you have looked at the docs, but it does seem possible https://spacy.io/api/textcategorizer

"If you want to perform multi-label classification and predict zero, one or more true labels per document, use the textcat_multilabel component instead."

[santhoshlinga]

Thank you @bobbywilson0 for your comments

I am working on descriptions that range from single-line statements to 3-5 paragraphs. To handle this, I have approached it with Named Entity Recognition (NER) to extract the information. To control false positives, I have refined this approach. Working with multi-label text classification did not meet my expectations.

Currently, I have streamlined the process to run NER on each description. Based on the NER output, I run text classification for each entity. While the results are excellent, this process is time-consuming. I am now focusing on improving performance to reduce the processing time.

[bobbywilson0]

Looks like you're making nice progress, I just commented hoping to help someone new out, but seems like you know what you're doing! 😄

[santhoshlinga]

Currently, I have NER running with my descriptions divided into chunks of 100, with a NER batch size of 30. This update has reduced the processing time from 19 hours to 7 hours by using chunk processing instead of individual execution for each Description.

After getting results from each chunk of descriptions, I collect all the entity values as different chunks and then execute the all the result phrases with a batch size of 50 since these are smaller in size.

Now, I am looking for any better approaches. Please let me know if you have tried any.