Matching subsets

[rkennedy-argus]

Apologies if this is addressed in the documentation. I wasn't able to find to find an answer to this if it is.

I have records from one source with multiple street addresses, which I would like to match with records from a second source with just one street address. It's not clear to me if Zingg can handle this sort of matching where the column count differs (e.g. source one has Address1, Address2, …, AddressN and source two just has Address).

It's not limited to street addresses. I'd also like to do similar matching of things like DNS domains, email addresses, etc where the two record sets have differing counts and I'd like to match on the presence of an overlap.

But I'm unsure how to express that in my data and/or in my configuration.

[sonalgoyal]

In the Community version of Zingg, it would be best to make multiple rows out of addresses and do the fuzzy matching. Regarding DNS etc, are these exact matches?

[rkennedy-argus]

In the Community version of Zingg, it would be best to make multiple rows out of addresses and do the fuzzy matching.

Hmm, we'll have multiple multi-valued fields (e.g. addresses and domains). So, in order to find matches we'll need to create {number of addresses} * {number of domains} combinations of rows for comparison. Not impossible, but it's going to add a fair bit of overhead to multiple stages in the process (generating the rows, performing the matching in Zingg, collating the results).

This (entity resolution) isn't my typical day-to-day thing. Is this commonplace for folks performing entity resolution? Or am I doing something a bit unconventional and should perhaps reconsider my approach?

Also, you qualified your answer with "In the Community version of Zingg". Is there something in the Enterprise version that would change or influence your answer in another direction?

Regarding DNS etc, are these exact matches?

Probably, yes. My concern was less about comparing domains and more about including domains as an example of another multi-valued field type present in our records to demonstrate that we'd have to generate all combinations across N discrete fields, which may significantly increase the number of rows we're comparing.

[sonalgoyal]

Can you please help me understand the question a bit more. Examples would be great!

1. How much data do you have?
2. How many multi valued fields do you have? You had mentioned that you have one source with only address but the other with address 1, addr2, addr3 ... Now it seems it is address and domains? And addr1, domain1, addr2, domain 1..addr1, domain 2..?
3. Are each of the multi values fields contributing to the match? Like you want address and domain together to quailify a match? Or are domains to be matched separately ?

[rkennedy-argus]

1. How much data do you have?

We have a public dataset with 2.5-3M records in it, which we'll be joining with customer-specific datasets with (likely) 100s to 1,000s of records.

2. How many multi valued fields do you have?

In the public dataset each record includes:

• A name
• 0…7 street addresses
• 0…N URLs, which we plan to boil down to just the domains of the hostnames

In the customer dataset, each record includes:

• A name
• 0…4 street addresses
• 1…N email addresses, which we plan to boil down to just the domains

3. Are each of the multi values fields contributing to the match? Like you want address and domain together to quailify a match? Or are domains to be matched separately ?

They would work together. The hope is that we'd have sufficient domain coverage and accuracy in the public dataset to achieve an exact match that we feel reasonably confident about. That's sometimes tricky (e.g. if a shared domain is referenced), so street addresses become a backup option for us. Street addresses have their own issues (fuzzy matching for one, but also our records are likely to have a lot of addresses, not all of which may be enumerated in the public dataset and even if they are, could be shared addresses). The last ditch, if we can't get a convincing match from either the domain(s) or street address(es) is to match on name. Names for our types of record are difficult, however, owing to issues of fuzziness (abbreviations, nicknames, etc) and overlapping of "common" names.

The hope right now is to get our public dataset and a sample of a customer dataset into a format that we can feed to Zingg to evaluate how the linking process goes. Hence the questions about modeling our data with these multi-valued fields.

Apologies that some of the above is (intentionally) fairly vague (e.g. what types of "names" we're working with). We're not openly discussing what exactly we're up to just yet.

[sonalgoyal]

Thank you. My recommendation for the community version would be doing name, addr, domain structure and build multiple records out of one. Your dataset is not huge, number of fields less and you are anyways doing a link, not a match. Use fuzzy match types for all name and address. Not sure if domains have typos etc, but check between exact and fuzzy.

FWIW, I put up some documentation for the relate feature in enterprise. Just in case you think you are probably better off with 2 models - name and address, and name and domain models. Then bring them all togther with relate. https://docs.zingg.ai/latest/stepbystep/relations