Understanding Unexpected Behavior in Exact Match String Field Due to Training Data Labeling

·Sep 06, 2024 11:11 AM

Hey, could someone please verify whether my understanding of the drivers for slightly unexpected behaviour I observed is correct? The situation: I have a string field (my_field) where an exact match is required and I set the matching strategy to exact in the config. The output contains a number of z clusters where my_field is the only non-null field for all records but the values in my_field are different for all records. They're similar (e.g. my_field = 'value1' and my_field = 'value2') but for this field, similarity is insignificant unless they're identical. My assumption: Examples in the training data must have caused this. Most likely, it contains a pair/pairs that were labelled as a match (for other reasons than the my_field values) that had similar my_field values. Does this sound right or am I overlooking something here?

13 comments

· Sorted by Oldest

Nitish J.
·
Hi Thordis, "my_field is the only non-null field for all records but the values in my_field are different for all records." are these kind of records part of a same cluster?
Sonal G.
·
Adding to the conversation here. Seems that near similar m_field values with other attributes being null got clustered together? Thordis you are right, do check the labels and use the updateLabel phase if necessary.

Thordis

are these kind of records part of a same cluster?

yes, exactly: I have a number of clusters like this (with a z_minScore of ~0.996)

z_cluster   |  my_field   | other_fields
-------------------------------------------
    1       |    abc      |     null
    1       |    abd      |     null
    1       |    abe      |     null
... etc.

ok, thanks for confirming - I'll try that out. Thanks both!

✅1

Sonal G.
·
if your other fields are likely to contain a lot of nulls, you may want to use the is null or blank match type on them as well. without that, nulls will be treated as matches which may also be confusing the algos.
Thordis
·
ah good point! I overlooked this and thought that nulls were ignored when matching. I'll give this a go too. Thank you!
✅1
Thordis
·
I made these updates and now I have the opposite problem: I have records with identical my_field values and everything else null and they end up in different cluster:
z_cluster | my_field | other_fields ------------------------------------------- 1 | abc | null 2 | abc | null 3 | abc | null ... etc.
I'm going to try to add examples to the training data that mark an example of these as a matche but I was surprised to see this behaviour. Any ideas what could cause this?
Sonal G.
·
Seems like the null was being taken as a match earlier, and hence they matched. Ading few such samples (10-15) in the training samples and running 1-2 rounds of training should steer it in the right direction.
Thordis
·
ok, thanks. I'll give that a go
👍1
Thordis
·
Hey, one more follow up question: when using the null_or_blank strategy in combination with the exact strategy, is this the right way to configure it?
"matchType" : "exact,null_or_blank"
I'm still seeing a lot of fuzzy matches like I was encountering at the top of this post despite adding a number of examples to the training data to try and fix this. I'm planning on adding more, but just wanted to double check first that there wasn't something funky going on (e.g. something about how I've configured this meaning that the exact part was ignored or similar)
Sonal G.
·
yes that seems right. did you also run a few rounds of findTrainingData and label on top of the training samples?
Thordis
·
ok, great. Thanks for confirming! For this model, I'm actually constructing the training data myself so I skip straight to the training phase. This is because I found the findTrainingData phase didn't identify a lot of matches for previous models created for the same data. I'd therefore have to run findTrainingData so often that it was taking up a lot of time. I'm still experimenting with different permutations of options, but for now I think I'll stick with constructing the training data myself (the accuracy improved a decent amount after the last iteration, so I'll try at least one more)
✅1
Sonal G.
·
if you have training data or build one, it is still a good practice to run ftd and label a few rounds so the params can be learn accurately.
Thordis
·
ah interesting - I missed that so I appreciate the advice. I'll give that a go. Thanks!
👍1

Nitish J.
·
Hi Thordis, "my_field is the only non-null field for all records but the values in my_field are different for all records." are these kind of records part of a same cluster?
Sonal G.
·
Adding to the conversation here. Seems that near similar m_field values with other attributes being null got clustered together? Thordis you are right, do check the labels and use the updateLabel phase if necessary.

Thordis

are these kind of records part of a same cluster?

yes, exactly: I have a number of clusters like this (with a z_minScore of ~0.996)

z_cluster   |  my_field   | other_fields
-------------------------------------------
    1       |    abc      |     null
    1       |    abd      |     null
    1       |    abe      |     null
... etc.

ok, thanks for confirming - I'll try that out. Thanks both!

✅1

Sonal G.
·
if your other fields are likely to contain a lot of nulls, you may want to use the is null or blank match type on them as well. without that, nulls will be treated as matches which may also be confusing the algos.
Thordis
·
ah good point! I overlooked this and thought that nulls were ignored when matching. I'll give this a go too. Thank you!
✅1
Thordis
·
I made these updates and now I have the opposite problem: I have records with identical my_field values and everything else null and they end up in different cluster:
z_cluster | my_field | other_fields ------------------------------------------- 1 | abc | null 2 | abc | null 3 | abc | null ... etc.
I'm going to try to add examples to the training data that mark an example of these as a matche but I was surprised to see this behaviour. Any ideas what could cause this?
Sonal G.
·
Seems like the null was being taken as a match earlier, and hence they matched. Ading few such samples (10-15) in the training samples and running 1-2 rounds of training should steer it in the right direction.
Thordis
·
ok, thanks. I'll give that a go
👍1
Thordis
·
Hey, one more follow up question: when using the null_or_blank strategy in combination with the exact strategy, is this the right way to configure it?
"matchType" : "exact,null_or_blank"
I'm still seeing a lot of fuzzy matches like I was encountering at the top of this post despite adding a number of examples to the training data to try and fix this. I'm planning on adding more, but just wanted to double check first that there wasn't something funky going on (e.g. something about how I've configured this meaning that the exact part was ignored or similar)
Sonal G.
·
yes that seems right. did you also run a few rounds of findTrainingData and label on top of the training samples?
Thordis
·
ok, great. Thanks for confirming! For this model, I'm actually constructing the training data myself so I skip straight to the training phase. This is because I found the findTrainingData phase didn't identify a lot of matches for previous models created for the same data. I'd therefore have to run findTrainingData so often that it was taking up a lot of time. I'm still experimenting with different permutations of options, but for now I think I'll stick with constructing the training data myself (the accuracy improved a decent amount after the last iteration, so I'll try at least one more)
✅1
Sonal G.
·
if you have training data or build one, it is still a good practice to run ftd and label a few rounds so the params can be learn accurately.
Thordis
·
ah interesting - I missed that so I appreciate the advice. I'll give that a go. Thanks!
👍1