Thordis

ah interesting - I missed that so I appreciate the advice. I'll give that a go. Thanks!

ok, great. Thanks for confirming! For this model, I'm actually constructing the training data myself so I skip straight to the training phase. This is because I found the findTrainingData phase didn't identify a lot of matches for previous models created for the same data. I'd therefore have to run findTrainingData so often that it was taking up a lot of time. I'm still experimenting with different permutations of options, but for now I think I'll stick with constructing the training data myself (the accuracy improved a decent amount after the last iteration, so I'll try at least one more)

Hey, one more follow up question: when using the null_or_blank strategy in combination with the exact strategy, is this the right way to configure it?

"matchType" : "exact,null_or_blank"

I'm still seeing a lot of fuzzy matches like I was encountering at the top of this post despite adding a number of examples to the training data to try and fix this. I'm planning on adding more, but just wanted to double check first that there wasn't something funky going on (e.g. something about how I've configured this meaning that the exact part was ignored or similar)

ok, thanks. I'll give that a go

I made these updates and now I have the opposite problem: I have records with identical my_field values and everything else null and they end up in different cluster:

z_cluster   |  my_field   | other_fields
-------------------------------------------
    1       |    abc      |     null
    2       |    abc      |     null
    3       |    abc      |     null
... etc.

I'm going to try to add examples to the training data that mark an example of these as a matche but I was surprised to see this behaviour. Any ideas what could cause this?

ah good point! I overlooked this and thought that nulls were ignored when matching. I'll give this a go too. Thank you!

are these kind of records part of a same cluster?

yes, exactly: I have a number of clusters like this (with a z_minScore of ~0.996)

z_cluster   |  my_field   | other_fields
-------------------------------------------
    1       |    abc      |     null
    1       |    abd      |     null
    1       |    abe      |     null
... etc.

ok, thanks for confirming - I'll try that out. Thanks both!

thanks for the clear answer. I'll add examples like this to the training set in that case