R J. | Zingg Community

Commented on Best Practices for Training Multiple Models in Dat...·Posted inHelp Zingg

Hi it’s me again, I have a model I’m confident about and it’s working fine, I’m using 2 fuzzy match data points and one exact. I’m having issues when trying to find duplicates in large datasets, it’s taking a long time to finish “match” task, is there a guideline I can refer to for best cluster setups? Or how to handle large datasets? My dataset is about 160M rows. Thanks

Commented on Best Practices for Training Multiple Models in Dat...·Posted inHelp Zingg

R J.

thank you!

Commented on Best Practices for Training Multiple Models in Dat...·Posted inHelp Zingg

R J.

Hi this fixed the issue, I do have a follow up question. Once I've trained the model and executted "trainMatch", "link" and "match", if I decide I need to keep training the model, besides deleting cluster and cluster_members what other tables or directories should be removed? Thanks

Commented on Best Practices for Training Multiple Models in Dat...·Posted inHelp Zingg

R J.

Hi, I am using different model IDs, restarting the cluster did fix the issue. Thanks for your help!!!

Posted in Help Zingg·

R J.

Best Practices for Training Multiple Models in Databricks Using Shared Initial and Incremental Datasets

What is the best way to train different models (in databricks) using the same initial and incremental datasets with the goal of finding the better performing model? I've modified the ncvoters notebooks to use with my data, I've tried creating 3 separate directories and altering the names of the initial and incremental tables so each is saved in the correct path for each different directory, but I encounter some issues, I ran the Label Training section 5 times and labeled 25 negative pairs the first round and 80 positive pairs total the following rounds. When using trainMatch.execute() I get the following error: "zingg.common.client.ZinggClientException: Unable to train as insufficient training data found. Training data has 0 matches and 25 non matches. Please run findTrainingData and label till you have sufficient labelled data to build the models". It seems that when looking for the labeled pairs it's only finding the first set of pairs that were labeled and not the following ones. I checked the MARKED_DIR and it's correct, I can load the data and see all the pairs labeled correctly too. Do the tables have to be named initial and incremental always? What am I missing?

9Comments