Aniello G.

Hi all, I have a question about the labelling file. Regarding the no match entries, is it better to group them into similar pairs that don’t match, for example:

z_cluster,z_isMatch,firstname,lastname,jobtitle,..
123,0,James,Sallivan,IT Specialist,..
123,0,Anthony,Sallivan,Data Engineer,..
456,0,Frank,Williams,Project Manager,..
456,0,Franco,William,Sr Project Manager,..

or is it also fine to have larger no match groups within the same z_cluster, since none of them match anyway, for example:

z_cluster,z_isMatch,firstname,lastname,jobtitle,..
123,0,James,Sallivan,IT Specialist,..
123,0,Anthony,Sallivan,Data Engineer,..
123,0,Frank,Williams,Project Manager,..
123,0,Franco,William,Sr Project Manager,..

Which of the two options would make the training more effective? Intuitively, I would go for the first option, but I’m wondering just in case... Thanks.

Hi all, We are attempting to run Zingg 0.4.0 on the newly released AWS Glue 5.0, but we are encountering issues with the Zingg pipes, specifically after the trainMatch phase, during access to the output pipe. The logs indicate that, with large datasets, the workers fail to communicate with the driver, causing the SparkSession to terminate. We have been trying to fine-tune various Spark parameters but have not had any success so far. It appears this may be related to the Spark setup, although these issues did not occur on Glue 4.0 when using the package compiled for Spark 3.3.x. It would be great to hear if anyone in the community has faced similar issues with Glue 5.0 and managed to resolve them. Thank you.