Sonal G.

i mean the patch needs to support Zingg running on Spark locally on user machines.

hmm..maybe a bug then. if you can, please do try the local version and submit a pr. that will be very helpful

that looks good to me Umang D.. However, doesnt the SparkContext created automatically have the checkpoint dir when you pass it in the conf? The current code only sets the default when no value is found.

Sure, thanks!

Aniello G. any pointers here for emr?

Yes please check that the checkpoint folder is set on the spark config/context in emr before running Zingg.

ZinggDir is for the models and not for checkpoint folders

It seems to be complaining about the spark checkpoint directory, which is set to local /tmp/checkpoint by default by Zingg. Is there a way to change this before running Zingg in emr?

you could definitely try the debug logs for Matcher class and see how many comparisons it is doing.

Are all your cores occupied? Do you have like 8 cores or more?