Rajesh P.

Merry Christmas and Happy New Year to Zingg too... 🎉

Thanks for acknowledging ! 🙂

will keep posted here if anything i can find out

yes Sonal... I understand... but first why selective properties are getting shown in the output log... that needs to be investigated

yes...

Sonal G., i have tried to pass the --conf option as well and passed the spark zingg.conf properties file. It is not able to run... But, when i tried to run using --properties-file by passing the zingg.conf file, it is able to pickup few spark properties ( like spark.executor.memory, spark.executor.cores ...) , but not all the spark properties (like spark.serializer ) mentioned in the zingg.conf file... (you can see in the screenshot above)

yeah Sonal G., EMR too takes the properties like spark-submit command... checked here https://repost.aws/knowledge-center/emr-set-spark-parameters

Yes I am running on AWS EMR, and to the above note, i believe the arguments, like --conf, --properties-file would still hold true right while running the zingg job right ?

yes, --conf is specific to the config.json.env file that we are passing, which contains the field definitions stuff. --properties-file is for the zingg.conf file as mentioned here in Zingg docs https://blog.infostrux.com/identity-resolution-with-zingg-ai-snowflake-and-aws-emr-for-the-canadian-football-league-22cf0850ab53