Prem A.

Commented on Error initializing SparkContext due to spark.kryos...·Posted inHelp Zingg

Prem A.

is it work in aws?

Commented on Error initializing SparkContext due to spark.kryos...·Posted inHelp Zingg

Prem A.

80m records with 8-10 fields took less than 2 hours on 1 driver(128 GB RAM, 32 cores), 8 workers(224 GB RAM, 64 cores). This is a user-reported stat without any optimization.

Commented on Error initializing SparkContext due to spark.kryos...·Posted inHelp Zingg

Prem A.

thanks Sonal run completed successfully with 20L

Commented on Error initializing SparkContext due to spark.kryos...·Posted inHelp Zingg

Prem A.

or 0.001

Commented on Error initializing SparkContext due to spark.kryos...·Posted inHelp Zingg

Prem A.

you mean 0.1

Commented on Error initializing SparkContext due to spark.kryos...·Posted inHelp Zingg

Prem A.

"labelDataSampleSize" : 0.5,

Commented on Error initializing SparkContext due to spark.kryos...·Posted inHelp Zingg

Prem A.

job struck this line

Commented on Error initializing SparkContext due to spark.kryos...·Posted inHelp Zingg

Prem A.

2024-04-02 12:42:26,854 [main] INFO zingg.TrainingDataFinder - Writing uncertain pairs 2024-04-02 12:42:27,834 [dag-scheduler-event-loop] WARN org.apache.spark.scheduler.DAGScheduler - Broadcasting large task binary with size 12.9 MiB [Stage 201:=================================================>(9999 + 1) / 10000]

Commented on Error initializing SparkContext due to spark.kryos...·Posted inHelp Zingg

Prem A.

44 matches

Commented on Error initializing SparkContext due to spark.kryos...·Posted inHelp Zingg

Prem A.

training data 1K records