Ángel Á.

Interesting question indeed. I'd like to analyze it too.

Most likely it’s not going to provide any clues, but what about spawning a thread in the notebook to get the Python threads’ stack traces? I wrote about how to do it in this article: https://blog.devgenius.io/apache-spark-wtf-written-in-pyspark-34591c5f32bf

You mentioned a notebook. Are you using PySpark?

sorry, I saw you already included the version

That's weird. What version of Spark you're using? Could you send us a threaddump of the driver and one of the executors with active tasks running?

could you share some code example of what they're executing? could you provide number of executors, cores and memory settings?

hmm...not sure about that ... I mean, if my hypothesis is correct, that may work ... not because it's the cause but only because it's going to execute faster so the time settings are not applied. But maybe in the future ...as the data increases, you get the same errors back.

I would play around a bit with these parameters:

# Extend BlockManager timeout
--conf spark.storage.blockManagerSlaveTimeoutMs=300000  # 5 min, or more

# Extend general network timeout
--conf spark.network.timeout=300s

# Prevent executors from being killed quickly if dynamic allocation is on
--conf spark.dynamicAllocation.executorIdleTimeout=300s