Patrick S.

Hey just wanted to post here as well that we resolved this. This is due to a spark issue in the latest fabric release where large physical plans can cause out of memory issues when using adaptive query execution. They will fix in future releases. Workaround: Use this at the beginning of your code especially when matching large and complex datasets. Without this, match fails every time for us now. spark.conf.set("spark.sql.adaptive.enabled", "false")

Yes pyspark

Hi Angel thanks for replying. I tried on multiple occasions to get the thread dumps but the spark UI becomes unavailable when the match notebook is active and running. I did download the event logs after the session is disconnected, they are about 3GBs in size and don't really indicate any glaring issues. Any other options or ideas? I did log a ticket with Microsoft, as this was working fine before their November release. No update from them Yet.

Yes - this appears to happen when you have a schema enabled lakehouse and attempt to run zingg from the notebook. It presents this error word for word and is repeatable. The schema lakehouse feature is in public preview. My hope is that once this feature is in GA that the functionality works correctly. https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-schemas