Luke B.

Commented on Consistent Connection Reset by Peer Errors During...·Posted inHelp Zingg

I still don't have the details...all on my list to collect though. I definitely appreciate the ideas

Commented on Consistent Connection Reset by Peer Errors During...·Posted inHelp Zingg

this is in Databricks, so I think we just need to get a bigger cluster, or enabling autoscaling. I don't think directly modifying Dynamic Allocation is advisable in this case... in vanilla spark outside of Databricks, that would make sense to me

Commented on Consistent Connection Reset by Peer Errors During...·Posted inHelp Zingg

Luke B.

that's a good question to add to my list... still hoping to get a meeting with them this afternoon

Commented on Consistent Connection Reset by Peer Errors During...·Posted inHelp Zingg

Luke B.

I'm trying to get a meeting with them to view the sparkUI, but I haven't seen it yet. Do you have any back-of-the-envelope guidance for appropriate cluster sizing for 50m records?

Commented on Consistent Connection Reset by Peer Errors During...·Posted inHelp Zingg

Luke B.

they just updated me that they are trying on 50 million records all of a sudden (up from 5 million I think). so I'm confident they are using a very undersized cluster.

Commented on Consistent Connection Reset by Peer Errors During...·Posted inHelp Zingg

Luke B.

does this indicate a data skew problem that is causing a worker to OOM or something? Or is this maybe just a sign they need more partitioning or a bigger cluster?

Posted in Help Zingg·

Luke B.

Consistent Connection Reset by Peer Errors During trainMatch Execution with IOException Stack Trace

My customer is hitting this error consistently when running trainMatch: In the driver logs, we see stack traces like this:

25/11/04 20:23:36 WARN TransportChannelHandler: Exception in connection from /<redact>:40654
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
	at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:254)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:750)
25/11/04 20:23:36 ERROR TransportResponseHandler: Still have 15 requests outstanding when connection from /<redact>:40654 is closed
25/11/04 20:23:36 WARN BlockManagerMasterEndpoint: Error trying to remove broadcast 31337 from block manager BlockManagerId(7, <redact>, 34471, None)
java.io.IOException: Connection reset by peer
	at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
	at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
	at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
	at sun.nio.ch.IOUtil.read(IOUtil.java:192)
	at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
	at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:254)
	at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1132)
	at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:357)
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:151)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.lang.Thread.run(Thread.java:750)

14Comments

Commented on Customer experiencing increasing runtime in findTr...·Posted inHelp Zingg

Luke B.

sounds like the problem is resolved... it isn't clear if he made the suggested change or if something transient was happening... either way, we are on to a new error now. I'll start a new thread below

Commented on How to Configure Jars, Wheels, and Runtimes in the...·Posted inHelp Zingg

Luke B.

yea, sorry, meant to come back and tell you the same... the trial accounts are what we call "serverless", so you cannot spin up classic compute. this is a problem for zingg, b/c zingg installs as a jar, and you are not able to install custom jars on serverless compute

Commented on How to Configure Jars, Wheels, and Runtimes in the...·Posted inHelp Zingg

Luke B.

from there, are you able to see the "all-purpose compute" tab?