Spark 通过HIVE ON HBASE表读取数据源,报错:unread block data
具体错误信息示例如下:
18/03/19 20:46:54 ERROR Executor: Exception in task 0.0 in stage 0.0 (TID 0)
java.lang.IllegalStateException: unread block dataat java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2400)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1379)at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:722)
18/03/19 20:46:54 WARN TaskSetManager: Lost task 0.0 in stage 0.0 (TID 0, localhost): java.lang.IllegalStateException: unread block dataat java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2400)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1379)at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1970)at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1894)at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1777)at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1347)at java.io.ObjectInputStream.readObject(ObjectInputStream.java:369)at org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:75)at org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:114)at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:253)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:722)
问题原因分析:该问题查了好长时间(开始以为是数据源有问题;后来各种查、各种试,发现是缺少jar包,因为是CDH环境,通过增加spark/jars下面的jar即可:
cd /opt/...../CDH/spark/jars/ln -s /opt/...../CDH/jars/hive-hbase-handler-***.jar hive-hbase-handler***.jar
总结:引起该问题的原因有很多,最好由简入繁、从最简单直接的原因查起,逐一排除,提高排查问题效率。
相关的解决方案:
方法一:spark-submit --jars
根据spark官网,在提交任务的时候指定--jars,用逗号分开。这样做的缺点是每次都要指定jar包,如果jar包少的话可以这么做,但是如果多的话会很麻烦。可以封装一个sh脚本做目录/引用文件扫描也可以。
spark-submit --master yarn-client --jars ***.jar,***.jar(你的jar包,用逗号分隔) myjar.jar
方法二:extraClassPath
提交时在spark-default中设定参数,将所有需要的jar包考到一个文件里,然后在参数中指定该目录就可以了,较上一个方便很多:
spark.executor.extraClassPath=/extlib/*
spark.driver.extraClassPath=/extlib/*
# 修改为自己的目录
方法二:fat-jar
如果你还是觉得第二种麻烦,这种方法是将所有依赖的jar,包括你写的代码全部打包在一起(fat-jar),就是提交的时候比较慢,毕竟大啊。还有可能和系统jar冲突。自己注意解决把。
参考链接:
- setting-spark-classpaths-on-ec2-spark-driver-extraclasspath-and-spark-executor
- Spark任务提交jar包依赖解决方案