当前位置: 首页 > 后端技术 > Python

Oozie(HUE)调度Spark2

时间:2023-03-25 19:41:06 Python

EnvironmentJDK1.8.0Hadoop2.6.0Scala2.11.8Spark2.1.2Oozie4.1Hue3.9yarnlocalmodeEnterWorkspace进入lib目录,上传jar和配置文件拖拽SparkProgram选择刚才的lib目录进行填充在jar名称中,点击添加确认,填写主要业务类的名称,并配置参数。单击小齿轮可查看其他参数。保存配置并提交。运行yarncluster模式,进入Workspace。进入lib目录,上传jar和配置文件。name填写完整的HDFS路径hdfs://localcluster/user/hue/oozie/workspaces/hue-oozie-1570773494.4/lib/DataWarehouse-1.0-SNAPSHOT.jar填写主要业务类的名称,点击减号要删除FILES,配置参数hdfs://localcluster/user/hue/oozie/workspaces/hue-oozie-1570773494.4/lib/DataWarehouse-1.0-SNAPSHOT.jardw.user.qhy.wc.WordCount--properties-filespark.properties点击小齿轮查看其他ParameterChangeclienttocluster保存配置提交并运行OozieHTTP接口OozieWebServicesAPI官方文档workflow.xml操作失败,错误信息[${wf:errorMessage(wf:lastErrorNode())}]<org.apache.oozie.action.hadoop.SparkMain],main()抛出异常,尝试添加(hdfs://localcluster/user/hue/oozie/workspaces/hue-oozie-1570758098.65/lib/DataWarehouse-1.0-SNAPSHOT.jar)多次到分布式缓存。可以参考这篇文章的处理方法:java.lang.IllegalArgumentException:Attempttoadd(custom-jar-with-spark-code.jar)multipletimestothedistributedcache报类似下面的Error(kryo)java。io.IOException:java.lang.NullPointerExceptionjava.io.EOFExceptioncom.esotericsoftware.kryo.KryoException可能是kryoserializer使用不当造成的,最简单的解决方法是spark.serializer=org。apache.spark.serializer.KryoSerializer改回默认的spark.serializer=org.apache.spark.serializer.JavaSerializer进一步参考本文解决方案:Spark2的序列化(JavaSerializer/KryoSerializer)本文来自walkersnapshot