spark WARNTaskSchedulerImpl:Initial job has not accepted any

spark在提交任务时,出现如下错误:

15/03/26 22:29:36 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory15/03/26 22:29:51 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory15/03/26 22:30:06 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory15/03/26 22:30:21 WARN TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient memory

从警告信息上看,初始化job时没有获取到任何资源;提示检查集群,确保workers可以被注册并有足够的内存资源。

如上问题产生的原因是多方面的,可能原因如下:

1.因为提交任务的节点不能和spark工作节点交互,因为提交完任务后提交任务节点上会起一个进程,展示任务进度,大多端口为4044,工作节点需要反馈进度给该该端口,所以如果主机名或者IP在hosts中配置不正确。所以检查下主机名和ip是否配置正确

2.有可能是内存不足

检查内存

conf.set("spark.executor.memory", "3000m")

Make sure to setSPARK_LOCAL_IPandSPARK_MASTER_IP.

查看8080端口,确保一些workers保持Alive状态,确保 some cores 是可利用的。

3.driver在不停地工作,,一直处于running状态,而从前端8080页面又可看出程序的状态一直是WAITING。submit所在的shell输出driver信息后便退出了,目前还未得到程序运行结果,这显然是因为该程序处于WAITING状态,还未运行。而导致这一结果,应该跟日志的输出有关。值得一提的是,到目前为止,出了更改了对master ip和port的配置以及worker的配置外,其他深入的配置还需进一步研究。

如果我们想要更多的玫瑰花,就必须种植更多的玫瑰树。

spark WARNTaskSchedulerImpl:Initial job has not accepted any

相关文章:

你感兴趣的文章:

标签云: