使用Ganglia监控Spark

由于软件协议问题,Spark build默认不包含Ganglia的编译,使用Ganglia监控Spark首先需要重新编译Spark,将GangliaSink编译到Spark中。

使用sbt编译时,加入 -Pspark-ganglia-lgpl

build/sbt -Pyarn -Phadoop-2.4 -Phive -Phive-thriftserver -Pspark-ganglia-lgpl assembly  

附:Ganglia的安装配置

安装epel

rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm  

修改epel配置, 修改文件“/etc/yum.repos.d/epel.repo”, 将baseurl的注释取消, mirrorlist注释掉

中心节点安装(yum install -y xxx):

gmetad: ganglia-gmetad  
gmond: ganglia-gmond  
ganglia-web: ganglia-web  
Apache: httpd  
php: php  

被监测节点:

gmond:ganglia-gmond  

中心节点Ganglia-web配置

ln -s /usr/share/ganglia /var/www/html  
chown -R apache:apache /var/www/html/ganglia  
chmod -R 755 /var/www/html/ganglia  

访问权限:修改/etc/httpd/conf.d/ganglia.conf

<Location /ganglia>  
  Order deny,allow 
  # Deny from all 
  # Allow from 127.0.0.1 
  # Allow from ::1 
  Allow from all 
# Allow from .example.com 
</Location>  

rrd数据库权限

chown -R ganglia:ganglia /var/lib/ganglia/rrds  

配置/etc/ganglia/gmetad.conf

data_source “ClusterName” masterIP:8649  

配置/etc/ganglia/gmond.conf

cluster {  
  name = “ClusterName” 
  owner = “ganglia"
  ... 
} 
udp_send_channel {  
  # the host who gather this cluster's monitoring data and send these data   to gmetad node 
  host = masterIP
  port = 8649 
} 
udp_recv_channel {  
  port = 8649 
} 
tcp_accept_channel {  
  port = 8649 
}

中心节点启动

service httpd start  
service gmetad start  
service gmond start  

自启动

chkconfig httpd on  
chkconfig gmetad on  
chkconfig gmond on  

被检测节点

service gmond start  
chkconfig gmond on