前言：blackbox_exporter是Prometheus官方提供的exporter之一，主要提供http、dns、tcp、icmp的监控数据采集。Consul主要提供服务发现、健康检查等功能，本次集成主要使用服务发现功能。本文主要实现，基于consul_sd_config&consul的prometheus服务发现，实现网络设备ping监控，站点可用性监控，证书相关信息监控。安装环境：k8sconsulPrometheusblackbox_exporter1:Consul安装1.1:使用helm安装consulBash#添加consulhelmsourcehelmrepoaddhashicorphttps://helm.releases.hashicorp.com#installconsulhelm-nconsulinstall\--setstorageClass=alicloud-disk-efficiency\consulhashicorp/consul\--version=0.32.11.2:查看服务安装状态Bash[root@xxxxxxxxconsul_install]#kubectl-nconsulgetpodsNAMEREADYSTATUSRESTARTSAGEconsul-consul-9lxfc1/1Running06d1hconsul-领事-ntqcf1/1运行06d1hconsul-consul-q7c6f1/1运行06d1hconsul-consul-server-01/1运行06d1hconsul-consul-server-11/1运行06d1hconsul-consul-server-21/1Running06d1h1.3:nginx-ingressconsul_ingress.ymlBash#consul.xxxxxx.cn----->替换为正确的域名apiVersion:networking.k8s.io/v1kind:Ingressmetadata:name:consul-ingressnamespace:consul注释:kubernetes.io/ingress.class:nginxnginx.ingress.kubernetes.io/rewrite-target:/spec:rules:-host:consul.xxxxxx.cnhttp:paths:-path:/pathType:Prefix后端:service:name:consul-consul-uiport:number:80执行部署Bashkubectlapply-fconsul_ingress.yml1.4:accesstest2:Blackbox_export2.1:blackboxinstallblackbox-exporter-config.yamlBashapiVersion:v1kind:ConfigMapmetadata:name:blackbox-exporterlabels:app:blackbox-exporterdata:blackbox.yml:|-modules:##----------DNS检测配置----------dns_tcp:prober:dnsdns:transport_protocol:"tcp"preferred_ip_protocol:"ip4"query_name:"kubernetes.default.svc.cluster.local"#用于检测域名可用URLquery_type:"A"##----------TCP检测模块配置------------tcp_connect:prober:tcptimeout:5s##----------ICMP检测配置------------ping:亲ber:icmptimeout:5sicmp:preferred_ip_protocol:"ip4"##----------HTTPGET2xx检测模块配置----------http_get_2xx:prober:httptimeout:10shttp:method:GETpreferred_ip_protocol:"ip4"valid_http_versions:["HTTP/1.1","HTTP/2"]valid_status_codes:[200]#已验证的HTTP状态码，默认为2xxno_follow_redirects:false#不跟随重定向##----------HTTPGET3xx检测模块配置------------http_get_3xx:prober:httptimeout:10shttp:method:GETpreferred_ip_protocol:"ip4"valid_http_versions:["HTTP/1.1","HTTP/2"]valid_status_codes:[301,302,304,305,306,307]#验证的HTTP状态码，默认为2xxno_follow_redirects:false#不跟随重定向##----------HTTPPOST监控模块------------http_post_2xx:prober:httptimeout:10shttp:method:POSTpreferred_ip_protocol:"ip4"valid_http_versions:["HTTP/1.1","HTTP/2"]#headers:#HTTP头设置#Content-Type:application/json#body:'{}'#请实体设置blackbox-exporter-deploy.yamlBashapiVersion：v1kind：Servicemetadata：名称：blackbox-exporter标签：k8s-app：blackbox-exporterspec：类型：ClusterIP端口：-名称：http端口：9115targetPort：9115选择器：k8s-app：blackbox-exporter---apiVersion:apps/v1kind:Deploymentmetadata:name:blackbox-exporterlabels:k8s-app:blackbox-exporterspec:replicas:1selector:matchLabels:k8s-app:blackbox-exporter模板：元数据：标签：k8s-app:blackbox-exporter规范：容器：-名称：黑盒导出器图像：舞会/黑盒导出器：v0.19.0参数：---config.file=/etc/blackbox_exporter/blackbox.yml---web.listen-address=:9115---log.level=infoports:-name:httpcontainerPort:9115resources:limits:cpu:3memory:6000Mirequests:cpu:100mmemory:50MilivenessProbe:tcpSocket:port:9115initialDelaySeconds:5timeoutSeconds:5periodSeconds:10successThreshold:1failureThreshold:3readinessProbe:tcpSocket:port:9115initialDelaySeconds:5timeoutSeconds:5periodSeconds:10successThreshold:1failureThreshold:3volumeMounts:-name:configmountPath:/etc/blackbox_exportervolumes:-name:configconfigMap:name:blackbox-exporterdefaultMode:420执行安装Bashkubectlapply-fblackbox-exporter-deploy.yamlkubectlapply-fblackbox-exporter-config.yaml2.2：nginxingressblackbox-exporterblackbox_ingress.ymlBashapiVersion:networking.k8s.io/v1kind:Ingressmetadata:name:blackbox-ingress命名空间:监控注解:kubernetes.io/ingress.class:nginxnginx.ingress.kubernetes.io/rewrite-target:/spec:rules:-host:blackbox-devops.lululemon.cnhttp:paths:-path:/pathType:Prefixbackend:service:name:blackbox-exporterport:number:9115执行安装Bashkubectlapply-fblackbox_ingress.yml3:romtheus添加服务动态发现Bash#####http_get_2xx数据获取-job_name:http_get_2xxparams:module:-http_get_2xxscrape_interval:2sscrape_timeout:2smetrics_path:/probeconsul_sd_configs:#consul服务地址-server:consul-consul-server.consul.svc.cluster.local:8500tag_separator:','services:-http_get_2xxrelabel_configs:-source_labels:['__meta_consul_service_address']target_label:__param_target-source_labels:['__meta_consul_service_address']target_label:instance-target_label:__address__##blackbox-export地址替换：blackbox-exporter.monitoring.svc.cluster.local:9115#######icmp配置-job_name:blackbox_icmpparams:module:-pingscrape_interval:2sscrape_timeout:2smetrics_path:/probeconsul_sd_configs:#consul服务地址-server:consul-consul-server.consul.svc.cluster.local:8500tag_separator:','服务：-pingrelabel_configs：-source_labels：['__meta_consul_service_address']target_label：__param_target-source_labels：['__meta_consul_service_address']target_label：实例-target_label：__address__##blackbox-export地址替换：blackbox-exporter.monitoring.svc.cluster。位置al:91154:添加icmp监控4.1:添加监控地址到consulicmp_listbash192.168.1.1192.168.1.2add_consul_service_icmp.shBash#!/usr/bin/envbaship_addr=$1iftest"$ip_addr";thencurl-XPUT-d'{"id":"icmp_'${ip_addr}'","name":"ping","address":"'${ip_addr}'","port":443,"Meta":{"env":"prod","team":"network","project":"network","owner":"Mike"},"tags":["node"],"checks":[{"http":"http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval":"15s"}]}'\http://consul-consul-server:8500/v1/agent/service/registerelseecho"Pleaseenterparameters"fiaddservicepingBashforiin`caticmp_list`;dobashadd_consul_service_icmp.sh$i;done4.2:查看consul服务4.3:删除ping监控地址脚本bash#!/usr/bin/envbaship_addr=$1curl-XPUThttp://consul-consul-server:8500/v1/agent/service/deregister/icmp_${ip_addr}5:添加http_get_2xx5.1:添加监控域名domain_name_listBashwwww.baidu.comwwww.1111.comwwww.2222.comadd_consul_service_http_get_2xx.shBash#!/usr/bin/envbashservice_name=$1iftest"$service_name";thencurl-XPUT-d'{"id":"http_get_2xx_'${service_name}'","name":"http_get_2xx","address":"https://'${service_name}'","port":443,"Meta":{"env":"prod","team":"web","project":"web","owner":"Devops"},"tags":["node"],"checks":[{"http":"http://blackbox-exporter.monitoring.svc.cluster.local:9115/","interval":"15s"}]}'\http://consul-consul-server:8500/v1/agent/service/registerelseecho"Pleaseenterparameters"fiaddservicehttp_get_2xxBashforiin`catdomain_name_list`;dobashadd_consul_service_http_get_2xx.sh$i;done5.2:查看consul服务5.3:删除域名监控脚本del_consul_service_http_get_2xx.shBash#!/usr/bin/envbaship_addr=$1curl-XPUThttp://consul-consul-server:8500/v1/agent/service/deregister/http_get_2xx_${ip_addr}6：查看prometheus监控总结：采用以上方案，黑盒监控自建cmdb平台很容易集成到自动化监控中，不需要过多的人工干预，可以节省大量的人力成本。grafana的配置这里就不介绍了，通过谷歌来完成。
