当前位置: 首页 > Linux

Centos7安装ELK进行日志收集(elasticsearch、logstash、kibana)

时间:2023-04-06 21:52:55 Linux

不用说,每个人都听说过麋鹿。最近自己建了一套日志收集工具,供大家参考:1、安装elasticsearch、logstash、kibana。强烈建议在es官网安装:今天是2019.4.9,目前最新版本是6.7.1,三个都是用6.7.1版本安装的(版本要一样,否则有会各种疑难杂症)elasticsearch:https://www.elastic.co/downlo...logstash:https://www.elastic.co/cn/dow...kibana:https://www.elastic.co/cn/dow...其中es的安装教程我写在这篇文章中:https://segmentfault.com/a/11...,下面介绍logstash和kibana的安装过程1.logstash:下载后##进入安装包所在目录,解压tar-xflogstash-6.7.1.tar.gz##切换到bin目录cd/logstash-6.7.1/bin##编辑或创建一个启动配置文件,然后使用这个配置文件启动viminput_flter_output.confinput{file{path=>"/crawler/jenkins/*.log"##生成日志目录type=>"cml"##Indextypestart_position=>"beginning"##在开头输入原始日志信息}stdin{}}filter{}output{elasticsearch{action=>"index"hosts=>"www.iamcrawler.cn:9500"##输出到elasticsearch上面index=>"log-%{+yyyy.MM.dd}"##生成日志时间索引}}#最后保存退出##在bin目录下,再写一个启动batvimrun.batshlogstash-finput_flter_output.conf&#最后保存退出然后运行run.bat即可[root@iamcrawlerbin]#shrun.bat##正常会出现以下情况:[root@iamcrawlerbin]#SendingLogstashlogsto/crawler/logstash/现在通过log4j2.properties[2019-04-09T13:11:40,120][WARN][logstash.config.source.multilocal]配置的logstash-6.7.1/logs忽略“pipelines.yml”文件,因为模块或命令指定了行选项[2019-04-09T13:11:40,138][INFO][logstash.runner]StartingLogstash{"logstash.version"=>"6.7.1"}[2019-04-09T13:11:50,041][信息][logstash.pipeline]启动管道{:pipeline_id=>"main","pipeline.workers"=>2,"pipeline.batch.size"=>125,"pipeline.batch.delay"=>50}[2019-04-09T13:11:50,697][INFO][logstash.outputs.elasticsearch]Elasticsearch池URL已更新{:changes=>{:removed=>[],:added=>[http://www.iamcrawler.cn:9500/]}}[2019-04-09T13:11:51,065][WARN][logstash.outputs.elasticsearch]恢复连接到ES实例{:url=>"http://www.iamcrawler.cn:9500/"}[2019-04-09T13:11:51,191][INFO][logstash.outputs.elasticsearch]ES输出版本确定{:es_version=>6}[2019-04-09T13:11:51,196][WARN][logstash.outputs.elasticsearch]检测到6.x及以上集群:`type`事件字段不会用于确定文档_type{:es_version=>6}[2019-04-09T13:11:51,232][INFO][logstash.outputs.elasticsearch]使用默认映射模板[2019-04-09T13:11:51,253][INFO][logstash.outputs.elasticsearch]新增Elasticsearch输出{:class=>"LogStash::Outputs::ElasticSearch",:hosts=>["//www.iamcrawler.cn:9500"]}[2019-04-09T13:11:51,287][INFO][logstash.outputs.elasticsearch]正在尝试安装模板{:manage_template=>{"template"=>"logstash-*","version"=>60001,"settings"=>{"index.refresh_interval"=>"5s"},"mappings"=>{"_default_"=>{"dynamic_templates"=>[{"message_field"=>{"路径_match"=>"message","match_mapping_type"=>"string","mapping"=>{"type"=>"text","norms"=>false}}},{"string_fields"=>{"match"=>"*","match_mapping_type"=>"string","mapping"=>{"type"=>"text","norms"=>false,"fields"=>{"keyword"=>{"type"=>"keyword","ignore_above"=>256}}}}}],"properties"=>{"@timestamp"=>{"type"=>"date"},"@version"=>{"type"=>"keyword"},"geoip"=>{"dynamic"=>true,"properties"=>{"ip"=>{"type"=>"ip"},"location"=>{"type"=>"geo_point"},"latitude"=>{"type"=>"half_float"},"longitude"=>{"type"=>"half_float"}}}}}}}}[2019-04-09T13:11:51,399][INFO][logstash.outputs.elasticsearch]将elasticsearch模板安装到_template/logstash[2019-04-09T13:11:51,783][INFO][logstash.inputs.file]没有设置sincedb_path,根据“路径”设置生成一个{:sincedb_path=>"/crawler/logstash/logstash-6.7.1/data/plugins/inputs/file/.sincedb_6677650ec826fa62a735f6625357dead",:path=>["/crawler/jenkins/*.log"]}[2019-04-09T13:11:51,896][INFO][logstash.pipeline]管道成功启动{:pipeline_id=>"main",:thread=>"#"}[2019-04-09T13:11:52,010][INFO][filewatch.observingtail]开始,创建Discoverer,Watchwithfile和sincedb集合[2019-04-09T13:11:52,033][INFO][logstash.agent]管道运行{:count=>1,:running_pipelines=>[:main],:non_running_pipelines=>[]}[2019-04-09T13:11:52,723][INFO][logstash.agent]SuccessfullystartedLogstashAPIendpoint{:port=>9600}二、Kibana安装Kibana安装比较简单,下载后进入安装目录的config目录,如:cd/crawler/kibana/kibana-6.7.1-linux-x86_64/configvimkibana.yml##添加如下命令server.host:0.0.0.0elasticsearch.url:"http://localhost:9500"#这里是es的http地址##进入kibana目录,运行如下命令后台运行kibana。