当前位置: 首页 > Linux

Logstash7.x安装及数据导入(阿里云Ubuntu)

时间:2023-04-06 05:46:14 Linux

接上一篇:Kibana7.x安装及界面预览(阿里云Ubuntu)系统环境操作系统:Ubuntu18.04LTS(阿里云)系统IP#内网,私有地址172.内.内.内#外网,公网地址112.外层。外。Elasticsearch版本:7.2Kibana版本:7.2Logstash版本:7.2安装配置官方文档:https://www.elastic.co/guide/...Logstash不同于Elasticsearch和Kibana,需要分开安装Java环境~$sudoaptinstalldefault-jdk~$java-versionopenjdkversion"11.0.3"2019-04-16OpenJDKRuntimeEnvironment(build11.0.3+7-Ubuntu-1ubuntu218.04.1)OpenJDK64-BitServerVM(build11.0.3+7-Ubuntu-1ubuntu218.04.1,混合模式,分享)下载logstashwgethttps://artifacts.elastic.co/downloads/logstash/logstash-7.2.0.tar.gz解压logstashtar-zxflogstash-7.2.0。tar.gz后续操作在logstash-7.2.0目录下cdlogstash-7.2.0/到groupslens下载MovieLens测试数据集wgethttp://files.grouplens.org/datasets/movielens/ml-latest-small.zip解压测试数据集unzipml-latest-small.zip创建并编辑logstash.conf文件,添加如下内容(Ruby语法)input{file{path=>"/home/walker/es/ml-latest-small/movies.csv"start_position=>"beginning"sincedb_path=>"/dev/null"}}filter{csv{separator=>","columns=>["id","content","genre"]}mutate{split=>{"genre"=>"|"}remove_field=>["path","host","@timestamp","message"]}mutate{split=>["content","("]add_field=>{"title"=>"%{[content][0]}"}add_field=>{"year"=>"%{[content][2]}"}}mutate{convert=>{"year"=>"integer"}strip=>["title"]remove_field=>["path","host","@timestamp","message","content"]}}output{elasticsearch{hosts=>"http://172.18.193.52:9200"index=>"movies"document_id=>"%{id}"}stdout{}}导入数据,注意导入后不会自动退出,使用Ctrl-C手动退出#个人路径不同./bin/logstash-f/home/walker/es/ml-latest-small/logstash.conf管理查看数据(Index相当于关系数据库的Table)关系数据库与ElasticsearchDevtools的抽象类比查看文档总数(即数据总数)#查看总数据量GET/movies/_count#consoleoutput{"count":9743,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0}}Viewdatawithid1#Viewdatawithid1GET/movies/_doc/1#Consoleoutput{"_index":"movies","_type":"_doc",“_id”:“1”,“_version”:1,“_seq_no”:121,“_primary_term”:1,“found”:true,“_source”:{“@version”:“1”,“genre”:["Adventure","Animation","Children","Comedy","Fantasy"],"year":1995,"id":"1","title":"ToyStory"}}删除电影数据(Elasticsearch的Index可以类比关系数据库的Table)#删除命令DELETE/movies#控制台输出{"acknowledged":true}walker的目录结构$tree/home/walker/es/-L2/home/walker/es/├──elasticsearch-7.2.0│├──bin│├──配置│├──数据│├──jdk│├──lib│├──LICENSE.txt│├──logs│├──模块│├──NOTICE.txt│├──插件│└──README.textile├──elasticsearch-7.2.0-linux-x86_64.tar.gz├──kibana-7.2.0-linux-x86_64│├──bin│├──built_assets│├──config│├──data│├──LICENSE.txt│├──node│├──node_modules│├──NOTICE.txt│├──优化│├──package.json│├──插件│├──README.txt│├──src│├──目标│├──webpackShims│└──x-pack├──kibana-7.2.0-linux-x86_64.tar.gz├──logstash-7.2.0│├──bin│├──配置│├──贡献者│├──数据│├──Gemfile│├──Gemfile。锁│├──lib│├──LICENSE.txt│├──日志│├──logstash-core│├──logstash-core-plugin-api│├──modules│├──NOTICE.TXT│├──工具│├──供应商│└──x-pack├──logstash-7.2.0.tar.gz├──ml-latest-small│├──links.csv│├──logstash.conf│├──movies.csv│├──ratings.csv│├──README.txt│└──tags.csv└──ml-latest-small.zip本文为阮一鸣《Elasticsearch核心技术与实战》的学习笔记。