ELK日志环境入门-02之Logstash官方源码部署
一、ELK 简介
ELK 是一个开源的实时日志分析平台，它主要由 Elasticsearch、Logstash 和 Kiabana 三部分组成。
二、Logstash概述
Logstash 主要用于收集服务器日志，它是一个开源数据收集引擎，具有实时管道功能。Logstash 可以动态地将来自不同数据源的数据统一起来，并将数据标准化到您所选择的目的地。
Logstash 收集数据的过程主要分为以下三个部分：
- 输入：数据（包含但不限于日志）往往都是以不同的形式、格式存储在不同的系统中，而 Logstash 支持从多种数据源中收集数据（File、Syslog、MySQL、消息中间件等等）。
- 过滤器：实时解析和转换数据，识别已命名的字段以构建结构，并将它们转换成通用格式。
- 输出：Elasticsearch 并非存储的唯一选择，Logstash 提供很多输出选择。
三、ELK 平台搭建基于Logstash官方源码部署
本节主要介绍搭建 ELK 日志平台，包括安装 Indexer 角色的 Logstash，Elasticsearch 以及 Kibana 三个组件。完成本小节，您需要做如下准备：
- 一台 Linux/CentOS 机器或虚拟机，作为入门教程，此处省略了 Elasticsearch 集群的搭建，将 Logstash(Indexer)、Elasticsearch 以及 Kibana 安装在不同机器上，即ELK分布式部署。
- 在 Linux/CentOS 上安装 JDK，注意 Logstash 要求 JDK 在 1.8 版本或以上。
- ELK官方组件下载，您可以在 ELK源码下载。
- 本文详细演示Logstash官方源码部署
- Elasticsearch官方源码部署，请移步《ELK日志环境入门-01之Elasticsearch官方源码部署》
- Kibana官方源码部署，请移步《ELK日志环境入门-03之Kibana官方源码部署》
四、Logstash官方源码部署
1、安装依赖环境
[root@node opt]# wget https://download.oracle.com/otn/java/jdk/8u301-b09/d3c52aa6bfa54d3ca74e617f18309292/jdk-8u301-linux-x64.rpm?AuthParam=1630374823_29c8a62eebc8754fe26188019ff8acfc
[root@node opt]# rpm -ivh jdk-8u301-linux-x64.rpm
正在升级/安装...
1:jdk1.8-2000:1.8.0_301-fcs ################################# [100%]2、下载elastic官网最新版本Logstash-7.14.0
[root@node ~]# ll /opt/
总用量 426068
-rw-r--r-- 1 root root 321737210 8月 31 09:49 kibana-7.14.0-linux-x86_64.tar.gz
3、创建并解压至自定义目录
[root@node ~]# tar xf /opt/logstash-7.14.0-linux-x86_64.tar.gz -C /usr/local/elk/
4、 修改 logstash .yml 文件配置
http.host: 10.100.202.102 //局域网ip或者公网ip5、举例创建常见TCP日志收集管道
[root@node ~]# vim /usr/local/elk/logstash-7.14.0/config/tcp.conf## 添加如下内容
input{
tcp{
type => "tcp"
mode => "server"
port => "9601"
host => "10.100.202.102" //logstash主机IP
codec => json_lines
}
}
output{
elasticsearch{
hosts =>["10.100.202.100:9200"] //elasticserach主机IP
index=>"tcp_log-%{+YYYY.MM.dd}.log"
#user => "elastic"
#password => "BH@yihubai!"
}
}
6、启动验证服务是否正常
[root@node ~]# /usr/local/elk/logstash-7.14.0/bin/logstash -f /usr/local/elk/logstash-7.14.0/config/tcp.conf &
[root@node ~]# ss -lnt
[root@node ~]# ps aux|grep java
root 64558 13.1 31.6 3216980 589984 pts/0 Sl 15:38 1:44 /usr/java/jdk1.8.0_301-amd64/bin/java -Xms512m -Xmx512m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -cp /usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/checker-compat-qual-2.0.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/commons-codec-1.14.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/commons-logging-1.2.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/error_prone_annotations-2.1.3.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/guava-24.1.1-jre.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jackson-databind-2.9.10.8.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jackson-dataformat-yaml-2.9.10.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/janino-3.1.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/javassist-3.26.0-GA.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jruby-complete-9.2.19.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/log4j-1.2-api-2.14.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/log4j-api-2.14.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/log4j-core-2.14.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/log4j-jcl-2.14.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/log4j-slf4j-impl-2.14.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/logstash-core.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/slf4j-api-1.7.30.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/snakeyaml-1.23.jar org.logstash.Logstash -f /usr/local/elk/logstash-7.14.0/config/tcp.conf
7、创建Logstash开机自启动脚本
[root@node ~]# vim /etc/systemd/system/logstash.service
## 粘贴如下内容
[Unit]
Description=logstash
[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/elk/logstash-7.14.0/bin/logstash -f /usr/local/elk/logstash-7.14.0/config/tcp.conf
Restart=always
LimitNOFILE=16384
[Install]
WantedBy=multi-user.target授权脚本777权限
[root@node ~]# chmod 777 /etc/systemd/system/logstash.service
常用命令
systemctl enable logstash
systemctl start logstash
systemctl status logstash # 查看状态
本文是原创文章，采用 CC BY-NC-ND 4.0 协议，完整转载请注明来自 Lukey
阅读建议
评论
匿名评论
隐私政策
你无需删除空行，直接评论以获取最佳展示效果
