ELK日志环境入门-02之Logstash官方源码部署

一、ELK 简介

ELK 是一个开源的实时日志分析平台，它主要由 Elasticsearch、Logstash 和 Kiabana 三部分组成。

二、Logstash概述

Logstash 主要用于收集服务器日志，它是一个开源数据收集引擎，具有实时管道功能。Logstash 可以动态地将来自不同数据源的数据统一起来，并将数据标准化到您所选择的目的地。

Logstash 收集数据的过程主要分为以下三个部分：

输入：数据（包含但不限于日志）往往都是以不同的形式、格式存储在不同的系统中，而 Logstash 支持从多种数据源中收集数据（File、Syslog、MySQL、消息中间件等等）。
过滤器：实时解析和转换数据，识别已命名的字段以构建结构，并将它们转换成通用格式。
输出：Elasticsearch 并非存储的唯一选择，Logstash 提供很多输出选择。

三、ELK 平台搭建基于Logstash官方源码部署

本节主要介绍搭建 ELK 日志平台，包括安装 Indexer 角色的 Logstash，Elasticsearch 以及 Kibana 三个组件。完成本小节，您需要做如下准备：

一台 Linux/CentOS 机器或虚拟机，作为入门教程，此处省略了 Elasticsearch 集群的搭建，将 Logstash(Indexer)、Elasticsearch 以及 Kibana 安装在不同机器上，即ELK分布式部署。
在 Linux/CentOS 上安装 JDK，注意 Logstash 要求 JDK 在 1.8 版本或以上。
ELK官方组件下载，您可以在 ELK源码下载。
本文详细演示Logstash官方源码部署
Elasticsearch官方源码部署，请移步《ELK日志环境入门-01之Elasticsearch官方源码部署》
Kibana官方源码部署，请移步《ELK日志环境入门-03之Kibana官方源码部署》

四、Logstash官方源码部署

1、安装依赖环境

[root@node opt]# wget https://download.oracle.com/otn/java/jdk/8u301-b09/d3c52aa6bfa54d3ca74e617f18309292/jdk-8u301-linux-x64.rpm?AuthParam=1630374823_29c8a62eebc8754fe26188019ff8acfc

[root@node opt]# rpm -ivh jdk-8u301-linux-x64.rpm

正在升级/安装...
   1:jdk1.8-2000:1.8.0_301-fcs        ################################# [100%]

2、下载elastic官网最新版本Logstash-7.14.0

[root@node ~]# ll /opt/
总用量 426068
-rw-r--r--  1 root root 321737210 8月  31 09:49 kibana-7.14.0-linux-x86_64.tar.gz

3、创建并解压至自定义目录

[root@node ~]# tar xf /opt/logstash-7.14.0-linux-x86_64.tar.gz -C /usr/local/elk/

4、修改 logstash .yml 文件配置

http.host: 10.100.202.102     //局域网ip或者公网ip

5、举例创建常见TCP日志收集管道

[root@node ~]# vim /usr/local/elk/logstash-7.14.0/config/tcp.conf

## 添加如下内容
input{
  tcp{
   type => "tcp"
   mode => "server"
   port => "9601"
   host => "10.100.202.102"     //logstash主机IP
   codec => json_lines
  }
}

output{
     elasticsearch{
     hosts =>["10.100.202.100:9200"]     //elasticserach主机IP
     index=>"tcp_log-%{+YYYY.MM.dd}.log"
     #user => "elastic"
     #password => "BH@yihubai!"
    }
}

6、启动验证服务是否正常

[root@node ~]# /usr/local/elk/logstash-7.14.0/bin/logstash -f /usr/local/elk/logstash-7.14.0/config/tcp.conf &
[root@node ~]# ss -lnt

[root@node ~]# ps aux|grep java

root      64558 13.1 31.6 3216980 589984 pts/0  Sl   15:38   1:44 /usr/java/jdk1.8.0_301-amd64/bin/java -Xms512m -Xmx512m -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djruby.compile.invokedynamic=true -Djruby.jit.threshold=0 -Djruby.regexp.interruptible=true -XX:+HeapDumpOnOutOfMemoryError -Djava.security.egd=file:/dev/urandom -Dlog4j2.isThreadContextMapInheritable=true -cp /usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/animal-sniffer-annotations-1.14.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/checker-compat-qual-2.0.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/commons-codec-1.14.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/commons-compiler-3.1.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/commons-logging-1.2.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/error_prone_annotations-2.1.3.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/google-java-format-1.1.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/gradle-license-report-0.7.1.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/guava-24.1.1-jre.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/j2objc-annotations-1.1.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jackson-annotations-2.9.10.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jackson-core-2.9.10.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jackson-databind-2.9.10.8.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jackson-dataformat-cbor-2.9.10.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jackson-dataformat-yaml-2.9.10.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/janino-3.1.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/javassist-3.26.0-GA.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jruby-complete-9.2.19.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/jsr305-1.3.9.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/log4j-1.2-api-2.14.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/log4j-api-2.14.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/log4j-core-2.14.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/log4j-jcl-2.14.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/log4j-slf4j-impl-2.14.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/logstash-core.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.commands-3.6.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.contenttype-3.4.100.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.expressions-3.4.300.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.filesystem-1.3.100.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.jobs-3.5.100.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.resources-3.7.100.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.core.runtime-3.7.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.equinox.app-1.3.100.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.equinox.common-3.6.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.equinox.preferences-3.4.1.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.equinox.registry-3.5.101.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.jdt.core-3.10.0.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.osgi-3.7.1.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/org.eclipse.text-3.5.101.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/reflections-0.9.11.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/slf4j-api-1.7.30.jar:/usr/local/elk/logstash-7.14.0/logstash-core/lib/jars/snakeyaml-1.23.jar org.logstash.Logstash -f /usr/local/elk/logstash-7.14.0/config/tcp.conf

7、创建Logstash开机自启动脚本

[root@node ~]# vim /etc/systemd/system/logstash.service

## 粘贴如下内容

[Unit]
Description=logstash

[Service]
Type=simple
User=root
Group=root
ExecStart=/usr/local/elk/logstash-7.14.0/bin/logstash -f /usr/local/elk/logstash-7.14.0/config/tcp.conf
Restart=always
LimitNOFILE=16384

[Install]
WantedBy=multi-user.target

授权脚本777权限

[root@node ~]# chmod 777 /etc/systemd/system/logstash.service

常用命令

systemctl enable logstash
systemctl start logstash
systemctl status logstash # 查看状态