Logstash Docker 部署 安装 logstash-output-jdbc
前置步骤参考：https://blog.csdn.net/weixin_44121790/article/details/141305720
问题：
今天使用docker 部署logstash，遇到无法运行的问题，原因是因为配置问题使用了 logstash-output-jdbc ，但是镜像默认没有安装。
配置文件如下：
#注释方法#####
input {syslog {port => "5044"}
}
filter {ruby {code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)"}if("WinFileService" in [message] or "FtpFileTransfer" in [message]){grok{match =>{"message"=>"%{WORD:opt_type} Event: %{DATA:opt_name}, Path: %{DATA:path}, File/Folder: %{WORD:file_type}, Size: %{BASE16FLOAT:size} %{WORD:size_unit}, User: %{DATA:username}, IP: %{IPV4:opt_ip}"}}grok{match=>{"path"=>"%{GREEDYDATA}/%{GREEDYDATA:file_name}"}}if("KB" in [size_unit]){ruby{code =>"event.set('size_byte', event.get('size').to_i * 1024)"}}if("MB" in [size_unit]){ruby{code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024)" }} if("GB" in [size_unit]){ruby{code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024 * 1024)"}}if("Bytes" in [size_unit]){ruby{code =>"event.set('size_byte', event.get('size').to_i)"}}}
}output {stdout {}if("FtpFileTransfer" in [opt_type]){jdbc{driver_jar_path => "/usr/share/logstash/config/jar/mysql-connector-j-8.4.0.jar"driver_class => "com.mysql.cj.jdbc.Driver"connection_string => "jdbc:mysql://192.168.10.23:3306/database_name?user=root&password=xxxxxxx&useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true"statement => ["insert into t_ftp_log(level, event, full_path, file_size, event_time,app_source,user_name,source_ip,file_name) VALUES (?,?,?,?,?,?,?,?,?)","[log][syslog][priority]","opt_name","path","size_byte","@timestamp","[host][hostname]","username","opt_ip","file_name"]} }
}解决方法：
编辑logstash.yml
指定 config路径
http.host: "0.0.0.0"
# xpack.monitoring.elasticsearch.hosts: [ "http://127.0.0.1:9200" ]
path.logs: /usr/share/logstash/logs
path.config: /usr/share/logstash/config/conf.d/*.conf
重点
在此config路径下创建一个简单的config文件logstash.conf，内容如下：
input {syslog {port => 5044}
}output {stdout {codec => rubydebug}
}
然后启动容器
docker run --name logstash-sample -p 5044:5044 -p 9600:9600 -v /opt/docker/logstash/config:/usr/share/logstash/config -v /opt/docker/logstash/data:/usr/share/logstash/data -v /opt/docker/logstash/pipeline:/usr/share/logstash/pipeline -d docker.elastic.co/logstash/logstash:8.14.1使用docker logs [container]查看是否启动成功，如果启动成功，则使用如下命令进入容器
docker exec -it --user root logstash-syslog-nas /bin/bash
进入容器后，需修改Gemfile source 源为国内镜像，否则可能会无法获取安装包
source "https://gems.ruby-china.com"
然后执行命令安装 logstash-output-jdbc
bin/logstash-plugin install logstash-output-jdbc
安装成功后，退出容器。记得配置 mysql-connector-j-8.4.0.jar路径，参考文章开头的配置文件
exit
重点
进入映射配置文件路径
cd /usr/share/logstash/config/conf.d/
将一开始配置的文件logstash.conf 替换为最初的配置：
#注释方法#####
input {syslog {port => "5044"}
}
filter {ruby {code => "event.set('timestamp', event.get('@timestamp').time.localtime + 8*60*60)"}if("WinFileService" in [message] or "FtpFileTransfer" in [message]){grok{match =>{"message"=>"%{WORD:opt_type} Event: %{DATA:opt_name}, Path: %{DATA:path}, File/Folder: %{WORD:file_type}, Size: %{BASE16FLOAT:size} %{WORD:size_unit}, User: %{DATA:username}, IP: %{IPV4:opt_ip}"}}grok{match=>{"path"=>"%{GREEDYDATA}/%{GREEDYDATA:file_name}"}}if("KB" in [size_unit]){ruby{code =>"event.set('size_byte', event.get('size').to_i * 1024)"}}if("MB" in [size_unit]){ruby{code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024)" }} if("GB" in [size_unit]){ruby{code =>"event.set('size_byte', event.get('size').to_i * 1024 * 1024 * 1024)"}}if("Bytes" in [size_unit]){ruby{code =>"event.set('size_byte', event.get('size').to_i)"}}}
}output {stdout {}if("FtpFileTransfer" in [opt_type]){jdbc{driver_jar_path => "/usr/share/logstash/config/jar/mysql-connector-j-8.4.0.jar"driver_class => "com.mysql.cj.jdbc.Driver"connection_string => "jdbc:mysql://192.168.10.23:3306/database_name?user=root&password=xxxxxxx&useUnicode=true&characterEncoding=utf8&zeroDateTimeBehavior=convertToNull&useSSL=true&serverTimezone=GMT%2B8&autoReconnect=true&rewriteBatchedStatements=true"statement => ["insert into t_ftp_log(level, event, full_path, file_size, event_time,app_source,user_name,source_ip,file_name) VALUES (?,?,?,?,?,?,?,?,?)","[log][syslog][priority]","opt_name","path","size_byte","@timestamp","[host][hostname]","username","opt_ip","file_name"]} }
}
然后重启容器
docker restart logstash
恭喜
大功告成！
