一、收集Nginx日志
1、Nginx配置成Json格式日志
#编辑nginx主配置文件 log_format logstash_json '{"@timestamp":"$time_local",' '"remote_addr":"$remote_addr",' '"remote_user":"$remote_user",' '"body_bytes_sent":"$body_bytes_sent",' '"request_time":"$request_time",' '"status":"$status",' '"request":"$request",' '"request_method":"$request_method",' '"http_referrer":"$http_referer",' '"body_bytes_sent":"$body_bytes_sent",' '"http_x_forwarded_for":"$http_x_forwarded_for",' '"http_user_agent":"$http_user_agent"}'; access_log /var/log/nginx/access.log logstash_json;
2、重启Nginx服务查看Json格式日志
{"@timestamp":"07/Oct/2018:17:08:49 +0800", "remote_addr":"192.168.1.68", "remote_user":"-", "body_bytes_sent":"612", "request_time":"0.000", "status":"200", "request":"GET / HTTP/1.1", "request_method":"GET", "http_referrer":"-", "body_bytes_sent":"612", "http_x_forwarded_for":"-", "http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"}
3、logstash中增加Nginx配置
input { file { path => "/var/log/nginx/access.log" type => "nginx-accesslog" start_position => "beginning" } } output { if [type] == "nginx-accesslog" { elasticsearch { hosts => ["192.168.1.235:9200"] index => "nginx-accesslog-%{+YYYY.MM.dd}" } } }
4、测试配置文件
bin/logstash -f /opt/logstash-6.4.1/config/syslog.conf -t #显示以下内容说明没问题 Configuration OK
二、收集Tomcat日志
1、修改tomcat/conf/server.xml结尾部分的日志配置
<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs" prefix="tomcat_access" suffix=".log" pattern="{"client":"%h","client user":"%l", "authenticated":"%u", "access time":"%t", "method":"%r", "status":"%s","send bytes" :"%b","Query?string":"%q","partner":"%{Referer} i","Agent version":"%{User-Agent}i"}"/>
2、重启Tomcat,查看日志格式
{"client":"192.168.1.68", "client user":"-", "authenticated":"-", "access time":"[07/Oct/2018:15:54:00 +0800]", "method":"GET / HTTP/1.1", "status":"200","send bytes":"11250", "Query?string":"", "partner":"-","Agent version":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"}
3、logstash中增加Tomcat配置
input{ file { path => "/app/tomcat/logs/tomcat_access.*.log" type => "tomcatlog" start_position => "beginning" stat_interval => "5" } } output{ if[type] == "tomcatlog" { elasticsearch { hosts => ["192.168.1.235:9200"] index => "tomcatlog-%{+YYYY.MM.dd}" } } }
4、测试配置文件
bin/logstash -f /opt/logstash-6.4.1/config/tomcat.conf -t #显示以下内容说明没问题 Configuration OK
三、运行Logstash服务并查看Kibana
#运行配置下所有匹配后缀为.conf的文件 bin/logstash -f "config/*.conf"
1、访问Kibana,设置索引
Tomcat同样步骤设置索引并查看数据:
注意:logstash中的配置文件如果有使用if[type]判断,所有配置都需要写判断,否则数据会混乱。