一、收集Nginx日志

1、Nginx配置成Json格式日志

#编辑nginx主配置文件
log_format logstash_json '{"@timestamp":"$time_local",'
    '"remote_addr":"$remote_addr",'
    '"remote_user":"$remote_user",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"request_time":"$request_time",'
    '"status":"$status",'
    '"request":"$request",'
    '"request_method":"$request_method",'
    '"http_referrer":"$http_referer",'
    '"body_bytes_sent":"$body_bytes_sent",'
    '"http_x_forwarded_for":"$http_x_forwarded_for",'
    '"http_user_agent":"$http_user_agent"}';

access_log  /var/log/nginx/access.log  logstash_json;

2、重启Nginx服务查看Json格式日志

{"@timestamp":"07/Oct/2018:17:08:49 +0800",
"remote_addr":"192.168.1.68",
"remote_user":"-",
"body_bytes_sent":"612",
"request_time":"0.000",
"status":"200",
"request":"GET / HTTP/1.1",
"request_method":"GET",
"http_referrer":"-",
"body_bytes_sent":"612",
"http_x_forwarded_for":"-",
"http_user_agent":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"}

3、logstash中增加Nginx配置

input {
    file {
        path => "/var/log/nginx/access.log"
        type => "nginx-accesslog"
        start_position => "beginning"
    }    
}

output {
    if [type] == "nginx-accesslog" {
        elasticsearch {
        hosts => ["192.168.1.235:9200"]
        index => "nginx-accesslog-%{+YYYY.MM.dd}"
        }
    }
}

4、测试配置文件

bin/logstash -f /opt/logstash-6.4.1/config/syslog.conf -t

#显示以下内容说明没问题
Configuration OK

二、收集Tomcat日志

1、修改tomcat/conf/server.xml结尾部分的日志配置

<Valve className="org.apache.catalina.valves.AccessLogValve" directory="logs"
    prefix="tomcat_access" suffix=".log"
    pattern="{&quot;client&quot;:&quot;%h&quot;,&quot;client user&quot;:&quot;%l&quot;,
&quot;authenticated&quot;:&quot;%u&quot;, &quot;access time&quot;:&quot;%t&quot;,
&quot;method&quot;:&quot;%r&quot;, &quot;status&quot;:&quot;%s&quot;,&quot;send bytes&quot;
:&quot;%b&quot;,&quot;Query?string&quot;:&quot;%q&quot;,&quot;partner&quot;:&quot;%{Referer}
i&quot;,&quot;Agent version&quot;:&quot;%{User-Agent}i&quot;}"/>

2、重启Tomcat，查看日志格式

{"client":"192.168.1.68",
"client user":"-", 
"authenticated":"-", 
"access time":"[07/Oct/2018:15:54:00 +0800]",
 "method":"GET / HTTP/1.1", 
"status":"200","send bytes":"11250",
"Query?string":"",
"partner":"-","Agent version":"Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/69.0.3497.100 Safari/537.36"}

3、logstash中增加Tomcat配置

input{
    file {
        path => "/app/tomcat/logs/tomcat_access.*.log"
        type => "tomcatlog"
        start_position => "beginning"
        stat_interval => "5"
    }
}

output{
    if[type] == "tomcatlog" {
        elasticsearch {
            hosts => ["192.168.1.235:9200"]
            index => "tomcatlog-%{+YYYY.MM.dd}"
        }
    }
}

4、测试配置文件

bin/logstash -f /opt/logstash-6.4.1/config/tomcat.conf -t

#显示以下内容说明没问题
Configuration OK

三、运行Logstash服务并查看Kibana

#运行配置下所有匹配后缀为.conf的文件

bin/logstash -f "config/*.conf"

1、访问Kibana，设置索引

Tomcat同样步骤设置索引并查看数据：

注意：logstash中的配置文件如果有使用if[type]判断，所有配置都需要写判断，否则数据会混乱。