使用 Elasticsearch+logstash 存储获取实时日志【Cdn Realtime Analytics】

发表于 2014-07-02 Disqus：

安装的我就不写了。主要说下方案 nginx 实时吐日志给syslog-ng via pipe syslog-ng 向logstash 推送日志 via internet udp logstash 把日志塞进elasticsearch 并index 发送方： nginx.conf [bash] # … log_format real_time ‘- $time_iso8601 $host $request_time $status $bytes_sent’; server { listen 80; server_name my_test_rt; access_log /dev/realtime.pipe real_time; location /{ proxy_pass http://backend.com; } } # … [/bash] syslog-ng.conf [bash] source s_pipe { pipe(“/dev/realtime.pipe”); }; destination d_udp { udp(“127.0.0.1” port(9999) template (“$MSG\n”) ); }; log {source(s_pipe); destination(d_udp); }; [/bash] [bash] #创建一个管道： makefifo /dev/realtime.pipe #先启动syslog-ng #不然nginx启动时会卡住 service syslog-ng start service nginx start [/bash] 接收方： /etc/logstash/conf.d/rt.conf [bash] input { udp { port =>9999 } } filter { grok { pattern => [“%{TIMESTAMP_ISO8601:timestamp} %{IPORHOST:host} %{IPORHOST:domain} %{NUMBER:request_time} %{NUMBER:status} %{NUMBER:bytes_sent}” ] } mutate { remove_field => [ “message”, “@version” ] } } output { elasticsearch { host => “127.0.0.1” flush_size => 1 index => “rt-%{+YYYY.MM.dd.HH.mm}” } } [/bash] 把logstash 和 elasticsearch 都启动。整个体系就运转起来了