Filebeat：ELK
协议栈的新成员，一个轻量级开源日志文件数据搜集器，基于 Logstash-Forwarder 源代码开发，是对它的替代。在需要采集日志数据的
server 上安装 Filebeat，并指定日志目录或日志文件后，Filebeat 就能读取数据，迅速发送到 Logstash 进行解析，亦或直接发送到 Elasticsearch 进行集中式存储和分析。
Filebeat使用golang编写，Logstash使用 ruby 、java跑在jvm上比较大。所以生产上推荐使用Filebeat。
Filebeat的使用比较简单。我这里 给我前公司的配置。
filebeat:
prospectors:
-
paths:
- /path/to/mysql-slow.log
multiline:
pattern: "^# Time:"
negate: true
match: after
document_type: db-mysql-slow
input_type: log
fields:
type: db-mysql-slow
tags: mysql.ds
encoding: plain
fields_under_root: true
scan_frequency: "10s"
harvester_buffer_size: 16384
tail_files: true
backoff: "3s"
-
paths:
- /path/to/head3.log
document_type: db-mysql-error
input_type: log
fields:
type: db-mysql-error
tags: mysql.ds
encoding: plain
fields_under_root: true
scan_frequency: "10s"
harvester_buffer_size: 16384
tail_files: true
backoff: "3s"
spool_size: 1000
idle_timeout: "10s"
registry_file: /opt/opbin/filebeat/registry
output:
redis:
host: "10.103.11.18"
port: 6379
save_topology: true
index: "logstash"
db: 0
db_topology: 1
timeout: 5
reconnect_interval: 1
logging:
level: info
to_files: true
to_syslog: false
files:
path: /opt/opbin/filebeat/mybeat
name: mybeat.log
rotateeverybytes: 10485760
keepfiles: 5
下面给出logstash的对应配置：
beats {
port => 5044
}
}
filter {
grok {
match => [ "message", "(?m)^# User@Host: %{USER:query_user}\[[^\]]+\] @ (?:(?<query_host>\S*) )?\[(?:%{IP:query_ip})?\]\s*Id: %{ NUMBER :id:int}\s+# Query_time: %{NUMBER:query_time: float }\s+Lock_time: %{NUMBER:lock_time:float}\s+Rows_sent: %{NUMBER:rows_sent:int}\s+Rows_examined: %{NUMBER:rows_examined:int}\s*(?:use %{DATA:database};\s*)?SET timestamp=%{NUMBER:timestamp};\s*(?<query>(?<action>\w+)\s+.*)" ]
}
grok {
match => { "message" => "# Time: " }
add_tag => [ "drop" ]
tag_on_failure =>[]
}
if "drop" in [tags] {
drop {}
}
date {
match => [ "timestamp", "UNIX", "YYYY-MM-dd HH:mm:ss"]
remove_field => [ "timestamp" ]
}
}
output {
elasticsearch {
hosts => "192.168.1.63:9200"
manage_template => false
index => "%{[@metadata][beat]}-%{[type]}-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}
ps:在前公司做了大量的ELK相关工作，后续会分享出来。谢谢
