1 默认配置
1.1 filebeat
filebeat-7.17.yml,从网关中下载k8s的配置,指定es和kibana的配置
通过kibana查询可以查询到日志了,但此时还不知道具体怎么用。
1.2 kibana
在Discover中创建索引格式:filebeat-*,得到如下图,可以看出acc-statement-server的日志最多。但里面的字段太多了,下面应该怎么看呢?
再跟进查看日志,filebeat应该是每一样记录一次,这个浪费了很多存储空间。另外排查问题也并不好查。
2 多行合并输出
如果使用默认的配置,每一行日志,就会产生一条记录。
2.1 filebeat
增加多行规则匹配
设置索引,符合条件走自己的索引,否则则为默认索引
output.elasticsearch:hosts: ['10.101.10.2:9200','10.101.10.3:9200','10.101.10.4:9200']username: ${ELASTICSEARCH_USERNAME}password: ${ELASTICSEARCH_PASSWORD}indices:- index: acc-accountbook-server-%{+yyyy.MM.dd}when.contains:kubernetes.container.name: acc-accountbook-server- index: acc-analysis-server-%{+yyyy.MM.dd}when.contains:kubernetes.container.name: acc-analysis-serverindex: filebeat-7.17.25-%{+yyyy.MM.dd}
在kibana中跟进日志,发现少部分日志输出成功,大多失败,这是什么原因呢?
为什么有些可以添加数据,有些不能呢
调试发现,我在索引前面加上了eayc就可以了,看来问题就出现在索引策略
2.2 logback
上面的时间分割,是需要logback配置与之对应。如我的系统日志打印出来的是这个,那么filebeat中就无法实现多行合并了。
如下图修改logback配置。
2.3 pipeline
在elasticsearch中创建pipeline,参考了【ELK】到【EFK】,【Filebeat】收集【SpringBoot】日志,但最终还是放弃了,还是按照k8s自带的格式,这样便于处理,不需要非得自定义格式。
PUT _ingest/pipeline/eayc_log_pipeline
{"description": "岁月云日志管道","processors": [{"remove": {"field": "agent.name","ignore_missing": true}},{"remove": {"field": "agent.ephemeral_id","ignore_missing": true}},{"remove": {"field": "log.file.path","ignore_missing": true}},{"remove": {"field": "input.type","ignore_missing": true}},{"remove": {"field": "kubernetes.node.labels.kubernetes_io/hostname","ignore_missing": true}},{"remove": {"field": "kubernetes.labels.k8s_kuboard_cn/layer","ignore_missing": true}},{"remove": {"field": "kubernetes.deployment.name","ignore_missing": true}},{"remove": {"field": "container.runtime","ignore_missing": true}},{"remove": {"field": "ecs.version","ignore_missing": true}},{"remove": {"field": "host.architecture","ignore_missing": true}},{"remove": {"field": "host.containerized","ignore_missing": true}},{"remove": {"field": "host.mac","ignore_missing": true}},{"remove": {"field": "host.os.codename","ignore_missing": true}},{"remove": {"field": "host.os.name","ignore_missing": true}},{"remove": {"field": "host.os.platform","ignore_missing": true}},{"remove": {"field": "kubernetes.labels.k8s_kuboard_cn/name","ignore_missing": true}},{"remove": {"field": "kubernetes.labels.pod-template-hash","ignore_missing": true}},{"remove": {"field": "kubernetes.namespace_uid","ignore_missing": true}},{"remove": {"field": "kubernetes.node.labels.beta_kubernetes_io/arch","ignore_missing": true}},{"remove": {"field": "kubernetes.node.labels.beta_kubernetes_io/os","ignore_missing": true}},{"remove": {"field": "log.flags","ignore_missing": true}},{"remove": {"field": "log.offset","ignore_missing": true}},{"remove": {"field": "kubernetes.container.id","ignore_missing": true}},{"remove": {"field": "kubernetes.pod.uid","ignore_missing": true}}]
}
创建索引策略
PUT _ilm/policy/eayc_log_policy
{"policy": {"phases": {"hot": {"min_age": "0ms","actions": {"rollover": {"max_size": "50gb","max_age": "30d"}}},"delete": {"min_age": "90d","actions": {"delete": {}}}}}
}
2.4 elasticsearch
在k8s中启动filebeat中,查看filebeat的日志发现
2024-10-29T07:55:33.935Z ERROR [elasticsearch] elasticsearch/client.go:226 failed to perform any bulk index operations: 500 Internal Server Error: {"error":{"root_cause":[{"type":"illegal_state_exception","reason":"There are no ingest nodes in this cluster, unable to forward request to an ingest node."}],"type":"illegal_state_exception","reason":"There are no ingest nodes in this cluster, unable to forward request to an ingest node."},"status":500}
则需要在elasticsearch.yml中增加配置
node.roles: [ingest]
创建组合模板
PUT _component_template/filebeat_settings
{"template": {"settings": {"number_of_shards": 1,"number_of_replicas": 1}}
}PUT _component_template/filebeat_mappings
{"template": {"mappings": {"properties": {"@timestamp": {"type": "date"},"message": {"type": "text"}}}}
}PUT _component_template/eayc_mappings
{"template": {"mappings": {"properties": {"@timestamp": {"type": "date"},"message": {"type": "text"},"custom_field": {"type": "keyword"}}}}
}PUT _component_template/acc_mappings
{"template": {"mappings": {"properties": {"@timestamp": {"type": "date"},"message": {"type": "text"},"custom_field": {"type": "keyword"}}}}
}PUT _index_template/filebeat
{"index_patterns": ["filebeat-*"],"composed_of": ["filebeat_settings", "filebeat_mappings"],"priority": 100,"template": {"settings": {"index.lifecycle.name": "eayc_log_policy","index.lifecycle.rollover_alias": "filebeat-write"}}
}PUT _index_template/eayc
{"index_patterns": ["eayc-*"],"composed_of": ["filebeat_settings", "eayc_mappings"],"priority": 100,"template": {"settings": {"index.lifecycle.name": "eayc_log_policy","index.lifecycle.rollover_alias": "filebeat-write"}}
}PUT _index_template/acc
{"index_patterns": ["acc-*"],"composed_of": ["filebeat_settings", "acc_mappings"],"priority": 100,"template": {"settings": {"index.lifecycle.name": "eayc_log_policy","index.lifecycle.rollover_alias": "filebeat-write"}}
}
接着再看acc添加进去了
再看日志数据出来了