KubeSphere使用外部ES进行日志收集（多行日志）

news/2024/11/27 23:44:06/

环境

kubesphere ： v3.3.1

Docker：20.10.8

Fluent-Bit：2.0.6-2.0.8

ES+Kibana：7.9.3

Docker日志示例

{"log":"2023-01-10 11:32:50.021 -  INFO --- [scheduling-1] traceId: p6spy : 1|conn-0|statement|SELECT  fd_id AS id,fd_user_id AS userId,fd_specific_user AS specificUser,fd_home_assessment AS homeAssessment,fd_home_assessment_time AS homeAssessmentTime,fd_end_home_assessment AS endHomeAssessment,fd_end_home_assessment_time AS endHomeAssessmentTime,fd_daily_assessment AS dailyAssessment,fd_daily_count AS dailyCount,fd_daily_assessment_time AS dailyAssessmentTime,fd_sort AS sort,fd_delete_flag AS deleteFlag,fd_update_time AS updateTime,fd_create_time AS createTime,fd_version AS version,fd_org_id AS orgId  FROM t_patient_assessment     WHERE (fd_daily_assessment = 1) \n","stream":"stdout","time":"2023-01-10T03:32:50.021311904Z"}
{"log":"2023-01-10 11:32:50.022 -  INFO --- [scheduling-1] traceId: com.gjwlyy.covid.core.job.DailyReminderJob : \u003c\u003c\u003c 当前是用户1gkfb79bq39pe8mi37jknh81crt5k82m隔离的-23天 \n","stream":"stdout","time":"2023-01-10T03:32:50.023079165Z"}
{"log":"2023-01-10 11:32:50.025 - ERROR --- [scheduling-1] traceId: org.springframework.scheduling.support.TaskUtils$LoggingErrorHandler : Unexpected error occurred in scheduled task java.lang.NumberFormatException: For input string: \"[{\"relation\":\"or\",\"rule\":\"=\",\"value\":\"0\"}]\"\n","stream":"stdout","time":"2023-01-10T03:32:50.025703129Z"}
{"log":"\u0009at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)\n","stream":"stdout","time":"2023-01-10T03:32:50.025727938Z"}
{"log":"\u0009at java.lang.Long.parseLong(Long.java:589)\n","stream":"stdout","time":"2023-01-10T03:32:50.025733862Z"}
{"log":"\u0009at java.lang.Long.parseLong(Long.java:631)\n","stream":"stdout","time":"2023-01-10T03:32:50.025737051Z"}
{"log":"\u0009at com.gjwlyy.covid.core.job.DailyReminderJob.lambda$dealPatientDailyJob$1(DailyReminderJob.java:116)\n","stream":"stdout","time":"2023-01-10T03:32:50.025740047Z"}
{"log":"\u0009at java.util.ArrayList.forEach(ArrayList.java:1257)\n","stream":"stdout","time":"2023-01-10T03:32:50.025743184Z"}
{"log":"\u0009at com.gjwlyy.covid.core.job.DailyReminderJob.lambda$dealPatientDailyJob$2(DailyReminderJob.java:110)\n","stream":"stdout","time":"2023-01-10T03:32:50.025747023Z"}
{"log":"\u0009at java.util.ArrayList.forEach(ArrayList.java:1257)\n","stream":"stdout","time":"2023-01-10T03:32:50.025750286Z"}
{"log":"\u0009at com.gjwlyy.covid.core.job.DailyReminderJob.lambda$dealPatientDailyJob$3(DailyReminderJob.java:109)\n","stream":"stdout","time":"2023-01-10T03:32:50.025753216Z"}
{"log":"\u0009at java.util.ArrayList.forEach(ArrayList.java:1257)\n","stream":"stdout","time":"2023-01-10T03:32:50.025756256Z"}
{"log":"\u0009at com.gjwlyy.covid.core.job.DailyReminderJob.dealPatientDailyJob(DailyReminderJob.java:106)\n","stream":"stdout","time":"2023-01-10T03:32:50.025759298Z"}
{"log":"\u0009at com.gjwlyy.covid.core.job.DailyReminderJob.dailyJob(DailyReminderJob.java:61)\n","stream":"stdout","time":"2023-01-10T03:32:50.025762317Z"}
{"log":"\u0009at com.gjwlyy.covid.core.job.DailyReminderJob$$FastClassBySpringCGLIB$$34fb91e0.invoke(\u003cgenerated\u003e)\n","stream":"stdout","time":"2023-01-10T03:32:50.025765223Z"}
{"log":"\u0009at org.springframework.cglib.proxy.MethodProxy.invoke(MethodProxy.java:218)\n","stream":"stdout","time":"2023-01-10T03:32:50.025768376Z"}
{"log":"\u0009at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:688)\n","stream":"stdout","time":"2023-01-10T03:32:50.025771336Z"}
{"log":"\u0009at com.gjwlyy.covid.core.job.DailyReminderJob$$EnhancerBySpringCGLIB$$7af1cd70.dailyJob(\u003cgenerated\u003e)\n","stream":"stdout","time":"2023-01-10T03:32:50.025774261Z"}
{"log":"\u0009at sun.reflect.GeneratedMethodAccessor167.invoke(Unknown Source)\n","stream":"stdout","time":"2023-01-10T03:32:50.025777303Z"}
{"log":"\u0009at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n","stream":"stdout","time":"2023-01-10T03:32:50.025780146Z"}
{"log":"\u0009at java.lang.reflect.Method.invoke(Method.java:498)\n","stream":"stdout","time":"2023-01-10T03:32:50.025783075Z"}
{"log":"\u0009at org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:84)\n","stream":"stdout","time":"2023-01-10T03:32:50.025785872Z"}
{"log":"\u0009at org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)\n","stream":"stdout","time":"2023-01-10T03:32:50.025788727Z"}
{"log":"\u0009at org.springframework.scheduling.concurrent.ReschedulingRunnable.run(ReschedulingRunnable.java:93)\n","stream":"stdout","time":"2023-01-10T03:32:50.025798209Z"}
{"log":"\u0009at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)\n","stream":"stdout","time":"2023-01-10T03:32:50.02580237Z"}
{"log":"\u0009at java.util.concurrent.FutureTask.run(FutureTask.java:266)\n","stream":"stdout","time":"2023-01-10T03:32:50.02580537Z"}
{"log":"\u0009at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)\n","stream":"stdout","time":"2023-01-10T03:32:50.025808239Z"}
{"log":"\u0009at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)\n","stream":"stdout","time":"2023-01-10T03:32:50.025811169Z"}
{"log":"\u0009at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n","stream":"stdout","time":"2023-01-10T03:32:50.025814181Z"}
{"log":"\u0009at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n","stream":"stdout","time":"2023-01-10T03:32:50.025817118Z"}
{"log":"\u0009at java.lang.Thread.run(Thread.java:748)\n","stream":"stdout","time":"2023-01-10T03:32:50.025820003Z"}
{"log":"\n","stream":"stdout","time":"2023-01-10T03:32:50.025822812Z"}
{"log":"2023-01-10 11:33:00.001 -  INFO --- [scheduling-1] traceId: com.gjwlyy.covid.core.job.DailyReminderJob : \u003c\u003c\u003c dailyJob 患者日程定时任务开始.请求参数 \n","stream":"stdout","time":"2023-01-10T03:33:00.001296014Z"}
{"log":"2023-01-10 11:33:00.007 -  INFO --- [scheduling-1] traceId: p6spy : 1|conn-0|statement|SELECT  fd_id AS id,fd_schedule_name AS scheduleName,fd_schedule_trigger_time AS scheduleTriggerTime,fd_status AS status,fd_sort AS sort,fd_delete_flag AS deleteFlag,fd_update_time AS updateTime,fd_create_time AS createTime,fd_version AS version,fd_org_id AS orgId  FROM t_management_schedule     WHERE (fd_delete_flag = false AND fd_status = true) \n","stream":"stdout","time":"2023-01-10T03:33:00.007959022Z"}
{"log":"2023-01-10 11:33:00.008 -  INFO --- [scheduling-1] traceId: com.gjwlyy.covid.core.job.DailyReminderJob : \u003c\u003c\u003c 定时任务涉及的管理日程记录id:[100099] \n","stream":"stdout","time":"2023-01-10T03:33:00.008512022Z"}

安装Fluent-Bit

kubectl create ns fluent-bit

helm repo add fluent https://fluent.github.io/helm-charts

helm upgrade --install fluent-bit fluent/fluent-bit -n fluent-bit

配置Fluent-Bit

编写文件custom_parsers.conf

[PARSER]Name java_multi_line    #起个名Key_name log   #默认的Key_name为logFormat regex   #固定值Regex (?<log>^{"log":"\d{4}-\d{2}-\d{2}\ \d{2}:\d{2}:\d{2}\.\d{3}\ .*})   #匹配log字段（时间点，例如：2023-01-09 03:59:24.617）
[PARSER]Name docker_no_timeFormat jsonTime_Keep OffTime_Key timeTime_Format %Y-%m-%dT%H:%M:%S.%L

编写文件fluent-bit.conf

[SERVICE]Daemon OffFlush 1Log_Level infoParsers_File parsers.confParsers_File custom_parsers.confHTTP_Server OnHTTP_Listen 0.0.0.0HTTP_Port 2020Health_Check On[INPUT]Name tailPath /var/log/containers/*.log    #收集日志的地址Exclude_Path /var/log/containers/*_kube*-system_*.log   #排除不想收集的日志Docker_Mode On   #如果启用，插件将重新组合分离的Docker日志行，然后将它们传递给上面配置的任何解析器。该模式不能与Multiline 模式同时使用。 （重要）Docker_Mode_Flush   5   #等待时间(以秒为单位)，以处理排队的多行消息  （重要）Docker_Mode_Parser java_multi_line  #指定处理器的名称，custom_parsers.conf里面定义的 （重要）Parser docker# multiline.parser docker, criTag kube.*Mem_Buf_Limit 5MBSkip_Long_Lines On
[FILTER]Name kubernetesMatch kube.*Annotations Off   #不收集注解Labels Off    #不收集标签Merge_Log Off   #当启用时，它检查日志字段内容是否是JSON字符串映射，如果是，它将映射字段作为日志结构的一部分追加。Keep_Log OffK8S-Logging.Parser On   #允许Kubernetes Pods建议预定义的解析K8S-Logging.Exclude On   #允许Kubernetes Pods从日志处理器中排除它们的日志[FILTER]Name    nest   #操作嵌套数据Match    kube.*Operation    lift   #通过lift模式，从记录的将指定map中的key value都提取出来放到上一层Nested_under    kubernetes   #指定需要提取的map名Add_prefix    kubernetes_    #添加前缀[Filter]Name    modifyMatch    kube.*Remove    stream     #移除字段streamRemove    kubernetes_docker_idRemove    kubernetes_pod_idRemove    kubernetes_hostRemove    kubernetes_container_hashRename    kubernetes_namespace_name namespace_name  #修改字段名称Rename    kubernetes_pod_name pod_nameRename    kubernetes_container_name container_nameRename    kubernetes_container_image container_image[OUTPUT]Name esHost 192.168.31.253Port 9200HTTP_User elasticHTTP_Passwd 6ygXXXXXXXXXXXXXLogstash_Format OnReplace_Dots On  #替换“.”变为“_”Logstash_Prefix kubesphere  #索引名称前缀Retry_Limit False

效果展示


http://www.ppmy.cn/news/10915.html

相关文章

融合注意力模块SE基于轻量级yolov5s实践路面坑洼目标检测系统

融合注意力模块SE基于轻量级yolov5s实践路面坑洼目标检测系统

在很多的项目实战中验证分析注意力机制的加入对于模型最终性能的提升发挥着积极正向的作用&#xff0c;在我之前的一些文章里面也做过了一些尝试&#xff0c;这里主要是想基于轻量级的s系列模型来开发构建路面坑洼检测系统&#xff0c;在模型中加入SE注意力模块&#xff0c;以期…
阅读更多...
React相关扩展一（setState、lazyLoad、Hooks相关）（九）

React相关扩展一（setState、lazyLoad、Hooks相关）（九）

系列文章目录 第一章&#xff1a;React基础知识&#xff08;React基本使用、JSX语法、React模块化与组件化&#xff09;&#xff08;一&#xff09; 第二章&#xff1a;React基础知识&#xff08;组件实例三大核心属性state、props、refs&#xff09;&#xff08;二&#xff0…
阅读更多...
【寒假每日一题】DAY1.水仙花数

【寒假每日一题】DAY1.水仙花数

一、题目描述 求0&#xff5e;100000之间的所有“水仙花数”并输出。 什么是水仙花数&#xff1a; “水仙花数”是指一个n位数&#xff0c;其各位数字的n次方之和确好等于该数本身&#xff0c;如:153&#xff1d;1^3&#xff0b;5^3&#xff0b;3^3&#xff0c;则153是一个“水…
阅读更多...
STM32 TIM PWM高阶操作：刹车及状态约束

STM32 TIM PWM高阶操作：刹车及状态约束

STM32 TIM PWM高阶操作&#xff1a;刹车及状态约束 刹车及状态约束是STM32 TIM PWM控制里面比较复杂的一部分&#xff0c;涉及到PWM波形产生前&#xff0c;中&#xff0c;后的管脚状态输出。 这里先引入两个描述&#xff0c;一个是“半高阻”&#xff0c;意思是STM32管脚输出…
阅读更多...
vue+xlsx实现表格的导入导出:

vue+xlsx实现表格的导入导出:

文章目录一、vue前端使用xlsx和 xlsx-style 导出excel&#xff0c;并修改样式:1、改造后效果&#xff1a;2、实现&#xff1a;3、引入库xlsx-style4、excelUtil.js文件二、前端xlsx插件怎么设置导出的excel列宽自适应&#xff1f;2-1、效果2-2、效果三、xlsx插件&#xff0c;导…
阅读更多...
元年科技2022回顾：奋楫扬帆数字化，转型升级立潮头

元年科技2022回顾：奋楫扬帆数字化，转型升级立潮头

当下数字经济无疑以最具创新、辐射力最广泛的经济形态迸发着无限活力&#xff0c;这也意味着数字化转型的浪潮风起云涌。作为数字化转型的主力军&#xff0c;元年科技肩负重任奋楫扬帆&#xff0c;在沧海中勇往直前。 回顾2022年&#xff0c;元年科技在不断走向深入的数字化转…
阅读更多...
Vue3——第四章（响应式基础：reactive、ref）

Vue3——第四章（响应式基础：reactive、ref）

一、用reactive()声明响应式状态 我们可以使用 reactive() 函数创建一个响应式对象或数组&#xff1a; 响应式对象其实是 JavaScript Proxy&#xff0c;其行为表现与一般对象相似。不同之处在于 Vue 能够跟踪对响应式对象属性的访问与更改操作。 要在组件模板中使用响应式状…
阅读更多...
【Linux】Linux 项目自动化构建工具 -- make/makefile

【Linux】Linux 项目自动化构建工具 -- make/makefile

&#x1f451;作者主页&#xff1a;进击的安度因 &#x1f3e0;学习社区&#xff1a;进击的安度因&#xff08;个人社区&#xff09; &#x1f4d6;专栏链接&#xff1a;Linux 文章目录一、前言二、概念三、demo 实现四、原理与规则1、依赖关系和依赖方法① 感性理解② 深层理解…
阅读更多...
最新文章

Copyright @ 2022~2023