haproxy程序崩溃问题处理

embedded/2024/12/22 22:56:06/

背景：
线上一k8s环境告警出节点失联，通过排查和k8s的api建立链接失败，检查发现haproxy出现了重启，对应的日志显示出程序运行崩溃，这个情况根据日志追溯，发现曾多次崩溃，后续也在其他k8s环境也有出现。崩溃日志如下：

{“log”:“[NOTICE] 107/003719 (1) : New worker #1 (7) forked\n”,“stream”:“stderr”,“time”:“2024-04-17T00:37:19.824357956Z”}
{“log”:“Thread 4 is about to kill the process.\n”,“stream”:“stderr”,“time”:“2024-05-21T22:07:34.11114335Z”}
{“log”:" Thread 1 : act=0 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111167358Z”}
{“log”:" stuck=0 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.11117337Z”}
{“log”:" cpu_ns: poll=151330165980 now=152417714721 diff=1087548741\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111177689Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111182066Z”}
{“log”:" \u003eThread 2 : act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.1111862Z”}
{“log”:" stuck=1 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111190734Z”}
{“log”:" cpu_ns: poll=149926470027 now=151612708615 diff=1686238588\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111194835Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111198839Z”}
{“log”:" call trace(8):\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111202879Z”}
{“log”:" | 0x55d4fef705f3 [89 c5 85 c0 0f 85 e3 01]: ha_thread_dump+0x253/0x458\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111206706Z”}
{“log”:" | 0x55d4fef70866 [48 8b 05 23 16 1d 00 48]: debug_handler+0x66/0x10e\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111210628Z”}
{“log”:" | 0x7f1e0f216730 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0x12730\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111214557Z”}
{“log”:" | 0x55d4fef59040 [0f 85 f4 02 00 00 48 b8]: process_runnable_tasks+0x3b0/0x75b\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111218605Z”}
{“log”:" Thread 3 : act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111222599Z”}
{“log”:" stuck=0 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111226627Z”}
{“log”:" cpu_ns: poll=1009086009481 now=1010561461521 diff=1475452040\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111230556Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111250608Z”}
{“log”:“*\u003eThread 4 : act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n”,“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111254956Z”}
{“log”:" stuck=1 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111259161Z”}
{“log”:" cpu_ns: poll=979476082232 now=981672729209 diff=2196646977\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111263123Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111267109Z”}
{“log”:" call trace(12):\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111271203Z”}
{“log”:" | 0x55d4fef91e77 [48 83 c4 10 5b 5d 41 5c]: wdt_handler+0x107/0x114\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111275151Z”}
{“log”:" | 0x7f1e0f216730 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0x12730\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111279082Z”}
{“log”:" | 0x55d4fef59033 [48 be 00 00 00 00 ff ff]: process_runnable_tasks+0x3a3/0x75b\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111282975Z”}
{“log”:" Thread 5 : act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.11128691Z”}
{“log”:" stuck=0 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111291316Z”}
{“log”:" cpu_ns: poll=148278385582 now=149700349268 diff=1421963686\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111295205Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111299086Z”}
{“log”:" \u003eThread 6 : act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111302863Z”}
{“log”:" stuck=1 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111306964Z”}
{“log”:" cpu_ns: poll=148756467035 now=150175517887 diff=1419050852\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111310773Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111314681Z”}
{“log”:" call trace(8):\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111318453Z”}
{“log”:" | 0x55d4fef705f3 [89 c5 85 c0 0f 85 e3 01]: ha_thread_dump+0x253/0x458\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111322535Z”}
{“log”:" | 0x55d4fef70866 [48 8b 05 23 16 1d 00 48]: debug_handler+0x66/0x10e\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111327669Z”}
{“log”:" | 0x7f1e0f216730 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0x12730\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111331797Z”}
{“log”:" | 0x55d4fef59030 [48 8b 02 48 be 00 00 00]: process_runnable_tasks+0x3a0/0x75b\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111335699Z”}
{“log”:" Thread 7 : act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111339745Z”}
{“log”:" stuck=0 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111343626Z”}
{“log”:" cpu_ns: poll=1000893082213 now=1002083221782 diff=1190139569\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111347437Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111351316Z”}
{“log”:" Thread 8 : act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111355091Z”}
{“log”:" stuck=0 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111358959Z”}
{“log”:" cpu_ns: poll=1101754150895 now=1103240639735 diff=1486488840\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111362785Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111366666Z”}
{“log”:" \u003eThread 9 : act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111370483Z”}
{“log”:" stuck=1 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111374385Z”}
{“log”:" cpu_ns: poll=1058224033381 now=1060422202804 diff=2198169423\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.11138071Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111384792Z”}
{“log”:" call trace(8):\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111388591Z”}
{“log”:" | 0x55d4fef705f3 [89 c5 85 c0 0f 85 e3 01]: ha_thread_dump+0x253/0x458\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111392324Z”}
{“log”:" | 0x55d4fef70866 [48 8b 05 23 16 1d 00 48]: debug_handler+0x66/0x10e\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.11139622Z”}
{“log”:" | 0x7f1e0f216730 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0x12730\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111400117Z”}
{“log”:" | 0x55d4fef59030 [48 8b 02 48 be 00 00 00]: process_runnable_tasks+0x3a0/0x75b\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111403998Z”}
{“log”:" Thread 10: act=0 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111407939Z”}
{“log”:" stuck=0 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111411786Z”}
{“log”:" cpu_ns: poll=148533347928 now=149605151681 diff=1071803753\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111415572Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111419455Z”}
{“log”:" Thread 11: act=0 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111423233Z”}
{“log”:" stuck=0 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111427075Z”}
{“log”:" cpu_ns: poll=154241567225 now=155736607041 diff=1495039816\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111430903Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.11143479Z”}
{“log”:" Thread 12: act=0 glob=0 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111438562Z”}
{“log”:" stuck=0 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111442448Z”}
{“log”:" cpu_ns: poll=157497641277 now=158546184939 diff=1048543662\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111446268Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111450241Z”}
{“log”:" \u003eThread 13: act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111454025Z”}
{“log”:" stuck=1 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111457891Z”}
{“log”:" cpu_ns: poll=149848571365 now=152047468944 diff=2198897579\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111461694Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.11146556Z”}
{“log”:" call trace(8):\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111469321Z”}
{“log”:" | 0x55d4fef705f3 [89 c5 85 c0 0f 85 e3 01]: ha_thread_dump+0x253/0x458\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111473445Z”}
{“log”:" | 0x55d4fef70866 [48 8b 05 23 16 1d 00 48]: debug_handler+0x66/0x10e\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111477443Z”}
{“log”:" | 0x7f1e0f216730 [48 c7 c0 0f 00 00 00 0f]: libpthread:+0x12730\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111481323Z”}
{“log”:" | 0x55d4fef59033 [48 be 00 00 00 00 ff ff]: process_runnable_tasks+0x3a3/0x75b\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111485223Z”}
{“log”:" \u003eThread 14: act=1 glob=1 wq=0 rq=0 tl=0 tlsz=0 rqsz=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111489139Z”}
{“log”:" stuck=1 prof=0 harmless=0 wantrdv=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111493058Z”}
{“log”:" cpu_ns: poll=149437998131 now=151636962981 diff=2198964850\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111496852Z”}
{“log”:" curr_task=0\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111500735Z”}
{“log”:" call trace(8):\n",“stream”:“stderr”,“time”:“2024-05-21T22:07:34.111506879Z”}

解决办法：

查询相关haproxy官网显示，确实存在这个问题，需要将haproxy升级到2.8.4以上，最终自己做了一个haproxy 2.9.9版本的镜像，推送到k8s环境进行观察和高可用测试，发现并无问题后，更新至生产环境，后面就没出现过次问题。

下载镜像地址：
haproxy:v2.9.9


http://www.ppmy.cn/embedded/118316.html

相关文章

JAVA集合框架 类型包装器

JAVA集合框架 类型包装器

Collection接口 用于集合类的接口 boolean add() 将对象添加给集合，如果添加对象之后，集合确实发生了变化，则返回true，否则false。若集合中已经有该对象，则返回false 一个特别的方法Iterator iterator() 该方法用于返…
阅读更多...
CAT1 RTU软硬件设计开源资料分析（MQTT协议+Modbus协议+GNSS定位版本 ）

CAT1 RTU软硬件设计开源资料分析（MQTT协议+Modbus协议+GNSS定位版本 ）

01 CAT1 RTU方案简介： 远程终端单元( Remote Terminal Unit，RTU)，一种针对通信距离较长和工业现场环境恶劣而设计的具有模块化结构的、特殊的计算机测控单元，它将末端检测仪表和执行机构与远程控制中心相连接。 奇迹MQTT RTUGN…
阅读更多...
人工智能-大语言模型-微调技术-LoRA及背后原理简介

人工智能-大语言模型-微调技术-LoRA及背后原理简介

1. 《LORA: LOW-RANK ADAPTATION OF LARGE LANGUAGE MODELS》 LORA: 大型语言模型的低秩适应 摘要： 随着大规模预训练模型的发展，全参数微调变得越来越不可行。本文提出了一种名为LoRA（低秩适应）的方法，通过在Transf…
阅读更多...
算法：斐波那契数列

算法：斐波那契数列

题目描述&#xff1a; 大家都知道斐波那契数列&#xff0c;现在要求输入一个整数 n&#xff0c;请你输出斐波那契数列的第 n 项。 n<39 问题分析&#xff1a; 可以肯定的是这一题通过递归的方式是肯定能做出来&#xff0c;但是这样会有一个很大的问题&#xff0c;那就是递…
阅读更多...
建造者模式__c#

建造者模式__c#

目录 调用 指挥者 抽象建造者 建造者 定义具体产品 调用 用指挥者指挥建造者建造产品 在指挥者这里组装成产品 namespace _建造者模式 {internal class Program{static void Main(string[] args){Builder buildernew JiangHuaiBuilder();//建造者Director director new…
阅读更多...
等保测评：企业如何构建安全的网络架构

等保测评：企业如何构建安全的网络架构

等保测评的目的和重要性 等保测评&#xff08;信息安全等级保护测评&#xff09;是企业构建安全网络架构的重要依据。它通过对信息系统的安全等级进行评估&#xff0c;帮助企业识别潜在的安全风险&#xff0c;提供科学的安全保护建议&#xff0c;确保网络系统的稳定和可靠运行。…
阅读更多...
Apache APISIX学习（1）：介绍、docker启动

Apache APISIX学习（1）：介绍、docker启动

一、介绍 Apache APISIX 是一个动态、实时、高性能的 API 网关&#xff0c; 提供负载均衡、动态上游、灰度发布、服务熔断、身份认证、可观测性等丰富的流量管理功能。你可以把 Apache APISIX 当做流量入口&#xff0c;来处理所有的业务数据&#xff0c;包括动态路由、动态上游…
阅读更多...
第24届工博会盛大开启，大腾智能携前沿数字化解决方案亮相

第24届工博会盛大开启，大腾智能携前沿数字化解决方案亮相

2024年9月24日&#xff0c;第24届中国国际工业博览会(ICIF China 2024)&#xff08;以下简称“工博会”&#xff09;在上海盛大开幕。本届工博会以“新质领航 数字赋能”为主题&#xff0c;覆盖工业自动化、智能制造等多个领域&#xff0c;汇聚全球制造业的精英与前沿技术&…
阅读更多...
最新文章

Copyright @ 2022~2023