ORACLE 19C RAC DIAG进程消耗大量内存的分析

devtools/2024/9/23 0:59:00/

近期一个ORACLE 19C的RAC环境，多次出现数据库实例的后台进程DIAG消耗很多内存（达到20G+），节点1、节点2都出现过次问题。
问题分析：通过对DIAG进程TRACE分析，结合在ORACLE官方后台进行问题、BUG查询匹配，该进程占用内存的现象不是持续增大，而是突然增大。目前分析是心跳网络不稳定时DIAG进程产生大量TRACE时的内存泄漏，导致DIAG进程占用很多内存（mos:Database Ora_dia0_* Process Consumes High Memory Utilization (文档 ID 2787583.1),此问题一般为私网的性能较差导致）。
处理办法：
1、之前是对DIAG进程人工KILL后释放内存（进程KILL后会自动启动不影响业务）；后续考虑增加脚本在后台KILL此进程。
2、增加主机层的OSW监控工具监控网络性能，监控数据保留15天，持续监控分析。

相关日志如下：

1、消耗很多内存的DIAG进行信息截图，截图如下：

2、分析DIAG进程的TRACE日志，大量的IPCLW开头的日志

*** 2024-04-12T21:20:03.747180+08:00
2024-04-12 21:20:03.747 :kjzgrcfg_finalize(): Reconfig finished inc:8 state:5
2024-04-12 21:20:03.789 :kjzgrcfg_msghdlr(): Step 1 remote_inc:8 prop_inc:8 state:5 master:0
IPCLW:[0.123777436]{E}[WAIT]:PROTO: [1712928004190595]RETRANS DBG local acnh 0x7fcfc4721ca8 dump:
IPCLW:[0.123777437]{-}[WAIT]:UTIL: [1712928004190595]  ACNH 0x7fcfc4721ca8 State: 1 SMSN: 69887638 PKT(69906133.325238309) # Pending: 4096
IPCLW:[0.123777438]{-}[WAIT]:UTIL: [1712928004190595]   Peer: DIA0.KSXP_diag.67881 AckSeq: 325234213
IPCLW:[0.123777439]{-}[WAIT]:UTIL: [1712928004190595]   Flags: 0x20000000 IHint: 0x5e64fe2a0000001f THint: 0x3218cc490000001c 
IPCLW:[0.123777440]{-}[WAIT]:UTIL: [1712928004190595]   Local Address: 169.254.12.225:60038 Remote Address: 169.254.1.19:51696
IPCLW:[0.123777441]{-}[WAIT]:UTIL: [1712928004190595]   Remote PID: ver 0 flags 1 trans 2 tos 0 opts 0 xdata3 28b1 xdata2 e85e640e
IPCLW:[0.123777442]{-}[WAIT]:UTIL: [1712928004190595]             : mmsz 32768 mmr 4096 mms 4096 xdata 101630a2
IPCLW:[0.123777443]{-}[WAIT]:UTIL: [1712928004190595]   IVPort: 63416 TVPort: 12450 IMPT: 47425 RMPT: 10417   Pending Sends: Yes Unacked Sends: Yes 
IPCLW:[0.123777444]{-}[WAIT]:UTIL: [1712928004190595]   Send Engine Queued: Yes sshdl -1 ssts 0 rtts 1712928004190837 snderrchk 5 creqcnt 1 credits 0/8
IPCLW:[0.123777445]{-}[WAIT]:UTIL: [1712928004190595]   Unackd Messages 69902037 -> 69906132. SSEQ 325234213 Send Time: INVALID TIME SMSN # Xmits: 3977664 EMSN INVALID TIME
IPCLW:[0.123777446]{-}[WAIT]:UTIL: [1712928004190595]  Pending send queue: 
IPCLW:[0.123777447]{-}[WAIT]:UTIL: [1712928004190595]    [0] mbuf 0x7fcfc47467d0 MSN 69902037 Seq 325234213 -> 325234214 # XMits: 3977664
IPCLW:[0.123777448]{-}[WAIT]:UTIL: [1712928004190595]    [1] mbuf 0x7fcfc4746bf0 MSN 69902038 Seq 325234214 -> 325234215 # XMits: 3977663
IPCLW:[0.123777449]{-}[WAIT]:UTIL: [1712928004190595]    [2] mbuf 0x7fcfc47463b0 MSN 69902039 Seq 325234215 -> 325234216 # XMits: 3977662
IPCLW:[0.123777450]{-}[WAIT]:UTIL: [1712928004190595]    [3] mbuf 0x7fcfc4745f90 MSN 69902040 Seq 325234216 -> 325234217 # XMits: 3977661
IPCLW:[0.123777451]{-}[WAIT]:UTIL: [1712928004190595]    [4] mbuf 0x7fcfc4745b70 MSN 69902041 Seq 325234217 -> 325234218 # XMits: 3977660
IPCLW:[0.123777452]{-}[WAIT]:UTIL: [1712928004190595]    [5] mbuf 0x7fcfc4745750 MSN 69902042 Seq 325234218 -> 325234219 # XMits: 3977659
IPCLW:[0.123777453]{-}[WAIT]:UTIL: [1712928004190595]    [6] mbuf 0x7fcfc4745330 MSN 69902043 Seq 325234219 -> 325234220 # XMits: 3977658
IPCLW:[0.123777454]{-}[WAIT]:UTIL: [1712928004190595]    [7] mbuf 0x7fcfc4744f10 MSN 69902044 Seq 325234220 -> 325234221 # XMits: 3977657
IPCLW:[0.123777455]{-}[WAIT]:UTIL: [1712928004190595]    [8] mbuf 0x7fcfc4744af0 MSN 69902045 Seq 325234221 -> 325234222 # XMits: 0
IPCLW:[0.123777456]{-}[WAIT]:UTIL: [1712928004190595]    [9] mbuf 0x7fcfc47446d0 MSN 69902046 Seq 325234222 -> 325234223 # XMits: 0
IPCLW:[0.123777457]{-}[WAIT]:UTIL: [1712928004190595]    [10] mbuf 0x7fcfc47442b0 MSN 69902047 Seq 325234223 -> 325234224 # XMits: 0
IPCLW:[0.123777458]{-}[WAIT]:UTIL: [1712928004190595]    [11] mbuf 0x7fcfc4743e90 MSN 69902048 Seq 325234224 -> 325234225 # XMits: 0
IPCLW:[0.123777459]{-}[WAIT]:UTIL: [1712928004190595]    [12] mbuf 0x7fcfc4743a70 MSN 69902049 Seq 325234225 -> 325234226 # XMits: 0
IPCLW:[0.123777460]{-}[WAIT]:UTIL: [1712928004190595]    [13] mbuf 0x7fcfc4743650 MSN 69902050 Seq 325234226 -> 325234227 # XMits: 0
IPCLW:[0.123777461]{-}[WAIT]:UTIL: [1712928004190595]    [14] mbuf 0x7fcfc4743230 MSN 69902051 Seq 325234227 -> 325234228 # XMits: 0
IPCLW:[0.123777462]{-}[WAIT]:UTIL: [1712928004190595]    [15] mbuf 0x7fcfc4742e10 MSN 69902052 Seq 325234228 -> 325234229 # XMits: 0
IPCLW:[0.123777463]{-}[WAIT]:UTIL: [1712928004190595]    [16] mbuf 0x7fcfc47429f0 MSN 69902053 Seq 325234229 -> 325234230 # XMits: 0
IPCLW:[0.123777464]{-}[WAIT]:UTIL: [1712928004190595]    [17] mbuf 0x7fcfc47425d0 MSN 69902054 Seq 325234230 -> 325234231 # XMits: 0
IPCLW:[0.123777465]{-}[WAIT]:UTIL: [1712928004190595]    [18] mbuf 0x7fcfc47421b0 MSN 69902055 Seq 325234231 -> 325234232 # XMits: 0
IPCLW:[0.123777466]{-}[WAIT]:UTIL: [1712928004190595]    [19] mbuf 0x7fcfc4741d90 MSN 69902056 Seq 325234232 -> 325234233 # XMits: 0
IPCLW:[0.123777467]{-}[WAIT]:UTIL: [1712928004190595]    [20] mbuf 0x7fcfc4741970 MSN 69902057 Seq 325234233 -> 325234234 # XMits: 0
IPCLW:[0.123777468]{-}[WAIT]:UTIL: [1712928004190595]    [21] mbuf 0x7fcfc4741550 MSN 69902058 Seq 325234234 -> 325234235 # XMits: 0
IPCLW:[0.123777469]{-}[WAIT]:UTIL: [1712928004190595]    [22] mbuf 0x7fcfc4741130 MSN 69902059 Seq 325234235 -> 325234236 # XMits: 0
IPCLW:[0.123777470]{-}[WAIT]:UTIL: [1712928004190595]    [23] mbuf 0x7fcfc4740d10 MSN 69902060 Seq 325234236 -> 325234237 # XMits: 0
IPCLW:[0.123777471]{-}[WAIT]:UTIL: [1712928004190595]    [24] mbuf 0x7fcfc47408f0 MSN 69902061 Seq 325234237 -> 325234238 # XMits: 0
IPCLW:[0.123777472]{-}[WAIT]:UTIL: [1712928004190595]    [25] mbuf 0x7fcfc47404d0 MSN 69902062 Seq 325234238 -> 325234239 # XMits: 0
IPCLW:[0.123777473]{-}[WAIT]:UTIL: [1712928004190595]    [26] mbuf 0x7fcfc47400b0 MSN 69902063 Seq 325234239 -> 325234240 # XMits: 0
IPCLW:[0.123777474]{-}[WAIT]:UTIL: [1712928004190595]    [27] mbuf 0x7fcfc473fc90 MSN 69902064 Seq 325234240 -> 325234241 # XMits: 0
IPCLW:[0.123777475]{-}[WAIT]:UTIL: [1712928004190595]    [28] mbuf 0x7fcfc473f870 MSN 69902065 Seq 325234241 -> 325234242 # XMits: 0
IPCLW:[0.123777476]{-}[WAIT]:UTIL: [1712928004190595]    [29] mbuf 0x7fcfc473f450 MSN 69902066 Seq 325234242 -> 325234243 # XMits: 0
IPCLW:[0.123777477]{-}[WAIT]:UTIL: [1712928004190595]    [30] mbuf 0x7fcfc473f030 MSN 69902067 Seq 325234243 -> 325234244 # XMits: 0
IPCLW:[0.123777478]{-}[WAIT]:UTIL: [1712928004190595]    [31] mbuf 0x7fcfc473ec10 MSN 69902068 Seq 325234244 -> 325234245 # XMits: 0
IPCLW:[0.123777479]{-}[WAIT]:UTIL: [1712928004190595]    [32] mbuf 0x7fcfc473e7f0 MSN 69902069 Seq 325234245 -> 325234246 # XMits: 0
IPCLW:[0.123777480]{-}[WAIT]:UTIL: [1712928004190595]    [33] mbuf 0x7fcfc473e3d0 MSN 69902070 Seq 325234246 -> 325234247 # XMits: 0
IPCLW:[0.123777481]{-}[WAIT]:UTIL: [1712928004190595]    [34] mbuf 0x7fcfc473dfb0 MSN 69902071 Seq 325234247 -> 325234248 # XMits: 0
IPCLW:[0.123777482]{-}[WAIT]:UTIL: [1712928004190595]    [35] mbuf 0x7fcfc473db90 MSN 69902072 Seq 325234248 -> 325234249 # XMits: 0
IPCLW:[0.123777483]{-}[WAIT]:UTIL: [1712928004190595]    [36] mbuf 0x7fcfc473d770 MSN 69902073 Seq 325234249 -> 325234250 # XMits: 0
IPCLW:[0.123777484]{-}[WAIT]:UTIL: [1712928004190595]    [37] mbuf 0x7fcfc473d350 MSN 69902074 Seq 325234250 -> 325234251 # XMits: 0
IPCLW:[0.123777485]{-}[WAIT]:UTIL: [1712928004190595]    [38] mbuf 0x7fcfc473cf30 MSN 69902075 Seq 325234251 -> 325234252 # XMits: 0
IPCLW:[0.123777486]{-}[WAIT]:UTIL: [1712928004190595]    [39] mbuf 0x7fcfc473cb10 MSN 69902076 Seq 325234252 -> 325234253 # XMits: 0
IPCLW:[0.123777487]{-}[WAIT]:UTIL: [1712928004190595]    [40] mbuf 0x7fcfc473c6f0 MSN 69902077 Seq 325234253 -> 325234254 # XMits: 0
IPCLW:[0.123777488]{-}[WAIT]:UTIL: [1712928004190595]    [41] mbuf 0x7fcfc473c2d0 MSN 69902078 Seq 325234254 -> 325234255 # XMits: 0
IPCLW:[0.123777489]{-}[WAIT]:UTIL: [1712928004190595]    [42] mbuf 0x7fcfc473beb0 MSN 69902079 Seq 325234255 -> 325234256 # XMits: 0
IPCLW:[0.123777490]{-}[WAIT]:UTIL: [1712928004190595]    [43] mbuf 0x7fcfc473ba90 MSN 69902080 Seq 325234256 -> 325234257 # XMits: 0
IPCLW:[0.123777491]{-}[WAIT]:UTIL: [1712928004190595]    [44] mbuf 0x7fcfc473b670 MSN 69902081 Seq 325234257 -> 325234258 # XMits: 0
IPCLW:[0.123777492]{-}[WAIT]:UTIL: [1712928004190595]    [45] mbuf 0x7fcfc473b250 MSN 69902082 Seq 325234258 -> 325234259 # XMits: 0
IPCLW:[0.123777493]{-}[WAIT]:UTIL: [1712928004190595]    [46] mbuf 0x7fcfc473ae30 MSN 69902083 Seq 325234259 -> 325234260 # XMits: 0
IPCLW:[0.123777494]{-}[WAIT]:UTIL: [1712928004190595]    [47] mbuf 0x7fcfc473aa10 MSN 69902084 Seq 325234260 -> 325234261 # XMits: 0
IPCLW:[0.123777495]{-}[WAIT]:UTIL: [1712928004190595]    [48] mbuf 0x7fcfc473a5f0 MSN 69902085 Seq 325234261 -> 325234262 # XMits: 0
IPCLW:[0.123777496]{-}[WAIT]:UTIL: [1712928004190595]    [49] mbuf 0x7fcfc473a1d0 MSN 69902086 Seq 325234262 -> 325234263 # XMits: 0
IPCLW:[0.123777497]{-}[WAIT]:UTIL: [1712928004190595]    [50] mbuf 0x7fcfc4739db0 MSN 69902087 Seq 325234263 -> 325234264 # XMits: 0
IPCLW:[0.123777498]{-}[WAIT]:UTIL: [1712928004190595]    [51] mbuf 0x7fcfc4739990 MSN 69902088 Seq 325234264 -> 325234265 # XMits: 0
IPCLW:[0.123777499]{-}[WAIT]:UTIL: [1712928004190595]    [52] mbuf 0x7fcfc4739570 MSN 69902089 Seq 325234265 -> 325234266 # XMits: 0
IPCLW:[0.123777500]{-}[WAIT]:UTIL: [1712928004190595]    [53] mbuf 0x7fcfc4739150 MSN 69902090 Seq 325234266 -> 325234267 # XMits: 0
IPCLW:[0.123777501]{-}[WAIT]:UTIL: [1712928004190595]    [54] mbuf 0x7fcfc4738d30 MSN 69902091 Seq 325234267 -> 325234268 # XMits: 0
IPCLW:[0.123777502]{-}[WAIT]:UTIL: [1712928004190595]    [55] mbuf 0x7fcfc4738910 MSN 69902092 Seq 325234268 -> 325234269 # XMits: 0
 

3、MOS文档中关于此问题描述

Oracle Database - Enterprise Edition - Version 19.8.0.0.0 and later
Database ora_dia0_* process consumes High Memory utilization
<Instance>_dia0_21668_base_5.trc
==============================
*** 2021-06-07T23:02:45.794273-07:00 (CDB$ROOT(1))
2021-06-07 23:02:45.794 :kjzgrcfg_finalize(): Reconfig finished inc:31 state:5
2021-06-07 23:02:45.800 :kjzgrcfg_msghdlr(): Step 1 remote_inc:31 prop_inc:31 state:5 master:2
IPCLW:[0.124755339]{E}[WAIT]:PROTO: [1623132166001387]RETRANS DBG local acnh 0x7fe9aa139530 dump:
IPCLW:[0.124755340]{-}[WAIT]:UTIL: [1623132166001387] ACNH 0x7fe9aa139530 State: 1 SMSN:
1655824697 PKT(1662191011.496113055) # Pending: 4096 >>>>>>>>>>>>>>>>>>>>>>>
IPCLW:[0.124755341]{-}[WAIT]:UTIL: [1623132166001387] Peer: DIA0.KSXP_diag.30620 AckSeq: 496108959
IPCLW:[0.124755342]{-}[WAIT]:UTIL: [1623132166001387] Flags: 0x20000000 IHint: 0x1d07ad590000001f
THint: 0x52fe03630000001f
IPCLW:[0.124755343]{-}[WAIT]:UTIL: [1623132166001387] Local Address: 192.xxx.xx.120:39061 Remote
Address: 192.xxx.12.xxx:35575 >>>>>>>>>>>>>>>>>>>>>>>>>
IPCLW:[0.124755344]{-}[WAIT]:UTIL: [1623132166001387] Remote PID: ver 0 flags 1 trans 2 tos 0 opts 0
xdata3 8227 xdata2 b8300501
IPCLW:[0.124755345]{-}[WAIT]:UTIL: [1623132166001387] : mmsz 32768 mmr 4096 mms 4096 xdata
b883f95f
IPCLW:[0.124755346]{-}[WAIT]:UTIL: [1623132166001387] IVPort: 55103 TVPort: 63839 IMPT: 52824
RMPT: 33319 Pending Sends: Yes Unacked Sends: Yes >>>>>>>>>>>>>>>>>>>>
IPCLW:[0.124755347]{-}[WAIT]:UTIL: [1623132166001387] Send Engine Queued: Yes sshdl -1 ssts 0 rtts
1623132166002134 snderrchk 5 creqcnt 1 credits 0/8
IPCLW:[0.124755348]{-}[WAIT]:UTIL: [1623132166001387] Unackd Messages 1662186915 ->
1662191010. SSEQ 496108959 Send Time: INVALID TIME SMSN # Xmits: 4227520 EMSN INVALID TIME
>>>>>>>>>
IPCLW:[0.124755349]{-}[WAIT]:UTIL: [1623132166001387
MOS文档中的案例信息与TRACE是匹配的，官方文档对此解释是心跳网络异常，需要检查处理网络问题：
CAUSE
Packet Reassembly Failure at Interconnect
SOLUTION
Involve System Admin for fixing the packet reassembly failure/Packet drop at interconnect.
4、检查网络状态，有少量报错，符合文档上说的现象。

http://www.ppmy.cn/devtools/40026.html

相关文章

Java设计模式-工厂

Java设计模式-工厂

Java设计模式中&#xff0c;工厂模式主要包括普通工厂模式以及抽象工厂模式&#xff0c;普通工厂模式是用于制造输出不同类型的对象&#xff0c;抽象工厂模式是用于制造输出不同类型的普通工厂&#xff0c;本文主要描述工厂模式的基本用法。 如上所示&#xff0c;使用普通工厂模…
阅读更多...
【经验总结】Vue2中的全局变量（store

【经验总结】Vue2中的全局变量（store

需求场景 需要在vue中存储一个可变的&#xff0c;可读写的全局变量在不同的js、页面中均可调用和读写 技术&#xff1a;使用vue的store 用法总结 一、定义变量 1、找到vue的/src/store路径&#xff0c;在modules文件夹下创建文件&#xff08;这里便于测试创建demo.js&…
阅读更多...
查看ubuntu当前路径的剩余存储空间

查看ubuntu当前路径的剩余存储空间

要查看Ubuntu当前路径所在磁盘分区的剩余存储空间&#xff0c;应该使用df命令&#xff0c;而不是du命令&#xff0c;因为df命令能显示磁盘分区的使用情况&#xff0c;包括总容量、已用空间和可用空间。为了使输出更易于阅读&#xff0c;可以加上-h选项。如果你还想知道特定挂载…
阅读更多...
2024.05.10作业

2024.05.10作业

TCP服务器 头文件 #ifndef WIDGET_H #define WIDGET_H#include <QWidget> #include <QTcpServer> #include <QTcpSocket> #include <QList> #include <QMessageBox> #include <QDebug>QT_BEGIN_NAMESPACE namespace Ui { class Widget; …
阅读更多...
光通信行业专业术语解析大全

光通信行业专业术语解析大全

近期公司内部开展了一期学习交流会&#xff0c;各位同事收获颇多&#xff0c;特别是关于一些专业术语的简称&#xff0c;大家都觉得非常有意思&#xff01;小编马不停蹄的整理出来分享给大家&#xff0c;希望可以给学习光通信知识的萌新们小小助力&#xff01; 以下是光通信行…
阅读更多...
【excel】统计单元格内数字/字符串的数量

【excel】统计单元格内数字/字符串的数量

文章目录 一、问题二、步骤&#xff08;一&#xff09;将A1中的数字分解出来&#xff0c;在不同的单元格中显示&#xff08;二&#xff09;统计每个数字出现的个数&#xff08;三&#xff09;去重 三、尾巴 一、问题 单元格中有如下数值&#xff1a;12345234534545&#xff0c…
阅读更多...
（Java）心得：LeetCode——11.盛最多水的容器

（Java）心得：LeetCode——11.盛最多水的容器

一、原题 给定一个长度为 n 的整数数组 height 。有 n 条垂线&#xff0c;第 i 条线的两个端点是 (i, 0) 和 (i, height[i]) 。 找出其中的两条线&#xff0c;使得它们与 x 轴共同构成的容器可以容纳最多的水。 返回容器可以储存的最大水量。 说明&#xff1a;你不能倾斜容…
阅读更多...
Vivado功耗之散热模型详解

Vivado功耗之散热模型详解

目录 一、前言 二、热模型 三、结温计算 四、散热器计算 五、参考资料 一、前言 ​在功耗评估中&#xff0c;Vivado中report power流程以及XPE中都有关于environment的配置&#xff0c;该界面配置涉及到多个晦涩的概念&#xff0c;之前的文章中也对相关概念进行过翻译&…
阅读更多...
最新文章

Copyright @ 2022~2023