You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-02 - 04:02
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack0slot2.osadl.org (updated Tue Dec 02, 2025 00:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1090808996055,4cyclictest1347299-21latency_hist05:27:537
1090799995452,2cyclictest386931-21kworker/u16:4+flush-8:003:37:544
109048825320,29sleep70-21swapper/703:22:217
1090808995247,4cyclictest1680289-21latency_hist08:12:537
109028724822,7sleep50-21swapper/503:19:395
108084924130,7sleep60-21swapper/603:18:036
1090808994035,4cyclictest1670163-21latency_hist08:07:537
1090802994037,3cyclictest1448039-21kworker/u16:0+flush-8:007:03:185
1090802994037,3cyclictest1448039-21kworker/u16:0+flush-8:007:03:185
1090808993934,4cyclictest1660110-21latency_hist08:02:537
109025923828,7sleep40-21swapper/403:19:134
1090802993735,2cyclictest1078631-21kworker/u16:5+events_unbound04:30:465
1090802993634,2cyclictest1155820-21kworker/u16:4+events_unbound04:03:255
1090808993529,5cyclictest1226207-21latency_hist04:27:547
1090804993533,2cyclictest1670560-21kworker/u16:4+events_unbound08:18:176
1090802993532,3cyclictest1578335-21kworker/u16:5+flush-8:007:53:185
1090802993532,2cyclictest1558182-21kworker/u16:2+events_unbound07:17:545
1090802993331,2cyclictest1428826-21kworker/u16:3+events_unbound06:44:465
1090802993331,2cyclictest1296755-21kworker/u16:2+flush-8:006:51:105
1090799993331,2cyclictest1406797-21kworker/u16:1+flush-8:006:13:194
1090802993230,2cyclictest1014463-21kworker/u16:3+flush-8:004:14:065
1090799993229,2cyclictest1275869-21kworker/u16:1+flush-8:005:07:494
1090802993129,2cyclictest1720699-21kworker/u16:0+events_unbound08:49:455
1090802993129,2cyclictest1558182-21kworker/u16:2+events_unbound07:17:465
1090802993129,2cyclictest1417539-21kworker/u16:0+flush-8:006:04:145
1090802993129,2cyclictest1406797-21kworker/u16:1+flush-8:005:57:105
1090802993129,2cyclictest1296755-21kworker/u16:2+events_unbound05:44:115
1090802993129,2cyclictest1296755-21kworker/u16:2+events_unbound05:34:585
1090802993129,2cyclictest1296755-21kworker/u16:2+events_unbound05:34:585
1090802993129,2cyclictest1155820-21kworker/u16:4+events_unbound03:57:155
1090802993129,2cyclictest1078631-21kworker/u16:5+events_unbound04:49:075
1090802993129,2cyclictest1078631-21kworker/u16:5+events_unbound04:49:075
1090802993129,2cyclictest1078631-21kworker/u16:5+events_unbound04:40:095
1090802993129,2cyclictest1078631-21kworker/u16:5+events_unbound03:36:075
1090799993129,2cyclictest1670560-21kworker/u16:4+flush-8:008:08:174
1090804993028,2cyclictest1014463-21kworker/u16:3+flush-8:004:58:106
1090802993028,2cyclictest1679330-21kworker/u16:5+events_unbound08:19:495
1090802993028,2cyclictest1308969-21kworker/u16:3+flush-8:005:24:185
1090802993028,2cyclictest1078631-21kworker/u16:5+events_unbound04:11:355
1090802993028,2cyclictest1078631-21kworker/u16:5+events_unbound03:31:595
1090808992928,0cyclictest0-21swapper/707:40:057
1090808992923,5cyclictest1317159-21cat05:12:547
1090804992926,2cyclictest1448039-21kworker/u16:0+flush-8:007:08:086
1090804992926,2cyclictest1078631-21kworker/u16:5+flush-8:004:33:106
1090802992927,2cyclictest1589502-21kworker/u16:1+events_unbound08:30:265
1090802992927,2cyclictest1428826-21kworker/u16:3+events_unbound07:51:215
1090802992927,2cyclictest1428826-21kworker/u16:3+events_unbound07:51:215
1090802992927,2cyclictest1397652-21kworker/u16:0+events_unbound05:52:265
1090802992927,2cyclictest1308969-21kworker/u16:3+flush-8:005:18:095
1090802992927,2cyclictest1014463-21kworker/u16:3+flush-8:003:50:265
1090808992827,0cyclictest0-21swapper/706:32:067
1090808992827,0cyclictest0-21swapper/706:26:187
109080899280,26cyclictest589235-21systemd-journal08:32:477
1090802992826,2cyclictest1679330-21kworker/u16:5+events_unbound08:11:465
1090802992826,2cyclictest1558182-21kworker/u16:2+events_unbound07:36:065
1090802992826,2cyclictest1297138-21kworker/u16:4+events_unbound05:06:465
1090802992826,2cyclictest1078631-21kworker/u16:5+events_unbound04:25:145
1090802992826,2cyclictest1014463-21kworker/u16:3+events_unbound03:25:105
1090799992826,2cyclictest1014463-21kworker/u16:3+flush-8:003:28:214
1090808992726,0cyclictest0-21swapper/706:09:387
1090804992724,2cyclictest1014463-21kworker/u16:3+flush-8:004:23:186
1090802992725,2cyclictest1448039-21kworker/u16:0+events_unbound06:30:585
1090802992725,2cyclictest1406797-21kworker/u16:1+flush-8:006:01:135
1090802992725,2cyclictest1308969-21kworker/u16:3+flush-8:005:14:135
1090802992725,2cyclictest1296755-21kworker/u16:2+flush-8:006:22:305
1090802992725,2cyclictest1296755-21kworker/u16:2+flush-8:006:22:305
1090802992725,2cyclictest1296755-21kworker/u16:2+events_unbound05:39:225
1090802992725,2cyclictest1014463-21kworker/u16:3+events_unbound04:53:585
1090808992622,3cyclictest1445844-21kthreadcore06:13:157
1090802992624,2cyclictest1589502-21kworker/u16:1+events_unbound08:24:505
1090802992624,2cyclictest1558182-21kworker/u16:2+flush-8:008:03:075
1090802992624,2cyclictest1498690-21kworker/u16:1+events_unbound07:11:465
1090802992624,2cyclictest1428826-21kworker/u16:3+events_unbound07:44:585
1090802992624,2cyclictest1227428-21kworker/u16:0+flush-8:005:09:585
1090802992624,2cyclictest1227428-21kworker/u16:0+events_unbound04:34:175
1090802992624,2cyclictest1155820-21kworker/u16:4+flush-8:004:00:415
1090799992624,2cyclictest1679330-21kworker/u16:5+flush-8:008:23:094
1090799992624,2cyclictest1448039-21kworker/u16:0+flush-8:006:23:214
1090799992624,2cyclictest1275869-21kworker/u16:1+events_unbound05:08:184
1090808992525,0cyclictest0-21swapper/707:44:217
1090808992524,0cyclictest0-21swapper/708:21:097
1090808992524,0cyclictest0-21swapper/707:27:417
1090802992523,2cyclictest1720737-21kworker/u16:2+events_unbound08:34:385
1090802992523,2cyclictest1720737-21kworker/u16:2+events_unbound08:34:385
1090802992523,2cyclictest1687006-21kworker/u16:6+events_unbound08:45:175
1090802992523,2cyclictest1589502-21kworker/u16:1+flush-8:008:14:585
1090802992523,2cyclictest1589502-21kworker/u16:1+events_unbound07:28:255
1090802992523,2cyclictest1578335-21kworker/u16:5+flush-8:007:59:115
1090799992523,2cyclictest1078631-21kworker/u16:5+flush-8:004:12:534
1090808992423,0cyclictest0-21swapper/705:37:317
1090808992423,0cyclictest0-21swapper/705:37:317
1090808992423,0cyclictest0-21swapper/705:03:547
1090808992418,4cyclictest589235-21systemd-journal03:52:067
1090804992422,2cyclictest386931-21kworker/u16:4+events_unbound03:33:096
1090802992422,2cyclictest1428826-21kworker/u16:3+events_unbound07:27:385
1090802992422,2cyclictest1377477-21kworker/u16:4+flush-8:006:08:195
1090802992422,2cyclictest1377477-21kworker/u16:4+events_unbound06:17:055
1090802992422,2cyclictest1125127-21kworker/u16:2+events_unbound03:45:385
1090799992422,2cyclictest1428826-21kworker/u16:3+flush-8:006:38:094
1090799992421,2cyclictest1125127-21kworker/u16:2+flush-8:004:39:504
1090799992420,3cyclictest1428826-21kworker/u16:3+flush-8:007:49:164
1090799992420,3cyclictest1428826-21kworker/u16:3+flush-8:007:49:164
1090808992322,0cyclictest0-21swapper/707:57:297
1090808992322,0cyclictest0-21swapper/705:54:067
1090808992322,0cyclictest0-21swapper/704:12:587
1090808992322,0cyclictest0-21swapper/704:00:197
1090802992321,2cyclictest1519148-21kworker/u16:4+flush-8:007:01:135
1090802992321,2cyclictest1519148-21kworker/u16:4+events_unbound06:56:575
1090802992321,2cyclictest1428826-21kworker/u16:3+flush-8:007:42:225
1090802992321,2cyclictest1296755-21kworker/u16:2+events_unbound05:01:395
1090802992321,2cyclictest1155820-21kworker/u16:4+events_unbound04:19:225
1090802992320,2cyclictest1448039-21kworker/u16:0+flush-8:006:23:045
1090799992321,2cyclictest1428826-21kworker/u16:3+flush-8:006:43:114
1090799992321,2cyclictest1367165-21kworker/u16:1+flush-8:005:43:104
1090808992222,0cyclictest0-21swapper/707:12:507
1090808992221,0cyclictest0-21swapper/706:56:307
1090808992219,2cyclictest1145672-21latency_hist03:47:547
109080899220,21cyclictest589235-21systemd-journal04:58:097
1090804992220,2cyclictest1346363-21kworker/u16:0+flush-8:005:23:216
1090802992220,2cyclictest1720699-21kworker/u16:0+events_unbound08:39:005
1090799992220,2cyclictest1679330-21kworker/u16:5+flush-8:008:22:524
1090799992220,2cyclictest1558182-21kworker/u16:2+events_unbound07:13:104
1090799992219,2cyclictest1227428-21kworker/u16:0+flush-8:004:57:394
1090799992218,3cyclictest1720737-21kworker/u16:2+flush-8:008:40:434
1090808992120,0cyclictest0-21swapper/708:50:457
1090808992120,0cyclictest0-21swapper/707:19:397
1090808992120,0cyclictest0-21swapper/706:51:057
1090808992120,0cyclictest0-21swapper/703:36:267
109080899210,19cyclictest27177-21gmain04:51:057
109080899210,19cyclictest27177-21gmain04:51:057
1090804992119,2cyclictest1275869-21kworker/u16:1+events_unbound05:08:226
1090802992119,2cyclictest1428826-21kworker/u16:3+events_unbound06:42:205
1090802992119,2cyclictest1014463-21kworker/u16:3+flush-8:004:43:235
1090799992119,2cyclictest1346363-21kworker/u16:0+flush-8:005:33:174
1090799992119,2cyclictest1346363-21kworker/u16:0+flush-8:005:33:174
1090799992119,2cyclictest1014463-21kworker/u16:3+events_unbound04:23:114
1090799992117,3cyclictest1377477-21kworker/u16:4+flush-8:006:19:074
1090799992117,3cyclictest1377477-21kworker/u16:4+flush-8:006:19:074
1090808992019,0cyclictest0-21swapper/705:16:077
1090808992019,0cyclictest0-21swapper/704:36:127
1090808992019,0cyclictest0-21swapper/704:05:017
1090808992019,0cyclictest0-21swapper/703:41:187
1090808992019,0cyclictest0-21swapper/703:26:427
1090802992018,2cyclictest1448039-21kworker/u16:0+flush-8:006:33:245
1090802992018,2cyclictest1346363-21kworker/u16:0+events_unbound05:31:175
1090802992018,2cyclictest1078631-21kworker/u16:5+events_unbound03:39:265
1090799992018,2cyclictest1014463-21kworker/u16:3+events_unbound03:23:174
1090799992017,2cyclictest1558182-21kworker/u16:2+events_unbound07:43:164
1090808991919,0cyclictest0-21swapper/706:47:217
1090808991918,0cyclictest0-21swapper/707:01:257
1090808991918,0cyclictest0-21swapper/705:48:037
1090808991913,5cyclictest1256577-21latency_hist04:42:537
109080899190,17cyclictest0-21swapper/707:48:197
109080899190,17cyclictest0-21swapper/707:48:197
1090804991917,2cyclictest1448039-21kworker/u16:0+flush-8:006:53:086
1090799991917,2cyclictest1720699-21kworker/u16:0+flush-8:008:47:534
1090799991916,2cyclictest1558182-21kworker/u16:2+flush-8:008:03:094
1090808991817,0cyclictest0-21swapper/708:33:447
1090808991817,0cyclictest0-21swapper/708:33:447
1090808991813,4cyclictest589235-21systemd-journal04:20:447
109080899180,16cyclictest1545599-21cut07:03:147
109080899180,16cyclictest1545599-21cut07:03:147
1090804991811,6cyclictest0-21swapper/608:33:066
1090804991811,6cyclictest0-21swapper/608:33:066
1090799991816,2cyclictest1078631-21kworker/u16:5+flush-8:004:48:094
1090799991816,2cyclictest1078631-21kworker/u16:5+flush-8:004:48:094
1090799991816,2cyclictest1014463-21kworker/u16:3+events_unbound04:13:214
1090799991815,2cyclictest1428826-21kworker/u16:3+events_unbound06:28:124
1090808991717,0cyclictest0-21swapper/705:59:407
1090808991716,0cyclictest0-21swapper/706:05:497
1090808991716,0cyclictest0-21swapper/704:56:097
1090808991715,1cyclictest0-21swapper/708:23:127
1090808991714,2cyclictest4175659-21snmpd08:39:457
1090804991714,2cyclictest1589502-21kworker/u16:1+flush-8:007:53:096
1090804991714,2cyclictest1014463-21kworker/u16:3+flush-8:004:03:126
1090799991714,2cyclictest1296755-21kworker/u16:2+flush-8:005:23:144
1090799991714,2cyclictest1155820-21kworker/u16:4+flush-8:004:02:544
1090799991714,2cyclictest1155820-21kworker/u16:4+flush-8:003:58:144
1090799991714,2cyclictest1125127-21kworker/u16:2+flush-8:004:43:174
1090808991615,0cyclictest0-21swapper/707:28:117
1090808991615,0cyclictest0-21swapper/706:20:327
1090808991615,0cyclictest0-21swapper/706:20:327
1090808991615,0cyclictest0-21swapper/705:30:227
1090808991615,0cyclictest0-21swapper/704:28:107
1090808991615,0cyclictest0-21swapper/703:53:317
109080899160,2cyclictest4175659-21snmpd04:43:547
1090804991614,2cyclictest1327128-21kworker/u16:4+flush-8:005:28:226
1090799991613,2cyclictest1155820-21kworker/u16:4+flush-8:004:18:224
1090808991514,0cyclictest0-21swapper/705:47:387
109080899150,14cyclictest1488464-21latency_hist06:37:537
109080899150,14cyclictest0-21swapper/708:13:197
1090804991513,2cyclictest1589502-21kworker/u16:1+flush-8:007:31:306
1090804991511,3cyclictest1448039-21kworker/u16:0+flush-8:006:38:206
1090799991513,2cyclictest1125127-21kworker/u16:2+events_unbound03:48:184
1090799991513,2cyclictest1014463-21kworker/u16:3+events_unbound04:58:224
109079999150,12cyclictest0-21swapper/407:19:104
109080899148,5cyclictest1609649-21latency_hist07:37:527
109080499140,1cyclictest0-21swapper/607:03:196
109080499140,1cyclictest0-21swapper/607:03:196
109080499140,14cyclictest0-21swapper/608:23:246
109080499140,14cyclictest0-21swapper/604:13:236
109080499140,13cyclictest0-21swapper/608:28:066
1090799991413,1cyclictest0-21swapper/408:12:564
1090799991412,2cyclictest1448039-21kworker/u16:0+events_unbound06:33:184
1090799991412,2cyclictest1417539-21kworker/u16:0+flush-8:006:03:184
1090799991411,2cyclictest1687006-21kworker/u16:6+flush-8:008:28:234
1090799991411,2cyclictest1589502-21kworker/u16:1+flush-8:007:53:084
1090799991411,2cyclictest1406797-21kworker/u16:1+flush-8:006:08:134
109079999140,1cyclictest0-21swapper/407:38:154
109080899139,3cyclictest4175659-21snmpd04:11:087
1090808991310,2cyclictest96-21ksoftirqd/705:38:187
1090804991312,1cyclictest0-21swapper/607:51:156
1090804991312,1cyclictest0-21swapper/607:51:156
1090804991311,2cyclictest1720737-21kworker/u16:2+flush-8:008:38:176
109080499130,13cyclictest0-21swapper/606:28:386
109080499130,13cyclictest0-21swapper/605:06:116
1090799991311,2cyclictest1720699-21kworker/u16:0+events_unbound08:48:174
109079999130,13cyclictest0-21swapper/405:51:034
109079999130,13cyclictest0-21swapper/405:16:034
109080899129,2cyclictest4175659-21snmpd06:38:177
109080899122,9cyclictest1337237-21sh05:22:537
1090808991211,0cyclictest0-21swapper/703:28:157
109080499127,4cyclictest1570422-21cat07:18:116
1090804991211,1cyclictest0-21swapper/608:02:456
1090804991211,1cyclictest0-21swapper/603:23:156
109080499120,12cyclictest0-21swapper/608:16:476
109080499120,0cyclictest0-21swapper/608:09:236
109079999129,2cyclictest1428826-21kworker/u16:3+flush-8:007:08:244
109079999129,2cyclictest1078631-21kworker/u16:5+flush-8:004:33:124
109079999127,4cyclictest1651158-21cat07:58:084
1090799991211,1cyclictest0-21swapper/404:28:464
1090799991210,2cyclictest1428826-21kworker/u16:3+events_unbound06:58:134
1090799991210,2cyclictest1296755-21kworker/u16:2+flush-8:006:48:264
109079999120,12cyclictest0-21swapper/405:58:564
109079999120,12cyclictest0-21swapper/405:18:444
109080899118,2cyclictest4175659-21snmpd08:44:367
109080899116,4cyclictest96-21ksoftirqd/707:13:137
109080499118,2cyclictest1478910-21cpuspeed_turbos06:33:076
109080499117,3cyclictest1742172-21cat08:43:066
1090804991110,1cyclictest0-21swapper/606:02:006
1090804991110,1cyclictest0-21swapper/605:46:236
1090804991110,1cyclictest0-21swapper/605:41:116
1090804991110,1cyclictest0-21swapper/604:19:136
109080499110,11cyclictest0-21swapper/608:52:066
109080499110,11cyclictest0-21swapper/605:21:106
109080499110,11cyclictest0-21swapper/603:43:546
109080499110,0cyclictest0-21swapper/604:54:416
109079999119,2cyclictest1377477-21kworker/u16:4+flush-8:005:53:204
1090799991110,1cyclictest0-21swapper/406:53:574
109079999110,11cyclictest0-21swapper/403:44:284
109080499109,1cyclictest0-21swapper/607:37:226
109080499109,1cyclictest0-21swapper/606:43:096
109080499109,1cyclictest0-21swapper/606:26:546
109080499109,1cyclictest0-21swapper/604:52:006
109080499109,1cyclictest0-21swapper/604:52:006
109080499109,1cyclictest0-21swapper/604:41:456
109080499100,0cyclictest0-21swapper/605:51:366
109079999109,1cyclictest0-21swapper/407:26:444
109079999109,1cyclictest0-21swapper/405:29:184
10908049998,1cyclictest0-21swapper/607:47:096
10908049998,1cyclictest0-21swapper/607:42:166
10908049998,1cyclictest0-21swapper/607:17:316
10908049998,1cyclictest0-21swapper/606:03:176
10908049996,3cyclictest1428826-21kworker/u16:3+flush-8:006:08:106
10908049996,2cyclictest1186926-21tune2fs04:08:096
10908049996,2cyclictest0-21swapper/606:48:166
10908049995,3cyclictest1449267-21cat06:18:086
10908049995,3cyclictest1449267-21cat06:18:086
10908049995,3cyclictest1166728-21cat03:58:096
10908049990,8cyclictest1398316-21cut05:53:086
10907999998,1cyclictest0-21swapper/403:57:424
10907999996,2cyclictest1578335-21kworker/u16:5+flush-8:007:36:144
10908049987,1cyclictest0-21swapper/605:37:436
10908049987,1cyclictest0-21swapper/605:37:436
10908049986,2cyclictest102-21kauditd06:58:086
10908049985,2cyclictest0-21swapper/603:53:136
10908049984,3cyclictest1580535-21iostat_ios07:23:096
10908049984,3cyclictest1326778-21unixbench_singl05:13:246
10908049984,3cyclictest1150962-21kthreadcore03:48:146
10908049984,3cyclictest1104924-21ntp_states03:27:586
10908049983,4cyclictest1229413-21cut04:28:136
10908049983,4cyclictest1127135-21cut03:38:136
10907999984,3cyclictest1598654-21ntp_states07:28:194
10907999983,4cyclictest1730489-21unixbench-2d08:33:204
10907999983,4cyclictest1730489-21unixbench-2d08:33:204
10907999983,4cyclictest1372339-21cut05:38:154
10908049975,1cyclictest0-21swapper/606:13:096
10908049973,3cyclictest1660908-21expr08:03:056
10908049973,3cyclictest1257035-21cat04:43:086
10907999973,3cyclictest1540844-21cut07:03:104
10907999973,3cyclictest1540844-21cut07:03:104
10907999973,3cyclictest1125661-21cpuspeed_turbos03:38:084
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional