You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-15 - 12:23
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack0slot2.osadl.org (updated Tue Jul 15, 2025 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
86777099350,33cyclictest1364753-21latency_hist07:32:157
86777099340,32cyclictest974017-21latency_hist04:22:167
867759993432,2cyclictest1086208-21kworker/u16:0+flush-8:005:22:455
86777099330,31cyclictest1015231-21latency_hist04:42:157
867759993329,3cyclictest90340-21kworker/u16:3+flush-8:004:02:315
86777099320,30cyclictest1436767-21latency_hist08:07:157
867759993028,2cyclictest1138631-21kworker/u16:0+flush-8:006:17:305
867770992928,0cyclictest0-21swapper/707:57:107
867770992926,2cyclictest1127347-21ntpq05:32:437
867759992927,2cyclictest867262-21kworker/u16:2+flush-8:004:42:305
867759992927,2cyclictest867262-21kworker/u16:2+flush-8:004:42:305
86777099280,26cyclictest1097405-21cat05:22:097
867766992826,2cyclictest1549909-21kworker/u16:0+events_unbound09:02:136
867766992826,2cyclictest1138631-21kworker/u16:0+events_unbound06:17:446
867759992826,2cyclictest1138631-21kworker/u16:0+flush-8:005:47:355
867766992725,2cyclictest1385501-21kworker/u16:2+events_unbound08:31:506
867759992725,2cyclictest1086208-21kworker/u16:0+events_unbound05:27:295
86777099260,24cyclictest1365474-21sendmail07:32:167
867766992624,2cyclictest1272546-21kworker/u16:1+events_unbound08:06:186
867766992624,2cyclictest1148885-21kworker/u16:5+flush-8:005:59:226
867766992623,2cyclictest912628-21kworker/u16:0+flush-8:004:17:196
867766992523,2cyclictest1457503-21kworker/u16:3+events_unbound08:57:146
867766992523,2cyclictest1281332-21kworker/u16:5+flush-8:006:54:506
867766992523,2cyclictest1281332-21kworker/u16:5+events_unbound07:04:266
867766992523,2cyclictest1281332-21kworker/u16:5+events_unbound07:04:266
867766992523,2cyclictest1272546-21kworker/u16:1+flush-8:007:41:256
867766992523,2cyclictest1180424-21kworker/u16:1+events_unbound06:17:096
867766992523,2cyclictest1117830-21kworker/u16:3+events_unbound06:24:466
867766992522,2cyclictest1138631-21kworker/u16:0+events_unbound06:47:176
867766992422,2cyclictest867262-21kworker/u16:2+events_unbound05:15:436
867766992422,2cyclictest702961-21kworker/u16:4+events_unbound03:45:266
867766992422,2cyclictest702961-21kworker/u16:4+events_unbound03:34:256
867766992422,2cyclictest1281332-21kworker/u16:5+events_unbound07:23:146
867766992422,2cyclictest1138631-21kworker/u16:0+flush-8:005:48:186
867766992422,2cyclictest1138631-21kworker/u16:0+events_unbound07:56:146
867766992422,2cyclictest112242-21kworker/u16:1+flush-8:003:39:256
867759992422,2cyclictest1272546-21kworker/u16:1+events_unbound07:03:345
867759992422,2cyclictest1272546-21kworker/u16:1+events_unbound07:03:345
86777099230,21cyclictest1272167-21latency_hist06:47:147
867766992321,2cyclictest974250-21kworker/u16:1+events_unbound05:22:146
867766992321,2cyclictest1426187-21kworker/u16:4+events_unbound08:11:306
867766992321,2cyclictest1314009-21kworker/u16:3+events_unbound07:09:146
867766992321,2cyclictest112242-21kworker/u16:1+events_unbound03:49:346
867766992320,2cyclictest974250-21kworker/u16:1+flush-8:004:36:466
867759992321,2cyclictest1385501-21kworker/u16:2+events_unbound08:22:095
867759992321,2cyclictest1272546-21kworker/u16:1+events_unbound07:23:065
867759992321,2cyclictest1138631-21kworker/u16:0+events_unbound08:01:305
867759992321,2cyclictest1138631-21kworker/u16:0+events_unbound07:01:055
867759992321,2cyclictest1127971-21kworker/u16:4+events_unbound05:33:425
867759992320,3cyclictest867262-21kworker/u16:2+events_unbound03:57:335
867766992220,2cyclictest1281332-21kworker/u16:5+events_unbound07:19:536
867766992220,2cyclictest1272546-21kworker/u16:1+flush-8:007:29:496
867766992220,2cyclictest1138631-21kworker/u16:0+flush-8:007:12:436
867766992220,2cyclictest1138631-21kworker/u16:0+events_unbound06:29:536
867766992220,2cyclictest1086208-21kworker/u16:0+flush-8:005:31:026
867759992220,2cyclictest702961-21kworker/u16:4+events_unbound04:13:285
867759992220,2cyclictest1426187-21kworker/u16:4+events_unbound08:26:215
867759992220,2cyclictest1426187-21kworker/u16:4+events_unbound08:15:505
867759992220,2cyclictest1281332-21kworker/u16:5+events_unbound07:30:575
867759992220,2cyclictest1138631-21kworker/u16:0+events_unbound06:44:015
867759992220,2cyclictest1045698-21kworker/u16:4+events_unbound04:57:335
867770992119,1cyclictest1292790-21awk06:57:147
867766992119,2cyclictest1385501-21kworker/u16:2+events_unbound08:18:256
867766992119,2cyclictest1281332-21kworker/u16:5+flush-8:007:47:316
867766992119,2cyclictest1281332-21kworker/u16:5+flush-8:007:47:316
867759992119,2cyclictest1281332-21kworker/u16:5+flush-8:007:10:015
867759992119,2cyclictest1148885-21kworker/u16:5+events_unbound05:53:325
867759992119,2cyclictest1026229-21kworker/u16:0+events_unbound04:49:105
867770992019,0cyclictest0-21swapper/704:42:397
867770992019,0cyclictest0-21swapper/704:42:397
86777099200,18cyclictest1519177-21latency_hist08:47:147
867766992018,2cyclictest1281332-21kworker/u16:5+flush-8:007:32:536
867766992018,2cyclictest1138631-21kworker/u16:0+flush-8:006:09:266
867766992018,2cyclictest1117830-21kworker/u16:3+events_unbound06:04:036
867766992018,2cyclictest1026229-21kworker/u16:0+events_unbound04:47:546
867759992018,2cyclictest1477997-21kworker/u16:5+events_unbound08:32:545
867759992018,2cyclictest1138631-21kworker/u16:0+events_unbound06:35:315
867770991917,1cyclictest1117996-21munin-run05:32:157
867770991917,1cyclictest0-21swapper/703:47:457
86777099190,17cyclictest1313-21snmpd06:00:247
867766991917,2cyclictest983332-21kworker/u16:5+events_unbound04:39:186
867766991917,2cyclictest912628-21kworker/u16:0+flush-8:004:12:506
867766991917,2cyclictest867262-21kworker/u16:2+events_unbound06:38:256
867766991917,2cyclictest867262-21kworker/u16:2+events_unbound05:10:266
867766991917,2cyclictest1272546-21kworker/u16:1+flush-8:007:42:366
867759991917,2cyclictest974250-21kworker/u16:1+flush-8:005:17:585
867759991917,2cyclictest867262-21kworker/u16:2+events_unbound04:19:205
867759991917,2cyclictest1385501-21kworker/u16:2+flush-8:008:09:455
867759991917,2cyclictest112242-21kworker/u16:1+events_unbound03:41:255
867759991915,3cyclictest1045698-21kworker/u16:4+flush-8:005:07:415
14857262190,1sleep40-21swapper/408:27:354
867770991817,0cyclictest0-21swapper/704:32:187
867770991815,2cyclictest1539763-21munin-run08:57:147
86777099180,17cyclictest1313273-21latency_hist07:07:157
86777099180,17cyclictest1313273-21latency_hist07:07:157
867766991816,2cyclictest974250-21kworker/u16:1+events_unbound05:36:016
867766991816,2cyclictest912628-21kworker/u16:0+events_unbound04:12:146
867766991816,2cyclictest867262-21kworker/u16:2+events_unbound04:03:546
867766991815,2cyclictest1385501-21kworker/u16:2+flush-8:008:42:306
867759991816,2cyclictest867262-21kworker/u16:2+events_unbound04:36:425
867759991816,2cyclictest1457503-21kworker/u16:3+events_unbound08:47:125
867759991816,2cyclictest1426187-21kworker/u16:4+events_unbound08:04:565
867759991816,2cyclictest1385501-21kworker/u16:2+events_unbound08:51:285
867759991816,2cyclictest1281332-21kworker/u16:5+flush-8:007:39:475
867759991816,2cyclictest112242-21kworker/u16:1+events_unbound03:52:085
867759991816,2cyclictest1117830-21kworker/u16:3+events_unbound06:22:575
867759991815,2cyclictest702961-21kworker/u16:4+events_unbound03:42:245
867770991715,1cyclictest922828-21wc03:57:157
867770991715,1cyclictest922828-21wc03:57:157
867770991715,1cyclictest0-21swapper/705:42:457
867766991715,2cyclictest867262-21kworker/u16:2+events_unbound05:05:146
867766991715,2cyclictest1180424-21kworker/u16:1+events_unbound06:35:256
867766991715,2cyclictest1138631-21kworker/u16:0+flush-8:005:45:226
867766991714,2cyclictest90340-21kworker/u16:3+flush-8:004:01:266
867759991715,2cyclictest90340-21kworker/u16:3+events_unbound03:33:565
867759991715,2cyclictest1415069-21kworker/u16:3+events_unbound07:53:205
867759991715,2cyclictest1148885-21kworker/u16:5+events_unbound05:57:445
867759991715,2cyclictest1138631-21kworker/u16:0+flush-8:007:21:135
867759991715,2cyclictest1138631-21kworker/u16:0+events_unbound07:34:425
867759991715,2cyclictest1138631-21kworker/u16:0+events_unbound07:13:595
867759991715,2cyclictest1138631-21kworker/u16:0+events_unbound06:41:565
867759991715,2cyclictest1138631-21kworker/u16:0+events_unbound06:06:385
867759991715,2cyclictest1117830-21kworker/u16:3+flush-8:006:30:415
867759991713,3cyclictest1457503-21kworker/u16:3+flush-8:008:27:395
867744991716,1cyclictest0-21swapper/406:53:584
867744991715,1cyclictest0-21swapper/403:47:364
867770991614,1cyclictest1210518-21awk06:17:147
86777099160,14cyclictest1313-21snmpd03:37:507
86777099160,14cyclictest1169450-21latency_hist05:57:157
867766991614,2cyclictest912628-21kworker/u16:0+events_unbound04:25:496
867766991614,2cyclictest1477997-21kworker/u16:5+events_unbound08:39:516
867766991614,2cyclictest1148885-21kworker/u16:5+events_unbound05:53:266
867759991614,2cyclictest974250-21kworker/u16:1+events_unbound05:39:325
867759991614,2cyclictest867262-21kworker/u16:2+events_unbound04:12:045
867759991614,2cyclictest1477997-21kworker/u16:5+events_unbound08:39:395
867759991614,2cyclictest1117830-21kworker/u16:3+events_unbound05:47:075
867759991614,2cyclictest1086208-21kworker/u16:0+events_unbound05:14:485
867759991614,2cyclictest1045698-21kworker/u16:4+events_unbound05:05:225
867759991613,2cyclictest702961-21kworker/u16:4+events_unbound04:22:245
86775999160,15cyclictest741rcuc/508:52:355
867744991615,1cyclictest0-21swapper/408:53:424
867744991615,1cyclictest0-21swapper/408:50:384
867744991615,1cyclictest0-21swapper/408:33:134
867744991615,1cyclictest0-21swapper/408:27:094
867744991615,1cyclictest0-21swapper/408:22:024
867744991615,1cyclictest0-21swapper/408:13:504
867744991615,1cyclictest0-21swapper/408:11:474
867744991615,1cyclictest0-21swapper/407:53:214
867744991615,1cyclictest0-21swapper/407:42:064
867744991615,1cyclictest0-21swapper/407:36:584
867744991615,1cyclictest0-21swapper/407:30:504
867744991615,1cyclictest0-21swapper/407:22:384
867744991615,1cyclictest0-21swapper/407:09:204
867744991615,1cyclictest0-21swapper/406:41:414
867744991615,1cyclictest0-21swapper/406:33:294
867744991615,1cyclictest0-21swapper/406:31:264
867744991615,1cyclictest0-21swapper/406:20:114
867744991615,1cyclictest0-21swapper/406:17:064
867744991615,1cyclictest0-21swapper/406:07:534
867744991615,1cyclictest0-21swapper/405:55:364
867744991615,1cyclictest0-21swapper/405:51:304
867744991615,1cyclictest0-21swapper/405:39:134
867744991615,1cyclictest0-21swapper/405:36:094
867744991615,1cyclictest0-21swapper/405:28:594
867744991615,1cyclictest0-21swapper/405:23:524
867744991615,1cyclictest0-21swapper/405:21:494
867744991615,1cyclictest0-21swapper/405:14:394
867744991615,1cyclictest0-21swapper/405:02:214
867744991615,1cyclictest0-21swapper/405:00:184
867744991615,1cyclictest0-21swapper/404:48:014
867744991615,1cyclictest0-21swapper/404:47:004
867744991615,1cyclictest0-21swapper/404:47:004
867744991615,1cyclictest0-21swapper/404:37:474
867744991615,1cyclictest0-21swapper/404:33:414
867744991615,1cyclictest0-21swapper/404:30:374
867744991615,1cyclictest0-21swapper/404:22:254
867744991615,1cyclictest0-21swapper/404:13:124
867744991615,1cyclictest0-21swapper/404:10:084
867744991615,1cyclictest0-21swapper/404:07:044
867744991615,1cyclictest0-21swapper/403:58:524
867744991615,1cyclictest0-21swapper/403:53:454
867744991615,1cyclictest0-21swapper/403:53:454
867744991613,2cyclictest1059-21lldpd03:45:374
86774499160,16cyclictest0-21swapper/408:57:434
867770991513,1cyclictest1076940-21basename05:12:157
86777099150,14cyclictest1508936-21latency_hist08:42:147
86777099150,14cyclictest1313-21snmpd07:51:427
86777099150,14cyclictest1313-21snmpd07:51:427
867766991513,2cyclictest974250-21kworker/u16:1+events_unbound05:37:276
867766991513,2cyclictest90340-21kworker/u16:3+flush-8:004:31:186
867766991513,2cyclictest1457503-21kworker/u16:3+flush-8:008:32:576
867766991513,2cyclictest1457503-21kworker/u16:3+events_unbound08:47:366
867766991513,2cyclictest1426187-21kworker/u16:4+events_unbound08:16:186
867766991513,2cyclictest1138631-21kworker/u16:0+events_unbound07:02:026
867766991513,2cyclictest1086208-21kworker/u16:0+events_unbound05:22:496
867759991513,2cyclictest974250-21kworker/u16:1+events_unbound04:38:375
867759991513,2cyclictest912628-21kworker/u16:0+flush-8:003:53:105
867759991513,2cyclictest912628-21kworker/u16:0+flush-8:003:53:105
867759991513,2cyclictest867262-21kworker/u16:2+flush-8:006:48:015
867759991513,2cyclictest867262-21kworker/u16:2+events_unbound06:16:005
867759991513,2cyclictest1477997-21kworker/u16:5+events_unbound08:58:365
867759991513,2cyclictest1385501-21kworker/u16:2+events_unbound07:45:485
867759991513,2cyclictest1272546-21kworker/u16:1+events_unbound07:48:485
867759991513,2cyclictest1272546-21kworker/u16:1+events_unbound07:48:485
867759991513,2cyclictest1138631-21kworker/u16:0+flush-8:006:10:485
867744991514,1cyclictest0-21swapper/408:42:264
867744991514,1cyclictest0-21swapper/408:40:234
867744991514,1cyclictest0-21swapper/406:46:484
867744991514,1cyclictest0-21swapper/404:21:244
86777099148,5cyclictest1087183-21turbostat05:17:147
867770991413,0cyclictest0-21swapper/705:22:167
867770991412,1cyclictest1066628-21turbostat05:07:157
86777099140,13cyclictest1344166-21turbostat07:22:147
86777099140,13cyclictest1282458-21turbostat06:52:147
86777099140,12cyclictest1354464-21latency_hist07:27:157
867766991412,2cyclictest90340-21kworker/u16:3+flush-8:003:55:596
867766991412,2cyclictest90340-21kworker/u16:3+flush-8:003:55:596
867766991412,2cyclictest1385501-21kworker/u16:2+events_unbound08:24:556
867759991412,2cyclictest912628-21kworker/u16:0+events_unbound04:29:405
867744991413,1cyclictest0-21swapper/405:44:204
86774499140,14cyclictest0-21swapper/407:02:034
8889942130,1sleep40-21swapper/403:37:384
86777099139,3cyclictest1241350-21idleruntime-cro06:32:157
86777099136,6cyclictest27204-21gmain05:50:537
86777099135,7cyclictest27430-21gmain04:07:297
867770991312,0cyclictest0-21swapper/706:37:507
867770991310,2cyclictest1385294-21munin-run07:42:147
867770991310,2cyclictest1323570-21munin-run07:12:147
86777099130,2cyclictest1477958-21munin-plugin-st08:27:147
86777099130,11cyclictest1332932-21ntpq07:12:417
86777099130,11cyclictest1313-21snmpd03:45:337
867766991311,2cyclictest974250-21kworker/u16:1+events_unbound05:00:176
867766991311,2cyclictest90340-21kworker/u16:3+events_unbound04:45:266
867766991311,2cyclictest90340-21kworker/u16:3+events_unbound04:45:266
867766991311,2cyclictest1045698-21kworker/u16:4+flush-8:004:54:106
867766991310,2cyclictest0-21swapper/606:42:316
867759991311,2cyclictest1272546-21kworker/u16:1+flush-8:006:52:305
867744991312,1cyclictest0-21swapper/407:48:144
867744991312,1cyclictest0-21swapper/407:48:144
867744991312,1cyclictest0-21swapper/406:02:464
86777099128,3cyclictest27204-21gmain04:02:417
86777099126,5cyclictest27642-21gmain07:57:417
86777099126,5cyclictest27430-21gmain08:10:537
86777099121,10cyclictest1026530-21cstates04:47:297
867770991211,0cyclictest0-21swapper/704:52:297
867770991211,0cyclictest0-21swapper/704:01:527
86777099120,4cyclictest1199922-21sshd-session06:07:517
86777099120,11cyclictest1313-21snmpd05:40:027
86777099120,11cyclictest1313-21snmpd04:25:197
86777099120,11cyclictest1313-21snmpd04:15:097
86777099120,10cyclictest1313-21snmpd08:32:307
86777099120,10cyclictest1313-21snmpd07:47:147
867759991210,2cyclictest1045698-21kworker/u16:4+events_unbound04:54:075
86774499129,3cyclictest0-21swapper/404:57:144
86774499128,4cyclictest0-21swapper/405:07:294
86774499128,1cyclictest0-21swapper/408:05:394
867744991211,1cyclictest0-21swapper/407:18:324
867744991211,1cyclictest0-21swapper/406:23:154
867744991211,1cyclictest0-21swapper/405:59:424
86774499120,12cyclictest0-21swapper/403:33:104
86777099116,5cyclictest27642-21gmain06:18:137
86777099116,4cyclictest27763-21gvfs-afc-volume08:47:367
86777099115,5cyclictest976-21in:imjournal08:27:327
867770991111,0cyclictest0-21swapper/706:05:477
86777099110,10cyclictest1313-21snmpd08:57:197
86777099110,10cyclictest1313-21snmpd08:17:427
86777099110,10cyclictest1313-21snmpd06:58:317
86777099110,10cyclictest1313-21snmpd03:32:477
86774499119,2cyclictest0-21swapper/407:58:294
86774499119,2cyclictest0-21swapper/407:45:104
86774499119,2cyclictest0-21swapper/406:50:544
867744991110,1cyclictest0-21swapper/407:14:274
867744991110,1cyclictest0-21swapper/407:06:564
867744991110,1cyclictest0-21swapper/407:06:564
86777099100,9cyclictest1456973-21sshd-session08:12:517
86777099100,9cyclictest0-21swapper/706:32:357
86777099100,3cyclictest1313-21snmpd04:27:397
86777099100,1cyclictest0-21swapper/706:22:347
86777099100,1cyclictest0-21swapper/704:57:407
86776699108,2cyclictest1385501-21kworker/u16:2+events_unbound08:01:016
8677709950,4cyclictest868047-21latency_hist03:32:177
8677669932,1cyclictest0-21swapper/603:32:176
8677449930,2cyclictest1-21systemd03:32:174
8677599910,1cyclictest0-21swapper/503:32:175
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional