You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-29 - 01:41
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack0slot2.osadl.org (updated Sun Jun 29, 2025 00:43:35)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
68254826050,7sleep50-21swapper/503:30:355
68243925318,30sleep70-21swapper/703:29:007
68257624939,7sleep40-21swapper/403:30:584
683045993331,2cyclictest862399-21kworker/u16:1+events_unbound04:58:344
683048993230,2cyclictest1018219-21kworker/u16:4+events_unbound06:54:295
68269123222,7sleep60-21swapper/603:32:366
683051993130,0cyclictest0-21swapper/706:38:497
68305199310,29cyclictest925210-21latency_hist05:33:197
68305199310,29cyclictest1347896-21latency_hist08:58:187
683045993129,2cyclictest758224-21kworker/u16:5+flush-8:004:33:494
683045993128,2cyclictest1153586-21kworker/u16:4+events_unbound07:23:444
683051993029,0cyclictest0-21swapper/708:38:197
68305199300,28cyclictest882942-21latency_hist05:08:197
683048993028,2cyclictest684456-21kworker/u16:0+events_unbound03:39:545
683045993028,2cyclictest758224-21kworker/u16:5+events_unbound04:43:374
683045993027,2cyclictest862399-21kworker/u16:1+events_unbound05:33:334
683048992927,2cyclictest893368-21kworker/u16:4+events_unbound05:57:545
683048992927,2cyclictest863630-21kworker/u16:3+events_unbound05:59:525
683048992927,2cyclictest863630-21kworker/u16:3+events_unbound05:10:545
683048992927,2cyclictest862399-21kworker/u16:1+events_unbound05:40:545
683048992927,2cyclictest748544-21kworker/u16:3+events_unbound04:24:535
683048992927,2cyclictest684456-21kworker/u16:0+events_unbound03:43:295
683048992927,2cyclictest1306747-21kworker/u16:2+events_unbound08:45:535
683048992927,2cyclictest1224953-21kworker/u16:3+events_unbound08:00:525
683048992927,2cyclictest1153586-21kworker/u16:4+events_unbound07:43:565
683048992927,2cyclictest1153586-21kworker/u16:4+events_unbound07:27:295
683048992927,2cyclictest1152811-21kworker/u16:2+events_unbound07:50:525
683045992927,2cyclictest702178-21kworker/u16:5+events_unbound03:48:224
683045992925,3cyclictest1061549-21kworker/u16:0+flush-8:007:13:204
683051992827,0cyclictest0-21swapper/706:33:347
683048992826,2cyclictest988435-21kworker/u16:0+events_unbound06:28:165
683048992826,2cyclictest758224-21kworker/u16:5+events_unbound04:34:535
683048992826,2cyclictest758224-21kworker/u16:5+events_unbound04:22:345
683048992826,2cyclictest1337422-21kworker/u16:4+events_unbound08:54:295
683048992826,2cyclictest1121989-21kworker/u16:1+events_unbound07:37:135
683048992826,2cyclictest1121989-21kworker/u16:1+events_unbound07:37:135
683048992826,2cyclictest1061549-21kworker/u16:0+events_unbound07:07:295
68305199270,25cyclictest1183079-21latency_hist07:38:197
68305199270,25cyclictest1183079-21latency_hist07:38:197
683048992725,2cyclictest893368-21kworker/u16:4+events_unbound05:27:295
683048992725,2cyclictest863630-21kworker/u16:3+flush-8:006:36:295
683048992725,2cyclictest862399-21kworker/u16:1+events_unbound05:35:305
683048992725,2cyclictest748544-21kworker/u16:3+events_unbound04:45:525
683048992725,2cyclictest726123-21kworker/u16:1+events_unbound04:17:295
683048992725,2cyclictest1306747-21kworker/u16:2+events_unbound08:49:455
683045992723,3cyclictest992208-21kworker/u16:1+flush-8:006:18:444
683051992624,1cyclictest0-21swapper/706:13:447
683050992624,2cyclictest1264470-21kworker/u16:1+events_unbound08:38:496
683048992624,2cyclictest820146-21kworker/u16:0+events_unbound05:04:535
683048992624,2cyclictest758224-21kworker/u16:5+flush-8:004:53:535
683048992624,2cyclictest684456-21kworker/u16:0+events_unbound03:56:335
683048992624,2cyclictest1275974-21kworker/u16:0+events_unbound08:34:525
683048992624,2cyclictest1121989-21kworker/u16:1+events_unbound07:40:015
683045992625,1cyclictest1285034-21kworker/u16:5+events_unbound08:33:364
683045992624,2cyclictest1109965-21kworker/u16:2+events_unbound07:03:524
683051992524,0cyclictest0-21swapper/708:18:527
683051992524,0cyclictest0-21swapper/708:18:527
683051992523,1cyclictest956180-21munin-run05:48:187
683051992522,2cyclictest852015-21munin-run04:53:197
683050992522,2cyclictest1061549-21kworker/u16:0+flush-8:006:53:196
683050992522,2cyclictest1061549-21kworker/u16:0+flush-8:006:53:196
683050992521,3cyclictest1042887-21kworker/u16:1+flush-8:006:43:376
683048992523,2cyclictest684456-21kworker/u16:0+events_unbound03:48:485
683048992523,2cyclictest684456-21kworker/u16:0+events_unbound03:37:455
683048992523,2cyclictest1240744-21kworker/u16:2+events_unbound08:09:535
683048992523,2cyclictest1121989-21kworker/u16:1+events_unbound07:13:175
683045992523,2cyclictest1264470-21kworker/u16:1+flush-8:008:28:484
683051992423,0cyclictest0-21swapper/707:28:497
683051992422,1cyclictest1007779-21awk06:13:197
683050992422,2cyclictest988435-21kworker/u16:0+events_unbound06:23:196
683048992423,1cyclictest1223880-21kworker/u16:0+events_unbound08:07:425
683048992422,2cyclictest863630-21kworker/u16:3+events_unbound07:01:535
683048992422,2cyclictest862399-21kworker/u16:1+events_unbound05:43:295
683048992422,2cyclictest1158862-21kworker/u16:5+events_unbound07:29:535
683048992422,2cyclictest1121989-21kworker/u16:1+events_unbound07:16:445
683048992422,2cyclictest1027439-21kworker/u16:5+events_unbound06:44:495
683048992422,2cyclictest1027439-21kworker/u16:5+events_unbound06:42:535
683048992422,2cyclictest1018219-21kworker/u16:4+events_unbound06:21:295
683048992421,2cyclictest1264470-21kworker/u16:1+events_unbound08:38:295
683045992422,2cyclictest820146-21kworker/u16:0+flush-8:004:53:454
683045992422,2cyclictest1240744-21kworker/u16:2+flush-8:008:23:484
683051992320,2cyclictest893215-21munin-run05:13:197
683048992321,2cyclictest893368-21kworker/u16:4+events_unbound05:15:305
683048992321,2cyclictest893368-21kworker/u16:4+events_unbound05:15:305
683048992321,2cyclictest863630-21kworker/u16:3+events_unbound05:22:345
683048992321,2cyclictest726123-21kworker/u16:1+flush-8:003:58:415
683048992321,2cyclictest1337422-21kworker/u16:4+events_unbound09:02:535
683048992321,2cyclictest1264470-21kworker/u16:1+events_unbound08:25:255
683048992321,2cyclictest1121989-21kworker/u16:1+flush-8:007:54:285
683048992321,2cyclictest1121989-21kworker/u16:1+events_unbound07:22:205
683045992321,2cyclictest862399-21kworker/u16:1+events_unbound05:53:214
68305199220,21cyclictest1327289-21latency_hist08:48:197
683048992220,2cyclictest1224953-21kworker/u16:3+events_unbound08:15:255
683045992220,2cyclictest863630-21kworker/u16:3+events_unbound06:08:144
683045992220,2cyclictest684456-21kworker/u16:0+events_unbound04:08:484
683045992220,2cyclictest1152811-21kworker/u16:2+flush-8:007:28:524
683045992219,2cyclictest992208-21kworker/u16:1+events_unbound06:08:204
683051992119,1cyclictest1121269-21awk07:08:197
683051992118,2cyclictest790217-21idleruntime-cro04:23:197
68305199210,20cyclictest1313-21snmpd03:53:217
683050992118,3cyclictest726123-21kworker/u16:1+flush-8:004:18:196
683048992119,2cyclictest863630-21kworker/u16:3+events_unbound06:05:345
683048992119,2cyclictest863630-21kworker/u16:3+events_unbound05:30:455
683048992119,2cyclictest840740-21kworker/u16:2+events_unbound05:02:505
683048992119,2cyclictest820146-21kworker/u16:0+flush-8:004:42:295
683048992119,2cyclictest820146-21kworker/u16:0+events_unbound04:52:505
683048992119,2cyclictest729814-21kworker/u16:2+events_unbound04:03:435
683048992119,2cyclictest684456-21kworker/u16:0+flush-8:004:09:535
683048992119,2cyclictest1275974-21kworker/u16:0+events_unbound08:30:345
683045992119,2cyclictest1153586-21kworker/u16:4+events_unbound07:33:504
683045992119,2cyclictest1153586-21kworker/u16:4+events_unbound07:33:504
683045992118,2cyclictest642007-21kworker/u16:4+events_unbound03:33:454
683045992118,2cyclictest1121989-21kworker/u16:1+flush-8:008:08:344
683045992018,2cyclictest758224-21kworker/u16:5+flush-8:004:38:334
683051991918,0cyclictest0-21swapper/707:18:487
683051991918,0cyclictest0-21swapper/705:53:417
683050991915,3cyclictest758224-21kworker/u16:5+events_unbound04:43:336
683048991917,2cyclictest863630-21kworker/u16:3+events_unbound06:09:095
683048991917,2cyclictest1027439-21kworker/u16:5+events_unbound06:31:205
683045991917,2cyclictest893368-21kworker/u16:4+events_unbound05:38:324
683051991814,2cyclictest1286071-21munin-run08:28:187
68305199180,17cyclictest1313-21snmpd04:30:157
68305199180,17cyclictest1313-21snmpd04:30:157
68305199180,16cyclictest894299-21latency_hist05:18:197
68305199180,16cyclictest894299-21latency_hist05:18:197
683050991815,3cyclictest1042887-21kworker/u16:1+flush-8:006:33:456
683048991816,2cyclictest992208-21kworker/u16:1+events_unbound06:14:255
683048991816,2cyclictest1061549-21kworker/u16:0+flush-8:006:48:555
683048991816,2cyclictest1061549-21kworker/u16:0+flush-8:006:48:555
683051991717,0cyclictest0-21swapper/707:13:467
683051991712,4cyclictest96-21ksoftirqd/704:13:197
68305199170,16cyclictest987166-21latency_hist06:03:197
68305199170,16cyclictest1313-21snmpd08:31:277
68305199170,15cyclictest1121968-21exim407:08:207
683050991715,2cyclictest863630-21kworker/u16:3+events_unbound05:18:456
683050991715,2cyclictest1275974-21kworker/u16:0+events_unbound08:33:526
683048991714,2cyclictest1240744-21kworker/u16:2+events_unbound08:18:315
683048991714,2cyclictest1240744-21kworker/u16:2+events_unbound08:18:315
683051991615,0cyclictest0-21swapper/707:38:207
683051991614,1cyclictest800539-21awk04:28:197
683051991614,1cyclictest728259-21munin-run03:53:197
683051991613,2cyclictest707612-21munin-run03:43:197
683051991613,2cyclictest1239980-21cut08:03:447
683050991613,2cyclictest729814-21kworker/u16:2+events_unbound03:58:226
683048991614,2cyclictest893368-21kworker/u16:4+events_unbound05:50:215
683051991514,0cyclictest0-21swapper/706:03:447
683051991513,1cyclictest841721-21awk04:48:197
68305199150,14cyclictest1313-21snmpd07:48:537
68305199150,13cyclictest1039449-21sendmail06:28:207
68305099150,1cyclictest0-21swapper/607:23:406
683045991513,2cyclictest748544-21kworker/u16:3+flush-8:004:23:144
683045991513,2cyclictest1152811-21kworker/u16:2+flush-8:007:38:414
683045991512,2cyclictest1027439-21kworker/u16:5+events_unbound06:23:484
68305199147,6cyclictest27642-21gmain03:33:497
68305199145,7cyclictest27763-21gvfs-afc-volume07:53:397
683051991412,1cyclictest966531-21awk05:53:187
683051991412,1cyclictest1110933-21awk07:03:197
68305199140,1cyclictest0-21swapper/704:38:457
68305199140,13cyclictest1313-21snmpd04:37:367
68305199140,12cyclictest1313-21snmpd06:25:187
68305199140,12cyclictest1313-21snmpd06:18:577
683050991413,1cyclictest0-21swapper/603:48:196
683050991412,2cyclictest702178-21kworker/u16:5+events_unbound03:53:216
683050991412,2cyclictest684456-21kworker/u16:0+flush-8:003:43:196
683050991410,3cyclictest966569-21kworker/u16:5+flush-8:005:53:446
683048991412,2cyclictest749593-21kworker/u16:4+events_unbound04:31:025
683048991412,2cyclictest749593-21kworker/u16:4+events_unbound04:31:025
683045991412,2cyclictest988435-21kworker/u16:0+events_unbound06:13:484
683045991412,2cyclictest863630-21kworker/u16:3+events_unbound05:18:324
68304599140,1cyclictest0-21swapper/408:48:544
68305199139,3cyclictest1265437-21munin-run08:18:187
68305199136,6cyclictest27904-21gmain05:33:327
683051991313,0cyclictest0-21swapper/708:08:517
683051991312,0cyclictest0-21swapper/707:26:347
683051991312,0cyclictest0-21swapper/704:18:147
683051991311,1cyclictest0-21swapper/706:53:447
68305199130,2cyclictest710081-21cat03:43:387
68305199130,12cyclictest96-21ksoftirqd/706:43:337
683050991312,1cyclictest0-21swapper/607:34:516
683050991312,1cyclictest0-21swapper/607:34:516
683050991311,2cyclictest862399-21kworker/u16:1+events_unbound05:13:196
683050991311,2cyclictest1224953-21kworker/u16:3+flush-8:008:03:196
683045991312,1cyclictest0-21swapper/405:13:524
683045991312,1cyclictest0-21swapper/405:13:524
683045991312,1cyclictest0-21swapper/404:06:434
683045991311,2cyclictest749593-21kworker/u16:4+events_unbound04:28:534
683045991311,2cyclictest749593-21kworker/u16:4+events_unbound04:28:534
683045991311,2cyclictest1275974-21kworker/u16:0+events_unbound08:53:484
683045991311,2cyclictest1223880-21kworker/u16:0+events_unbound08:03:544
683045991311,2cyclictest1061549-21kworker/u16:0+events_unbound07:08:514
683045991310,3cyclictest726123-21kworker/u16:1+flush-8:004:23:344
683045991310,2cyclictest726123-21kworker/u16:1+flush-8:003:53:354
68305199126,5cyclictest27763-21gvfs-afc-volume08:48:417
68305199120,3cyclictest1313-21snmpd05:18:367
68305199120,11cyclictest96-21ksoftirqd/704:53:427
68305199120,11cyclictest1313-21snmpd07:47:147
68305099129,2cyclictest893368-21kworker/u16:4+events_unbound05:38:346
683050991211,1cyclictest0-21swapper/608:30:306
683050991211,1cyclictest0-21swapper/606:56:526
683050991211,1cyclictest0-21swapper/605:30:386
683050991211,1cyclictest0-21swapper/605:23:426
683050991210,2cyclictest988435-21kworker/u16:0+events_unbound06:28:376
68305099120,1cyclictest0-21swapper/607:32:016
68305099120,1cyclictest0-21swapper/607:17:596
68305099120,11cyclictest0-21swapper/605:14:056
68305099120,11cyclictest0-21swapper/605:14:056
683045991211,1cyclictest0-21swapper/406:43:154
683045991211,1cyclictest0-21swapper/405:28:324
683045991210,2cyclictest893368-21kworker/u16:4+events_unbound05:23:214
683045991210,2cyclictest863630-21kworker/u16:3+flush-8:007:18:504
683045991210,2cyclictest862399-21kworker/u16:1+events_unbound05:48:364
683045991210,2cyclictest1240744-21kworker/u16:2+events_unbound08:13:484
68304599120,1cyclictest0-21swapper/403:44:574
68304599120,12cyclictest0-21swapper/407:58:174
68304599120,12cyclictest0-21swapper/406:45:184
683051991111,0cyclictest0-21swapper/704:07:187
68305199110,9cyclictest1313-21snmpd05:38:357
68305199110,10cyclictest1313-21snmpd09:00:317
68305199110,10cyclictest1313-21snmpd08:42:167
68305199110,10cyclictest1313-21snmpd08:01:487
68305199110,10cyclictest1313-21snmpd05:25:387
68305199110,10cyclictest1313-21snmpd04:58:227
68305199110,10cyclictest1313-21snmpd03:59:297
68305099119,2cyclictest748544-21kworker/u16:3+events_unbound04:28:386
68305099119,2cyclictest748544-21kworker/u16:3+events_unbound04:28:386
683050991110,1cyclictest0-21swapper/608:48:296
683050991110,1cyclictest0-21swapper/608:24:026
683050991110,1cyclictest0-21swapper/608:22:096
683050991110,1cyclictest0-21swapper/608:22:096
683050991110,1cyclictest0-21swapper/607:49:506
683050991110,1cyclictest0-21swapper/606:40:476
683050991110,1cyclictest0-21swapper/604:53:446
683050991110,1cyclictest0-21swapper/604:49:576
683050991110,1cyclictest0-21swapper/604:28:126
683050991110,1cyclictest0-21swapper/604:09:086
683050991110,1cyclictest0-21swapper/604:03:566
68305099110,1cyclictest0-21swapper/606:18:056
68305099110,1cyclictest0-21swapper/605:07:376
68305099110,11cyclictest0-21swapper/608:56:036
68305099110,11cyclictest0-21swapper/608:10:106
68305099110,11cyclictest0-21swapper/607:21:466
68305099110,11cyclictest0-21swapper/605:47:496
68304599117,3cyclictest1318522-21cat08:43:364
683045991110,1cyclictest0-21swapper/406:33:004
683045991110,1cyclictest0-21swapper/405:59:074
683045991110,1cyclictest0-21swapper/405:12:554
683045991110,1cyclictest0-21swapper/405:05:314
68304599110,11cyclictest0-21swapper/408:20:314
68304599110,11cyclictest0-21swapper/408:20:314
683051991010,0cyclictest0-21swapper/706:48:547
683051991010,0cyclictest0-21swapper/706:48:547
68305099109,1cyclictest0-21swapper/608:43:456
68305099109,1cyclictest0-21swapper/608:05:366
68305099109,1cyclictest0-21swapper/606:06:536
68305099109,1cyclictest0-21swapper/605:02:066
68305099107,2cyclictest977610-21/usr/sbin/munin05:58:456
68305099107,2cyclictest1120230-21needreboot07:03:476
68305099100,9cyclictest0-21swapper/605:48:506
68304599107,3cyclictest1224953-21kworker/u16:3+flush-8:007:58:444
68304599106,3cyclictest1346979-21kworker/u16:5+flush-8:008:58:504
68304599106,3cyclictest1307451-21munin-node08:38:324
68304599105,4cyclictest1102832-21cat06:58:384
68304599105,4cyclictest1091511-21df06:53:354
6830509998,1cyclictest0-21swapper/604:40:396
6830509998,1cyclictest0-21swapper/604:37:596
6830509990,8cyclictest685863-21kthreadcore03:33:386
6830459998,1cyclictest0-21swapper/405:43:494
6830509987,1cyclictest0-21swapper/607:47:566
6830509987,1cyclictest0-21swapper/605:33:386
6830509983,4cyclictest781952-21tr04:18:406
6830509983,4cyclictest1123998-21kthreadcore07:08:436
6830509983,4cyclictest1000691-21cut06:08:406
6830509980,7cyclictest0-21swapper/606:23:426
6830509980,1cyclictest0-21swapper/608:13:536
6830509980,1cyclictest0-21swapper/606:58:376
6830459987,1cyclictest0-21swapper/404:14:364
6830459986,2cyclictest726123-21kworker/u16:1+flush-8:003:58:454
6830459985,3cyclictest1027439-21kworker/u16:5+flush-8:006:35:204
6830459984,3cyclictest698773-21interrupts03:38:364
6830459984,3cyclictest1089929-21users06:48:524
6830459984,3cyclictest1089929-21users06:48:524
6830459983,4cyclictest844072-21cut04:48:394
6830509976,1cyclictest0-21swapper/607:38:386
6830509975,1cyclictest0-21swapper/607:53:386
6830509972,4cyclictest1352263-21tr08:58:386
6830459973,3cyclictest1194458-21expr07:43:374
6830509962,3cyclictest718822-21cat03:48:356
6830459962,3cyclictest1204647-21grep07:48:334
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional