You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-28 - 23:03
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot2.osadl.org (updated Sat Jun 28, 2025 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
209889999370,35cyclictest2380695-21latency_hist17:48:217
2098882993634,2cyclictest2526697-21kworker/u16:1+events_unbound20:23:545
2098882993432,2cyclictest2142958-21kworker/u16:0+events_unbound16:43:515
2098882993330,2cyclictest2021296-21kworker/u16:2+flush-8:015:33:425
2098882993128,2cyclictest2142958-21kworker/u16:0+events_unbound16:03:515
2098882993028,2cyclictest2699844-21kworker/u16:2+events_unbound20:47:545
2098882993028,2cyclictest2331104-21kworker/u16:1+events_unbound17:35:555
2098882993028,2cyclictest2085688-21kworker/u16:1+events_unbound15:51:325
2098882993028,2cyclictest2085688-21kworker/u16:1+events_unbound15:51:325
2098882992927,2cyclictest2607304-21kworker/u16:4+events_unbound19:47:315
2098882992927,2cyclictest2607304-21kworker/u16:4+events_unbound19:47:315
2098882992927,2cyclictest2526697-21kworker/u16:1+events_unbound19:39:425
2098882992927,2cyclictest2377691-21kworker/u16:2+events_unbound18:38:005
2098882992927,2cyclictest2267506-21kworker/u16:1+events_unbound17:16:315
2098882992926,2cyclictest2174933-21kworker/u16:2+flush-8:016:08:325
209889999280,26cyclictest2287980-21latency_hist17:03:227
2098893992825,2cyclictest2184145-21kworker/u16:4+events_unbound16:53:396
2098882992826,2cyclictest2586599-21kworker/u16:3+events_unbound20:51:475
2098882992826,2cyclictest2331104-21kworker/u16:1+events_unbound17:29:365
2098882992826,2cyclictest2184145-21kworker/u16:4+events_unbound16:54:305
2098882992826,2cyclictest2085688-21kworker/u16:1+events_unbound15:42:235
2098893992725,2cyclictest2184145-21kworker/u16:4+flush-8:016:53:216
2098882992725,2cyclictest2586599-21kworker/u16:3+events_unbound20:11:315
2098882992725,2cyclictest2377691-21kworker/u16:2+events_unbound18:45:435
2098882992725,2cyclictest2377691-21kworker/u16:2+events_unbound17:47:315
2098882992725,2cyclictest2319130-21kworker/u16:0+events_unbound17:19:365
2098882992725,2cyclictest2085688-21kworker/u16:1+events_unbound15:44:325
2098882992725,2cyclictest2068033-21kworker/u16:3+events_unbound17:06:565
2098882992724,2cyclictest2709793-21kworker/u16:4+events_unbound20:34:545
2098882992724,2cyclictest2278425-21kworker/u16:2+events_unbound17:23:315
2098882992724,2cyclictest2278425-21kworker/u16:2+events_unbound17:23:315
2098882992624,2cyclictest2699844-21kworker/u16:2+events_unbound20:39:555
2098882992624,2cyclictest2524740-21kworker/u16:0+events_unbound19:26:545
2098882992624,2cyclictest2524740-21kworker/u16:0+events_unbound18:59:465
2098882992624,2cyclictest2412304-21kworker/u16:3+flush-8:019:16:315
2098882992624,2cyclictest2402006-21kworker/u16:0+events_unbound18:11:555
2098882992624,2cyclictest2402006-21kworker/u16:0+events_unbound18:11:555
2098882992624,2cyclictest2377691-21kworker/u16:2+events_unbound18:39:325
2098882992624,2cyclictest2365489-21kworker/u16:0+events_unbound17:42:545
2098882992624,2cyclictest2184145-21kworker/u16:4+events_unbound16:29:315
2098882992623,2cyclictest2184145-21kworker/u16:4+flush-8:016:38:315
2098882992523,2cyclictest2759042-21kworker/u16:4+events_unbound20:55:145
2098882992523,2cyclictest2699844-21kworker/u16:2+events_unbound20:32:305
2098882992523,2cyclictest2586599-21kworker/u16:3+events_unbound19:54:315
2098882992523,2cyclictest2412304-21kworker/u16:3+events_unbound19:11:315
2098882992523,2cyclictest2412304-21kworker/u16:3+events_unbound18:28:305
2098882992523,2cyclictest2184145-21kworker/u16:4+events_unbound16:52:325
2098882992523,2cyclictest2085688-21kworker/u16:1+events_unbound16:01:325
2098882992522,2cyclictest2659337-21kworker/u16:0+events_unbound20:14:315
2098882992422,2cyclictest2526697-21kworker/u16:1+events_unbound19:31:555
2098882992422,2cyclictest2377691-21kworker/u16:2+flush-8:018:03:595
2098882992422,2cyclictest2267506-21kworker/u16:1+events_unbound17:11:285
26114022230,0chrt0-21swapper/419:38:464
2098899992321,1cyclictest2411567-21munin-run18:03:207
2098882992321,2cyclictest2751113-21kworker/u16:1+events_unbound20:59:555
2098882992321,2cyclictest2659337-21kworker/u16:0+events_unbound20:04:315
2098882992321,2cyclictest2586599-21kworker/u16:3+events_unbound20:21:315
2098882992321,2cyclictest2184145-21kworker/u16:4+flush-8:016:24:325
2098882992220,2cyclictest2402006-21kworker/u16:0+events_unbound17:58:305
2098882992220,2cyclictest2068033-21kworker/u16:3+events_unbound16:20:235
2098899992119,1cyclictest0-21swapper/719:28:477
209889999210,15cyclictest1313-21snmpd20:23:137
2098882992119,2cyclictest2068033-21kworker/u16:3+events_unbound17:02:325
209889999200,14cyclictest27975-21passimd15:39:517
209889999200,14cyclictest2761546-21latency_hist20:53:217
2098882992018,2cyclictest2586599-21kworker/u16:3+flush-8:019:58:545
2098882992018,2cyclictest2412304-21kworker/u16:3+events_unbound18:21:225
2098882992018,2cyclictest2377691-21kworker/u16:2+flush-8:018:48:595
2098882992018,2cyclictest2377691-21kworker/u16:2+events_unbound17:56:465
2098882992018,2cyclictest2377691-21kworker/u16:2+events_unbound17:48:305
2098882992018,2cyclictest2068033-21kworker/u16:3+events_unbound16:14:325
2098899991916,2cyclictest2638060-21munin-run19:53:207
2098899991911,2cyclictest2205549-21munin-run16:23:217
209889999190,17cyclictest2658603-21latency_hist20:03:217
2098882991916,2cyclictest2524740-21kworker/u16:0+flush-8:019:18:445
209889999187,5cyclictest27204-21gmain17:29:427
2098899991816,1cyclictest0-21swapper/715:58:487
2098899991811,2cyclictest2545402-21munin-run19:08:217
2098899991810,2cyclictest2143780-21munin-run15:53:217
2098899991810,2cyclictest2143780-21munin-run15:53:217
2098893991816,2cyclictest2085688-21kworker/u16:1+events_unbound15:40:476
2098882991816,2cyclictest2526697-21kworker/u16:1+events_unbound19:05:435
2098882991816,2cyclictest2412304-21kworker/u16:3+events_unbound18:27:475
209889999170,15cyclictest2627764-21latency_hist19:48:217
209889999170,15cyclictest2627764-21latency_hist19:48:217
209889999170,15cyclictest1313-21snmpd18:10:017
209889999170,15cyclictest1313-21snmpd18:10:017
2098893991715,2cyclictest2526697-21kworker/u16:1+flush-8:020:08:526
2098893991715,2cyclictest2402006-21kworker/u16:0+events_unbound17:58:416
2098893991715,2cyclictest2319130-21kworker/u16:0+flush-8:017:28:216
2098893991715,2cyclictest2319130-21kworker/u16:0+flush-8:017:28:216
2098893991714,2cyclictest2607304-21kworker/u16:4+flush-8:019:43:516
2098893991714,2cyclictest2607304-21kworker/u16:4+flush-8:019:43:516
2098882991715,2cyclictest2526697-21kworker/u16:1+events_unbound19:33:405
2098877991710,6cyclictest2122725-21pool-/usr/libex15:39:504
209889999160,14cyclictest2607136-21turbostat19:38:207
2098882991615,1cyclictest2377691-21kworker/u16:2+events_unbound18:57:275
2098882991614,2cyclictest2526697-21kworker/u16:1+events_unbound19:50:215
2098882991614,2cyclictest2412304-21kworker/u16:3+events_unbound18:15:425
209887799160,16cyclictest0-21swapper/420:28:484
209887799160,14cyclictest0-21swapper/419:03:484
209889999159,5cyclictest2504152-21turbostat18:48:207
2098899991511,0cyclictest0-21swapper/716:17:337
2098899991511,0cyclictest0-21swapper/715:56:067
209889999150,14cyclictest2318871-21turbostat17:18:217
209889999150,13cyclictest2782164-21latency_hist21:03:217
2098893991513,2cyclictest2377691-21kworker/u16:2+flush-8:018:03:516
2098893991512,2cyclictest2524740-21kworker/u16:0+flush-8:019:23:416
2098893991512,2cyclictest2229802-21kthreadcore16:33:406
2098893991511,3cyclictest2524740-21kworker/u16:0+flush-8:019:28:536
266872821412,1sleep70-21swapper/720:06:317
2098899991412,1cyclictest2329167-21munin-run17:23:217
2098899991411,0cyclictest0-21swapper/718:17:597
2098899991410,2cyclictest2166471-21kthreadcore16:03:407
209889999140,2cyclictest2729755-21ntpq20:33:507
209889999140,13cyclictest2689472-21latency_hist20:18:207
209889999140,13cyclictest1313-21snmpd18:39:137
2098882991412,2cyclictest2085688-21kworker/u16:1+flush-8:015:55:595
209887799140,14cyclictest0-21swapper/419:33:504
209889999136,6cyclictest27177-21gmain18:07:547
2098899991313,0cyclictest0-21swapper/717:27:327
2098899991313,0cyclictest0-21swapper/717:27:327
2098899991312,0cyclictest0-21swapper/720:23:477
2098899991312,0cyclictest0-21swapper/717:38:257
2098899991311,1cyclictest2555752-21awk19:13:217
209889999130,3cyclictest2132520-21ntpq15:43:517
209889999130,11cyclictest2560584-21kthreadcore19:13:457
209889999130,11cyclictest1313-21snmpd16:33:407
2098893991311,2cyclictest0-21swapper/618:53:366
209889399130,13cyclictest0-21swapper/616:11:366
2098882991311,2cyclictest2184145-21kworker/u16:4+events_unbound16:36:355
209887799131,11cyclictest0-21swapper/420:58:384
209887799130,13cyclictest0-21swapper/418:39:314
209889999128,3cyclictest27177-21gmain20:43:227
209889999126,5cyclictest27763-21gvfs-afc-volume16:28:377
209889999125,6cyclictest27642-21gmain17:51:547
2098899991211,0cyclictest0-21swapper/717:36:137
2098899991210,1cyclictest2720421-21idleruntime-cro20:33:207
209889999120,2cyclictest2299013-21/usr/sbin/munin17:08:497
209889999120,1cyclictest0-21swapper/716:23:377
209889999120,11cyclictest2676207-21cut20:08:447
209889999120,11cyclictest1313-21snmpd17:06:327
209889999120,10cyclictest2247545-21/usr/sbin/munin16:43:417
2098893991211,1cyclictest0-21swapper/617:46:306
2098893991211,1cyclictest0-21swapper/617:32:486
2098893991211,1cyclictest0-21swapper/616:07:026
2098893991211,1cyclictest0-21swapper/615:49:326
2098893991211,1cyclictest0-21swapper/615:49:326
209889399120,1cyclictest0-21swapper/618:36:196
209889399120,1cyclictest0-21swapper/618:18:126
209889399120,1cyclictest0-21swapper/616:02:286
209889399120,12cyclictest0-21swapper/617:13:246
209889399120,12cyclictest0-21swapper/616:39:596
209889399120,11cyclictest0-21swapper/620:43:336
209887799129,3cyclictest0-21swapper/415:53:314
2098877991211,1cyclictest0-21swapper/420:27:024
2098877991211,1cyclictest0-21swapper/420:07:014
2098877991211,1cyclictest0-21swapper/418:30:514
2098877991211,1cyclictest0-21swapper/417:20:324
2098877991210,2cyclictest0-21swapper/417:41:494
209887799120,9cyclictest0-21swapper/418:33:384
209887799120,8cyclictest0-21swapper/419:28:534
209887799120,1cyclictest0-21swapper/418:51:394
209887799120,1cyclictest0-21swapper/418:02:284
209887799120,1cyclictest0-21swapper/415:59:594
209887799120,12cyclictest0-21swapper/417:01:284
209889999110,9cyclictest1313-21snmpd19:23:367
209889999110,9cyclictest1313-21snmpd16:08:367
209889999110,3cyclictest2646262-21kthreadcore19:53:447
209889999110,3cyclictest2613933-21kthreadcore19:38:487
209889999110,3cyclictest2460841-21kthreadcore18:23:487
209889999110,2cyclictest2463734-21/usr/sbin/munin18:28:487
209889999110,10cyclictest2277693-21latency_hist16:58:227
209889999110,10cyclictest2099906-21cat15:33:427
209889999110,10cyclictest1313-21snmpd20:56:177
209889999110,10cyclictest1313-21snmpd20:41:357
209889999110,10cyclictest1313-21snmpd18:21:077
209889399119,2cyclictest2586599-21kworker/u16:3+events_unbound19:58:396
209889399119,2cyclictest2526697-21kworker/u16:1+events_unbound20:23:386
209889399119,2cyclictest2390691-21kworker/u16:5+events_unbound17:53:426
209889399119,2cyclictest2377691-21kworker/u16:2+events_unbound18:38:396
209889399117,3cyclictest2526697-21kworker/u16:1+flush-8:019:13:476
2098893991110,1cyclictest0-21swapper/617:52:306
2098893991110,1cyclictest0-21swapper/617:40:316
2098893991110,1cyclictest0-21swapper/617:20:586
2098893991110,1cyclictest0-21swapper/617:09:566
2098893991110,1cyclictest0-21swapper/616:21:426
209889399110,11cyclictest0-21swapper/620:08:046
209889399110,11cyclictest0-21swapper/619:55:566
209889399110,11cyclictest0-21swapper/619:50:446
209889399110,11cyclictest0-21swapper/619:22:126
209889399110,11cyclictest0-21swapper/618:50:126
209889399110,11cyclictest0-21swapper/617:00:106
209889399110,11cyclictest0-21swapper/615:47:396
209887799119,2cyclictest0-21swapper/418:13:404
209887799116,4cyclictest2266930-21systemd-tmpfile16:48:544
2098877991110,1cyclictest0-21swapper/420:39:574
2098877991110,1cyclictest0-21swapper/420:18:224
2098877991110,1cyclictest0-21swapper/420:00:044
2098877991110,1cyclictest0-21swapper/419:44:284
2098877991110,1cyclictest0-21swapper/419:44:284
2098877991110,1cyclictest0-21swapper/417:43:334
2098877991110,1cyclictest0-21swapper/415:45:104
209887799110,11cyclictest0-21swapper/420:52:534
209887799110,11cyclictest0-21swapper/416:40:204
209887799110,10cyclictest0-21swapper/419:13:424
209887799110,10cyclictest0-21swapper/418:53:364
209889999107,2cyclictest27642-21gmain18:49:227
209889999100,9cyclictest1313-21snmpd19:02:087
209889999100,1cyclictest0-21swapper/719:18:457
209889999100,1cyclictest0-21swapper/716:48:537
209889399109,1cyclictest0-21swapper/620:58:126
209889399109,1cyclictest0-21swapper/620:35:496
209889399109,1cyclictest0-21swapper/619:36:046
209889399109,1cyclictest0-21swapper/619:02:206
209889399109,1cyclictest0-21swapper/618:46:446
209889399106,3cyclictest2377691-21kworker/u16:2+flush-8:018:23:516
209887799109,1cyclictest0-21swapper/419:02:044
209887799109,1cyclictest0-21swapper/418:27:234
209887799109,1cyclictest0-21swapper/417:11:144
209887799109,1cyclictest0-21swapper/416:28:314
209887799109,1cyclictest0-21swapper/416:11:014
209887799100,3cyclictest2647697-21sed19:53:514
20988999990,8cyclictest1313-21snmpd18:54:147
20988999990,8cyclictest1313-21snmpd16:39:057
20988999990,8cyclictest0-21swapper/717:53:417
20988999990,2cyclictest1313-21snmpd18:36:447
20988939998,1cyclictest0-21swapper/620:48:456
20988939998,1cyclictest0-21swapper/620:19:256
20988939997,2cyclictest2174933-21kworker/u16:2+flush-8:016:13:536
20988939997,1cyclictest0-21swapper/618:28:366
20988939995,3cyclictest2551929-21kthreadcore19:08:446
20988779998,1cyclictest0-21swapper/418:48:114
20988779998,1cyclictest0-21swapper/418:11:564
20988779998,1cyclictest0-21swapper/418:11:564
20988779998,1cyclictest0-21swapper/416:57:034
20988779998,1cyclictest0-21swapper/416:48:134
20988779995,3cyclictest2330681-21cat17:23:404
20988779995,3cyclictest2330681-21cat17:23:404
20988779995,3cyclictest2310140-21tr17:13:404
20988779995,3cyclictest2227663-21tune2fs16:33:374
20988779991,8cyclictest0-21swapper/418:03:404
20988939987,1cyclictest0-21swapper/620:29:506
20988939987,1cyclictest0-21swapper/619:05:486
20988939987,1cyclictest0-21swapper/618:21:406
20988939987,1cyclictest0-21swapper/616:32:446
20988939987,1cyclictest0-21swapper/615:35:596
20988939986,2cyclictest2586599-21kworker/u16:3+flush-8:020:58:476
20988939986,2cyclictest2331104-21kworker/u16:1+flush-8:017:33:516
20988939986,1cyclictest0-21swapper/620:13:486
20988939986,1cyclictest0-21swapper/616:23:416
20988939983,4cyclictest2423387-21cat18:08:396
20988939983,4cyclictest2423387-21cat18:08:396
20988779987,1cyclictest0-21swapper/420:33:584
20988779987,1cyclictest0-21swapper/419:49:404
20988779987,1cyclictest0-21swapper/419:19:244
20988779987,1cyclictest0-21swapper/418:19:394
20988779987,1cyclictest0-21swapper/417:31:344
20988779987,1cyclictest0-21swapper/416:16:324
20988779983,4cyclictest2211194-21cut16:23:444
20988779983,4cyclictest2196537-21cut16:18:364
20988939976,1cyclictest0-21swapper/619:39:326
20988939974,2cyclictest2289758-21irqcore17:03:406
20988939973,3cyclictest2731777-21expr20:38:386
20988939973,3cyclictest2146584-21cut15:53:416
20988779976,1cyclictest0-21swapper/415:34:174
20988779975,1cyclictest0-21swapper/417:48:444
20988779975,1cyclictest0-21swapper/417:33:354
20988779973,3cyclictest2546915-21cat19:08:394
20988779973,3cyclictest2165340-21tr16:03:354
20988779973,3cyclictest2135731-21tr15:48:404
20988779973,3cyclictest2135731-21tr15:48:404
20988779972,4cyclictest2680691-21if_p4p120:13:414
20988779972,4cyclictest2672330-21cut20:08:414
20988779972,4cyclictest2582212-21cut19:23:474
20988779972,4cyclictest2392918-21sed17:53:444
20988779972,4cyclictest2289194-21expr17:03:364
20988779970,1cyclictest0-21swapper/420:53:374
20988999960,5cyclictest2099086-21cat15:33:237
20988939963,2cyclictest2247629-21cpuspeed16:43:376
20988779960,1cyclictest0-21swapper/420:43:404
20988939930,1cyclictest0-21swapper/615:33:236
20988779920,1cyclictest0-21swapper/415:33:234
20988829910,1cyclictest0-21swapper/515:33:235
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional