You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-25 - 06:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack1slot2s.osadl.org (updated Tue Nov 25, 2025 00:43:23)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2913501060,0irq/25-eth10-21swapper/318:36:153
2920324130,0sleep00-21swapper/018:37:450
2909623524,0sleep10-21swapper/118:36:301
2297623526,0sleep20-21swapper/218:34:352
294319976,0cyclictest16221-21kthreadcore20:04:482
8225060,0irq/24-0000:00:32104-21cat00:04:302
35160,0ktimersoftd/30-21swapper/321:46:253
339960,0migration/39412-21munin-node21:04:503
339960,0migration/3698-21sh23:22:443
339960,0migration/34677-21kthreadcore23:44:493
339960,0migration/327102-21sh21:37:343
339960,0migration/323139-21unixbench-2d18:54:543
339960,0migration/320716-21sh23:14:473
339960,0migration/318435-21sh21:30:413
339960,0migration/318266-21sh21:50:543
339960,0migration/31783-21runrttasks20:40:453
339960,0migration/31739-21munin-node23:24:433
294319960,0cyclictest0-21swapper/223:13:122
294259965,0cyclictest12453-21tune2fs20:49:440
28415060,0irq/27-eth0-tx-21198-21munin-node20:34:530
28415060,0irq/27-eth0-tx-0-21swapper/223:56:362
28415060,0irq/27-eth0-tx-0-21swapper/223:42:322
28415060,0irq/27-eth0-tx-0-21swapper/222:54:552
28415060,0irq/27-eth0-tx-0-21swapper/222:48:332
28415060,0irq/27-eth0-tx-0-21swapper/222:39:422
28415060,0irq/27-eth0-tx-0-21swapper/222:14:322
28415060,0irq/27-eth0-tx-0-21swapper/222:02:582
28415060,0irq/27-eth0-tx-0-21swapper/221:47:482
28415060,0irq/27-eth0-tx-0-21swapper/221:29:222
28415060,0irq/27-eth0-tx-0-21swapper/221:23:042
28415060,0irq/27-eth0-tx-0-21swapper/019:23:260
28415060,0irq/27-eth0-tx-0-21swapper/019:14:480
28405060,0irq/26-eth0-rx-25403-21cat23:39:303
28405060,0irq/26-eth0-rx-20294-21iostat_ios19:44:483
28405060,0irq/26-eth0-rx-1671-21sensors_volt21:19:513
249960,0watchdog/29315-21processes20:44:532
179960,0migration/14822-21gltestperf21:04:461
149960,0migration/03598-21sh21:04:060
149960,0migration/024389-21munin-run23:59:300
149960,0migration/014493-21kthreadcore22:09:490
8225050,0irq/24-0000:00:25784-21latency_hist23:19:302
8225050,0irq/24-0000:00:0-21swapper/220:34:312
339950,0migration/39017-21sh23:49:423
339950,0migration/34695-21fschecks_time21:24:453
339950,0migration/332375-21open_files20:14:523
339950,0migration/332103-21fschecks_count19:29:453
339950,0migration/331941-21kthreadcore20:59:493
339950,0migration/331901-21taskset00:03:583
339950,0migration/327614-21munin-node22:19:433
339950,0migration/326891-21munin-node22:59:453
339950,0migration/326630-21munin-node20:34:493
339950,0migration/321972-21kthreadcore19:19:493
339950,0migration/320558-21idleruntime21:54:453
339950,0migration/320483-21munin-node22:14:483
339950,0migration/318156-21taskset22:54:303
339950,0migration/311963-21kthreadcore23:29:483
339950,0migration/311065-21taskset20:48:143
339950,0migration/31016-21sendmail_mailst21:59:513
294319955,0cyclictest0-21swapper/221:09:072
294319950,0cyclictest0-21swapper/223:17:272
294319950,0cyclictest0-21swapper/222:27:192
294319950,0cyclictest0-21swapper/221:40:482
294319950,0cyclictest0-21swapper/218:55:542
294279955,0cyclictest0-21swapper/123:34:541
294279955,0cyclictest0-21swapper/120:54:051
294279950,0cyclictest0-21swapper/123:54:271
294259950,0cyclictest0-21swapper/021:46:200
294259950,0cyclictest0-21swapper/021:41:080
294259950,0cyclictest0-21swapper/021:35:560
294259950,0cyclictest0-21swapper/021:21:160
28415050,0irq/27-eth0-tx-27856-21ssh20:39:340
28415050,0irq/27-eth0-tx-19230-21ssh20:53:072
28415050,0irq/27-eth0-tx-15461-21rt-features23:29:532
28415050,0irq/27-eth0-tx-10441-21munin-node22:49:542
28415050,0irq/27-eth0-tx-0-21swapper/223:51:592
28415050,0irq/27-eth0-tx-0-21swapper/223:44:592
28415050,0irq/27-eth0-tx-0-21swapper/223:35:072
28415050,0irq/27-eth0-tx-0-21swapper/223:25:482
28415050,0irq/27-eth0-tx-0-21swapper/223:04:322
28415050,0irq/27-eth0-tx-0-21swapper/222:30:202
28415050,0irq/27-eth0-tx-0-21swapper/222:22:402
28415050,0irq/27-eth0-tx-0-21swapper/222:10:322
28415050,0irq/27-eth0-tx-0-21swapper/222:09:142
28415050,0irq/27-eth0-tx-0-21swapper/221:54:542
28415050,0irq/27-eth0-tx-0-21swapper/221:34:512
28415050,0irq/27-eth0-tx-0-21swapper/221:30:192
28415050,0irq/27-eth0-tx-0-21swapper/221:16:572
28415050,0irq/27-eth0-tx-0-21swapper/221:01:022
28415050,0irq/27-eth0-tx-0-21swapper/220:56:452
28415050,0irq/27-eth0-tx-0-21swapper/220:42:262
28415050,0irq/27-eth0-tx-0-21swapper/019:54:490
28415050,0irq/27-eth0-tx-0-21swapper/018:49:420
28415050,0irq/27-eth0-tx-0-21swapper/018:44:420
28405050,0irq/26-eth0-rx-7222-21ssh00:08:593
28405050,0irq/26-eth0-rx-31261-21tune2fs19:04:473
28405050,0irq/26-eth0-rx-19872-21irqcore22:34:463
28405050,0irq/26-eth0-rx-18668-21ssh21:10:373
28405050,0irq/26-eth0-rx-11510-21port_sshpriv19:34:513
28405050,0irq/26-eth0-rx-0-21swapper/323:58:243
28405050,0irq/26-eth0-rx-0-21swapper/323:12:413
28405050,0irq/26-eth0-rx-0-21swapper/322:47:083
28405050,0irq/26-eth0-rx-0-21swapper/322:32:483
28405050,0irq/26-eth0-rx-0-21swapper/322:05:093
28405050,0irq/26-eth0-rx-0-21swapper/321:16:143
28405050,0irq/26-eth0-rx-0-21swapper/320:50:293
28405050,0irq/26-eth0-rx-0-21swapper/318:59:483
259950,0migration/213540-21kthreadcore21:49:462
259950,0migration/211390-21dump-pmu-power19:14:262
249950,0watchdog/226804-21diskstats22:59:452
249950,0watchdog/215522-21iostat20:29:452
249950,0watchdog/20-21swapper/222:36:092
249950,0watchdog/20-21swapper/200:07:532
179950,0migration/18703-21kthreadcore21:44:491
179950,0migration/17485-21kthreadcore20:44:491
179950,0migration/126115-21sh22:38:321
179950,0migration/110463-21latency_hist22:29:301
169950,0watchdog/17767-21pmu-power23:04:531
169950,0watchdog/16699-21kthreadcore23:24:491
169950,0watchdog/130947-21kthreadcore20:14:491
169950,0watchdog/123470-21chrt23:56:331
169950,0watchdog/121300-21grep19:19:491
169950,0watchdog/10-21swapper/122:31:131
169950,0watchdog/10-21swapper/119:44:491
149950,0migration/0889-21kthreadcore18:39:490
149950,0migration/07970-21kthreadcore19:09:490
149950,0migration/06221-21kthreadcore22:24:480
149950,0migration/0547-21taskset20:16:010
149950,0migration/04143-21kthreadcore23:04:490
149950,0migration/03509-21kthreadcore20:19:470
149950,0migration/032207-21munin-node22:59:540
149950,0migration/030639-21switchtime19:24:530
149950,0migration/030226-21taskset19:02:440
149950,0migration/030152-21kthreadcore21:59:460
149950,0migration/029986-21kthreadcore23:39:480
149950,0migration/029118-21kthreadcore22:19:470
149950,0migration/027077-21ksmtuned21:16:470
149950,0migration/026163-21sh22:17:350
149950,0migration/023236-21kthreadcore22:54:490
149950,0migration/020945-21kthreadcore23:14:470
149950,0migration/019640-21sh21:13:240
149950,0migration/017452-21kthreadcore20:29:460
149950,0migration/015596-21kthreadcore19:39:480
149950,0migration/014020-21kthreadcore23:49:510
149950,0migration/013272-21kthreadcore21:49:460
149950,0migration/011672-21kthreadcore23:29:480
149950,0migration/010364-21latency_hist22:49:300
149950,0migration/010251-21taskset22:07:170
149950,0migration/010047-21sh20:45:360
339940,0migration/37281-21kthreadcore23:04:503
339940,0migration/37013-21kthreadcore19:09:493
339940,0migration/35046-21kthreadcore19:54:493
339940,0migration/330033-21kthreadcore21:39:473
339940,0migration/329332-21kthreadcore19:24:473
339940,0migration/323780-21latency22:54:493
339940,0migration/320992-21latency_hist20:34:313
339940,0migration/315808-21kthreadcore20:04:483
294319943,0cyclictest6445-21snmpd20:09:482
294319943,0cyclictest6445-21snmpd20:09:482
294319943,0cyclictest6445-21snmpd20:00:032
294319943,0cyclictest6445-21snmpd19:24:482
294319943,0cyclictest6445-21snmpd19:16:032
294319943,0cyclictest4835-21awk18:44:452
294319943,0cyclictest3425-21lldpd19:41:252
294319943,0cyclictest3425-21lldpd19:05:552
294319943,0cyclictest30476-21awk18:39:462
294319943,0cyclictest27901-21awk20:14:472
294279944,0cyclictest27633-21diskmemload23:11:251
294279944,0cyclictest0-21swapper/123:30:011
294279944,0cyclictest0-21swapper/119:29:451
294279943,0cyclictest728-21awk00:04:461
294279943,0cyclictest28556-21awk21:59:441
294279943,0cyclictest27405-21awk22:59:481
294279943,0cyclictest22050-21awk20:34:441
294279943,0cyclictest1970-21awk23:44:461
294279943,0cyclictest15406-21awk20:29:441
294279943,0cyclictest14291-21sadc20:29:311
294279943,0cyclictest1047-21awk19:54:461
294279943,0cyclictest10424-21sadc18:49:311
294279940,0cyclictest0-21swapper/122:20:291
294259944,0cyclictest0-21swapper/021:25:510
294259943,0cyclictest743-21awk00:04:460
294259943,0cyclictest3425-21lldpd22:46:150
294259943,0cyclictest3425-21lldpd20:04:560
294259943,0cyclictest26925-21awk23:19:450
294259943,0cyclictest21072-21awk20:09:450
294259943,0cyclictest21072-21awk20:09:450
294259943,0cyclictest10994-21awk23:09:450
294259940,0cyclictest7893-21munin-node20:24:540
28415040,0irq/27-eth0-tx-12261-21ssh21:09:402
28415040,0irq/27-eth0-tx-0-21swapper/019:29:430
28405040,0irq/26-eth0-rx-0-21swapper/323:39:473
28405040,0irq/26-eth0-rx-0-21swapper/322:40:333
28405040,0irq/26-eth0-rx-0-21swapper/322:27:363
28405040,0irq/26-eth0-rx-0-21swapper/322:11:343
28405040,0irq/26-eth0-rx-0-21swapper/320:54:363
28405040,0irq/26-eth0-rx-0-21swapper/319:14:473
259940,0migration/27925-21kthreadcore19:34:472
259940,0migration/228857-21kthreadcore18:59:502
249940,0watchdog/225651-21seq19:48:532
249940,0watchdog/211459-21idleruntime18:49:452
249940,0watchdog/20-21swapper/220:24:452
249940,0watchdog/20-21swapper/219:29:492
249940,0watchdog/20-21swapper/219:19:492
179940,0migration/1683-21kthreadcore19:04:501
179940,0migration/130665-21irqcore18:39:471
179940,0migration/130592-21kthreadcore23:19:481
179940,0migration/129027-21kthreadcore22:39:471
179940,0migration/128106-21latency_hist20:59:301
179940,0migration/126530-21sh21:56:531
179940,0migration/124936-21latency_hist19:24:311
179940,0migration/122374-21kthreadcore18:54:491
179940,0migration/119895-21kthreadcore23:14:471
179940,0migration/114910-21kthreadcore21:09:491
179940,0migration/113620-21kthreadcore22:49:471
179940,0migration/113075-21kthreadcore19:14:471
179940,0migration/112574-21latency_hist19:39:311
179940,0migration/112089-21munin-node21:29:411
179940,0migration/111878-21aten_r1power_po21:49:411
169940,0watchdog/127633-21diskmemload22:54:411
169940,0watchdog/127161-21kthreadcore18:59:491
169940,0watchdog/126916-21ssh21:37:011
169940,0watchdog/10-21swapper/123:59:571
169940,0watchdog/10-21swapper/122:44:491
169940,0watchdog/10-21swapper/121:20:411
169940,0watchdog/10-21swapper/120:58:571
169940,0watchdog/10-21swapper/120:40:371
169940,0watchdog/10-21swapper/119:09:491
149940,0migration/020163-21aten_r1power_po20:54:420
339930,0migration/315367-21kthreadcore18:49:483
294319932,0cyclictest6445-21snmpd20:19:482
294319932,0cyclictest1125-21awk19:54:462
294279933,0cyclictest2831-21kworker/1:223:39:401
294279933,0cyclictest12797-21kworker/1:122:04:581
294279933,0cyclictest11869-21kworker/1:020:19:361
294279933,0cyclictest11869-21kworker/1:020:04:401
294279933,0cyclictest11869-21kworker/1:019:59:381
294279933,0cyclictest11869-21kworker/1:019:49:401
294279933,0cyclictest11869-21kworker/1:019:34:401
294279933,0cyclictest11869-21kworker/1:018:49:341
294279932,0cyclictest4986-21awk21:24:471
294259932,0cyclictest6445-21snmpd19:34:380
294259932,0cyclictest4271-21grep23:44:490
294259932,0cyclictest3941-21sensors19:04:550
294259932,0cyclictest27731-21awk22:39:450
294259932,0cyclictest20661-21awk21:54:460
294259932,0cyclictest18571-21awk23:34:460
294259932,0cyclictest15479-21kthreadcore21:29:470
294259932,0cyclictest11623-21awk22:29:470
294259930,0cyclictest27504-21grep19:49:490
294259930,0cyclictest17287-21munin-node18:54:490
28415030,0irq/27-eth0-tx-0-21swapper/019:44:450
28405030,0irq/26-eth0-rx-0-21swapper/320:24:473
28405030,0irq/26-eth0-rx-0-21swapper/320:19:413
28405030,0irq/26-eth0-rx-0-21swapper/319:59:463
28405030,0irq/26-eth0-rx-0-21swapper/319:39:513
28405030,0irq/26-eth0-rx-0-21swapper/318:44:423
28405030,0irq/26-eth0-rx-0-21swapper/318:39:433
249930,0watchdog/227754-21kthreadcore19:49:492
179930,0migration/120301-21runrttasks21:14:411
169930,0watchdog/10-21swapper/122:14:451
169930,0watchdog/10-21swapper/122:09:491
169930,0watchdog/10-21swapper/121:39:411
169930,0watchdog/10-21swapper/120:09:331
169930,0watchdog/10-21swapper/120:09:331
294259921,0cyclictest27633-21diskmemload21:04:380
294259921,0cyclictest18608-21latency_hist22:34:300
294259921,0cyclictest0-21swapper/023:59:430
294259920,0cyclictest0-21swapper/023:24:320
294259920,0cyclictest0-21swapper/019:59:420
28405020,0irq/26-eth0-rx-0-21swapper/320:09:423
28405020,0irq/26-eth0-rx-0-21swapper/320:09:423
28405020,0irq/26-eth0-rx-0-21swapper/319:49:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional