You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-04 - 14:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #2

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack1slot2s.osadl.org (updated Sun Jan 04, 2026 12:43:22)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
304450710,0irq/25-eth10-21swapper/307:07:383
2248723625,0sleep10-21swapper/107:04:011
1644323624,0sleep00-21swapper/007:02:560
2264823526,0sleep20-21swapper/207:05:542
228919974,0cyclictest30562-21munin-run07:37:540
4160,0ktimersoftd/023064-21cut07:07:540
4160,0ktimersoftd/00-21swapper/009:54:430
339960,0migration/35100-21needreboot11:13:133
339960,0migration/332433-21sh11:49:533
339960,0migration/331506-21sh11:12:293
339960,0migration/32070-21sh09:54:323
339960,0migration/316521-21sh11:59:133
339960,0migration/311972-21timerwakeupswit09:18:173
339960,0migration/31024-21sh12:09:453
29715060,0irq/26-eth0-rx-986-21grep10:13:133
29715060,0irq/26-eth0-rx-15702-21iostat_ios09:03:113
249960,0watchdog/224749-21cat12:23:112
228929966,0cyclictest21130-21diskmemload10:04:011
228929960,0cyclictest0-21swapper/111:33:161
228919965,0cyclictest30518-21ssh11:10:040
149960,0migration/07929-21taskset11:18:000
149960,0migration/07929-21taskset11:18:000
149960,0migration/06823-21sh11:15:490
149960,0migration/05253-21runrttasks10:54:360
149960,0migration/031219-21sh10:33:110
149960,0migration/018373-21aten2.4-expect12:03:050
149960,0migration/012945-21sh10:40:230
149960,0migration/012378-21sh10:21:260
4150,0ktimersoftd/00-21swapper/008:38:060
35150,0ktimersoftd/30-21swapper/309:35:583
339950,0migration/37187-21taskset11:34:173
339950,0migration/36886-21taskset10:57:573
339950,0migration/36582-21fschecks_time10:38:083
339950,0migration/35144-21switchtime08:03:173
339950,0migration/33863-21munin-node07:38:143
339950,0migration/330237-21munin-node10:33:063
339950,0migration/330221-21fschecks_count09:13:083
339950,0migration/328019-21kthreadcore07:33:123
339950,0migration/32780-21aten_r1power_vo12:13:093
339950,0migration/327790-21taskset09:09:213
339950,0migration/327179-21sshd10:09:233
339950,0migration/326769-21kthreadcore07:58:113
339950,0migration/320670-21taskset10:43:593
339950,0migration/320236-21kthreadcore07:28:103
339950,0migration/319931-21fschecks_time12:23:063
339950,0migration/318839-21diskmemload12:22:223
339950,0migration/317774-21kthreadcore11:03:113
339950,0migration/317091-21taskset07:26:333
339950,0migration/312878-21fschecks_time10:03:073
339950,0migration/312775-21munin-node07:48:083
339950,0migration/31239-21latency_hist08:52:543
29725050,0irq/27-eth0-tx-8011-21munin-node08:58:162
29725050,0irq/27-eth0-tx-5977-21munin-node09:18:062
29725050,0irq/27-eth0-tx-31581-21ssh11:30:142
29725050,0irq/27-eth0-tx-26878-21ssh10:08:392
29725050,0irq/27-eth0-tx-26802-21ssh09:51:102
29725050,0irq/27-eth0-tx-2637-21munin-node12:13:172
29725050,0irq/27-eth0-tx-18425-21munin-node07:53:122
29725050,0irq/27-eth0-tx-18312-21munin-node12:03:102
29725050,0irq/27-eth0-tx-17251-21munin-node11:43:122
29725050,0irq/27-eth0-tx-13438-21ssh10:41:382
29725050,0irq/27-eth0-tx-12393-21mdstat10:58:142
29725050,0irq/27-eth0-tx-11772-21munin-node07:48:152
29725050,0irq/27-eth0-tx-0-21swapper/211:58:042
29725050,0irq/27-eth0-tx-0-21swapper/211:33:422
29725050,0irq/27-eth0-tx-0-21swapper/211:22:532
29725050,0irq/27-eth0-tx-0-21swapper/211:22:532
29725050,0irq/27-eth0-tx-0-21swapper/210:43:372
29715050,0irq/26-eth0-rx-3137-21ssh12:32:203
29715050,0irq/26-eth0-rx-25285-21smartctl09:48:173
29715050,0irq/26-eth0-rx-24927-21proc_pri08:18:153
29715050,0irq/26-eth0-rx-23648-21cut12:03:123
29715050,0irq/26-eth0-rx-1683-21grep10:53:113
29715050,0irq/26-eth0-rx-16143-21kthreadcore09:43:123
29715050,0irq/26-eth0-rx-14470-21munin-node11:38:133
29715050,0irq/26-eth0-rx-0-21swapper/311:56:253
29715050,0irq/26-eth0-rx-0-21swapper/311:45:273
29715050,0irq/26-eth0-rx-0-21swapper/311:28:043
29715050,0irq/26-eth0-rx-0-21swapper/311:23:053
29715050,0irq/26-eth0-rx-0-21swapper/310:48:133
29715050,0irq/26-eth0-rx-0-21swapper/310:32:033
29715050,0irq/26-eth0-rx-0-21swapper/310:23:103
29715050,0irq/26-eth0-rx-0-21swapper/309:39:243
29715050,0irq/26-eth0-rx-0-21swapper/309:29:263
29715050,0irq/26-eth0-rx-0-21swapper/309:23:563
29715050,0irq/26-eth0-rx-0-21swapper/308:43:093
259950,0migration/28133-21kthreadcore07:43:122
259950,0migration/27930-21kthreadcore07:18:132
259950,0migration/25240-21sh10:18:082
259950,0migration/232035-21kthreadcore10:33:112
259950,0migration/230266-21kthreadcore08:48:122
259950,0migration/229631-21latency_hist09:12:542
259950,0migration/227173-21kthreadcore11:48:112
259950,0migration/223921-21latency_hist07:32:542
259950,0migration/222583-21kthreadcore08:18:122
259950,0migration/21944-21missed_timers10:13:142
259950,0migration/215540-21munin-node09:23:112
259950,0migration/210140-21sh09:41:112
249950,0watchdog/24281-21grep09:58:082
249950,0watchdog/229040-21kthreadcore07:33:132
249950,0watchdog/227584-21if_eth112:08:082
228939950,0cyclictest0-21swapper/211:23:592
228939950,0cyclictest0-21swapper/209:33:562
228929955,0cyclictest21130-21diskmemload12:08:531
228929955,0cyclictest0-21swapper/112:13:091
228929950,0cyclictest0-21swapper/111:56:541
228929950,0cyclictest0-21swapper/110:49:071
228919955,0cyclictest0-21swapper/011:38:080
228919955,0cyclictest0-21swapper/011:25:030
228919955,0cyclictest0-21swapper/009:25:320
228919952,0cyclictest1277-21munin-run11:52:530
228919950,0cyclictest0-21swapper/012:34:150
228919950,0cyclictest0-21swapper/012:09:490
228919950,0cyclictest0-21swapper/009:10:430
19150,0ktimersoftd/10-21swapper/109:54:431
19150,0ktimersoftd/10-21swapper/109:42:441
179950,0migration/19997-21irqcore11:38:101
179950,0migration/16282-21cpuspeed_turbos09:18:061
179950,0migration/15687-21kthreadcore12:33:111
179950,0migration/129838-21kthreadcore11:48:111
179950,0migration/126853-21sh12:26:301
179950,0migration/12518-21kthreadcore08:03:121
179950,0migration/118781-21kthreadcore11:23:101
179950,0migration/112828-21munin-node08:33:171
179950,0migration/110590-21fschecks_time11:58:081
179950,0migration/110027-21kthreadcore11:18:111
179950,0migration/110027-21kthreadcore11:18:111
149950,0migration/06699-21kthreadcore07:18:130
149950,0migration/0505-21aten_r1power_cu11:33:040
149950,0migration/029625-21turbostat10:32:540
149950,0migration/029064-21aten2.4-expect10:13:060
149950,0migration/029061-21kernelversion12:28:120
149950,0migration/028790-21kthreadcore08:48:110
149950,0migration/02544-21taskset12:12:540
149950,0migration/025343-21kthreadcore08:43:130
149950,0migration/025200-21fschecks_time11:28:070
149950,0migration/021109-21kthreadcore08:18:110
149950,0migration/019848-21munin-node09:48:070
149950,0migration/017684-21cpuspeed_turbos07:28:050
149950,0migration/013037-21kthreadcore09:43:110
149950,0migration/012704-21sh09:19:270
149950,0migration/012668-21kthreadcore07:23:100
149950,0migration/011257-21cpuspeed_turbos12:18:050
149950,0migration/011228-21kthreadcore10:58:130
149950,0migration/01094-21kthreadcore09:13:120
339940,0migration/39771-21sendmail_mailtr07:18:183
339940,0migration/330315-21kthreadcore08:48:123
339940,0migration/323864-21sh07:53:143
339940,0migration/323427-21df07:08:073
339940,0migration/315932-21kthreadcore08:38:103
339940,0migration/31342-21kthreadcore07:13:133
339940,0migration/311746-21sensors_fan08:08:153
29725040,0irq/27-eth0-tx-27555-21ssh09:08:222
29725040,0irq/27-eth0-tx-24392-21ssh09:28:592
29725040,0irq/27-eth0-tx-0-21swapper/212:33:142
29725040,0irq/27-eth0-tx-0-21swapper/212:28:402
29725040,0irq/27-eth0-tx-0-21swapper/212:19:492
29725040,0irq/27-eth0-tx-0-21swapper/211:53:062
29725040,0irq/27-eth0-tx-0-21swapper/211:37:592
29725040,0irq/27-eth0-tx-0-21swapper/211:12:562
29725040,0irq/27-eth0-tx-0-21swapper/210:28:112
29725040,0irq/27-eth0-tx-0-21swapper/210:23:162
29725040,0irq/27-eth0-tx-0-21swapper/209:54:432
29725040,0irq/27-eth0-tx-0-21swapper/209:05:062
29725040,0irq/27-eth0-tx-0-21swapper/208:13:082
29725040,0irq/27-eth0-tx-0-21swapper/207:13:092
29715040,0irq/26-eth0-rx-0-21swapper/312:32:573
29715040,0irq/26-eth0-rx-0-21swapper/311:18:053
29715040,0irq/26-eth0-rx-0-21swapper/311:18:053
29715040,0irq/26-eth0-rx-0-21swapper/310:18:513
29715040,0irq/26-eth0-rx-0-21swapper/309:58:063
29715040,0irq/26-eth0-rx-0-21swapper/308:33:043
29715040,0irq/26-eth0-rx-0-21swapper/307:43:513
259940,0migration/232152-21runrttasks08:24:292
259940,0migration/225438-21kthreadcore07:08:122
259940,0migration/223733-21irqcore10:48:112
259940,0migration/215541-21kthreadcore10:03:112
259940,0migration/213123-21kthreadcore07:23:102
249940,0watchdog/232684-21munin-node08:28:162
249940,0watchdog/220305-21kthreadcore11:03:122
249940,0watchdog/20-21swapper/211:08:122
249940,0watchdog/20-21swapper/207:38:092
228939943,0cyclictest3556-21lldpd09:43:002
228939943,0cyclictest3556-21lldpd08:43:002
228939943,0cyclictest3556-21lldpd08:33:002
228939943,0cyclictest3514-21snmpd08:41:472
228939943,0cyclictest32637-21awk08:03:102
228939943,0cyclictest3241-21irqbalance08:53:122
228939943,0cyclictest31787-21awk10:53:092
228939943,0cyclictest25932-21awk07:58:092
228929944,0cyclictest0-21swapper/111:46:021
228929944,0cyclictest0-21swapper/107:48:171
228929943,0cyclictest4947-21awk07:18:101
228929943,0cyclictest398-21awk11:13:081
228929943,0cyclictest3556-21lldpd10:27:321
228929943,0cyclictest3556-21lldpd08:48:301
228929943,0cyclictest3556-21lldpd07:45:001
228929943,0cyclictest3556-21lldpd07:25:001
228929943,0cyclictest3514-21snmpd09:44:461
228929943,0cyclictest24967-21awk07:33:091
228929943,0cyclictest18352-21awk07:28:081
228929943,0cyclictest16262-21awk11:03:091
228929940,0cyclictest0-21swapper/109:09:191
228919944,0cyclictest0-21swapper/011:55:090
228919944,0cyclictest0-21swapper/011:07:230
228919943,0cyclictest8864-21awk08:58:100
228919943,0cyclictest3241-21irqbalance10:25:220
228919943,0cyclictest3241-21irqbalance10:08:520
228919943,0cyclictest3241-21irqbalance09:33:020
228919943,0cyclictest3241-21irqbalance09:28:120
228919943,0cyclictest10862-21awk11:58:090
179940,0migration/16113-21cpuspeed_turbos10:38:061
179940,0migration/14379-21fschecks_time09:58:081
179940,0migration/130041-21cpuspeed_turbos07:13:081
179940,0migration/127589-21kthreadcore07:58:111
179940,0migration/123068-21kthreadcore10:08:121
179940,0migration/122079-21fschecks_count10:28:091
179940,0migration/121340-21kthreadcore07:53:121
179940,0migration/121019-21kthreadcore08:18:111
179940,0migration/119491-21kthreadcore09:23:141
179940,0migration/118468-21kthreadcore08:38:111
179940,0migration/118450-21aten_r1power_vo12:03:051
179940,0migration/117793-21kthreadcore10:43:111
179940,0migration/11584-21df_abs08:53:061
179940,0migration/115366-21fschecks_time09:03:091
179940,0migration/11317-21kthreadcore10:33:121
179940,0migration/112390-21kthreadcore12:18:101
179940,0migration/111457-21needreboot08:08:121
169940,0watchdog/16100-21ssh10:56:401
169940,0watchdog/14887-21ssh10:18:041
169940,0watchdog/131442-21fschecks_time07:38:091
169940,0watchdog/129053-21cut09:33:121
169940,0watchdog/128170-21kthreadcore07:08:131
169940,0watchdog/127784-21kthreadcore11:28:121
169940,0watchdog/127712-21ssh12:28:031
169940,0watchdog/126965-21kthreadcore11:08:121
169940,0watchdog/123424-21kthreadcore09:48:121
169940,0watchdog/110313-21cut08:58:121
169940,0watchdog/10-21swapper/110:13:001
169940,0watchdog/10-21swapper/109:13:121
169940,0watchdog/10-21swapper/108:43:121
169940,0watchdog/10-21swapper/108:28:161
149940,0migration/04872-21irqcore09:58:110
149940,0migration/04366-21kthreadcore08:28:110
149940,0migration/032211-21kthreadcore07:13:130
149940,0migration/026583-21kthreadcore10:48:120
149940,0migration/025315-21timerandwakeup12:23:170
149940,0migration/019596-21kthreadcore11:43:110
149940,0migration/016256-21kthreadcore08:13:140
149940,0migration/013094-21sh10:03:080
29715030,0irq/26-eth0-rx-0-21swapper/308:30:023
29715030,0irq/26-eth0-rx-0-21swapper/308:26:143
29715030,0irq/26-eth0-rx-0-21swapper/308:13:073
228939930,0cyclictest0-21swapper/208:08:112
228929933,0cyclictest3556-21lldpd08:24:301
228929933,0cyclictest3556-21lldpd08:15:301
228929930,0cyclictest7655-21perl10:58:091
228919933,0cyclictest0-21swapper/010:44:220
228919932,0cyclictest5829-21sadc08:07:540
228919932,0cyclictest5805-21grep07:43:090
228919932,0cyclictest3514-21snmpd08:55:040
228919932,0cyclictest3241-21irqbalance07:38:130
228919932,0cyclictest17428-21sensors07:48:140
228919930,0cyclictest3061-21diskstats09:38:070
228919930,0cyclictest26638-21perl08:23:080
169930,0watchdog/10-21swapper/109:29:081
149930,0migration/031498-21runrttasks08:02:420
149930,0migration/020855-21runrttasks09:04:580
29715020,0irq/26-eth0-rx-0-21swapper/308:58:073
228919921,0cyclictest18677-21grep07:53:060
228919921,0cyclictest0-21swapper/008:33:050
228919921,0cyclictest0-21swapper/008:08:050
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional