You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-02 - 16:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sat May 02, 2026 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23884221699,97sleep20-21swapper/207:06:112
239772141106,25sleep30-21swapper/307:07:253
239592139102,26sleep00-21swapper/007:07:100
239902138102,25sleep10-21swapper/107:07:351
239202770,9sleep122-21ksoftirqd/109:42:111
17582740,2sleep20-21swapper/209:25:312
196882650,1sleep10-21swapper/109:39:461
61642630,2sleep30-21swapper/309:29:533
21382610,3sleep122-21ksoftirqd/110:14:211
206912580,1sleep320690-21ssh10:51:073
54512560,1sleep20-21swapper/211:27:592
165362550,10chrt34-21ksoftirqd/311:35:343
327572510,4sleep122-21ksoftirqd/112:11:561
169512510,1sleep10-21swapper/110:49:091
73142500,1sleep30-21swapper/312:16:593
211162490,1sleep321117-21sh12:03:473
90482470,1sleep00-21swapper/011:55:010
258892460,2sleep20-21swapper/212:06:472
258892460,2sleep20-21swapper/212:06:472
128572450,2sleep10-21swapper/110:21:311
209022440,1sleep20-21swapper/211:39:582
126282430,1sleep30-21swapper/312:20:333
184572420,1sleep20-21swapper/210:25:262
162532420,12sleep012-21ksoftirqd/012:24:110
243102410,1sleep011183-21diskmemload11:41:370
326262340,2chrt0-21swapper/012:35:220
253882320,2sleep00-21swapper/012:06:170
253882320,2sleep00-21swapper/012:06:170
144032320,2chrt0-21swapper/112:22:191
143712300,4sleep30-21swapper/309:35:193
39542290,2sleep20-21swapper/207:35:242
24453992822,4cyclictest12777-21idleruntime-cro11:10:002
76922270,3sleep3331rcuc/309:30:213
2445499273,18cyclictest75-21kswapd010:35:113
2445499273,18cyclictest75-21kswapd010:35:113
24453992712,9cyclictest53050irq/142-enp3s0-12:20:112
183892270,3chrt0-21swapper/009:38:290
306312260,1sleep20-21swapper/210:10:292
290792260,3sleep10-21swapper/112:09:591
290792260,3sleep10-21swapper/112:09:591
2445299261,1cyclictest0-21swapper/107:14:281
2445199269,5cyclictest12-21ksoftirqd/009:44:180
2445199268,3cyclictest141rcu_preempt12:15:160
2445199266,10cyclictest12-21ksoftirqd/011:09:210
147302260,3chrt14728-21ssh11:10:241
2445499251,23cyclictest0-21swapper/307:13:533
2445399259,11cyclictest53050irq/142-enp3s0-09:15:102
2445199259,8cyclictest141rcu_preempt09:33:100
2445199258,3cyclictest141rcu_preempt11:24:380
24451992517,6cyclictest32633-21ssh10:12:280
24451992511,5cyclictest12-21ksoftirqd/012:27:360
2445199250,24cyclictest0-21swapper/007:13:590
2445499242,16cyclictest23528-21ssh10:30:133
24453992413,8cyclictest27592-21df_inode09:45:142
2445299242,15cyclictest640-21snmpd08:50:111
24452992416,6cyclictest16308-21awk11:35:251
24451992412,10cyclictest27494-21sh09:20:520
149392240,1sleep20-21swapper/209:11:582
46902230,2chrt0-21swapper/311:03:553
274832230,2sleep10-21swapper/111:44:471
24454992317,4cyclictest34-21ksoftirqd/308:10:003
24453992321,1cyclictest0-21swapper/207:13:482
2445299231,16cyclictest3920-21ssh12:15:031
2445199239,5cyclictest12-21ksoftirqd/010:38:320
2445199239,5cyclictest12-21ksoftirqd/010:38:310
2445199239,3cyclictest141rcu_preempt11:54:350
2445199239,3cyclictest141rcu_preempt09:50:490
2445199239,3cyclictest141rcu_preempt09:50:490
2445199239,3cyclictest141rcu_preempt09:45:370
2445199236,2cyclictest141rcu_preempt11:25:220
2445199235,5cyclictest12-21ksoftirqd/011:30:400
2445199235,2cyclictest141rcu_preempt12:00:150
24451992318,3cyclictest18950irq/143-ahci[0011:05:010
24451992315,6cyclictest30853-21python312:10:230
24451992310,8cyclictest52750irq/139-enp3s0-10:20:110
24451992310,2cyclictest141rcu_preempt10:01:150
247092220,1sleep10-21swapper/111:18:221
2445499225,5cyclictest0-21swapper/308:20:243
24454992214,6cyclictest7715-21smartctl09:00:173
24454992214,6cyclictest20699-21ssh10:27:383
24454992214,6cyclictest1408-21ls07:30:193
24453992212,7cyclictest11240-21cat10:45:012
2445399221,17cyclictest28729-21kworker/u8:011:30:122
2445299225,14cyclictest28758-21grep11:45:161
24452992213,7cyclictest6155-21python308:55:231
24452992212,8cyclictest29761-21cut07:20:221
24452992210,10cyclictest29539-21ssh11:21:261
2445199226,4cyclictest12-21ksoftirqd/010:40:200
2445199223,10cyclictest12-21ksoftirqd/012:33:290
2445199221,7cyclictest14910-21users11:10:250
24451992211,8cyclictest17863-21ntpq10:25:220
24451992211,2cyclictest141rcu_preempt10:19:050
172462220,3chrt24449-21cyclictest10:49:262
2445499218,10cyclictest9512-21unixbench_multi11:30:243
2445499212,14cyclictest3240-21ssh10:15:133
2445499211,7cyclictest28191-21ls11:20:223
24454992112,7cyclictest23325-21fw_packets12:05:163
24454992112,7cyclictest23325-21fw_packets12:05:163
2445499211,11cyclictest13219-21ssh09:10:253
2445399212,15cyclictest11183-21diskmemload12:35:132
24453992113,6cyclictest29306-21perl08:35:142
24453992112,7cyclictest26951-21awk09:20:252
24453992111,8cyclictest13683-21tune2fs11:10:152
24452992114,4cyclictest3639-21tr11:50:211
24452992113,6cyclictest9044-21seq07:46:581
24452992113,6cyclictest20704-21seq12:26:401
24452992112,6cyclictest52950irq/141-enp3s0-09:30:221
24452992110,9cyclictest636-21runrttasks11:30:461
2445199215,4cyclictest12-21ksoftirqd/011:39:230
2445199215,4cyclictest12-21ksoftirqd/009:27:210
24451992114,4cyclictest8367-21/usr/sbin/munin09:55:130
24451992113,6cyclictest560-21dbus-daemon07:44:590
24451992113,6cyclictest15452-21seq08:01:370
24451992111,4cyclictest12-21ksoftirqd/011:45:180
2445199210,12cyclictest131rcuc/010:30:140
79342200,2chrt0-21swapper/311:54:083
305822200,2chrt0-21swapper/212:10:202
290462200,9sleep122-21ksoftirqd/110:34:121
2445499202,13cyclictest25355-21ssh09:20:143
2445499201,7cyclictest177612sh10:01:123
2445499201,6cyclictest0-21swapper/307:20:123
24454992012,6cyclictest278-21systemd-journal11:30:003
24454992012,6cyclictest26410-21cut11:20:003
24454992012,6cyclictest14595-21ssh10:23:163
24454992012,6cyclictest14517-21python311:10:223
24454992011,6cyclictest560-21dbus-daemon10:00:003
24454992011,6cyclictest30541-21users10:10:243
24454992010,7cyclictest560-21dbus-daemon09:00:003
2445399202,12cyclictest25031-21apt-get10:55:102
24453992012,6cyclictest31108-21cat07:25:142
24453992012,6cyclictest31108-21cat07:25:142
24453992011,7cyclictest15147-21date08:00:242
24453992011,7cyclictest10003-21cat09:05:192
24453992011,6cyclictest706-21cat08:45:102
24453992011,6cyclictest5599-21cat12:15:242
24453992011,4cyclictest21372-21switchtime11:15:242
24452992015,3cyclictest20900-21interrupts08:15:171
24452992012,5cyclictest52950irq/141-enp3s0-11:04:571
24452992011,6cyclictest278-21systemd-journal09:05:001
24452992010,7cyclictest17786-21netstat10:25:211
2445199207,3cyclictest141rcu_preempt10:54:370
2445199207,2cyclictest141rcu_preempt09:11:230
2445199206,9cyclictest131rcuc/010:05:360
2445199206,3cyclictest141rcu_preempt09:18:160
2445199204,7cyclictest12-21ksoftirqd/008:16:590
24451992013,5cyclictest12-21ksoftirqd/007:25:000
24451992012,6cyclictest3370-21hostname07:35:180
24451992012,6cyclictest278-21systemd-journal07:50:010
24451992012,6cyclictest20923-21cat11:15:190
87082190,2sleep10-21swapper/111:05:581
2445499199,6cyclictest52850irq/140-enp3s0-09:15:113
2445499192,5cyclictest31527-21apt-get12:35:133
2445499192,12cyclictest75-21kswapd010:40:093
24454991911,6cyclictest31624-21cat08:40:173
2445499191,15cyclictest22274-21ssh11:40:203
24454991910,7cyclictest31167-21awk12:10:253
24454991910,6cyclictest12351-21sed07:55:173
24453991914,3cyclictest6500-21sh11:52:422
24453991914,3cyclictest19569-21sh12:02:152
24453991913,4cyclictest28193-21sh10:09:482
24453991913,4cyclictest23591-21sh11:40:542
24453991913,4cyclictest21435-21sh09:40:182
24453991913,4cyclictest21266-21sh10:51:422
24453991912,5cyclictest9105-21cat09:55:152
24453991912,5cyclictest14589-21ntpq09:35:212
24453991911,6cyclictest6067-21cut08:55:222
24453991911,6cyclictest27637-21cat11:20:182
24453991911,6cyclictest21654-21cat10:05:002
24453991911,6cyclictest18881-21awk12:25:222
24453991911,6cyclictest10950-21awk11:55:252
24453991911,6cyclictest10844-21seq09:32:462
24453991910,7cyclictest4952-21users10:15:262
2445299198,8cyclictest18825-21diskstats09:15:131
2445299193,10cyclictest32351-21apt-get09:25:121
2445299192,12cyclictest15984-21apt-get08:05:161
24452991914,3cyclictest4233-21cron10:40:001
24452991914,3cyclictest4233-21cron10:40:001
24452991913,4cyclictest31484-21sh09:24:481
24452991913,4cyclictest11733-21cpuspeed_turbos09:10:141
24452991913,4cyclictest11294-21sh09:56:341
24452991912,5cyclictest11069-21seq07:50:311
24452991911,6cyclictest6818-21seq07:43:321
24452991911,6cyclictest3688-21tune2fs10:15:161
24452991911,6cyclictest27144-21grep08:30:161
24452991911,6cyclictest25269-21cat10:55:121
24452991911,6cyclictest15061-21sed08:00:241
24452991911,6cyclictest13267-21seq07:56:161
2445299191,15cyclictest52950irq/141-enp3s0-11:55:121
24452991911,5cyclictest2699-21cut07:35:011
24452991911,4cyclictest24781-21cpu12:30:131
24452991910,7cyclictest28447-21seq07:19:131
2445199198,3cyclictest141rcu_preempt10:45:500
2445199195,4cyclictest141rcu_preempt08:55:000
2445199194,4cyclictest12-21ksoftirqd/010:56:370
24451991913,4cyclictest9039-21sh09:05:000
24451991912,5cyclictest312-21seq08:43:470
24451991912,5cyclictest30865-21sed08:40:010
24451991912,5cyclictest2562-21seq08:49:370
24451991911,6cyclictest2456-21seq07:34:000
24451991911,6cyclictest16448-21fw_packets08:05:190
74132180,2chrt0-21swapper/307:45:123
70762180,2sleep10-21swapper/109:54:071
70762180,2sleep10-21swapper/109:54:071
4852180,1sleep20-21swapper/211:00:222
2445499189,6cyclictest25257-21grep08:25:193
2445499182,12cyclictest9395-21cpuspeed_turbos11:55:133
2445499182,12cyclictest6268-21ssh11:05:043
2445499182,12cyclictest29723-21ssh11:45:253
2445499182,10cyclictest9298-21apt-get09:05:123
2445499181,6cyclictest22426-21df_inode10:05:143
2445499181,6cyclictest11489-21apt-get10:45:133
2445499181,3cyclictest0-21swapper/310:55:163
24454991813,3cyclictest10307-21missed_timers07:50:183
24454991812,4cyclictest18468-21interrupts12:25:173
24454991811,5cyclictest4932-21cut07:40:013
24454991811,5cyclictest32031-21which07:25:233
24454991811,5cyclictest32031-21which07:25:233
24454991811,5cyclictest29266-21seq09:45:433
24454991811,5cyclictest20488-21expr08:15:133
24454991811,5cyclictest1434-21sed08:45:183
24454991811,4cyclictest11183-21diskmemload09:42:153
2445499181,12cyclictest24743-21ssh12:30:123
24454991810,6cyclictest4910-21cat08:55:003
24454991810,6cyclictest278-21systemd-journal08:34:593
2445399189,7cyclictest2540-21sed09:50:182
2445399189,7cyclictest2540-21sed09:50:182
2445399189,6cyclictest4917-21cat08:55:002
2445399188,7cyclictest7754-21cat07:45:152
2445399186,9cyclictest24022-21kworker/u8:011:45:122
2445399182,11cyclictest28800-21apt-get07:20:132
2445399182,11cyclictest24725-21apt-get12:30:122
2445399182,10cyclictest30292-21apt-get10:35:132
2445399182,10cyclictest30292-21apt-get10:35:132
24453991813,3cyclictest12621-21open_files07:55:212
24453991812,4cyclictest25684-21sh10:30:512
24453991811,5cyclictest835-21cat07:30:142
24453991811,5cyclictest7484-21df09:00:142
24453991811,5cyclictest16985-21which08:05:252
24453991811,5cyclictest10569-21expr10:20:152
2445399181,11cyclictest26636-21apt-get07:15:102
24453991810,6cyclictest24917-21forks08:25:152
24453991810,6cyclictest10108-21smartctl07:50:162
2445299189,7cyclictest2074-21cat08:45:241
2445299182,11cyclictest15224-21apt-get10:00:121
2445299181,6cyclictest211582ssh10:51:361
24452991813,3cyclictest31962-21processes07:25:221
24452991813,3cyclictest31962-21processes07:25:211
24452991813,3cyclictest11179-21sh09:10:001
24452991812,4cyclictest3151-21sh11:25:401
24452991811,5cyclictest28488-21date10:10:011
24452991811,5cyclictest16098-21cat12:00:121
24452991810,6cyclictest5541-21cut12:40:001
24452991810,6cyclictest24147-21seq08:23:091
2445199188,2cyclictest12-21ksoftirqd/008:20:150
24451991813,3cyclictest9272-21/usr/sbin/munin09:05:140
24451991813,3cyclictest28113-21unixbench_multi08:30:250
24451991812,4cyclictest30914-21acpi07:25:120
24451991812,4cyclictest30914-21acpi07:25:120
24451991811,5cyclictest19010-21ntpq08:10:210
24451991810,6cyclictest13658-21grep08:00:000
2445499178,7cyclictest18942-21ls08:10:193
24454991712,3cyclictest4357-21sh09:51:263
24454991712,3cyclictest4357-21sh09:51:263
24454991712,3cyclictest26611-21/usr/sbin/munin07:15:173
24454991710,5cyclictest7133-21date07:45:013
24454991710,5cyclictest30264-21grep08:35:253
24454991710,5cyclictest14781-21awk08:00:203
2445399179,6cyclictest20330-21ls08:15:122
2445399178,7cyclictest23342-21ntp_states08:20:212
2445399172,10cyclictest18118-21apt-get08:10:142
24453991712,3cyclictest27263-21hddtemp_smartct08:30:162
24453991711,4cyclictest6037-21pmu-power07:40:232
24452991712,3cyclictest27772-21grep09:45:161
2445299171,14cyclictest5822-21ssh10:40:201
24452991710,5cyclictest19471-21cat08:10:261
2445199179,6cyclictest6278-21grep08:55:240
2445199179,6cyclictest10680-21tr07:50:210
24451991710,5cyclictest28395-21seq07:18:330
2445399168,6cyclictest32025-21ps08:40:222
2445299169,5cyclictest32202-21sed08:40:241
2445299162,9cyclictest2801-21apt-get07:35:121
2445299162,9cyclictest24567-21apt-get08:25:131
2445299162,10cyclictest28845-21apt-get08:35:131
2445199168,6cyclictest24698-21expr08:25:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional