You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-08 - 10:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Tue Sep 08, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5429210863,33sleep00-21swapper/019:08:180
528728760,9sleep30-21swapper/319:06:293
522428370,9sleep20-21swapper/219:05:392
553427767,6sleep10-21swapper/119:09:391
280652580,0sleep10-21swapper/119:55:151
565499240,2cyclictest0-21swapper/300:20:193
565099237,15cyclictest0-21swapper/222:01:242
564499225,15cyclictest31295-21diskmemload23:30:130
564499225,15cyclictest31295-21diskmemload23:30:120
5644992218,3cyclictest7622-21ssh00:08:430
565099217,13cyclictest56550irq/124-enp0s3120:31:192
565099210,20cyclictest56550irq/124-enp0s3120:22:232
5646992119,2cyclictest0-21swapper/122:42:221
565499204,9cyclictest0-21swapper/321:00:163
5650992017,3cyclictest0-21swapper/221:13:132
5650992016,4cyclictest0-21swapper/222:30:332
564699201,2cyclictest24571-21ssh23:59:491
564699200,1cyclictest0-21swapper/123:54:171
564499208,6cyclictest9625-21perf20:25:000
5654991916,2cyclictest0-21swapper/300:07:173
565099190,19cyclictest0-21swapper/221:26:442
5654991815,2cyclictest18355-21apt-get22:00:113
565499180,18cyclictest0-21swapper/323:57:533
565099181,16cyclictest21954-21apt-get23:00:132
5646991817,1cyclictest0-21swapper/122:08:001
5646991817,1cyclictest0-21swapper/100:01:411
5646991811,7cyclictest0-21swapper/122:49:321
564699181,15cyclictest0-21swapper/122:25:281
5650991716,1cyclictest0-21swapper/222:42:182
5650991714,3cyclictest22830-21kworker/u8:021:49:432
564699174,5cyclictest0-21swapper/120:20:121
5646991715,1cyclictest0-21swapper/123:45:211
564699171,15cyclictest31295-21diskmemload23:03:021
564699170,11cyclictest0-21swapper/123:05:141
564499170,16cyclictest0-21swapper/023:25:430
565499169,3cyclictest0-21swapper/322:32:443
565499161,1cyclictest0-21swapper/321:55:163
565099160,15cyclictest56550irq/124-enp0s3120:02:592
564699162,14cyclictest3658-21ssh22:10:201
5646991615,1cyclictest0-21swapper/119:35:141
5646991613,2cyclictest32492-21sh20:05:001
5644991613,1cyclictest0-21swapper/021:42:020
564499160,15cyclictest0-21swapper/021:19:190
564499160,15cyclictest0-21swapper/000:11:140
565099154,2cyclictest141rcu_preempt22:09:422
5650991512,3cyclictest0-21swapper/221:22:132
565099150,14cyclictest0-21swapper/223:15:142
5646991513,2cyclictest0-21swapper/122:51:461
564699150,14cyclictest0-21swapper/100:30:251
564499154,6cyclictest15210-21munin-run19:30:000
564499153,11cyclictest2191-21dump-pmu-power22:10:000
564499151,7cyclictest26273-21idleruntime-cro22:05:000
5644991510,3cyclictest27505-21cron19:55:000
564499150,3cyclictest0-21swapper/020:15:150
564499150,14cyclictest0-21swapper/022:52:130
565499145,8cyclictest31295-21diskmemload22:10:003
565499141,0cyclictest0-21swapper/321:51:333
565499140,13cyclictest3868-21sh23:08:343
565099143,1cyclictest141rcu_preempt00:25:232
5650991413,1cyclictest55550irq/126-enp2s0-00:38:492
5650991411,2cyclictest2140-21turbostat00:24:592
5650991410,3cyclictest3299-21ls22:30:012
5650991410,2cyclictest9905-21cut20:25:132
565099140,1cyclictest0-21swapper/221:35:452
565099140,14cyclictest0-21swapper/200:08:182
564699141,1cyclictest0-21swapper/121:10:131
564499142,2cyclictest141rcu_preempt21:58:340
5644991411,2cyclictest11073-21apache222:35:130
5644991410,3cyclictest25855-21sort20:55:240
5644991410,2cyclictest19547-21cut22:20:130
564499140,3cyclictest0-21swapper/022:15:310
564499140,1cyclictest0-21swapper/023:40:040
564499140,1cyclictest0-21swapper/022:33:480
565499139,4cyclictest0-21swapper/321:42:023
565499137,4cyclictest0-21swapper/300:12:043
565499134,5cyclictest39-21ksoftirqd/321:35:003
565499131,9cyclictest31295-21diskmemload22:23:283
5654991310,2cyclictest30639-21cut20:00:143
5654991310,2cyclictest18121-21grep21:20:283
5654991310,2cyclictest17035-21wc19:30:283
565499130,13cyclictest0-21swapper/323:54:403
565499130,12cyclictest0-21swapper/320:15:153
565099139,4cyclictest0-21swapper/223:10:432
565099139,3cyclictest5983-21grep19:10:152
565099138,4cyclictest4346-21kworker/u8:023:49:062
565099137,6cyclictest32-21ksoftirqd/222:47:492
565099136,4cyclictest0-21swapper/219:30:002
5650991310,2cyclictest9021-21head19:15:162
5650991310,2cyclictest26165-21sed19:50:182
5650991310,2cyclictest23651-21grep20:50:252
5650991310,2cyclictest18082-21wc22:00:002
5650991310,2cyclictest15898-21grep23:35:132
5650991310,2cyclictest13101-21head00:30:292
5650991310,2cyclictest11336-21grep19:20:162
5646991312,1cyclictest0-21swapper/122:15:161
5646991310,2cyclictest7386-21basename23:30:141
5646991310,2cyclictest7386-21basename23:30:131
5646991310,2cyclictest4327-21grep20:10:261
5646991310,2cyclictest27493-21grep00:20:171
5646991310,2cyclictest22659-21ssh23:20:001
5646991310,2cyclictest16779-21cron20:40:001
564699130,12cyclictest32473-21cat21:30:001
564699130,10cyclictest0-21swapper/122:37:491
5644991310,2cyclictest3501-21sed21:50:250
5644991310,2cyclictest32357-21cron21:30:000
5644991310,2cyclictest31643-21wc20:00:210
5644991310,2cyclictest30883-21sed23:05:160
5644991310,2cyclictest2810-21basename20:10:140
5644991310,2cyclictest2244-21ssh21:30:300
5644991310,2cyclictest12659-21ssh00:30:260
564499130,12cyclictest13559-21wc20:30:220
565499129,2cyclictest29431-21cut22:25:233
565499129,2cyclictest26078-21cut00:00:173
565499129,2cyclictest21099-21grep20:45:253
565499129,2cyclictest19079-21ssh22:20:003
565499129,2cyclictest16613-21cut23:35:163
565499129,2cyclictest15197-21dump-pmu-power19:30:003
565499129,2cyclictest13921-21grep20:30:253
565499129,2cyclictest10816-21ssh00:30:033
565499128,3cyclictest9467-21ssh22:14:123
565499128,3cyclictest7311-21sh23:49:273
565499128,3cyclictest28383-21taskset22:44:573
565499128,3cyclictest26732-21ssh23:40:513
565499128,3cyclictest22861-21rm23:00:173
565499128,3cyclictest17599-21ssh23:16:093
565499128,3cyclictest13543-21ssh23:14:313
565499123,4cyclictest0-21swapper/320:20:123
565499122,8cyclictest537-21snmpd23:20:323
5654991210,1cyclictest26097-21grep20:55:253
5654991210,1cyclictest25889-21tr21:45:153
565499120,11cyclictest8312-21aten2_r7power_e19:15:133
565499120,11cyclictest10519-21sh00:30:003
565499120,11cyclictest0-21swapper/323:32:173
565499120,11cyclictest0-21swapper/323:32:163
565499120,10cyclictest0-21swapper/322:47:183
565099129,3cyclictest0-21swapper/222:22:332
565099129,3cyclictest0-21swapper/221:32:132
565099129,2cyclictest5820-21grep20:15:192
565099129,2cyclictest4124-21head22:10:242
565099129,2cyclictest29963-21sed21:05:192
565099129,2cyclictest29022-21cat19:55:212
565099129,2cyclictest1842-21grep21:50:132
565099129,2cyclictest18172-21ls19:35:152
565099129,2cyclictest15437-21wc23:35:002
565099129,2cyclictest15437-21wc23:35:002
565099129,2cyclictest10855-21grep00:10:202
565099124,6cyclictest0-21swapper/222:16:242
565099123,5cyclictest225-21systemd-journal21:04:592
565099120,12cyclictest0-21swapper/222:38:102
564699129,2cyclictest8621-21head21:15:181
564699129,2cyclictest7121-21grep23:10:221
564699129,2cyclictest38150irq/131-snd_hda20:15:161
564699129,2cyclictest31507-21date21:10:001
564699129,2cyclictest27197-21ssh21:45:231
564699129,2cyclictest22134-21ssh23:38:491
564699129,2cyclictest21471-21ssh22:01:161
564699129,2cyclictest21172-21grep22:20:251
564699129,2cyclictest11738-21grep00:10:291
564699127,1cyclictest18934-21df_abs00:15:171
564699123,2cyclictest27484-21perf00:40:001
564699122,9cyclictest3066-21ssh23:27:041
564699122,7cyclictest17888-21ls22:00:001
564699121,10cyclictest19498-21sh21:21:291
5646991210,1cyclictest25613-21tr23:40:231
5646991210,1cyclictest22979-21ssh23:20:101
5646991210,1cyclictest14957-21grep22:55:221
564699120,2cyclictest0-21swapper/121:36:181
564699120,11cyclictest0-21swapper/120:55:121
564499129,2cyclictest9656-21tail21:35:180
564499129,2cyclictest8018-21apt23:50:000
564499129,2cyclictest32566-21cut21:10:170
564499129,2cyclictest29295-21wc22:25:230
564499129,2cyclictest28064-21wc21:00:230
564499129,2cyclictest27499-21tail00:20:170
564499129,2cyclictest20215-21tail20:45:160
564499129,2cyclictest17028-21sh23:35:180
564499129,2cyclictest16555-21cut19:30:210
564499129,2cyclictest15909-21grep20:35:230
564499127,4cyclictest21419-21latency_hist23:00:000
564499124,7cyclictest2475-21perf20:10:010
564499124,7cyclictest2475-21perf20:10:000
564499120,0cyclictest0-21swapper/023:51:390
565499119,2cyclictest7545-21tail19:10:283
565499118,2cyclictest9836-21chrt20:25:103
565499118,2cyclictest8586-21ssh21:35:043
565499118,2cyclictest5329-21tr22:50:193
565499118,2cyclictest4554-21chrt20:11:393
565499118,2cyclictest32620-21expr21:10:173
565499118,2cyclictest28465-21sh21:26:583
565499118,2cyclictest23642-21grep20:50:243
565499118,2cyclictest23172-21expr19:45:153
565499118,2cyclictest21524-21taskset19:40:213
565499118,2cyclictest19861-21ssh00:35:143
565499118,2cyclictest198112sleep319:38:253
565499118,2cyclictest18393-21grep00:15:143
565499118,2cyclictest18211-21tr20:40:203
565499118,2cyclictest15552-21cat20:35:203
565499118,2cyclictest14536-21sh22:35:423
565499118,2cyclictest13816-21tr22:55:163
565499118,2cyclictest126512sleep319:22:553
565499118,2cyclictest10368-21sh21:15:553
565499117,3cyclictest32765-21apache_volume20:05:143
565499117,3cyclictest32765-21apache_volume20:05:133
565499117,3cyclictest31231-21chrt23:25:053
565099119,1cyclictest18106-21sed21:40:172
565099118,2cyclictest9628-21if_enp2s023:50:232
565099118,2cyclictest7185-21ssh22:50:392
565099118,2cyclictest31475-21tr23:25:142
565099118,2cyclictest2944-21tr20:10:142
565099118,2cyclictest28716-21ssh23:23:192
565099118,2cyclictest25361-21ssh00:19:232
565099118,2cyclictest25083-21which23:40:182
565099118,2cyclictest2463-21grep20:10:012
565099118,2cyclictest2463-21grep20:10:002
565099118,2cyclictest21881-21sed19:40:242
565099118,2cyclictest19971-21ssh23:56:142
565099118,2cyclictest17836-21sed20:40:172
565099118,2cyclictest13408-21ssh22:55:092
565099118,2cyclictest10587-21ssh21:16:062
565099113,7cyclictest30350-21apt-get23:05:142
565099110,10cyclictest25252-21sh00:00:082
564699119,2cyclictest9719-21cat19:15:241
564699119,1cyclictest824-21grep20:05:161
564699119,1cyclictest824-21grep20:05:161
564699119,1cyclictest4494-21uniq22:30:171
564699119,1cyclictest400-21ssh21:30:131
564699119,1cyclictest29112sleep100:05:221
564699119,1cyclictest27098-21grep19:50:271
564699119,1cyclictest214712sleep120:49:071
564699119,1cyclictest18095-21missed_timers20:40:201
564699119,1cyclictest13940-21grep20:30:251
564699119,1cyclictest12820-21sh19:25:011
564699118,2cyclictest9327-21ssh21:54:561
564699118,2cyclictest6683-21ssh00:27:011
564699118,2cyclictest27573-21sed21:00:191
564699118,2cyclictest17581-21grep21:40:161
564499119,1cyclictest17100-21head21:20:190
564499118,2cyclictest4045-21grep00:25:220
564499118,2cyclictest31670-21ssh22:45:590
564499118,2cyclictest3064-21cstates22:10:170
564499118,2cyclictest25941-21grep21:45:150
564499118,2cyclictest25106-21ssh00:00:020
564499118,2cyclictest24498-21ssh23:00:520
564499118,2cyclictest21926-21head19:40:240
564499118,2cyclictest21400-21tail22:40:170
564499118,2cyclictest20462-21grep00:35:170
564499118,2cyclictest20210-21grep00:15:270
564499118,2cyclictest19704-21ssh23:17:480
564499118,2cyclictest17911-21grep19:35:140
564499118,2cyclictest17695-21cut20:40:180
564499118,2cyclictest16696-21ssh23:55:010
564499118,2cyclictest11448-21grep19:20:160
565499108,1cyclictest300482sleep319:59:303
565499108,1cyclictest25396-21bc19:50:123
565499107,2cyclictest30631-21sed21:05:273
565099108,1cyclictest17749-21cut19:35:012
565099107,2cyclictest25891-21ls20:55:242
565099107,2cyclictest19785-21cat20:45:142
565099103,6cyclictest14674-21apache_accesses20:35:132
564699109,1cyclictest14240-21ntpq19:25:201
564699108,2cyclictest23954-21taskset20:52:471
564699108,2cyclictest11991-21taskset20:29:341
564699108,1cyclictest22742-21cat19:45:001
564699108,1cyclictest20193-21seq19:40:071
564699108,1cyclictest173882sleep119:32:471
564699100,10cyclictest7014-21ntpq19:10:221
564499108,1cyclictest6167-21ssh23:10:110
564499108,1cyclictest6107-21grep19:10:160
564499107,2cyclictest30088-21taskset19:59:590
564499107,2cyclictest22978-21taskset23:20:100
564499107,2cyclictest22919-21wget19:45:150
564499107,2cyclictest22797-21grep20:50:150
564499100,9cyclictest0-21swapper/021:05:190
56509991,7cyclictest56550irq/124-enp0s3119:45:132
56449997,1cyclictest9870-21apache_volume20:25:120
56449997,1cyclictest82252sleep019:15:100
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional