You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-20 - 01:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Fri Dec 19, 2025 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24970210289,9sleep20-21swapper/207:07:512
2502028979,6sleep00-21swapper/007:08:300
2504528557,10sleep30-21swapper/307:08:493
2500628567,14sleep10-21swapper/107:08:191
2523099220,15cyclictest0-21swapper/107:10:141
2523099211,17cyclictest8796-21/usr/sbin/munin12:10:271
2523099207,8cyclictest38150irq/131-snd_hda11:55:141
2522599206,8cyclictest16902-21idleruntime-cro11:20:000
25233991816,1cyclictest15967-21apache_processe12:25:112
2523099175,3cyclictest141rcu_preempt07:20:001
25230991614,1cyclictest0-21swapper/110:20:021
2523099160,1cyclictest0-21swapper/108:05:141
2523699157,1cyclictest141rcu_preempt07:20:003
25236991512,2cyclictest21805-21idleruntime-cro08:10:003
25230991511,3cyclictest2632-21cron10:50:001
25230991511,3cyclictest11242-21sed11:05:201
2522599159,4cyclictest18738-21ls11:20:280
25225991514,1cyclictest0-21swapper/009:50:120
25225991511,3cyclictest32445-21ls09:35:260
25225991511,2cyclictest29063-21wc10:35:250
25225991511,2cyclictest29063-21wc10:35:250
2523699142,11cyclictest19598-21apt-get10:20:023
25236991411,2cyclictest6321-21cron08:45:003
25236991411,2cyclictest6321-21cron08:45:003
2523099149,3cyclictest4281-21wc09:45:201
2523099142,7cyclictest38150irq/131-snd_hda12:35:121
2523099140,12cyclictest32219-21munin-run10:45:001
25225991413,1cyclictest0-21swapper/010:12:090
25225991411,2cyclictest18242-21cut10:15:170
25225991411,2cyclictest15214-21grep12:20:260
25225991410,3cyclictest31810-21grep10:40:270
25225991410,2cyclictest24702-21tr08:15:120
25225991410,2cyclictest23110-21wc08:10:210
2523699131,5cyclictest0-21swapper/308:50:143
25236991310,2cyclictest30751-21cat09:35:003
25236991310,2cyclictest18249-21cut10:15:173
25236991310,2cyclictest15419-21sed09:00:263
25236991310,2cyclictest1053-21grep08:30:253
25233991313,0cyclictest0-21swapper/212:30:132
25233991312,1cyclictest0-21swapper/210:45:142
25233991310,2cyclictest32618-21sleep07:24:592
25233991310,2cyclictest27589-21uniq11:40:172
25233991310,2cyclictest27231-21grep09:25:192
25233991310,2cyclictest25087-21grep09:20:262
25233991310,2cyclictest23241-21wc09:20:012
25233991310,2cyclictest11033-21cat11:05:182
2523099139,2cyclictest10430-21grep10:00:121
2523099136,4cyclictest0-21swapper/108:30:161
25230991312,1cyclictest0-21swapper/107:35:261
25230991311,1cyclictest2838-21apt-get07:30:121
25230991310,2cyclictest9925-21cat11:05:011
25230991310,2cyclictest921-21grep11:50:261
25230991310,2cyclictest7322-21head08:45:161
25230991310,2cyclictest31097-21tr09:35:151
25230991310,2cyclictest28207-21tail10:35:161
25230991310,2cyclictest28207-21tail10:35:161
2522599139,3cyclictest16224-21tr12:25:130
2522599139,3cyclictest12300-21cat11:10:010
2522599136,6cyclictest13296-21gltestperf07:50:170
25225991311,1cyclictest9586-21tune2fs08:50:170
25225991310,2cyclictest24465-21wc09:20:200
25225991310,2cyclictest14520-21cron11:15:000
25225991310,2cyclictest1380-21uname10:45:200
2523699129,2cyclictest7640-21tr07:40:153
2523699129,2cyclictest735-21grep07:25:153
2523699129,2cyclictest4762-21cat09:45:263
2523699129,2cyclictest3453-21grep08:35:273
2523699129,2cyclictest32191-21grep09:35:233
2523699129,2cyclictest32122-21grep07:20:243
2523699129,2cyclictest27847-21sleep07:15:003
2523699129,2cyclictest25327-21grep10:30:143
2523699129,2cyclictest23334-21wc08:10:223
2523699129,2cyclictest22224-21grep11:30:163
2523699129,2cyclictest19426-21cron08:05:003
2523699129,2cyclictest10244-21tr07:45:143
2523399129,2cyclictest4940-21latency_hist07:35:002
2523399129,2cyclictest4812-21grep12:00:172
2523399129,2cyclictest23018-21tr08:10:202
2523399129,2cyclictest21912-21ls12:35:212
2523399128,3cyclictest26618-21cron08:20:002
2523399124,3cyclictest225-21systemd-journal07:20:002
2523399121,9cyclictest0-21swapper/210:00:002
25233991210,1cyclictest30978-21cut09:35:152
25233991210,1cyclictest282982sleep208:20:232
25233991210,1cyclictest17331-21grep09:05:212
25233991210,1cyclictest15570-21cut11:15:162
2523399120,11cyclictest18418-21grep11:20:232
2523099129,2cyclictest31258-21sed07:20:151
2523099129,2cyclictest30042-21cut08:25:161
2523099129,2cyclictest19430-21cron08:05:001
2523099129,2cyclictest18518-21wc11:20:241
2523099129,2cyclictest14000-21tr09:00:151
2523099129,2cyclictest13433-21cut10:05:161
2523099128,3cyclictest10224-21wc08:50:231
2523099128,2cyclictest15179-21grep10:10:131
2523099127,2cyclictest2248-21taskset07:26:041
2523099124,3cyclictest9978-21apt-get07:45:131
2523099121,5cyclictest225-21systemd-journal11:29:101
2523099120,11cyclictest0-21swapper/110:25:381
2522599129,2cyclictest7863-21tr07:40:160
2522599129,2cyclictest7151-21cut12:05:170
2522599129,2cyclictest27973-21wc08:20:200
2522599129,2cyclictest2789-21wc08:35:190
2522599129,2cyclictest27439-21grep07:10:270
2522599129,2cyclictest24248-21cron11:35:000
2522599129,2cyclictest13968-21head10:05:210
2522599129,2cyclictest13875-21tr09:00:140
2522599129,2cyclictest12290-21cut12:15:190
2522599129,2cyclictest10436-21grep12:10:250
2522599128,3cyclictest9619-21grep09:55:250
2522599128,3cyclictest4445-21wc10:50:250
2522599124,4cyclictest225-21systemd-journal07:30:140
2523699119,1cyclictest14322-21grep12:20:173
2523699118,2cyclictest4448-21grep07:30:253
2523699118,2cyclictest30385-21cat11:45:193
2523699118,2cyclictest26948-21taskset11:40:073
2523699118,2cyclictest26821-21cat08:20:003
2523699118,2cyclictest21672-21sh11:30:003
2523699118,2cyclictest19224-21head09:10:173
2523699118,2cyclictest18233-21turbostat.cron09:10:003
2523699118,2cyclictest1503-21head10:45:223
2523699118,2cyclictest14436-21grep10:05:263
2523699118,2cyclictest1409-21cron11:55:003
2523699118,2cyclictest13648-21grep07:50:203
2523699118,2cyclictest13615-21cat09:00:003
2523399119,1cyclictest8568-21munin-run12:10:002
2523399119,1cyclictest31983-21grep11:50:142
2523399119,1cyclictest28738-21grep09:30:132
2523399119,1cyclictest15339-21grep09:00:262
2523399119,1cyclictest12433-21grep12:15:212
2523399118,2cyclictest9535-21grep12:10:162
2523399118,2cyclictest7054-21expr08:45:142
2523399118,2cyclictest3822-21grep09:45:172
2523399118,2cyclictest32148-21grep08:30:142
2523399118,2cyclictest26176-21expr07:10:152
2523399118,2cyclictest24251-21cron11:35:002
2523399118,2cyclictest1779-21wc07:25:232
2523399118,2cyclictest17071-21tr08:00:002
2523399118,2cyclictest14677-21perf07:55:002
2523399118,2cyclictest13130-21grep11:10:182
2523399118,2cyclictest12960-21grep10:05:142
2523099118,2cyclictest7677-21head12:05:221
2523099118,2cyclictest5566-21tr10:55:151
2523099118,2cyclictest4945-21head08:40:171
2523099118,2cyclictest4945-21head08:40:171
2523099118,2cyclictest31364-21chrt11:48:541
2523099118,2cyclictest27681-21grep11:40:181
2523099118,2cyclictest27182-21tr08:20:151
2523099118,2cyclictest26410-21chrt08:17:221
2523099118,2cyclictest26242-21cut09:25:141
2523099118,2cyclictest26117-21sed10:30:201
2523099118,2cyclictest23273-21ls08:10:221
2523099118,2cyclictest19145-21hostname12:30:181
2523099118,2cyclictest18152-21cut10:15:161
2523099118,2cyclictest17999-21taskset09:07:201
2523099118,2cyclictest14335-21grep12:20:171
2523099118,2cyclictest12523-21cut11:10:161
2523099117,3cyclictest14665-21cron07:55:001
2523099117,3cyclictest12037-21cut08:55:181
2523099117,2cyclictest5699-21cut09:50:141
2523099117,2cyclictest15989-21wc07:55:201
2522599119,1cyclictest20294-21tail11:25:190
2522599118,2cyclictest8692-21cron08:50:000
2522599118,2cyclictest7918-21grep11:00:150
2522599118,2cyclictest7071-21taskset10:56:100
2522599118,2cyclictest6298-21tr07:35:200
2522599118,2cyclictest2793-21head11:55:200
2522599118,2cyclictest27772-21tail09:25:260
2522599118,2cyclictest25283-21sort10:30:140
2522599118,2cyclictest23209-21turbostat09:20:000
2522599118,2cyclictest23184-21grep10:25:170
2522599118,2cyclictest21405-21sed08:05:260
2522599118,2cyclictest19059-21grep12:30:170
2522599118,2cyclictest18268-21sed09:10:000
2522599118,2cyclictest16647-21grep07:55:250
2522599118,2cyclictest12102-21sleep07:50:000
2522599117,3cyclictest11107-21grep10:00:150
2522599117,2cyclictest29529-21tr08:25:140
2522599116,3cyclictest20166-21grep10:20:140
2522599113,4cyclictest225-21systemd-journal08:45:000
2522599113,4cyclictest225-21systemd-journal08:45:000
2523699108,2cyclictest22241-21cat12:35:253
2523699108,2cyclictest17221-21proc_pri12:25:203
2523699108,1cyclictest83902sleep312:07:593
2523699108,1cyclictest8155-21awk09:55:133
2523699108,1cyclictest4661-21taskset10:50:393
2523699108,1cyclictest40902sleep312:00:133
2523699108,1cyclictest3532-21taskset11:56:483
2523699108,1cyclictest26133-21apt-get09:25:133
2523699108,1cyclictest14974-21taskset07:55:133
2523699108,1cyclictest12270-21taskset10:02:373
2523699108,1cyclictest11578-21expr12:15:143
2523699107,2cyclictest6652-21wc09:50:183
2523699107,2cyclictest30003-21find10:40:013
2523699107,2cyclictest22725-21grep09:15:263
2523699107,2cyclictest22467-21apache_processe10:25:123
2523699107,2cyclictest17201-21chrt11:20:153
2523699107,2cyclictest15820-21cat11:15:193
2523699107,2cyclictest12546-21grep11:10:163
2523699107,2cyclictest10982-21uname11:05:183
2523699102,4cyclictest24125-21cut09:20:173
2523399108,2cyclictest5862-21expr09:50:132
2523399108,2cyclictest5679-21expr07:35:152
2523399108,2cyclictest27852-21expr10:35:142
2523399108,2cyclictest27852-21expr10:35:142
2523399108,2cyclictest23941-21threads10:25:262
2523399108,2cyclictest10144-21open_files08:50:222
2523399108,1cyclictest5396-21hostname10:55:142
2523399108,1cyclictest29382-21cut08:25:142
2523399108,1cyclictest24933-21cat11:35:152
2523399108,1cyclictest18901-21expr09:10:152
2523399108,1cyclictest1327-21expr09:40:172
2523399108,1cyclictest11716-21which10:00:212
2523399108,1cyclictest10075-21wget07:45:142
2523399107,2cyclictest8367-21head07:40:192
2523399107,2cyclictest31623-21cat11:50:002
2523399107,2cyclictest29934-21touch10:40:012
2523399107,2cyclictest25721-21grep10:30:162
2523399107,2cyclictest2377-21df08:35:162
2523399107,2cyclictest20585-21df10:20:152
2523399107,2cyclictest17308-21basename08:00:112
2523399107,2cyclictest15345-21tr10:10:142
2523399107,2cyclictest13636-21tr12:20:142
2523399105,2cyclictest20938-21grep08:05:222
2523399100,9cyclictest19361-21taskset10:17:102
2523399100,8cyclictest8252-21diskstats11:00:172
2523099108,1cyclictest79282sleep109:55:061
2523099108,1cyclictest5175-21grep12:00:201
2523099108,1cyclictest4128-21grep10:50:211
2523099108,1cyclictest229132sleep109:16:401
2523099108,1cyclictest19641-21grep09:10:211
2523099108,1cyclictest179312sleep112:28:161
2523099107,2cyclictest9450-21chrt07:41:521
2523099107,2cyclictest25204-21cut11:35:161
2523099107,2cyclictest25174-21grep09:20:271
2523099107,2cyclictest1725-21chrt09:40:201
2523099107,2cyclictest16371-21cat11:15:251
2523099104,2cyclictest225-21systemd-journal09:35:001
2523099100,5cyclictest2432-21fschecks_count08:35:171
2522599108,1cyclictest32440-21seq07:22:590
2522599108,1cyclictest29723-21grep07:15:260
2522599108,1cyclictest1577-21perf08:35:000
2522599107,2cyclictest31250-21chrt11:47:240
2522599107,2cyclictest26533-21chrt11:38:030
2522599107,2cyclictest12683-21tr08:55:240
2522599106,3cyclictest28602-21grep09:30:120
2522599100,9cyclictest4269-21grep12:00:140
2522599100,9cyclictest27543-21sed11:40:160
252369998,1cyclictest16499-21chrt10:10:233
252369997,1cyclictest85652sleep308:48:253
252369997,1cyclictest8004-21expr11:00:153
252369997,1cyclictest6244-21sed10:55:193
252369997,1cyclictest5167-21taskset07:35:063
252369997,1cyclictest27674-21head08:20:183
252369997,1cyclictest27562-21wget10:35:133
252369997,1cyclictest27562-21wget10:35:133
252369997,1cyclictest24817-21cut11:35:143
252369996,2cyclictest852-21cut09:40:153
252369996,2cyclictest29308-21aten2_r7power_c08:25:133
252369996,2cyclictest19128-21grep12:30:173
252339998,1cyclictest11311-21taskset08:55:132
252339997,1cyclictest28632sleep210:50:052
252339997,1cyclictest2411-21grep11:55:172
252339997,1cyclictest19635-21wget11:25:162
252339996,2cyclictest4189-21sshd08:40:102
252339996,2cyclictest4189-21sshd08:40:102
252309997,1cyclictest22342-21egrep11:30:161
252309997,1cyclictest129662sleep112:15:461
252259997,1cyclictest4708-21sort09:45:260
252259997,1cyclictest31862-21cut11:50:140
252259997,1cyclictest20949-21tr12:35:140
252259997,1cyclictest19350-21if_enp2s009:10:180
252259997,1cyclictest18036-21head08:00:160
252259995,3cyclictest707-21tr07:25:140
2251290,1sleep00-21swapper/009:40:280
252369986,1cyclictest9986-21ntp_states12:10:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional