You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-29 - 12:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Wed Apr 29, 2026 00:45:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
45112185171,8sleep10-21swapper/119:07:431
831921600,1sleep30-21swapper/321:20:463
45542141106,24sleep20-21swapper/219:08:182
27832139102,26sleep30-21swapper/319:05:053
43882137100,26sleep00-21swapper/019:06:060
74072650,2sleep30-21swapper/322:09:063
307102650,2sleep10-21swapper/122:25:331
65072600,4sleep06510-21wc20:25:220
12532600,2sleep20-21swapper/223:15:562
298842550,2sleep00-21swapper/022:25:200
182832540,2sleep30-21swapper/323:53:133
297942530,1sleep30-21swapper/322:25:203
264892530,1sleep10-21swapper/123:10:371
106152530,2sleep10-21swapper/121:23:051
8832510,1sleep0131rcuc/021:40:140
198022500,1sleep30-21swapper/323:30:163
300432480,5sleep2271rcuc/222:25:222
251082470,1sleep00-21swapper/019:55:160
197732470,1sleep10-21swapper/122:18:041
8392440,1sleep10-21swapper/123:15:301
4963994011,17cyclictest24738-21diskmemload21:49:113
112402400,1sleep10-21swapper/123:00:131
4963993911,13cyclictest24738-21diskmemload22:35:433
252382390,2sleep10-21swapper/123:34:481
496399371,21cyclictest0-21swapper/322:17:353
496399370,23cyclictest13486-21ssh23:01:003
65762340,2chrt0-21swapper/223:20:162
496399348,20cyclictest8759-21ssh22:10:123
4963993313,18cyclictest16573-21basename22:40:133
496399327,22cyclictest620-21cut23:40:173
496399322,23cyclictest27436-21apt-get00:00:123
496399318,20cyclictest7310-21expr19:15:133
496399317,21cyclictest20318-21expr19:45:123
496399312,25cyclictest75-21kswapd021:50:133
496399310,4cyclictest65412chrt22:55:453
198382310,2chrt0-21swapper/021:54:300
496399308,19cyclictest8792-21seq23:21:483
496399308,19cyclictest21384-21vmstat21:30:273
496399300,8cyclictest18151-21ssh21:28:573
496399300,6cyclictest8920-21ssh22:34:293
496399300,6cyclictest1876-21cut00:05:133
496399300,5cyclictest26715-21ssh22:23:173
496399300,4cyclictest23764-21sh21:56:293
496399300,4cyclictest23764-21sh21:56:293
496399300,27cyclictest29203-21cut00:25:193
312712300,2sleep30-21swapper/323:39:083
496399298,18cyclictest12556-21cpuspeed_turbos23:25:143
496399297,19cyclictest4245-21sed22:55:003
496399297,19cyclictest10471-21wc00:35:193
496399296,20cyclictest26763-21sed21:10:273
496399296,20cyclictest26763-21sed21:10:263
496399294,22cyclictest8828-21switchtime20:30:233
496399291,6cyclictest15013-21/usr/sbin/munin00:15:183
496399291,24cyclictest6487-21proc_pri20:25:223
4963992911,16cyclictest11499-21sh00:11:523
4963992911,16cyclictest11499-21sh00:11:513
496399290,7cyclictest3194-21ssh23:17:533
496399290,3cyclictest21206-21ssh23:07:023
496399290,2cyclictest85312chrt00:34:213
204322290,2sleep00-21swapper/022:42:340
496399283,22cyclictest23446-21sort22:45:113
496399283,20cyclictest0-21swapper/321:40:133
496399282,22cyclictest980-21timerandwakeup21:15:253
496399281,23cyclictest0-21swapper/321:35:123
496399280,3cyclictest6705-21taskset23:45:013
4962992821,5cyclictest28-21ksoftirqd/222:50:002
324512280,2chrt0-21swapper/122:50:581
126992280,13sleep012-21ksoftirqd/022:36:320
496399270,4cyclictest27463-21sed20:00:173
496399270,3cyclictest24421-21ssh23:57:253
496399270,3cyclictest24187-21timerwakeupswit21:05:253
496399270,2cyclictest174832sleep319:35:493
496099271,24cyclictest0-21swapper/019:13:070
196532270,2chrt0-21swapper/000:18:210
496399268,16cyclictest31150-21ssh22:02:093
496399267,16cyclictest0-21swapper/320:05:003
496399265,18cyclictest15212-21cut20:45:233
496399264,19cyclictest22479-21cpuspeed_turbos19:50:153
496399262,18cyclictest11657-21apt-get19:25:133
496399261,9cyclictest11974-21/usr/sbin/munin20:40:143
4963992614,10cyclictest20229-21sendmail20:59:593
4963992611,12cyclictest16176-21/usr/sbin/munin20:50:233
4963992610,13cyclictest2168-21switchtime20:15:233
496399260,3cyclictest24738-21diskmemload23:14:213
4962992611,12cyclictest14103-21apt-get21:50:122
496199261,24cyclictest0-21swapper/119:12:581
85182250,2sleep00-21swapper/020:30:190
496399255,17cyclictest31784-21cut20:10:163
496399254,18cyclictest3241-21grep20:20:133
496399254,18cyclictest3241-21grep20:20:133
496399251,9cyclictest22571-21cat00:20:183
496399251,8cyclictest25628-21/usr/sbin/munin19:55:223
496399251,4cyclictest10796-21sed19:20:253
496399251,23cyclictest0-21swapper/319:12:383
496399250,3cyclictest20820-21df21:00:133
496299251,23cyclictest0-21swapper/219:13:082
271762250,2sleep10-21swapper/122:23:451
101572250,5sleep012-21ksoftirqd/022:10:250
6772240,10sleep012-21ksoftirqd/000:04:250
50442240,3sleep25047-21apt-get20:25:002
50442240,3sleep25047-21apt-get20:25:002
4963992410,11cyclictest0-21swapper/319:40:113
496299243,3cyclictest141rcu_preempt23:05:222
4962992415,7cyclictest26677-21df21:35:132
4961992413,7cyclictest7147-21apt-get19:15:111
496199240,7cyclictest9143-21fschecks_count22:10:151
255802240,2sleep00-21swapper/021:10:160
255802240,2sleep00-21swapper/021:10:160
496399231,7cyclictest10430-21uname20:35:183
496399230,7cyclictest14772-21/usr/sbin/munin19:30:213
4962992314,6cyclictest19247-21sensors_temp19:40:232
4962992311,7cyclictest53050irq/142-enp3s0-21:10:132
4962992311,7cyclictest53050irq/142-enp3s0-21:10:122
4961992314,6cyclictest31932-21apt-get21:15:131
4961992313,6cyclictest584-21entropy23:40:161
496199231,10cyclictest151972sh23:50:241
93812220,2sleep10-21swapper/123:46:121
496299229,10cyclictest11168-21mailstats22:35:232
4962992214,6cyclictest15033-21ls19:30:242
4962992213,7cyclictest25302-21memory19:55:192
4962992213,7cyclictest21061-21hostname21:00:162
496199222,4cyclictest10503-21missed_timers00:35:191
4961992215,5cyclictest4148-21sh22:05:491
4961992215,4cyclictest5532-21sh22:55:211
496099226,13cyclictest550-21systemd-logind19:50:200
215022220,2sleep00-21swapper/021:30:350
496299219,6cyclictest13368-21forks21:25:162
496299218,11cyclictest28-21ksoftirqd/222:35:002
496299215,4cyclictest28-21ksoftirqd/221:55:442
496299215,4cyclictest28-21ksoftirqd/221:55:442
496299213,13cyclictest11203-21ssh23:00:132
496299212,14cyclictest2933-21ssh00:30:142
496299211,7cyclictest12757-21seq22:12:442
4962992113,6cyclictest830-21sensors_temp21:15:242
4962992113,6cyclictest12877-21pluginstate20:40:222
4962992111,8cyclictest309-21ssh00:28:052
4961992113,6cyclictest23057-21seq21:55:471
4961992113,6cyclictest23057-21seq21:55:471
4961992113,6cyclictest2051-21sendmail_mailqu20:15:211
4961992113,6cyclictest19209-21cron21:30:011
4961992112,7cyclictest560-21dbus-daemon21:55:001
4961992112,7cyclictest26458-21cron20:00:001
4961992112,7cyclictest16259-21df19:35:151
4961992112,6cyclictest29121-21threads22:00:241
496099217,3cyclictest235622sh21:56:180
496099217,3cyclictest235622sh21:56:170
496099212,6cyclictest636-21runrttasks22:20:100
496099212,14cyclictest32643-21ssh23:40:130
496099211,5cyclictest7231-21cstates23:45:140
4960992113,6cyclictest23635-21sed21:05:190
4960992112,6cyclictest3888-21cut00:30:210
4960992112,6cyclictest30991-21apt-key23:15:000
496299209,8cyclictest28-21ksoftirqd/223:56:372
496299209,3cyclictest141rcu_preempt22:59:242
496299209,3cyclictest141rcu_preempt00:04:392
496299206,3cyclictest141rcu_preempt23:42:292
496299206,11cyclictest15582-21perl00:15:162
4962992013,5cyclictest13783-21sensors23:25:242
4962992013,4cyclictest53050irq/142-enp3s0-23:50:112
4962992013,4cyclictest28-21ksoftirqd/219:50:182
4962992012,6cyclictest31808-21smartctl20:10:172
4962992012,6cyclictest27068-21seq20:00:142
4962992011,7cyclictest18805-21fw_packets20:55:172
496199205,6cyclictest22-21ksoftirqd/100:31:261
496199203,12cyclictest1732-21rm00:05:131
496199202,13cyclictest5489-21apt-get20:25:141
496199201,9cyclictest23493-21taskset22:45:121
4961992013,5cyclictest26822-21uniq21:35:141
4961992013,4cyclictest11153-21sh21:47:291
4961992012,6cyclictest8449-21date20:30:181
4961992012,6cyclictest27196-21fw_packets20:00:151
4961992012,6cyclictest20431-21cat21:00:001
4961992012,6cyclictest17927-21sensors_temp22:40:241
4961992012,6cyclictest14829-21cut00:15:001
4961992012,6cyclictest11994-21diskstats19:25:171
4961992012,5cyclictest636-21runrttasks00:03:501
496199201,17cyclictest31487-21dump-pmu-power21:15:001
496199201,17cyclictest31487-21dump-pmu-power21:15:001
4961992011,6cyclictest278-21systemd-journal23:30:001
4961992010,7cyclictest560-21dbus-daemon00:25:001
496099204,3cyclictest24738-21diskmemload23:19:430
496099204,3cyclictest24738-21diskmemload22:32:460
496099203,13cyclictest6258-21ssh21:20:130
496099201,5cyclictest21950-21/usr/sbin/munin23:55:220
4960992011,6cyclictest11279-21sort23:00:130
4962991914,3cyclictest28878-21processes22:00:212
4962991913,4cyclictest7617-21sh23:45:172
4962991913,4cyclictest3177-21sh21:41:132
4962991913,4cyclictest12496-21sh21:48:512
4962991912,5cyclictest9600-21apt-get00:35:112
4962991912,5cyclictest17038-21systemctl19:35:232
4962991911,6cyclictest8926-21sed20:30:252
4962991911,6cyclictest31890-21awk22:50:282
4962991911,6cyclictest23661-21ls22:20:252
4962991911,6cyclictest17432-21date20:50:232
4962991911,6cyclictest14282-21expr20:45:152
4962991911,6cyclictest10470-21sed20:35:172
496299191,15cyclictest24738-21diskmemload21:23:212
4962991910,6cyclictest29374-21cat20:05:132
496199199,6cyclictest18093-21apt-get19:40:121
496199198,4cyclictest24738-21diskmemload23:36:321
496199192,11cyclictest9309-21apt-get19:20:121
496199191,3cyclictest141rcu_preempt00:12:251
4961991913,4cyclictest13806-21cron20:45:011
4961991912,5cyclictest14823-21memory20:45:191
4961991912,4cyclictest24738-21diskmemload00:28:211
4961991912,4cyclictest17801-21sh23:05:061
4961991911,6cyclictest21750-21seq19:45:251
4961991911,6cyclictest10101-21cat22:35:131
4961991911,5cyclictest4482-21cat20:20:251
4961991911,5cyclictest4482-21cat20:20:251
496099193,3cyclictest141rcu_preempt00:05:250
4960991913,3cyclictest9841-21cpuspeed_turbos20:35:120
4960991912,5cyclictest9828-21sed00:10:260
4960991912,5cyclictest9828-21sed00:10:250
4960991911,6cyclictest16779-21grep19:35:200
4960991911,4cyclictest24738-21diskmemload23:30:240
4960991910,6cyclictest18441-21grep23:05:160
496299189,7cyclictest22131-21awk19:50:012
496299187,3cyclictest28-21ksoftirqd/222:40:172
496299186,3cyclictest141rcu_preempt00:10:142
496299186,3cyclictest141rcu_preempt00:10:132
496299183,7cyclictest141rcu_preempt00:05:132
496299181,8cyclictest28326-21ssh23:36:102
4962991813,3cyclictest6129-21sh22:07:492
4962991813,3cyclictest25117-21sh00:22:072
4962991813,3cyclictest19487-21sh21:30:052
4962991812,4cyclictest24121-21timerandwakeup21:05:252
4962991812,4cyclictest17807-21sh22:16:042
4962991811,5cyclictest24971-21ssh23:10:152
4962991811,5cyclictest19613-21tune2fs23:30:162
4962991811,5cyclictest12395-21sed19:25:192
4962991810,6cyclictest7141-21basename19:15:112
496199188,3cyclictest24738-21diskmemload23:56:251
496199182,4cyclictest141rcu_preempt21:30:141
4961991813,3cyclictest7527-21sh23:20:291
4961991812,4cyclictest2000-21mailstats21:40:251
4961991811,5cyclictest3377-21cat22:30:151
4961991811,5cyclictest31230-21basename20:10:111
4961991811,5cyclictest22846-21apt-get21:05:131
4961991811,5cyclictest17216-21ps20:50:211
4961991811,5cyclictest16581-21cat00:15:271
4961991811,5cyclictest13961-21cat19:30:131
4961991811,5cyclictest10958-21cat20:35:241
496099187,8cyclictest560-21dbus-daemon23:30:000
496099186,10cyclictest0-21swapper/022:15:160
4960991813,3cyclictest636-21runrttasks22:05:210
4960991813,3cyclictest4438-21/usr/sbin/munin22:55:250
4960991813,3cyclictest30679-21sh23:38:310
4960991813,3cyclictest27657-21/usr/sbin/munin22:00:220
4960991813,3cyclictest20233-21sh20:59:590
4960991813,3cyclictest19074-21pmu-power19:40:220
4960991813,3cyclictest15693-21latency_hist19:35:000
4960991812,4cyclictest31903-21/usr/sbin/munin21:15:160
4960991812,4cyclictest27772-21sh00:24:470
4960991812,4cyclictest24738-21diskmemload22:49:200
4960991811,5cyclictest3526-21cut20:20:160
4960991811,5cyclictest3526-21cut20:20:150
496099181,14cyclictest32427-21grep20:10:230
4960991810,5cyclictest52750irq/139-enp3s0-21:25:160
496299179,6cyclictest6438-21pluginstate20:25:212
496299178,7cyclictest9578-21ls19:20:132
496199179,6cyclictest29460-21forks20:05:151
4961991710,5cyclictest20768-21expr21:00:131
496199170,15cyclictest24294-21dump-pmu-power19:55:001
496099178,6cyclictest6136-21fschecks_count23:20:130
496099172,11cyclictest7379-21ssh21:45:120
496099172,10cyclictest28659-21apt-get20:05:030
4960991712,3cyclictest9283-21/usr/sbin/munin19:20:170
4960991712,3cyclictest30192-21/usr/sbin/munin22:50:210
4960991712,3cyclictest26294-21/usr/sbin/munin21:35:240
4960991712,3cyclictest24738-21diskmemload00:25:070
4960991712,3cyclictest11631-21/usr/sbin/munin19:25:230
4960991712,3cyclictest10336-21irqstats00:35:180
4960991711,5cyclictest12-21ksoftirqd/020:05:010
4960991710,4cyclictest15018-21rt-features23:50:210
496299169,5cyclictest2053-21sendmail_mailqu20:15:212
496099168,5cyclictest21716-21grep19:45:240
496099162,9cyclictest964-21apt-get20:15:100
496099162,8cyclictest20497-21apt-get21:00:120
4960991611,3cyclictest16758-21hddtemp_smartct20:50:160
4960991610,4cyclictest8102-21pmu-power19:15:210
4960991610,3cyclictest13817-21idleruntime-cro20:45:000
4960991510,3cyclictest14237-21cstates20:45:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional