You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-16 - 05:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue Jun 16, 2026 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8648221998,96sleep00-21swapper/019:05:440
8648221998,96sleep00-21swapper/019:05:440
87092190151,32sleep20-21swapper/219:06:312
87092190151,32sleep20-21swapper/219:06:302
8792217397,24sleep30-21swapper/319:07:353
8792217397,24sleep30-21swapper/319:07:353
87132140104,25sleep10-21swapper/119:06:341
87132140104,25sleep10-21swapper/119:06:331
153072590,1sleep00-21swapper/022:35:260
253252570,1chrt0-21swapper/022:43:480
55452550,2sleep20-21swapper/200:05:222
218142540,2sleep0131rcuc/023:30:000
98762530,2sleep30-21swapper/321:44:013
246072530,2chrt0-21swapper/121:55:091
127032530,2sleep30-21swapper/300:10:243
165532510,2sleep30-21swapper/322:12:413
288982490,2sleep10-21swapper/121:10:121
254362490,2sleep30-21swapper/322:19:553
202542450,2sleep10-21swapper/100:15:551
137542430,10sleep334-21ksoftirqd/300:35:243
311832420,1sleep30-21swapper/323:35:563
114002410,1sleep10-21swapper/121:20:231
61412380,3sleep122-21ksoftirqd/122:29:421
4232320,3sleep30-21swapper/323:13:453
252442310,3sleep20-21swapper/221:55:142
168452310,2sleep00-21swapper/022:12:590
109282310,2sleep10-21swapper/100:34:011
67852290,3chrt0-21swapper/322:05:193
188912290,2sleep30-21swapper/323:50:583
9453992815,9cyclictest15926-21df23:25:142
945099271,25cyclictest0-21swapper/119:11:381
226132260,1sleep30-21swapper/321:29:273
168192260,6sleep1211rcuc/123:25:231
945399253,8cyclictest31561-21apt-get22:00:102
945399251,23cyclictest0-21swapper/219:11:422
945399250,2cyclictest141rcu_preempt00:14:432
9454992414,7cyclictest30664-21df21:35:143
945499240,23cyclictest0-21swapper/319:11:333
9453992418,4cyclictest28-21ksoftirqd/222:30:002
9453992413,4cyclictest28-21ksoftirqd/220:45:212
9450992414,8cyclictest24809-21ssh00:20:121
945099241,11cyclictest23180-21perf21:30:001
9449992422,1cyclictest0-21swapper/019:11:380
944999241,9cyclictest1979450ssh21:50:310
9449992414,7cyclictest24845-21df19:45:130
944999241,10cyclictest77852sh00:07:020
143702240,2sleep00-21swapper/021:46:460
945499233,12cyclictest18011-21ssh00:15:103
945499231,9cyclictest29212rm00:27:393
945399231,3cyclictest141rcu_preempt21:25:002
9453992313,7cyclictest315-21sed00:25:222
945099232,14cyclictest21227-21ssh23:05:131
9450992318,3cyclictest211rcuc/100:00:001
945099231,16cyclictest4318-21ssh00:05:031
9449992313,8cyclictest27484-21ls19:50:180
84322230,3sleep0560-21dbus-daemon23:20:010
945499224,13cyclictest8855-21sh23:20:103
945499222,14cyclictest28516-21diskmemload22:00:043
9454992211,7cyclictest9012-21tr22:55:243
945399224,2cyclictest141rcu_preempt22:18:042
945399222,17cyclictest23870-21head19:40:232
9453992215,5cyclictest1726-21cron23:15:002
9453992213,6cyclictest7452-21sed22:05:252
945399220,12cyclictest0-21swapper/223:40:272
945099222,13cyclictest1136-21sh22:50:101
9450992212,6cyclictest4692-21cut21:15:221
9450992212,6cyclictest20226-21apt-get19:35:131
945099221,15cyclictest31419-21sh22:00:031
945099221,10cyclictest12570-21chrt23:46:221
9449992213,7cyclictest9798-21df_inode20:25:140
61422220,2sleep00-21swapper/022:53:430
296062220,2sleep10-21swapper/119:55:201
945499211,4cyclictest0-21swapper/300:08:223
9454992113,6cyclictest7004-21expr00:30:223
945399213,14cyclictest23287-21ntpq23:30:222
945399213,13cyclictest135591ssh00:35:222
9453992113,5cyclictest636-21runrttasks22:56:392
9453992113,5cyclictest278-21systemd-journal20:10:002
9453992110,9cyclictest560-21dbus-daemon19:30:002
9453992110,3cyclictest28-21ksoftirqd/221:12:322
945099212,14cyclictest18050-21taskset20:45:111
945099212,13cyclictest11311-21apt-get21:45:131
9450992112,7cyclictest2786-21cron23:39:591
944999219,4cyclictest84132rm21:42:320
9449992113,6cyclictest29604-21basename00:00:000
9449992113,6cyclictest17535-21seq20:41:490
9449992113,6cyclictest15945-21apt19:25:110
9449992113,6cyclictest12549-21cat00:35:130
9449992113,6cyclictest11341-21apt00:10:110
945499202,13cyclictest28516-21diskmemload22:30:123
9454992013,5cyclictest3925-21interrupts23:40:153
9454992012,6cyclictest23559-21ls20:55:233
9454992011,6cyclictest9457-21cat20:25:123
945499201,15cyclictest28516-21diskmemload22:41:293
945399209,9cyclictest27853-21ssh22:20:352
945399209,9cyclictest15967-21ssh22:36:042
945399209,9cyclictest14404-21ssh22:10:312
945399208,2cyclictest28-21ksoftirqd/200:00:012
945399203,5cyclictest28-21ksoftirqd/221:50:172
9453992016,2cyclictest18950irq/143-ahci[0020:29:592
9453992014,5cyclictest5866-21ssh22:53:272
9453992013,4cyclictest2757-21sed23:15:152
9453992012,6cyclictest7265-21date20:20:012
9453992012,6cyclictest3449-21awk21:15:122
9453992012,6cyclictest27825-21cat22:45:192
9453992012,6cyclictest22236-21ntp_kernel_pll_23:05:222
9453992011,8cyclictest9446-21cyclictest21:36:312
9453992010,8cyclictest16953-21ssh23:01:102
945099203,11cyclictest30316-21ssh21:35:121
945099203,11cyclictest2163-21taskset23:15:111
9450992014,4cyclictest22-21ksoftirqd/100:40:011
9450992013,5cyclictest17578-21grep23:50:191
9450992013,4cyclictest14501-21sh23:00:101
9450992012,6cyclictest18612-21smartctl21:50:161
9450992012,5cyclictest25174-21sed21:30:241
9450992011,6cyclictest21737-21grep22:40:261
9450992011,5cyclictest22-21ksoftirqd/100:05:001
9450992010,7cyclictest32230-21basename22:25:001
944999209,6cyclictest53050irq/142-enp3s0-21:10:120
944999209,5cyclictest20638-21seq00:16:180
944999204,15cyclictest30930-21ssh22:23:430
944999202,12cyclictest3204-21apt-get23:40:110
9449992015,3cyclictest27424-21sh23:10:010
9449992013,5cyclictest636-21runrttasks00:21:380
9449992013,5cyclictest1427-21ls20:05:200
9449992012,6cyclictest8334-21tune2fs22:55:170
9449992012,6cyclictest25634-21seq21:30:440
9449992012,6cyclictest22618-21cat23:30:160
9449992012,6cyclictest15791-21cut20:40:010
9449992012,6cyclictest11913-21users21:20:260
9449992012,6cyclictest11209-21readlink23:45:230
9449992012,5cyclictest11896-21cron00:35:010
944999200,18cyclictest3456-21fschecks_time20:10:160
24572200,2sleep00-21swapper/022:02:000
121122200,1sleep00-21swapper/022:09:570
945499199,7cyclictest560-21dbus-daemon23:05:253
945499199,7cyclictest14408-21cat22:35:193
945499192,12cyclictest23312-21apt-get23:55:123
945499191,9cyclictest23609-21ssh21:30:133
9454991914,3cyclictest21642-21unixbench_singl20:50:243
9454991912,4cyclictest28508-21idleruntime-cro21:10:003
9454991911,6cyclictest28900-21find21:10:113
9454991911,6cyclictest20130-21cat19:35:013
9454991911,6cyclictest18109-21cat20:45:113
9454991911,6cyclictest15906-21awk20:40:113
9454991911,6cyclictest1434-21ls22:50:133
9454991911,3cyclictest3044-21missed_timers23:15:193
9454991910,6cyclictest3916-21ntp_kernel_pll_20:10:213
945399199,8cyclictest28-21ksoftirqd/219:20:222
945399199,6cyclictest52950irq/141-enp3s0-22:30:132
945399199,6cyclictest52950irq/141-enp3s0-00:15:122
945399198,2cyclictest28-21ksoftirqd/220:35:262
945399198,2cyclictest28-21ksoftirqd/200:34:312
945399198,2cyclictest28-21ksoftirqd/200:04:152
945399197,3cyclictest141rcu_preempt20:30:162
945399195,2cyclictest28-21ksoftirqd/221:47:072
945399192,3cyclictest141rcu_preempt21:00:112
945399192,12cyclictest30957-21apt-get20:00:132
945399192,11cyclictest23586-21apt-get21:30:112
9453991914,3cyclictest7438-21sh21:41:322
9453991914,3cyclictest4545-21vmstat20:10:282
9453991914,3cyclictest17754-21missed_timers21:25:202
9453991913,4cyclictest14711-21sh23:48:312
9453991912,5cyclictest28-21ksoftirqd/219:40:002
9453991911,6cyclictest30353-21seq19:55:542
9453991911,6cyclictest23613-21cat20:55:242
9453991911,6cyclictest11712-21cat19:15:132
945399190,5cyclictest25223-21fschecks_time00:20:152
945099197,8cyclictest13609-21ssh22:35:121
945099197,3cyclictest1735-21cron23:15:001
945099192,12cyclictest28516-21diskmemload22:55:131
9450991913,4cyclictest28516-21diskmemload23:24:231
9450991911,6cyclictest5386-21date00:30:011
9450991911,6cyclictest26620-21apt-get21:05:111
9450991911,6cyclictest13601-21ntpq22:10:201
9450991911,5cyclictest22-21ksoftirqd/122:49:591
9450991910,7cyclictest12227-21memory00:10:191
944999192,13cyclictest3481-21ssh21:15:120
944999191,8cyclictest28523-21perf21:10:000
9449991913,4cyclictest4362-21sh22:27:520
9449991913,4cyclictest21185-21sh22:15:360
9449991913,4cyclictest20769-21sh23:52:520
9449991913,4cyclictest20493-21sh21:27:180
9449991913,4cyclictest17883-21sh23:02:050
9449991911,6cyclictest23424-21readlink20:55:220
9449991911,6cyclictest12440-21seq22:34:170
9449991910,7cyclictest12241-21seq20:30:170
9449991910,6cyclictest14153-21cat19:20:140
9449991910,6cyclictest12771-21which19:15:220
9449991910,5cyclictest53050irq/142-enp3s0-21:55:100
282722190,1sleep30-21swapper/322:45:263
280172190,6sleep028021-21cpuspeed_turbos23:10:130
945499182,4cyclictest141rcu_preempt00:00:133
945499182,11cyclictest28799-21apt-get19:55:133
945499181,7cyclictest1844-21grep22:25:263
945499181,3cyclictest21149-21ntp_states19:35:203
9454991813,3cyclictest30957-21latency_hist22:00:003
9454991813,3cyclictest0-21swapper/321:54:153
9454991812,4cyclictest27537-21sh00:21:303
9454991812,4cyclictest21261-21sh23:29:273
9454991812,4cyclictest17012-21processes19:25:223
9454991812,4cyclictest14650-21tr23:00:123
9454991811,5cyclictest52850irq/140-enp3s0-22:20:593
9454991811,5cyclictest14071-21expr19:20:143
9454991811,5cyclictest11829-21cat19:15:133
9454991811,5cyclictest10150-21/usr/sbin/munin21:20:153
9454991811,4cyclictest28516-21diskmemload21:16:493
9454991810,6cyclictest23818-21awk19:40:223
9454991810,6cyclictest10613-21cat23:45:163
9454991810,5cyclictest27360-21cat19:50:173
945399188,2cyclictest28-21ksoftirqd/220:50:222
945399186,3cyclictest141rcu_preempt23:35:182
945399182,11cyclictest24523-21apt-get19:45:122
9453991811,2cyclictest16586-21/usr/sbin/munin23:50:232
9453991810,6cyclictest26717-21wc19:50:132
9453991810,6cyclictest21374-21uname22:40:242
9453991810,5cyclictest8834-21acpi23:20:102
945099189,7cyclictest22513-21cut19:40:111
945099187,8cyclictest15925-21find19:25:111
945099182,13cyclictest0-21swapper/122:15:131
945099181,7cyclictest6658-21grep20:15:251
945099181,7cyclictest25997-21head19:45:251
945099181,4cyclictest0-21swapper/123:34:421
9450991811,5cyclictest4831-21sed23:40:241
9450991811,5cyclictest20883-21sed20:50:181
9450991811,5cyclictest19444-21sed19:30:231
9450991811,5cyclictest10364-21seq21:44:291
9450991810,6cyclictest4334-21date20:10:251
9450991810,6cyclictest23642-21grep20:55:241
9450991810,6cyclictest17240-21cat20:40:251
9450991810,6cyclictest11676-21cat20:30:131
9450991810,5cyclictest24838-21uniq21:00:151
944999189,6cyclictest29264-21gltestperf19:55:160
9449991813,3cyclictest9060-21cstates23:20:120
9449991813,3cyclictest3222-21sh21:39:030
9449991813,3cyclictest29579-21hwlatdetect23:35:170
9449991813,3cyclictest28962-21sh22:45:450
9449991811,5cyclictest30250-21expr00:00:140
9449991811,5cyclictest2508-21seq00:27:140
9449991811,5cyclictest22268-21cut19:40:000
9449991810,6cyclictest8769-21awk20:20:250
9449991810,6cyclictest6495-21cat20:15:240
9449991810,5cyclictest31712-21load20:00:180
945499179,6cyclictest26514-21ls21:05:003
945499179,6cyclictest26504-21cut19:50:003
945499179,6cyclictest13539-21date20:35:003
9454991711,4cyclictest24594-21sh23:31:033
9454991711,4cyclictest11701-21forks21:45:153
9454991710,5cyclictest8399-21gzip20:20:223
9454991710,5cyclictest31649-21expr20:00:183
9453991712,3cyclictest18058-21/usr/sbin/munin19:30:102
9453991710,5cyclictest7595-21expr20:20:132
9453991710,5cyclictest27517-21ntp_kernel_pll_21:05:202
9450991712,4cyclictest22-21ksoftirqd/120:24:591
9450991711,4cyclictest636-21runrttasks20:25:141
9450991711,4cyclictest22-21ksoftirqd/120:05:001
945099171,13cyclictest7765-21ssh22:30:181
9450991710,5cyclictest939-21expr20:05:161
9450991710,5cyclictest26890-21expr19:50:121
9450991710,5cyclictest15360-21seq19:20:511
9450991710,5cyclictest13954-21expr20:35:151
9450991710,5cyclictest11900-21expr19:15:131
945099170,8cyclictest8428-21ssh22:06:161
944999179,6cyclictest23000-21cat19:40:150
944999179,6cyclictest20064-21cat20:50:010
9449991713,2cyclictest20159-21/usr/sbin/munin20:50:210
9449991711,4cyclictest18068-21acpi19:30:100
9449991710,5cyclictest26328-21seq21:04:400
945499169,5cyclictest900-21expr20:05:153
945499161,3cyclictest5499-21diskstats20:15:153
945399169,5cyclictest16240-21df20:40:142
945499159,4cyclictest13932-21expr20:35:153
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional