You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-20 - 03:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackaslot8s.osadl.org (updated Fri Mar 20, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16356221297,94sleep20-21swapper/219:08:382
193821660,1sleep00-21swapper/023:09:200
163122152100,25sleep30-21swapper/319:08:033
16417214596,40sleep10-21swapper/119:09:261
163542139101,26sleep00-21swapper/019:08:350
65852670,1sleep36590-21awk00:00:243
176442650,3sleep30-21swapper/323:45:073
305422610,2sleep20-21swapper/222:17:492
77192580,2sleep10-21swapper/122:00:231
279622570,2sleep30-21swapper/323:28:213
123692560,1sleep312371-21sh00:29:413
188172530,1sleep00-21swapper/021:45:130
119182510,1sleep10-21swapper/100:05:031
174512500,2sleep10-21swapper/123:20:171
151452490,1sleep315147-21ssh23:42:503
53322480,2sleep0131rcuc/000:00:090
88432460,1sleep20-21swapper/222:50:122
80672460,2sleep30-21swapper/322:00:253
50392450,3sleep05038-21/usr/sbin/munin23:35:170
79302440,2sleep00-21swapper/000:25:250
178552430,1sleep10-21swapper/122:56:131
119382430,1sleep20-21swapper/221:15:212
143922420,1sleep10-21swapper/123:18:241
210092390,2chrt0-21swapper/321:46:123
225762350,2sleep20-21swapper/222:11:322
1674099353,12cyclictest25008-21cron00:15:003
204682340,1sleep00-21swapper/023:22:330
1674099341,12cyclictest14034-21sed00:30:193
162682340,2chrt211rcuc/122:06:531
192312330,4chrt331rcuc/322:33:523
16738993310,5cyclictest22-21ksoftirqd/123:08:381
280482320,2sleep30-21swapper/321:51:233
1674099325,11cyclictest24407-21ssh23:50:103
1674099324,13cyclictest4405-21/usr/sbin/munin21:10:143
1674099323,4cyclictest0-21swapper/322:50:003
1674099322,11cyclictest25326-21awk21:25:243
107242320,2sleep20-21swapper/221:39:212
1674099313,11cyclictest23268-21taskset00:37:103
1674099313,10cyclictest15141-21chrt23:19:103
1674099311,13cyclictest18018-21chrt22:08:393
1674099304,8cyclictest18096-21latency_hist21:45:013
1674099303,10cyclictest2621-21ssh22:20:353
1674099302,10cyclictest5740-21sed21:35:163
1674099302,10cyclictest24146-21ssh23:00:483
1674099301,12cyclictest32653-21fschecks_time00:20:153
1674099300,13cyclictest25059-21ssh22:37:483
1673899308,12cyclictest22-21ksoftirqd/121:41:091
1673899307,12cyclictest22-21ksoftirqd/123:47:061
16738993016,2cyclictest141rcu_preempt21:15:451
16738993012,5cyclictest22-21ksoftirqd/123:36:521
16738993011,5cyclictest22-21ksoftirqd/121:45:221
16738993010,5cyclictest22-21ksoftirqd/121:59:121
182282290,1chrt0-21swapper/022:08:530
1674099294,6cyclictest0-21swapper/321:17:053
1674099292,9cyclictest28517-21chrt22:15:463
1674099292,13cyclictest20994-21awk20:35:003
1674099292,11cyclictest21558-21ssh23:23:393
1674099292,11cyclictest12511-21ssh00:05:143
1673999297,4cyclictest28-21ksoftirqd/223:00:012
16739992914,4cyclictest28-21ksoftirqd/223:50:242
1673899299,10cyclictest22-21ksoftirqd/123:13:231
1673899297,11cyclictest22-21ksoftirqd/123:57:151
16738992910,4cyclictest22-21ksoftirqd/122:53:011
96712280,2sleep00-21swapper/023:15:100
1674099283,12cyclictest0-21swapper/321:55:123
1674099283,10cyclictest29616-21perf19:40:003
1674099281,9cyclictest3586-21diskmemload22:50:313
1674099280,15cyclictest0-21swapper/300:17:023
1674099280,10cyclictest0-21swapper/321:32:003
1673999284,4cyclictest141rcu_preempt22:30:112
16739992811,4cyclictest28-21ksoftirqd/223:33:432
1673899289,4cyclictest22-21ksoftirqd/100:38:491
1673899287,11cyclictest22-21ksoftirqd/121:52:141
1673899286,10cyclictest141rcu_preempt23:44:041
16738992816,10cyclictest11650-21dump-pmu-power00:05:001
16738992811,5cyclictest22-21ksoftirqd/121:24:241
16738992811,5cyclictest22-21ksoftirqd/121:24:241
16738992810,9cyclictest22-21ksoftirqd/121:12:341
16738992810,5cyclictest22-21ksoftirqd/100:31:411
113152280,2chrt0-21swapper/121:39:561
39422270,2sleep122-21ksoftirqd/100:22:571
284622270,2chrt0-21swapper/022:15:430
1674099274,6cyclictest15337-21apt-get22:55:123
1674099273,10cyclictest11506-21chrt22:27:503
1674099272,9cyclictest30603-21ssh22:41:383
1674099272,10cyclictest5932-21sed23:35:243
1674099272,10cyclictest30145-21sed23:05:223
1674099271,4cyclictest0-21swapper/320:45:003
1673999279,11cyclictest28-21ksoftirqd/221:10:142
1673999278,4cyclictest141rcu_preempt23:20:042
1673999278,10cyclictest6499-21kworker/2:022:25:122
1673999274,8cyclictest28-21ksoftirqd/220:45:112
1673999271,12cyclictest28-21ksoftirqd/221:34:552
16739992710,5cyclictest28-21ksoftirqd/200:37:062
1673899279,5cyclictest22-21ksoftirqd/122:37:051
1673899279,4cyclictest22-21ksoftirqd/122:31:311
1673899279,4cyclictest22-21ksoftirqd/100:10:471
1673899278,5cyclictest22-21ksoftirqd/123:26:251
1673899278,4cyclictest22-21ksoftirqd/123:02:291
1673899276,11cyclictest22-21ksoftirqd/122:12:391
16738992711,5cyclictest22-21ksoftirqd/100:25:201
16738992710,5cyclictest22-21ksoftirqd/123:53:081
73482260,2chrt0-21swapper/220:00:262
1674099262,5cyclictest0-21swapper/319:55:003
1674099261,9cyclictest20044-21cstates22:10:133
1674099261,6cyclictest0-21swapper/323:10:593
1674099260,15cyclictest0-21swapper/323:58:223
1673999269,5cyclictest28-21ksoftirqd/222:57:282
1673999267,4cyclictest141rcu_preempt22:00:002
1673999265,10cyclictest28-21ksoftirqd/222:38:442
1673899268,5cyclictest22-21ksoftirqd/122:28:581
1673899268,5cyclictest22-21ksoftirqd/100:16:481
1673899268,4cyclictest22-21ksoftirqd/121:31:231
16738992612,4cyclictest22-21ksoftirqd/122:45:151
16738992611,8cyclictest22-21ksoftirqd/123:34:551
16738992611,4cyclictest22-21ksoftirqd/122:41:231
16738992610,5cyclictest22-21ksoftirqd/122:21:291
1674099256,10cyclictest34-21ksoftirqd/321:07:443
1674099251,8cyclictest32625-21sh23:31:183
1674099251,8cyclictest17355-21fschecks_count20:25:163
1673999253,8cyclictest28-21ksoftirqd/223:05:102
1673999253,7cyclictest28-21ksoftirqd/223:35:252
1673999253,12cyclictest28-21ksoftirqd/200:15:112
1673999252,13cyclictest28-21ksoftirqd/200:32:092
16739992511,2cyclictest141rcu_preempt00:00:222
16739992510,8cyclictest141rcu_preempt23:59:322
16739992510,3cyclictest141rcu_preempt22:05:122
1673899259,5cyclictest22-21ksoftirqd/121:28:301
16738992514,9cyclictest30217-21sh22:17:291
92152240,2sleep20-21swapper/222:01:322
1674099243,9cyclictest16483-21chrt20:22:213
1673999248,4cyclictest28-21ksoftirqd/222:46:552
1673999246,4cyclictest28-21ksoftirqd/221:43:112
1673999242,6cyclictest20247-21ssh23:46:132
1673999242,13cyclictest28-21ksoftirqd/223:44:422
1673999241,2cyclictest75-21kswapd023:15:002
1673899248,5cyclictest22-21ksoftirqd/119:30:201
148002240,1sleep00-21swapper/021:17:480
98312230,1sleep20-21swapper/223:15:122
202862230,1sleep30-21swapper/321:21:323
202862230,1sleep30-21swapper/321:21:323
1674099234,6cyclictest1050-21users21:00:233
1674099233,7cyclictest34-21ksoftirqd/320:00:233
1674099230,8cyclictest30313-21head20:55:163
1673999238,3cyclictest141rcu_preempt21:20:262
1673999238,3cyclictest141rcu_preempt21:20:262
1673999235,5cyclictest28-21ksoftirqd/200:05:182
1673999233,4cyclictest28-21ksoftirqd/221:51:552
1673899238,4cyclictest22-21ksoftirqd/120:10:201
16738992317,4cyclictest22-21ksoftirqd/119:45:001
1673899230,7cyclictest23147-21idleruntime-cro20:40:001
1673799232,14cyclictest20367-21apt-get22:35:030
1674099224,16cyclictest16838-21cpuspeed_turbos19:10:123
1674099223,4cyclictest26123-21hddtemp_smartct19:30:143
1674099221,8cyclictest22450-21python319:20:213
16740992218,3cyclictest331rcuc/320:52:533
1674099220,8cyclictest21574-21diskstats20:35:133
1674099220,5cyclictest0-21swapper/319:55:163
1673999223,9cyclictest141rcu_preempt19:55:002
1673999222,9cyclictest141rcu_preempt23:25:552
16739992213,7cyclictest31374-21users19:40:242
16739992213,6cyclictest52950irq/141-enp3s0-00:20:122
16739992213,6cyclictest29297-21grep22:40:252
1673999221,12cyclictest28-21ksoftirqd/221:49:492
1673899228,4cyclictest22-21ksoftirqd/120:45:151
1673899228,4cyclictest141rcu_preempt19:35:181
1673899225,3cyclictest141rcu_preempt20:20:261
1673899224,2cyclictest141rcu_preempt21:05:241
16738992216,4cyclictest22-21ksoftirqd/120:30:001
16738992213,7cyclictest20791-21seq19:17:141
16738992212,7cyclictest469-21ntp_states19:45:191
16738992211,2cyclictest141rcu_preempt19:28:461
16737992213,6cyclictest25899-21df_inode19:30:120
16740992118,2cyclictest331rcuc/319:29:563
16740992116,3cyclictest10678-21tune2fs20:10:163
1674099211,4cyclictest0-21swapper/320:46:483
16740992111,7cyclictest8858-21grep20:05:173
1673999217,2cyclictest141rcu_preempt00:25:022
1673999216,4cyclictest28-21ksoftirqd/221:26:572
1673999213,11cyclictest616-21sendmail_mailqu21:00:202
1673999212,9cyclictest141rcu_preempt00:10:152
1673999211,6cyclictest28-21ksoftirqd/220:45:002
16739992115,4cyclictest25514-21latency_hist19:30:002
16739992113,6cyclictest28-21ksoftirqd/220:10:002
16739992112,7cyclictest278-21systemd-journal19:15:002
16739992111,7cyclictest5018-21sensors_temp19:55:212
1673999210,12cyclictest27983-21cstates20:50:142
1673799211,9cyclictest7438-21ssh22:25:130
1673799211,7cyclictest8913-21rm21:13:370
1673799211,4cyclictest0-21swapper/022:21:420
16737992113,6cyclictest16165-21sed20:20:250
16737992112,7cyclictest27093-21awk21:50:250
1674099202,13cyclictest0-21swapper/319:45:113
16740992012,6cyclictest19866-21sed19:15:173
1673999203,2cyclictest141rcu_preempt22:20:262
16739992012,6cyclictest13596-21sensors_temp20:15:232
16739992011,6cyclictest19436-21cat19:15:132
1673899209,4cyclictest22-21ksoftirqd/120:15:201
16738992013,5cyclictest22-21ksoftirqd/120:00:001
16738992013,5cyclictest22-21ksoftirqd/119:55:001
16738992011,7cyclictest29389-21seq20:52:561
16738992011,7cyclictest25024-21seq20:41:491
1673799209,8cyclictest25302-21grep21:25:230
1673799207,7cyclictest636-21runrttasks23:55:130
1673799201,6cyclictest8607-21apt-get22:50:110
16737992014,4cyclictest5909-21latency21:35:170
16737992012,6cyclictest422-21ntp_states21:00:180
16737992012,6cyclictest10618-21expr20:10:140
16737992011,7cyclictest24877-21cut23:50:140
16737992011,7cyclictest14560-21tune2fs22:30:140
16737992010,6cyclictest52750irq/139-enp3s0-23:00:130
15352200,1sleep00-21swapper/019:50:000
1674099192,3cyclictest34-21ksoftirqd/320:15:173
1674099190,6cyclictest34-21ksoftirqd/319:40:143
1673999199,7cyclictest17888-21ntpq20:25:222
1673999196,11cyclictest271rcuc/220:55:152
1673999193,11cyclictest14813-21apt-get20:20:122
1673999192,12cyclictest19100-21apt-get20:30:132
16739991911,6cyclictest26571-21which19:30:202
16739991911,5cyclictest29262-21seq19:35:482
16739991910,6cyclictest2919-21cat21:05:242
1673899195,4cyclictest141rcu_preempt20:05:231
1673899194,6cyclictest221ksoftirqd/119:20:201
16738991912,5cyclictest30922-21tr20:55:221
16738991911,5cyclictest6926-21ntp_kernel_pll_20:00:211
16738991910,6cyclictest19061-21seq20:30:011
1673799199,8cyclictest31237-21df_inode21:30:150
1673799199,7cyclictest22232-21awk19:20:200
1673799197,3cyclictest141rcu_preempt00:15:170
16737991914,3cyclictest31976-21sh21:55:050
16737991912,5cyclictest4059-21expr19:55:120
16737991912,5cyclictest27397-21seq20:49:510
16737991912,5cyclictest18296-21seq22:56:400
16737991912,4cyclictest3586-21diskmemload23:25:120
16737991911,6cyclictest20711-21seq20:31:370
16737991911,6cyclictest18721-21find00:10:100
1673799191,15cyclictest17748-21ssh23:45:120
16737991910,6cyclictest1516-21cron22:45:000
1673999182,12cyclictest32010-21apt-get19:45:122
1673999182,11cyclictest10307-21apt-get20:10:132
16739991812,4cyclictest22198-21ntp_offset19:20:192
16739991811,5cyclictest21771-21expr20:35:152
16738991810,6cyclictest16994-21cat19:10:121
1673799188,7cyclictest19862-21cat19:15:170
1673799183,9cyclictest19738-21apt-get22:10:110
1673799182,11cyclictest12044-21apt-get00:05:110
1673799182,11cyclictest11826-21apt-get21:40:120
1673799181,8cyclictest11009-21ssh23:40:110
16737991813,3cyclictest9217-21sh22:01:330
16737991813,3cyclictest29790-21/usr/sbin/munin20:55:220
16737991812,4cyclictest1946-21sh23:33:030
16737991811,5cyclictest31415-21seq19:40:450
16737991811,5cyclictest28805-21run-parts20:50:230
16737991811,5cyclictest13751-21sed00:30:160
16737991811,5cyclictest12653-21expr20:15:140
16737991811,4cyclictest3586-21diskmemload00:35:240
16737991810,6cyclictest4691-21grep23:10:250
16737991810,6cyclictest32036-21seq00:20:070
16737991810,6cyclictest2037-21cat21:05:150
16737991810,6cyclictest17097-21expr19:10:130
16737991810,5cyclictest3480-21cat22:45:240
16737991810,5cyclictest17917-21grep21:20:160
16737991810,5cyclictest17917-21grep21:20:160
1673899172,10cyclictest31970-21apt-get21:00:101
1673799179,5cyclictest6540-21grep20:00:170
1673799179,5cyclictest27893-21expr19:35:120
1673799179,5cyclictest278-21systemd-journal20:30:000
16737991712,3cyclictest23408-21/usr/sbin/munin19:25:120
16737991712,3cyclictest0-21swapper/020:05:130
1673799168,6cyclictest1940-21expr19:50:120
1673799168,5cyclictest22322-21sendmail_mailqu20:35:210
1673799157,6cyclictest23859-21perl20:40:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional