You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-31 - 05:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Mon Aug 31, 2026 00:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
56942147111,25sleep10-21swapper/119:07:431
55572140104,25sleep00-21swapper/019:05:580
55352140104,25sleep30-21swapper/319:05:403
55592139103,25sleep20-21swapper/219:06:012
592027457,10sleep10-21swapper/119:10:011
151482700,2sleep00-21swapper/022:14:330
107432630,2sleep20-21swapper/223:46:482
222722620,4sleep122275-21proc_pri23:55:221
163322570,4sleep0131rcuc/021:50:250
132772570,2sleep10-21swapper/123:25:131
194642550,1sleep10-21swapper/121:53:181
173932540,2sleep30-21swapper/323:27:523
218602520,4sleep3331rcuc/300:20:013
153252520,2sleep00-21swapper/021:25:320
37252500,2sleep10-21swapper/122:05:231
130732500,2sleep20-21swapper/220:40:172
227102480,0sleep2271rcuc/200:20:162
104402470,1sleep30-21swapper/320:35:123
13982460,2sleep10-21swapper/122:04:151
103862440,2chrt0-21swapper/222:10:222
247922430,1sleep00-21swapper/021:56:560
246542430,3chrt0-21swapper/021:05:270
246032430,2sleep30-21swapper/323:09:333
56842420,2chrt0-21swapper/323:43:293
6194994111,12cyclictest30878-21ssh22:50:123
119622410,1sleep10-21swapper/121:47:461
119622410,1sleep10-21swapper/121:47:451
102372410,1sleep30-21swapper/300:35:143
619499407,14cyclictest9395-21ssh21:21:213
619299393,16cyclictest915-21taskset23:40:111
619499368,12cyclictest29052-21ssh23:12:163
619499367,10cyclictest1-21systemd20:10:223
619499366,10cyclictest25036-21diskmemload21:25:513
6192993612,6cyclictest11526-21seq00:11:221
619499358,16cyclictest30685-21sh21:13:433
619499358,12cyclictest28408-21ssh00:24:563
619499358,11cyclictest25036-21diskmemload23:33:483
619499357,10cyclictest25036-21diskmemload00:31:173
619499355,11cyclictest1517-21sh00:04:483
619499354,12cyclictest25036-21diskmemload23:21:353
6194993510,9cyclictest15851-21ssh22:38:493
619499348,9cyclictest4273-21ssh00:05:493
619499348,7cyclictest2061-21rm21:15:473
619499348,11cyclictest24758-21head22:45:153
619499348,10cyclictest2395-21rm00:29:413
619499347,12cyclictest26915-21cstates23:35:153
619499347,10cyclictest18215-21ssh23:04:523
619499346,16cyclictest10682-1kworker/u9:122:42:233
619499346,10cyclictest25036-21diskmemload21:45:543
619499346,10cyclictest25036-21diskmemload21:45:543
619499345,15cyclictest0-21swapper/321:35:513
191682340,2sleep10-21swapper/100:17:221
619499339,9cyclictest23908-21ssh21:32:263
619499339,10cyclictest8267-21ssh22:32:593
619499338,11cyclictest30603-21ssh22:01:003
619499338,10cyclictest9595-21rm22:58:113
619499337,9cyclictest8045-21ssh22:09:093
619499336,10cyclictest25036-21diskmemload23:45:193
6192993312,4cyclictest1956-21sh21:15:401
6192993310,5cyclictest5465-21sed22:30:281
619499329,9cyclictest3572-21sh23:17:303
619499329,9cyclictest31963-21ssh22:25:583
619499329,9cyclictest26888-21ssh23:59:333
619499327,11cyclictest28279-21rm22:24:013
619499325,10cyclictest25036-21diskmemload22:14:573
619299321,8cyclictest26928-21chrt21:35:141
6192993213,3cyclictest3070-21sh22:29:291
619299321,14cyclictest24542-21mailstats19:50:211
619499317,9cyclictest25975-21sh21:58:073
619499315,9cyclictest3049-21sh21:40:333
619499315,9cyclictest0-21swapper/321:50:313
619499315,10cyclictest12925-21sh00:12:493
619299319,5cyclictest32144-21date21:15:011
6192993112,4cyclictest20781-21sh23:05:411
6192993110,7cyclictest16250-21sed00:40:011
6192993110,7cyclictest10743-21apt-get22:35:131
6192993110,6cyclictest27785-21expr20:00:131
317892310,1sleep10-21swapper/100:26:361
619499302,8cyclictest0-21swapper/322:18:263
619299309,6cyclictest624-21gdbus22:20:001
619299309,6cyclictest14364-21load21:25:201
6192993011,4cyclictest28941-21sh23:36:031
619299301,11cyclictest14576-21apt-get20:45:111
6192993010,5cyclictest18310-21ls19:35:241
619499294,14cyclictest0-21swapper/320:55:243
619299296,8cyclictest0-21swapper/123:15:561
6192992911,3cyclictest13468-21processes20:40:221
6192992910,5cyclictest22-21ksoftirqd/100:30:131
6192992910,5cyclictest22-21ksoftirqd/100:05:201
619499285,8cyclictest0-21swapper/323:50:473
619299287,8cyclictest0-21swapper/122:57:581
619299287,4cyclictest0-21swapper/119:40:141
619299282,7cyclictest19942-21apt-get23:30:131
6192992812,4cyclictest25036-21diskmemload22:12:021
6192992811,3cyclictest25336-21sh21:33:511
6192992810,3cyclictest1704-21sh22:51:591
6192992810,3cyclictest13642-21pmu-power19:25:221
6192992810,3cyclictest13642-21pmu-power19:25:211
619299278,5cyclictest22-21ksoftirqd/120:55:181
619299271,24cyclictest0-21swapper/119:16:321
619199271,24cyclictest0-21swapper/019:16:360
256842270,10sleep012-21ksoftirqd/021:10:110
84422260,3chrt0-21swapper/222:57:012
70972260,10sleep122-21ksoftirqd/121:44:371
619499269,14cyclictest13429-21pmu-power20:40:213
6194992617,4cyclictest14821-21apt-get19:30:123
619399261,23cyclictest0-21swapper/219:15:262
619299261,1cyclictest0-21swapper/119:21:361
6191992614,7cyclictest9756-21apt-get00:35:090
619499252,9cyclictest0-21swapper/319:10:243
619499251,23cyclictest0-21swapper/319:15:363
294862250,7sleep0121ksoftirqd/022:25:010
93772240,1sleep19378-21ssh23:45:281
619499248,14cyclictest4752-21ntpq20:20:213
6194992410,9cyclictest0-21swapper/319:45:113
619199243,15cyclictest26646-21ssh23:35:140
6194992315,4cyclictest27691-21apt-get20:00:113
6193992316,5cyclictest31866-21ssh22:50:202
6193992312,9cyclictest7410-21ssh21:44:562
619399231,10cyclictest174942cat20:50:182
619199236,3cyclictest141rcu_preempt22:45:090
619199231,7cyclictest640-21snmpd22:29:160
6191992316,5cyclictest27422-21sendmail19:59:590
6191992312,9cyclictest5683-21ssh23:43:290
237682230,1sleep00-21swapper/023:08:430
19052230,9sleep012-21ksoftirqd/021:15:380
6194992211,8cyclictest22203-21sed21:00:223
6194992210,9cyclictest18859-21munin-run19:40:013
6193992218,2cyclictest25036-21diskmemload22:40:202
6193992212,6cyclictest52750irq/139-enp3s0-21:10:192
619299223,13cyclictest24157-21ssh22:45:111
619199221,7cyclictest19509-21cron23:30:010
619499218,10cyclictest8471-21df20:30:153
6194992113,6cyclictest18597-21turbostat.cron20:55:003
6194992111,8cyclictest30300-21grep20:05:163
6194992111,7cyclictest2155-21tr20:15:153
6194992111,6cyclictest10546-21apt-get19:20:133
6194992110,9cyclictest16469-21sleep20:49:593
6194992110,8cyclictest25668-21tr19:55:143
6194992110,8cyclictest25296-21rm21:10:013
6194992110,8cyclictest23866-21grep19:50:153
6194992110,8cyclictest19578-21tune2fs19:40:163
619399213,12cyclictest14218-21apt-get21:50:042
6193992112,7cyclictest26974-21ls23:35:162
6193992112,7cyclictest13998-21perl21:25:172
619299211,4cyclictest0-21swapper/122:23:401
6192992112,7cyclictest22618-21cat00:20:161
6192992111,7cyclictest12444-21df_inode23:00:141
619199214,3cyclictest25036-21diskmemload22:15:240
619199212,2cyclictest25036-21diskmemload23:57:570
619199211,6cyclictest8852-21apt-get00:10:120
619199211,6cyclictest20192-21apt-get21:30:130
6191992113,6cyclictest31864-21seq00:02:430
6191992112,7cyclictest3757-21cut20:20:010
6191992112,7cyclictest23038-21seq19:48:510
619199210,7cyclictest25036-21diskmemload22:30:010
281082210,2sleep00-21swapper/023:11:200
619499209,8cyclictest12898-21expr19:25:133
619499209,8cyclictest12898-21expr19:25:133
6194992010,7cyclictest6403-21fschecks_time20:25:153
619399207,10cyclictest52750irq/139-enp3s0-21:55:192
619399204,12cyclictest0-21swapper/219:20:142
619399201,8cyclictest215052ssh00:19:442
6193992014,3cyclictest25036-21diskmemload22:27:392
6193992014,3cyclictest25036-21diskmemload00:13:162
6193992013,5cyclictest31443-21seq23:14:412
6193992013,5cyclictest27398-40mandb00:00:002
6193992013,4cyclictest32235-21sh21:15:032
6193992013,4cyclictest25036-21diskmemload21:46:172
6193992013,4cyclictest25036-21diskmemload21:46:162
6193992012,6cyclictest28-21ksoftirqd/220:20:212
6193992012,6cyclictest13107-21ntpq23:00:192
6193992012,5cyclictest25036-21diskmemload23:22:392
6193992011,7cyclictest9854-21cat00:35:122
619399201,13cyclictest13271-21sh23:25:132
619299201,8cyclictest29018-21processes00:00:231
6192992011,7cyclictest6079-21expr20:25:131
6192992011,6cyclictest26044-21grep19:55:171
619199204,3cyclictest25036-21diskmemload22:59:190
619199204,13cyclictest25084-21munin-node00:20:110
619199203,3cyclictest25036-21diskmemload23:52:410
619199203,3cyclictest25036-21diskmemload23:47:530
619199203,3cyclictest25036-21diskmemload00:06:360
619199203,14cyclictest6458-21grep20:25:140
619199202,12cyclictest32233-21runrttasks23:15:120
6191992018,1cyclictest0-21swapper/019:21:390
6191992015,3cyclictest9359-21turbostat00:35:000
6191992013,5cyclictest20189-21sed20:55:240
6191992010,7cyclictest0-21swapper/023:30:110
619199200,11cyclictest287672rm21:35:340
100072200,1sleep20-21swapper/221:21:572
619399194,2cyclictest25036-21diskmemload21:31:362
619399193,3cyclictest25036-21diskmemload22:20:482
619399192,12cyclictest28896-21ssh00:25:122
619399192,11cyclictest27819-21apt-get00:00:112
619399192,11cyclictest12680-21apt-get19:25:122
619399192,11cyclictest12680-21apt-get19:25:112
6193991913,4cyclictest636-21runrttasks22:16:362
6193991912,5cyclictest3050-21cut22:05:162
6193991912,5cyclictest20400-21seq20:55:522
6193991912,5cyclictest18429-21seq23:52:492
6193991912,4cyclictest5790-21sh23:19:432
6193991911,6cyclictest560-21dbus-daemon19:10:072
619399191,16cyclictest4657-21cut22:30:182
6193991911,6cyclictest27200-21seq19:57:212
6193991911,6cyclictest2443-21cat23:40:252
6193991911,6cyclictest21716-21cat21:00:172
6193991911,6cyclictest17548-21basename19:35:172
6193991911,6cyclictest10162-21date20:35:012
6193991911,5cyclictest32746-21cat20:10:192
6193991910,7cyclictest16434-21seq19:31:382
6193991910,6cyclictest52750irq/139-enp3s0-23:30:242
6192991912,5cyclictest26724-21seq21:58:511
6192991911,6cyclictest3261-21seq20:16:101
6192991911,6cyclictest32511-21expr20:10:161
6192991911,6cyclictest21612-21cat21:00:151
6192991911,6cyclictest17651-21awk20:50:211
6192991910,7cyclictest16748-21date19:35:001
619199193,3cyclictest25036-21diskmemload23:23:400
619199192,4cyclictest12-21ksoftirqd/000:29:050
619199191,8cyclictest6805-21sshd21:20:150
619199191,5cyclictest1177-21apt-get21:40:110
6191991913,4cyclictest12154-21cron20:40:010
6191991912,5cyclictest3205-21seq22:53:300
6191991911,6cyclictest8932-21wc20:30:200
6191991911,6cyclictest31733-21basename20:10:000
6191991911,6cyclictest17033-21cat19:35:130
6191991911,6cyclictest16664-21pluginstate00:15:230
6191991911,5cyclictest16343-21seq19:30:280
6191991910,6cyclictest16712-21apt-get20:50:120
6191991910,6cyclictest12369-21expr23:00:130
619199190,6cyclictest141rcu_preempt22:42:260
287652190,3sleep2271rcuc/221:35:342
619399188,7cyclictest3070-21unixbench-2d20:15:252
619399187,9cyclictest52750irq/139-enp3s0-22:45:112
619399182,11cyclictest2162-21apt-get00:05:142
6193991814,2cyclictest25036-21diskmemload00:32:052
6193991812,4cyclictest11514-21cut22:35:192
6193991811,5cyclictest19381-21expr19:40:132
6193991810,6cyclictest19055-21expr23:05:142
619299188,3cyclictest104412cstates20:35:121
619299187,9cyclictest0-21swapper/123:52:571
6192991813,3cyclictest23735-21idleruntime21:05:181
6192991812,4cyclictest22267-21sh22:43:301
6192991811,5cyclictest8153-21seq23:20:271
619299181,13cyclictest6780-21apt-get21:20:151
6192991810,6cyclictest5505-21seq20:22:131
6192991810,6cyclictest26555-21awk23:10:211
619199188,8cyclictest4166-21wc22:05:260
619199187,4cyclictest52950irq/141-enp3s0-22:00:130
619199181,4cyclictest141rcu_preempt21:45:180
619199181,4cyclictest141rcu_preempt21:45:170
6191991813,3cyclictest11504-21irqstats22:35:190
6191991812,4cyclictest20469-21sort19:40:240
6191991811,5cyclictest13137-21cut19:25:150
6191991811,5cyclictest13137-21cut19:25:140
6191991810,6cyclictest32302-21diskstats20:10:140
6191991810,5cyclictest6823-21idleruntime19:10:170
619399178,6cyclictest24114-21pmu-power21:05:212
619399174,2cyclictest141rcu_preempt22:00:042
619399171,4cyclictest0-21swapper/220:35:202
619399171,11cyclictest21257-21apt-get19:45:132
6193991710,5cyclictest28132-21grep20:00:162
6193991710,5cyclictest24115-21ls19:50:172
6193991710,5cyclictest16575-21date20:50:002
619299178,7cyclictest21594-21diskstats19:45:141
619199179,6cyclictest12898-21tune2fs20:40:140
619199178,6cyclictest28281-21idleruntime20:00:180
6191991713,2cyclictest24310-21ntp_states19:50:200
6191991710,5cyclictest15088-21seq20:45:170
619399169,5cyclictest30102-21expr20:05:132
619399161,6cyclictest5966-21/usr/sbin/munin20:25:202
619299161,11cyclictest8139-21apt-get20:30:131
6192991610,4cyclictest30452-21if_err_enp3s020:05:171
6191991612,3cyclictest3825-21/usr/sbin/munin20:20:160
6191991610,4cyclictest21081-21cpuspeed_turbos21:00:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional