You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-23 - 09:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu Apr 23, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4862183107,28sleep10-21swapper/119:06:061
4802160103,27sleep00-21swapper/019:06:010
8182159107,26sleep30-21swapper/319:09:543
313482158102,27sleep20-21swapper/219:05:072
714521250,2sleep30-21swapper/322:35:273
12852810,6chrt141rcu_preempt21:19:402
311602680,2sleep20-21swapper/223:17:522
239782610,2sleep30-21swapper/323:36:223
161202610,2sleep012-21ksoftirqd/023:30:250
82925438,8sleep20-21swapper/219:10:012
34162540,2sleep10-21swapper/100:10:071
310552540,2sleep00-21swapper/021:40:530
286782530,3chrt0-21swapper/020:10:250
96942510,1sleep20-21swapper/221:25:182
88992440,2chrt0-21swapper/322:13:193
6992440,3chrt0-21swapper/022:06:460
194872430,3sleep119460-21ssh22:45:151
201022420,1chrt0-21swapper/200:21:562
74262410,2chrt0-21swapper/200:36:362
313162380,2sleep30-21swapper/300:05:473
309472360,2sleep20-21swapper/221:40:472
302872340,2sleep10-21swapper/100:05:221
320032290,1sleep30-21swapper/321:17:573
110099291,5cyclictest121ksoftirqd/021:37:200
252762280,5chrt25259-21ssh22:50:000
110199282,17cyclictest25820-21ssh22:50:111
7892270,2sleep20-21swapper/222:30:522
1101992616,7cyclictest2929-21memory21:20:171
1100992613,10cyclictest2815-21sed23:20:240
179852250,3chrt0-21swapper/121:00:251
110199252,18cyclictest15685-21kworker/u8:223:10:111
106112250,8sleep012-21ksoftirqd/021:25:320
49522240,2sleep228-21ksoftirqd/221:45:332
325802240,1sleep00-21swapper/022:55:060
171452240,1sleep20-21swapper/221:55:162
132602240,2chrt0-21swapper/321:52:123
111562240,3chrt0-21swapper/123:26:531
1103992414,7cyclictest18042-21df22:20:133
110299241,10cyclictest252802latency_hist22:50:002
110099247,10cyclictest24391-21diskmemload22:25:130
1100992416,6cyclictest22707-21sed23:35:240
1100992415,6cyclictest1430-21sendmail-mta22:51:580
110399232,14cyclictest4539-21ssh00:35:123
1103992311,8cyclictest29439-21apt-get20:15:123
1103992311,7cyclictest640-21snmpd22:45:123
1103992310,4cyclictest29910-21ssh23:40:363
110299233,12cyclictest22140-21ssh21:35:102
1101992314,7cyclictest23035-21users23:35:261
1101992312,8cyclictest22922-21df00:00:161
1100992316,5cyclictest12-21ksoftirqd/023:55:010
1100992314,6cyclictest560-21dbus-daemon22:40:000
110399229,9cyclictest23581-21ssh00:25:133
110399222,13cyclictest27039-21ssh23:15:113
1103992216,4cyclictest34-21ksoftirqd/321:15:013
1103992216,4cyclictest18950irq/143-ahci[0022:35:003
110399221,10cyclictest111572sh23:26:533
110299221,9cyclictest141rcu_preempt22:55:002
1102992213,7cyclictest560-21dbus-daemon20:05:002
1102992213,7cyclictest560-21dbus-daemon20:05:002
110199227,3cyclictest141rcu_preempt23:15:181
110199227,12cyclictest22242-21processes21:10:221
1101992216,3cyclictest22081-21dump-pmu-power00:00:001
1101992214,6cyclictest14723-21ntp_states23:05:211
1101992213,7cyclictest13411-21seq21:52:211
1101992210,8cyclictest16379-21apt-get19:45:121
110099228,8cyclictest52950irq/141-enp3s0-23:55:010
110099228,5cyclictest12-21ksoftirqd/020:30:000
181672210,9sleep228-21ksoftirqd/223:56:032
110399212,13cyclictest75-21kswapd023:45:123
110399212,13cyclictest3534-21taskset00:10:133
1103992114,4cyclictest22823-21cpu20:00:143
1103992114,4cyclictest22823-21cpu20:00:143
1103992112,6cyclictest25523-21awk22:25:223
1103992111,7cyclictest6579-21sendmail23:00:003
1103992110,9cyclictest30489-21sh22:53:253
110299219,4cyclictest15862-21ssh22:42:212
110299211,8cyclictest7258-21users22:35:272
110299210,19cyclictest4930-21ssh23:46:222
1101992113,5cyclictest636-21runrttasks23:31:071
1101992112,7cyclictest6449-21cat22:35:201
1101992112,7cyclictest5478-21date20:35:001
110099219,8cyclictest30250-21apt-get00:30:130
110099219,8cyclictest30250-21apt-get00:30:120
110099217,3cyclictest164942cpu21:55:120
110099213,15cyclictest23200-21cut20:00:170
110099213,15cyclictest23200-21cut20:00:170
110099213,13cyclictest7974-21apt-get23:25:120
1100992112,7cyclictest7349-21seq20:38:170
1100992112,7cyclictest6017-21smartctl19:20:150
1100992112,6cyclictest17405-21fw_packets00:20:150
1100992111,8cyclictest32177-21sed20:20:160
268932200,2sleep00-21swapper/023:15:040
161112200,3sleep0560-21dbus-daemon21:00:000
110399204,14cyclictest12674-21head22:15:253
110399204,14cyclictest12674-21head22:15:253
1103992014,4cyclictest34-21ksoftirqd/320:55:003
1103992014,4cyclictest34-21ksoftirqd/320:40:003
1103992012,6cyclictest3806-21diskstats20:30:143
1103992012,5cyclictest2671-21switchtime23:20:243
1103992010,6cyclictest52850irq/140-enp3s0-00:15:113
110399200,5cyclictest0-21swapper/323:30:203
110299202,13cyclictest17707-21ssh22:20:122
110299202,12cyclictest26836-21apt-get20:10:032
1102992013,5cyclictest16042-21sed19:45:002
1102992012,6cyclictest20426-21basename19:55:002
1102992011,7cyclictest1658-21df_inode20:25:132
1102992011,6cyclictest16232-21cut21:30:182
1102992010,7cyclictest3921-21basename22:10:002
110199207,11cyclictest27977-21cat21:15:141
110199202,11cyclictest53050irq/142-enp3s0-23:40:131
1101992014,4cyclictest15082-21sh22:41:331
1101992014,4cyclictest13380-21sh00:16:551
1101992013,5cyclictest4705-21diskstats22:10:151
1101992013,4cyclictest5996-21sh21:46:371
1101992013,4cyclictest31097-21sh21:40:571
1101992013,3cyclictest53050irq/142-enp3s0-21:30:211
1101992012,6cyclictest4002-21seq22:34:071
1101992012,6cyclictest3025-21cron19:15:001
1101992012,6cyclictest17941-21ls21:55:241
1101992012,6cyclictest13896-21date19:40:011
1101992011,7cyclictest24505-21cat00:25:191
1101992011,6cyclictest9197-21tune2fs21:25:151
110199201,16cyclictest24373-21apt-get22:25:121
110099201,9cyclictest9355-21apt21:50:000
110099201,5cyclictest0-21swapper/021:10:220
1100992012,6cyclictest19024-21ls21:05:130
1100992011,7cyclictest19736-21cat19:50:240
1100992011,6cyclictest21386-21sed19:55:190
34982190,3sleep00-21swapper/023:45:220
321052190,4sleep032103-21ssh22:30:210
110399199,6cyclictest52850irq/140-enp3s0-21:35:113
110399193,10cyclictest23382-21ssh22:00:123
110399191,6cyclictest17124-21sensors_temp23:55:223
1103991911,6cyclictest29490-21which21:40:163
1103991911,6cyclictest19965-21wc23:10:003
1103991911,6cyclictest18536-21cut21:05:013
1103991911,6cyclictest18438-21date19:50:003
1103991911,6cyclictest15033-21cron21:30:003
1103991911,6cyclictest14831-21cat20:55:173
110299197,6cyclictest24999-21unixbench_multi22:00:252
110299192,12cyclictest21333-21apt-get23:35:132
110299191,5cyclictest15277-21latency_hist23:55:002
1102991914,3cyclictest7350-21sh23:24:472
1102991913,4cyclictest3514-21sh00:34:242
1102991913,4cyclictest3514-21sh00:34:242
1102991913,4cyclictest14429-21sh21:53:232
1102991912,5cyclictest3169-21pluginstate21:20:202
1102991912,5cyclictest28660-21smartctl23:40:172
1102991912,4cyclictest8117-21pmu-power23:00:212
1102991911,6cyclictest16720-21expr21:00:142
1102991911,6cyclictest16256-21ls23:30:262
1102991911,6cyclictest10207-21cat20:45:132
1102991911,6cyclictest10207-21cat20:45:132
1102991910,6cyclictest21278-21cat23:10:182
110199192,11cyclictest10981-21ssh22:15:111
110199192,11cyclictest10981-21ssh22:15:111
1101991914,3cyclictest18435-21kernelversion22:20:171
1101991914,3cyclictest1744-21sh00:32:381
1101991914,3cyclictest1744-21sh00:32:381
1101991913,4cyclictest6111-21sh23:23:321
1101991913,4cyclictest29469-21sh22:04:501
1101991912,4cyclictest2238-21acpi23:45:111
1101991911,6cyclictest7967-21tune2fs19:25:151
1101991911,6cyclictest6415-21cat20:35:191
1101991911,6cyclictest3626-21diskstats19:15:141
1101991911,6cyclictest15849-21seq20:56:431
1101991911,5cyclictest31432-21switchtime22:05:231
1101991911,5cyclictest2865-21seq20:25:371
1101991910,7cyclictest7672-21basename23:00:161
1101991910,6cyclictest25713-21cut20:05:181
110099192,5cyclictest0-21swapper/000:15:230
1100991913,3cyclictest27506-21sh00:03:420
1100991913,3cyclictest16705-21sh21:30:220
1100991911,6cyclictest3606-21sed21:20:230
1100991911,6cyclictest1889-21sed19:10:160
1100991911,6cyclictest17208-21if_enp3s021:00:170
110099191,14cyclictest28887-21ssh23:40:190
1100991910,7cyclictest14581-21sed23:05:190
110399189,7cyclictest2130-21ls20:25:183
110399188,7cyclictest7353-21sh23:00:133
110399188,7cyclictest560-21dbus-daemon20:45:003
110399187,9cyclictest0-21swapper/321:55:073
110399184,6cyclictest1098-21cyclictest22:05:193
110399181,5cyclictest28836-21munin-run00:05:003
1103991813,3cyclictest3313-21sendmail_mailst21:20:213
1103991812,4cyclictest4188-21open_inodes21:45:203
1103991812,4cyclictest331rcuc/322:40:263
1103991812,4cyclictest13768-21sendmail19:40:013
1103991811,5cyclictest5629-21expr19:20:123
1103991811,5cyclictest31664-21sort20:20:133
1103991811,5cyclictest20177-21awk21:05:253
1103991811,5cyclictest17137-21expr00:20:133
1103991811,5cyclictest16281-21cat21:30:183
1103991811,4cyclictest636-21runrttasks23:10:103
1103991811,4cyclictest636-21runrttasks00:33:213
1103991811,4cyclictest636-21runrttasks00:33:213
1103991810,6cyclictest26301-21grep20:05:243
1103991810,6cyclictest14496-21perl19:40:163
110299189,7cyclictest9682-21sed19:30:012
110299181,7cyclictest19259-21grep21:05:162
110299181,6cyclictest3897-21df_inode00:10:162
1102991813,3cyclictest4121-21irqstats20:30:172
1102991813,3cyclictest11501-21sh23:27:142
1102991813,3cyclictest10089-21sh00:15:072
1102991812,4cyclictest7899-21sh22:12:182
1102991812,4cyclictest25716-21missed_timers20:05:182
1102991812,4cyclictest14419-21sh22:17:092
1102991812,4cyclictest14419-21sh22:17:092
1102991811,5cyclictest4064-21cat19:15:182
110299181,15cyclictest28821-21sh00:05:002
1102991811,5cyclictest24598-21cat22:25:142
1102991811,4cyclictest636-21runrttasks22:55:242
1102991810,6cyclictest7928-21ls20:40:132
1102991810,6cyclictest5592-21cat20:35:112
1102991810,6cyclictest3233-21grep00:10:012
1102991810,6cyclictest24499-21cat00:25:192
1102991810,6cyclictest21861-21sed21:10:182
1102991810,6cyclictest20720-21sort19:55:142
1102991810,6cyclictest12182-21expr19:35:132
1102991810,5cyclictest16364-21acpi19:45:112
1102991810,5cyclictest12158-21cat20:50:122
110199189,6cyclictest19349-21needreboot19:50:191
110199189,5cyclictest53050irq/142-enp3s0-00:20:121
110199187,9cyclictest0-21swapper/123:52:361
110199182,11cyclictest7770-21apt-get20:40:131
1101991813,3cyclictest24662-21sh21:36:121
1101991813,3cyclictest14864-21needreboot19:40:191
1101991812,3cyclictest7747-21sh00:36:561
1101991810,6cyclictest825-21grep22:55:161
1101991810,6cyclictest20372-21seq21:06:541
1101991810,6cyclictest10305-21forks20:45:141
1101991810,6cyclictest10305-21forks20:45:141
1101991810,5cyclictest27178-21find20:10:011
110099189,6cyclictest278-21systemd-journal19:19:590
110099182,11cyclictest6986-21apt-get23:00:120
110099182,11cyclictest3472-21apt-get20:30:130
110099182,11cyclictest10983-21apt-get22:15:130
110099182,11cyclictest10983-21apt-get22:15:130
110099181,8cyclictest30120-21memory00:05:200
110099181,4cyclictest0-21swapper/022:13:180
1100991813,3cyclictest636-21runrttasks22:40:140
1100991812,4cyclictest26705-21sh00:26:510
1100991811,5cyclictest5582-21ntpq00:35:200
1100991811,5cyclictest17988-21seq19:46:370
110099181,15cyclictest141rcu_preempt20:15:200
1100991811,5cyclictest10909-21pluginstate20:45:210
1100991811,5cyclictest10909-21pluginstate20:45:210
1100991811,4cyclictest20628-21diskmemload21:52:260
1100991810,6cyclictest9747-21cut20:45:000
1100991810,6cyclictest8857-21sed19:25:240
1100991810,6cyclictest12943-21seq19:35:220
1100991810,6cyclictest12065-21seq20:50:090
110399172,9cyclictest27264-21apt-get20:10:033
110399172,11cyclictest75-21kswapd023:50:113
1103991710,5cyclictest20937-21cat19:55:143
110299179,6cyclictest6157-21sed19:20:162
110299179,6cyclictest10335-21cat19:30:172
110299178,7cyclictest1184-21sed20:25:012
1102991712,3cyclictest13497-21sh23:05:022
1102991710,5cyclictest14287-21ls20:55:132
110199179,6cyclictest11554-21seq19:34:201
110199178,6cyclictest5679-21expr19:20:121
110199177,7cyclictest20623-21seq19:55:041
110199175,10cyclictest29882-21fschecks_time20:15:151
1101991712,3cyclictest31577-21/usr/sbin/munin20:20:141
110099179,6cyclictest28620-21ls21:15:190
110099179,6cyclictest278-21systemd-journal20:10:000
110099179,5cyclictest14228-21expr19:40:130
110099177,7cyclictest3568-21cpu00:10:140
1100991712,3cyclictest20307-21/usr/sbin/munin23:10:210
1100991711,3cyclictest23172-21gdbus22:00:010
110099171,11cyclictest9731-21apt-get19:30:120
110399169,5cyclictest10409-21cut19:30:183
110399168,6cyclictest3588-21expr19:15:133
110399168,6cyclictest19168-21uname19:50:173
1103991610,3cyclictest11368-21grep20:45:253
1103991610,3cyclictest11368-21grep20:45:253
110299168,6cyclictest29986-21expr20:15:162
110199169,5cyclictest12937-21grep20:50:181
1101991611,3cyclictest23378-21hddtemp_smartct20:00:181
1101991611,3cyclictest23378-21hddtemp_smartct20:00:181
110099160,4cyclictest0-21swapper/022:20:200
1103991511,2cyclictest1114-21/usr/sbin/munin19:10:233
1103991510,3cyclictest7646-21cpuspeed_turbos19:25:133
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional