You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-20 - 09:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Mon Jul 20, 2026 00:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
262782161103,29sleep00-21swapper/019:05:590
265332159103,29sleep20-21swapper/219:09:182
265742158105,28sleep10-21swapper/119:09:491
263302156103,28sleep30-21swapper/319:06:383
451521210,2sleep10-21swapper/123:27:501
278352780,5chrt141rcu_preempt00:09:542
278352780,5chrt141rcu_preempt00:09:532
33382710,2sleep20-21swapper/223:26:402
131282690,2sleep30-21swapper/323:10:153
98882680,5sleep19895-21sh00:20:151
289032630,3sleep334-21ksoftirqd/321:20:323
172372590,2sleep10-21swapper/100:25:221
125512590,2sleep20-21swapper/219:50:172
11312580,2sleep10-21swapper/122:36:391
11312580,2sleep10-21swapper/122:36:381
218522550,2sleep20-21swapper/223:16:082
139182550,2sleep10-21swapper/123:35:131
119842550,1sleep10-21swapper/123:09:401
38542540,1sleep13858-21wc19:30:181
293492540,4sleep20-21swapper/222:57:572
145212540,2sleep214520-21df21:10:142
177012530,8sleep122-21ksoftirqd/100:01:211
122432490,2chrt0-21swapper/321:32:353
91152480,4chrt271rcuc/222:18:572
278662480,1sleep20-21swapper/221:20:222
52122470,2sleep3331rcuc/322:40:133
234192460,2sleep30-21swapper/322:29:533
276302450,1sleep30-21swapper/319:10:183
134262450,2chrt0-21swapper/123:10:171
142932430,4sleep20-21swapper/219:55:162
62732410,1sleep00-21swapper/021:52:180
62732410,1sleep00-21swapper/021:52:180
292952410,1sleep30-21swapper/323:21:513
292952410,1sleep30-21swapper/323:21:503
282252400,1sleep10-21swapper/123:45:201
203962350,1sleep20-21swapper/222:03:082
26862993312,15cyclictest15986-21ssh00:25:110
93602300,5sleep10-21swapper/121:55:121
26862993024,4cyclictest131rcuc/020:15:130
26862993024,4cyclictest131rcuc/020:15:120
91462280,2chrt0-21swapper/123:55:201
74332280,7sleep012-21ksoftirqd/022:17:150
26862992813,5cyclictest12-21ksoftirqd/022:35:010
26862992813,5cyclictest12-21ksoftirqd/022:35:000
26862992811,3cyclictest141rcu_preempt22:23:260
26862992810,4cyclictest12-21ksoftirqd/021:45:300
179932280,9sleep012-21ksoftirqd/022:00:410
109982280,1sleep20-21swapper/223:32:372
57482270,3sleep00-21swapper/023:53:010
2686399271,24cyclictest0-21swapper/119:18:451
2686299279,5cyclictest12-21ksoftirqd/023:29:480
2686299279,5cyclictest12-21ksoftirqd/022:56:530
2686299276,11cyclictest12-21ksoftirqd/023:39:560
32222260,2sleep10-21swapper/123:02:341
2686599261,23cyclictest0-21swapper/319:15:283
2686299269,4cyclictest12-21ksoftirqd/023:12:320
2686299269,4cyclictest12-21ksoftirqd/021:42:130
2686299268,4cyclictest12-21ksoftirqd/022:45:080
2686299267,3cyclictest12-21ksoftirqd/021:35:220
2686299265,3cyclictest141rcu_preempt00:01:220
2686299261,13cyclictest12-21ksoftirqd/023:24:550
2686299261,13cyclictest12-21ksoftirqd/023:24:540
2686299260,24cyclictest0-21swapper/019:16:320
2686299260,13cyclictest12-21ksoftirqd/000:31:520
3242250,2sleep20-21swapper/200:37:102
31362250,3sleep00-21swapper/020:45:000
2686299259,4cyclictest12-21ksoftirqd/023:59:460
2686299258,2cyclictest141rcu_preempt23:09:460
2686299257,9cyclictest12-21ksoftirqd/022:44:360
2686299256,5cyclictest141rcu_preempt22:30:150
2686299253,13cyclictest12-21ksoftirqd/021:20:240
26862992519,4cyclictest12-21ksoftirqd/021:35:000
26862992513,5cyclictest12-21ksoftirqd/022:25:180
269522240,2sleep10-21swapper/100:09:011
269522240,2sleep10-21swapper/100:09:001
26864992422,1cyclictest0-21swapper/219:16:322
26863992416,6cyclictest8619-21ssh21:54:431
26863992416,6cyclictest8619-21ssh21:54:421
2686299248,5cyclictest12-21ksoftirqd/021:16:260
2686299247,4cyclictest12-21ksoftirqd/000:11:270
2686299246,3cyclictest141rcu_preempt00:39:040
26862992418,4cyclictest12-21ksoftirqd/020:25:010
26862992412,4cyclictest12-21ksoftirqd/000:17:400
26862992410,5cyclictest12-21ksoftirqd/022:10:200
207612240,11sleep012-21ksoftirqd/023:40:130
2686599231,18cyclictest29225-21apt-get20:30:123
26863992313,7cyclictest10004-21df21:00:161
2686299238,3cyclictest141rcu_preempt22:51:300
2686299238,3cyclictest141rcu_preempt00:05:450
2686299238,3cyclictest141rcu_preempt00:05:440
2686299237,4cyclictest12-21ksoftirqd/021:59:030
2686299232,10cyclictest12-21ksoftirqd/023:16:190
2686299231,11cyclictest12-21ksoftirqd/023:31:260
26862992310,9cyclictest23284-21df22:05:160
275222220,2sleep20-21swapper/219:10:172
26865992213,7cyclictest22348-21ls21:15:253
2686499223,16cyclictest10022-21grep21:30:282
26864992217,3cyclictest22888-21ssh22:29:212
26864992210,7cyclictest23998-21ssh22:30:142
26863992212,6cyclictest53050irq/142-enp3s0-21:35:251
26862992217,3cyclictest12-21ksoftirqd/021:15:010
26862992215,5cyclictest12-21ksoftirqd/019:40:000
2686299221,11cyclictest12-21ksoftirqd/023:47:400
26862992210,4cyclictest12-21ksoftirqd/000:21:280
26862992210,2cyclictest141rcu_preempt21:26:300
44512210,2sleep00-21swapper/019:30:250
2686599219,5cyclictest34-21ksoftirqd/321:57:393
2686599211,5cyclictest0-21swapper/323:43:243
26865992114,5cyclictest25703-21cron23:20:003
26865992112,7cyclictest628-21cat23:25:143
26865992112,7cyclictest10166-21munin-run22:20:013
2686599211,17cyclictest52850irq/140-enp3s0-23:50:143
2686599210,13cyclictest13706-21diskmemload22:38:103
2686599210,13cyclictest13706-21diskmemload22:38:093
2686499213,11cyclictest640-21snmpd20:35:122
2686499212,14cyclictest22744-21apt-get22:05:032
2686499212,14cyclictest13706-21diskmemload22:45:032
26864992114,4cyclictest13706-21diskmemload22:50:242
26864992112,7cyclictest853-21seq22:12:222
26864992112,6cyclictest9118-21sed00:20:002
2686499210,7cyclictest13537-21ssh22:21:402
26863992116,3cyclictest22512-21cron20:15:001
26863992113,6cyclictest20199-21seq22:26:381
26863992113,6cyclictest15867-21cron20:00:001
26863992112,7cyclictest2813-21seq20:42:531
26863992111,6cyclictest53050irq/142-enp3s0-22:40:121
2686299219,5cyclictest12-21ksoftirqd/019:44:480
2686299212,14cyclictest31828-21ssh23:00:140
26862992115,4cyclictest12-21ksoftirqd/019:50:000
26862992114,5cyclictest12-21ksoftirqd/020:59:590
26862992112,6cyclictest278-21systemd-journal20:05:070
101652210,1sleep30-21swapper/323:07:503
320962200,3chrt0-21swapper/223:00:142
2686599208,10cyclictest25429-21expr22:55:133
2686599202,13cyclictest29336-21apt-get00:35:123
2686599202,13cyclictest18457-21ssh22:50:133
26865992013,4cyclictest28866-21sh22:33:383
26865992013,4cyclictest13706-21diskmemload00:15:063
26865992012,6cyclictest7403-21cat19:40:003
26865992011,7cyclictest7914-21diskstats19:40:163
26865992011,7cyclictest23515-21cat20:15:193
26865992011,7cyclictest23515-21cat20:15:183
2686499202,14cyclictest21312-21munin-run00:05:002
2686499202,12cyclictest1734-21apt-get21:25:122
2686499201,5cyclictest0-21swapper/221:57:402
26864992013,4cyclictest636-21runrttasks23:45:502
26864992013,4cyclictest0-21swapper/220:00:112
26864992012,5cyclictest21899-21grep23:40:232
26864992011,5cyclictest15111-21cpu21:35:142
2686399202,12cyclictest21777-21chrt21:40:111
2686399201,7cyclictest291492sh23:21:421
2686399201,7cyclictest291492sh23:21:411
26863992014,4cyclictest4684-21sh23:51:561
26863992014,4cyclictest26169-21sh00:32:141
26863992014,4cyclictest2181-21sh22:13:411
26863992014,4cyclictest21277-21rm22:51:321
26863992014,4cyclictest12731-21sh21:33:061
26863992013,5cyclictest726-21seq00:37:331
26863992012,6cyclictest28514-21grep00:10:141
26863992012,6cyclictest23502-21diskmemload23:41:421
26863992012,6cyclictest1577-21perl19:25:161
26863992011,6cyclictest11629-21ntpq22:20:201
2686299207,3cyclictest141rcu_preempt20:40:010
26862992011,6cyclictest13878-21date19:55:010
135832200,1sleep30-21swapper/322:45:313
2686599199,7cyclictest14619-21fschecks_time21:10:173
26865991914,3cyclictest52850irq/140-enp3s0-00:30:193
26865991914,3cyclictest21809-21apt00:05:173
26865991914,3cyclictest21809-21apt00:05:163
26865991913,4cyclictest9226-21sh00:20:063
26865991913,4cyclictest24496-21sh21:41:263
26865991913,4cyclictest21926-21sh00:29:423
26865991913,4cyclictest2051-21sh21:49:483
26865991913,4cyclictest18004-21sh22:00:413
26865991913,4cyclictest10491-21sh22:20:063
26865991912,5cyclictest8125-21ntp_kernel_pll_23:30:213
26865991911,6cyclictest29999-21sshd23:46:313
26865991911,6cyclictest29639-21apt22:10:103
26865991911,6cyclictest16448-21expr20:00:123
26865991911,6cyclictest12055-21cat21:05:143
26865991911,5cyclictest23946-21ntp_kernel_pll_22:05:223
2686499196,4cyclictest141rcu_preempt23:35:272
2686499193,10cyclictest10659-21python300:20:242
2686499192,13cyclictest75-21kswapd021:45:122
2686499192,11cyclictest640-21snmpd19:35:142
26864991912,4cyclictest32266-21needreboot19:20:212
26864991911,6cyclictest5119-21seq19:34:352
26864991911,6cyclictest23558-21sed00:30:202
26864991911,6cyclictest22123-21df21:40:142
26864991911,6cyclictest13424-21cat23:10:162
26864991910,7cyclictest27302-21cut23:20:222
26864991910,7cyclictest27302-21cut23:20:222
2686399199,7cyclictest31588-21cat19:20:141
26863991914,3cyclictest7130-21sh22:16:571
26863991914,3cyclictest30773-21sh21:46:011
26863991914,3cyclictest22881-21sh22:05:121
26863991913,4cyclictest26588-21sh22:31:191
26863991913,4cyclictest19229-21sh23:15:011
26863991913,4cyclictest16983-21sh22:48:581
26863991911,6cyclictest13838-21cut19:55:001
26863991911,5cyclictest5685-21cat19:35:161
2686399191,15cyclictest53050irq/142-enp3s0-21:20:141
2686299198,4cyclictest12-21ksoftirqd/020:50:170
2686299197,2cyclictest141rcu_preempt20:27:050
2686299197,2cyclictest141rcu_preempt19:29:030
2686299197,2cyclictest141rcu_preempt19:22:530
26862991913,4cyclictest30464-21threads20:30:240
26862991912,5cyclictest5064-21seq20:49:230
26862991912,5cyclictest18203-21seq20:04:360
26862991912,5cyclictest15375-21grep19:55:260
26862991912,5cyclictest13399-21seq21:06:150
26862991912,5cyclictest10018-21diskstats21:00:180
2686599182,11cyclictest3279-21apt-get19:30:143
2686599181,7cyclictest2466-21/usr/sbin/munin21:25:163
26865991813,3cyclictest5435-21cpuspeed_turbos20:50:133
2686599181,2cyclictest0-21swapper/321:00:193
26865991812,4cyclictest6416-21sh21:52:273
26865991812,4cyclictest6416-21sh21:52:273
26865991812,4cyclictest31642-21sh00:11:593
26865991812,4cyclictest19475-21sh00:03:083
26865991812,4cyclictest18601-21sh21:37:163
26865991811,5cyclictest8473-21pmu-power20:55:223
26865991811,5cyclictest1928-21ls20:40:183
26865991811,5cyclictest15263-21cat19:55:263
26865991811,5cyclictest14807-21ntp_kernel_err23:35:193
26865991811,5cyclictest1314-21expr19:25:143
26865991811,5cyclictest11610-21pluginstate19:50:003
26865991810,6cyclictest8410-21cat23:55:143
26865991810,6cyclictest31892-21ls23:00:153
26865991810,5cyclictest31427-21cpu19:20:143
26865991810,5cyclictest11946-21grep19:50:133
2686499188,6cyclictest2891-21threads23:50:242
2686499188,3cyclictest141rcu_preempt21:00:182
2686499184,7cyclictest6062-21apt-get23:05:132
2686499183,4cyclictest5380-21apt-get20:50:132
2686499182,11cyclictest8132-21apt-get23:55:122
2686499182,11cyclictest7578-21apt-get19:40:132
2686499181,7cyclictest7864-21ssh21:53:572
2686499181,7cyclictest7864-21ssh21:53:562
26864991813,3cyclictest2774-21sh22:38:172
26864991813,3cyclictest2774-21sh22:38:172
2686499181,14cyclictest27407-21df20:25:142
26864991811,4cyclictest21060-21sh21:15:152
26864991810,6cyclictest5715-21cat22:40:182
2686399187,9cyclictest0-21swapper/122:00:151
2686399182,11cyclictest20840-21apt-get21:15:131
26863991813,3cyclictest7273-21dump-pmu-power20:59:551
26863991813,3cyclictest25079-21/usr/sbin/munin22:55:121
26863991812,4cyclictest19875-21sh21:14:281
26863991811,5cyclictest4335-21awk00:15:251
26863991810,6cyclictest560-21dbus-daemon19:50:011
26863991810,6cyclictest3278-21cat21:25:241
26863991810,6cyclictest13748-21cat21:10:011
26863991810,5cyclictest29561-21diskstats20:30:161
2686299186,2cyclictest141rcu_preempt19:14:020
2686599172,11cyclictest27086-21apt-get20:25:123
2686599171,11cyclictest31364-21apt-get20:35:143
2686599171,11cyclictest18508-21apt-get20:05:183
26865991710,5cyclictest25170-21ls20:20:143
26865991710,5cyclictest20793-21cat20:10:133
2686499178,6cyclictest278-21systemd-journal20:17:002
2686499178,6cyclictest278-21systemd-journal20:17:002
26864991711,4cyclictest21835-21sed20:10:242
26864991711,4cyclictest13116-21sed21:05:252
26864991711,4cyclictest10417-21sed19:45:162
26864991710,5cyclictest16783-21latency00:25:162
2686399179,6cyclictest5866-21grep20:50:161
2686399179,6cyclictest3304-21cat20:45:101
2686399178,6cyclictest24915-21acpi20:20:101
2686399174,11cyclictest22-21ksoftirqd/120:00:121
26863991713,2cyclictest32522-21python320:35:241
26863991711,4cyclictest636-21runrttasks23:30:071
26863991711,4cyclictest28962-21cron20:30:001
26863991711,4cyclictest27369-21gltestperf19:10:161
26863991710,5cyclictest9239-21seq19:42:171
26863991710,5cyclictest23329-21expr20:15:171
26863991710,5cyclictest23329-21expr20:15:161
2686299175,3cyclictest141rcu_preempt20:10:120
26865991611,3cyclictest3214-21/usr/sbin/munin20:45:103
2686499168,6cyclictest3705-21gltestperf20:45:132
2686499162,11cyclictest1639-21cut20:40:152
2686499162,10cyclictest640-21snmpd20:55:252
26864991611,3cyclictest29425-21ntp_states00:10:222
26864991611,3cyclictest29200-21/usr/sbin/munin20:30:242
26864991611,3cyclictest19218-21latency20:05:242
26864991610,4cyclictest636-21runrttasks20:21:012
2686399161,13cyclictest19977-21sed20:05:301
2686499158,5cyclictest1562-21tune2fs19:25:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional