You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-01 - 07:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Mon Aug 31, 2026 12:45:17)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
253932173120,45sleep20-21swapper/207:09:032
25242217096,24sleep00-21swapper/007:07:050
253002162108,29sleep10-21swapper/107:07:501
253642160109,29sleep30-21swapper/307:08:403
692021190,2sleep30-21swapper/310:16:033
140692750,3chrt0-21swapper/009:34:460
269262660,2sleep30-21swapper/309:44:003
34682630,1chrt3467-21/usr/sbin/munin11:25:162
272572630,2sleep00-21swapper/009:20:130
56162590,3sleep012-21ksoftirqd/008:50:240
46802580,2sleep10-21swapper/111:02:251
127312560,3sleep00-21swapper/011:55:200
47772550,2sleep30-21swapper/309:50:423
14532540,1sleep10-21swapper/110:35:471
124682530,1sleep10-21swapper/111:31:171
112722500,3sleep122-21ksoftirqd/110:20:061
11212500,1sleep00-21swapper/007:25:250
136112450,2sleep00-21swapper/011:09:290
281272440,1sleep30-21swapper/308:30:013
183392430,2sleep10-21swapper/108:05:191
253072420,1sleep125308-21ssh12:28:481
215832400,1chrt0-21swapper/210:27:052
45222370,1chrt0-21swapper/212:13:282
35302360,2chrt0-21swapper/312:12:283
261462330,2chrt0-21swapper/009:19:530
19562330,8sleep012-21ksoftirqd/011:24:340
19562330,8sleep012-21ksoftirqd/011:24:340
193552330,8sleep012-21ksoftirqd/010:01:390
180212330,3sleep00-21swapper/009:13:310
76192320,2chrt0-21swapper/009:30:010
26402320,2chrt0-21swapper/311:25:013
25743993216,8cyclictest636-21runrttasks12:20:110
170372320,2chrt0-21swapper/010:24:180
6042310,1sleep10-21swapper/112:34:421
81552300,2chrt0-21swapper/209:54:022
274472290,3sleep327446-21ssh10:55:193
2574599291,6cyclictest4055-21cron07:35:002
2574399285,10cyclictest12-21ksoftirqd/012:14:080
2574499271,24cyclictest0-21swapper/107:17:481
25743992723,3cyclictest131rcuc/010:16:210
2574399270,8cyclictest12-21ksoftirqd/012:00:180
2574399270,25cyclictest0-21swapper/007:20:070
12442270,4sleep112690-21diskmemload12:10:261
25746992616,7cyclictest12208-21latency10:20:193
2574699261,23cyclictest0-21swapper/307:17:533
25745992611,12cyclictest5874-21awk08:50:252
25744992624,1cyclictest0-21swapper/107:23:021
25744992613,11cyclictest22-21ksoftirqd/109:10:001
25743992621,3cyclictest131rcuc/012:15:080
25743992614,3cyclictest12-21ksoftirqd/011:39:100
2574399261,1cyclictest0-21swapper/007:16:130
88522250,3sleep20-21swapper/207:45:142
25745992517,6cyclictest22457-21sh10:04:442
2574599250,6cyclictest7761-21ssh11:05:122
2574599250,23cyclictest0-21swapper/207:16:042
2574499250,24cyclictest0-21swapper/109:04:461
2574399257,8cyclictest141rcu_preempt11:32:230
2574399254,5cyclictest12-21ksoftirqd/010:05:210
25743992510,9cyclictest12-21ksoftirqd/010:11:370
25743992510,6cyclictest12-21ksoftirqd/008:01:160
25743992510,5cyclictest12-21ksoftirqd/011:26:310
2574399250,6cyclictest26935-21ssh12:05:250
2574399250,10cyclictest32179-21ssh12:33:500
186282250,2sleep20-21swapper/211:35:392
99702240,1sleep10-21swapper/111:05:531
25746992422,1cyclictest0-21swapper/307:20:423
25745992416,6cyclictest18084-21munin-run12:00:002
2574599240,1cyclictest0-21swapper/207:20:342
2574399247,9cyclictest12-21ksoftirqd/007:48:080
2574399246,9cyclictest12-21ksoftirqd/010:41:050
2574399245,8cyclictest12-21ksoftirqd/010:34:340
2574399245,7cyclictest12-21ksoftirqd/010:25:120
2574399245,11cyclictest12-21ksoftirqd/012:27:410
2574399243,6cyclictest12-21ksoftirqd/009:40:210
2574399242,3cyclictest12-21ksoftirqd/011:50:000
2574399241,7cyclictest131rcuc/011:17:240
2574399241,7cyclictest12-21ksoftirqd/010:35:120
25743992417,5cyclictest12-21ksoftirqd/009:00:000
25743992416,6cyclictest12-21ksoftirqd/008:00:010
25746992310,4cyclictest34-21ksoftirqd/312:30:203
2574699230,12cyclictest30471-21tr08:35:133
2574599237,4cyclictest66050lldpd09:46:492
2574599233,4cyclictest28-21ksoftirqd/212:15:232
25745992314,6cyclictest16273-21ntp_states08:00:212
25745992313,7cyclictest23271-21users09:40:252
2574499232,16cyclictest14752-21ssh11:10:091
2574399239,8cyclictest12-21ksoftirqd/010:48:220
2574399237,9cyclictest12-21ksoftirqd/007:52:380
2574399235,7cyclictest12-21ksoftirqd/011:03:010
2574399232,7cyclictest12-21ksoftirqd/011:40:240
25743992319,3cyclictest131rcuc/011:10:490
25743992319,3cyclictest131rcuc/010:50:180
25743992319,3cyclictest131rcuc/009:50:380
25743992317,4cyclictest12-21ksoftirqd/008:25:000
2574399230,7cyclictest9982-21ssh09:55:150
2574399230,6cyclictest4380-21file07:35:130
100452230,3sleep210022-21ssh11:30:152
2574699222,15cyclictest25344-21sshd12:05:123
25746992212,7cyclictest2489-21cut12:35:213
25746992210,3cyclictest210262latency12:25:193
2574499224,8cyclictest6242-21cron08:55:001
2574399228,4cyclictest12-21ksoftirqd/009:45:190
2574399225,8cyclictest12-21ksoftirqd/012:35:340
2574399221,7cyclictest12-21ksoftirqd/008:15:120
25743992212,2cyclictest141rcu_preempt08:33:230
25743992210,3cyclictest12-21ksoftirqd/010:56:030
2574399220,7cyclictest12-21ksoftirqd/011:50:190
218402220,10sleep334-21ksoftirqd/311:38:493
8942215,2sleep3141rcu_preempt11:46:293
2574699219,8cyclictest17487-21apt-get08:05:123
2574699212,14cyclictest14643-21apt-get11:10:043
2574699211,6cyclictest0-21swapper/308:45:013
2574699211,6cyclictest0-21swapper/308:00:243
2574699211,6cyclictest0-21swapper/307:50:193
2574699211,2cyclictest141rcu_preempt12:20:003
25746992112,6cyclictest4137-21basename08:50:003
25746992111,6cyclictest53050irq/142-enp3s0-10:45:133
2574599213,15cyclictest2587-21grep09:25:232
2574599211,6cyclictest20664-21latency09:15:172
25745992115,4cyclictest23194-21ls11:15:272
25745992112,7cyclictest560-21dbus-daemon09:35:002
25745992112,7cyclictest10942-21cron10:20:002
2574599210,7cyclictest66050lldpd07:52:472
2574499213,16cyclictest3344-21gltestperf09:50:151
25744992114,4cyclictest3874-21latency11:25:201
25744992113,6cyclictest3118-21ntp_kernel_err08:45:201
25744992113,6cyclictest10625-21ntpq09:55:201
25744992112,7cyclictest30539-21diskstats10:10:141
25744992111,6cyclictest22-21ksoftirqd/111:55:131
25744992110,3cyclictest141rcu_preempt10:05:001
2574399216,7cyclictest12-21ksoftirqd/008:11:140
2574399215,8cyclictest131rcuc/009:00:190
2574399211,7cyclictest12-21ksoftirqd/009:35:550
2574699209,7cyclictest939-21apt-get11:00:113
2574699203,4cyclictest0-21swapper/312:24:543
2574699203,12cyclictest7909-21apt-get09:30:103
2574699203,12cyclictest28393-21apt-get09:45:133
2574699201,6cyclictest10557-21/usr/sbin/munin09:55:203
2574699201,6cyclictest0-21swapper/311:52:583
2574699201,5cyclictest0-21swapper/311:57:073
25746992013,5cyclictest34-21ksoftirqd/309:30:003
25746992012,5cyclictest26548-21ssh10:06:513
25746992012,5cyclictest21959-21sh10:04:143
25746992012,5cyclictest1213-21unixbench_multi07:25:263
25746992011,6cyclictest53050irq/142-enp3s0-11:40:123
25746992011,6cyclictest16244-21basename11:35:003
25746992010,6cyclictest25401-21grep08:20:263
2574599209,7cyclictest12983-21apt-get07:55:132
2574599208,5cyclictest28-21ksoftirqd/212:30:452
2574599202,16cyclictest24482-21pmu-power10:05:202
2574599201,8cyclictest26653-21/usr/sbin/munin09:20:232
25745992014,4cyclictest27855-21sh10:31:332
25745992012,6cyclictest1403-21cut11:00:152
25745992012,6cyclictest13977-21expr12:20:152
25745992012,5cyclictest2283-21grep12:35:182
25745992011,7cyclictest8014-21users08:55:262
25745992011,6cyclictest30208-21cut11:45:012
2574599201,12cyclictest12690-21diskmemload11:11:362
25745992010,6cyclictest52750irq/139-enp3s0-10:55:132
2574499208,3cyclictest22-21ksoftirqd/111:35:591
2574499202,9cyclictest0-21swapper/107:35:131
2574499202,13cyclictest4540-21ssh10:15:111
25744992012,6cyclictest31557-21ntpq11:45:191
25744992012,6cyclictest27425-21cat08:25:241
25744992012,6cyclictest25046-21awk10:05:241
25744992012,6cyclictest23260-21seq12:03:221
25744992012,6cyclictest17030-21seq08:02:331
25744992011,7cyclictest26658-21wc10:55:141
25744992011,7cyclictest24602-21date10:30:001
25744992011,7cyclictest24209-21perl11:40:161
25744992011,7cyclictest19866-21expr08:10:131
25744992011,6cyclictest25168-21threads08:20:251
2574499201,10cyclictest20196-21cstates09:15:131
25744992010,7cyclictest278-21systemd-journal12:39:591
2574399203,5cyclictest12-21ksoftirqd/009:05:160
2574399201,7cyclictest12-21ksoftirqd/008:44:580
2574699194,8cyclictest13124-21apt-get09:10:123
2574699193,11cyclictest28433-21apt-get11:20:113
2574699193,11cyclictest28433-21apt-get11:20:113
2574699192,12cyclictest19871-21apt-get09:15:123
2574699192,12cyclictest18158-21ssh10:25:103
2574699191,5cyclictest0-21swapper/308:15:143
2574699191,4cyclictest0-21swapper/312:00:283
2574699191,4cyclictest0-21swapper/310:54:393
25746991913,4cyclictest31943-21sh10:10:273
25746991911,5cyclictest27042-21df09:20:113
25746991910,6cyclictest278-21systemd-journal07:14:563
25746991910,6cyclictest14127-21cron11:10:003
2574599199,4cyclictest28-21ksoftirqd/212:25:142
2574599198,8cyclictest30275-21cpuspeed_turbos10:10:122
2574599196,2cyclictest141rcu_preempt09:38:162
2574599195,11cyclictest19724-21cpuspeed_turbos08:10:132
2574599192,12cyclictest30546-21rm11:45:132
2574599192,10cyclictest31790-21taskset10:35:122
25745991914,3cyclictest6903-21sh11:50:402
25745991914,3cyclictest16719-21sh10:23:592
25745991914,3cyclictest15297-21sh10:46:022
25745991913,4cyclictest31580-21sh11:21:492
25745991913,4cyclictest31580-21sh11:21:482
25745991912,5cyclictest9769-21run-parts09:00:242
25745991912,5cyclictest21938-21cat08:15:142
25745991912,4cyclictest12690-21diskmemload12:01:112
25745991911,6cyclictest27159-21ls07:10:232
25745991911,6cyclictest26061-21cat08:25:112
25745991911,6cyclictest15501-21sh09:11:032
2574599191,12cyclictest141rcu_preempt09:55:122
2574499199,5cyclictest52850irq/140-enp3s0-09:40:121
2574499197,8cyclictest4928-21apt-get11:50:131
2574499197,3cyclictest141rcu_preempt09:45:251
2574499192,15cyclictest14866-21sh08:00:011
25744991913,4cyclictest14345-21idleruntime-cro09:35:011
25744991912,5cyclictest1973-21cron08:45:001
25744991912,5cyclictest14292-21seq09:10:221
25744991912,4cyclictest31698-21sh11:21:561
25744991912,4cyclictest31698-21sh11:21:561
25744991912,4cyclictest24661-21sh11:16:451
25744991911,6cyclictest8577-21cut07:45:011
25744991911,6cyclictest26233-21cat12:05:191
25744991911,6cyclictest16433-21head09:35:261
25744991911,6cyclictest11869-21ps07:50:231
2574399194,8cyclictest0-21swapper/008:39:140
2574399194,2cyclictest141rcu_preempt07:33:510
2574399193,9cyclictest12-21ksoftirqd/008:45:160
25743991913,3cyclictest26430-21interrupts07:10:160
2574399190,7cyclictest12-21ksoftirqd/007:40:220
2574699189,6cyclictest1-21systemd07:30:023
2574699187,7cyclictest31779-21apt-get10:35:123
2574699182,6cyclictest0-21swapper/309:05:003
2574699182,12cyclictest12983-21apt-get07:55:133
25746991813,3cyclictest26385-21switchtime10:30:253
25746991811,5cyclictest5212-21sed07:35:193
25746991811,4cyclictest21530-21idleruntime-cro08:15:003
2574699180,16cyclictest7410-21cut10:40:233
2574599183,2cyclictest141rcu_preempt12:05:122
2574599182,11cyclictest32513-21apt-get08:40:132
25745991811,5cyclictest7744-21ls10:40:262
25745991811,5cyclictest28390-21cat08:30:142
25745991811,5cyclictest20037-21expr10:50:142
25745991810,6cyclictest30710-21grep08:35:152
2574599180,16cyclictest2326-21expr08:45:142
2574499189,6cyclictest52850irq/140-enp3s0-12:20:111
2574499187,8cyclictest2170-21anacron07:30:131
2574499182,11cyclictest21764-21apt-get08:15:121
25744991813,3cyclictest2699-21sh09:25:241
25744991812,4cyclictest16251-21sh10:46:591
25744991811,5cyclictest9608-21seq10:42:081
2574499181,15cyclictest27115-21fschecks_count09:20:131
2574499181,14cyclictest6382-21ssh12:15:041
25744991810,6cyclictest25681-21cat10:30:191
25744991810,6cyclictest1314-21users07:25:271
2574399182,4cyclictest12-21ksoftirqd/008:05:270
25743991811,5cyclictest26162-21expr08:25:130
2574699171,5cyclictest0-21swapper/309:00:003
25746991713,3cyclictest53050irq/142-enp3s0-11:15:273
25746991711,4cyclictest8784-21cpuspeed_turbos07:45:143
25746991711,4cyclictest11489-21latency09:05:173
25746991711,3cyclictest12690-21diskmemload09:35:053
2574599179,5cyclictest5519-21mailstats07:35:232
2574599171,3cyclictest10804-21apt-get09:05:122
25745991710,5cyclictest6752-21expr07:40:132
25745991710,5cyclictest24115-21expr08:20:132
25745991710,5cyclictest1170-21cat07:25:252
2574499178,6cyclictest19743-21acpi10:50:121
25744991712,3cyclictest25758-21/usr/sbin/munin07:10:151
25744991710,5cyclictest8576-21date09:00:001
25744991710,5cyclictest28452-21expr08:30:141
2574699169,5cyclictest6686-21expr07:40:133
25746991611,3cyclictest5284-21ntp_offset08:50:203
25746991611,3cyclictest28191-21/usr/sbin/munin08:30:143
2574599161,2cyclictest0-21swapper/208:05:122
25744991611,3cyclictest8682-21/usr/sbin/munin07:45:131
25744991611,3cyclictest30113-21dump-pmu-power08:39:551
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional