You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-28 - 00:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Thu Aug 27, 2026 12:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17976216998,25sleep20-21swapper/207:05:072
197092167118,27sleep00-21swapper/007:07:500
197542138101,26sleep30-21swapper/307:08:253
196532137100,26sleep10-21swapper/107:07:061
319562600,2sleep00-21swapper/007:35:210
240632600,3sleep20-21swapper/211:21:402
19592590,3sleep01957-21ssh11:30:010
167102580,1sleep00-21swapper/010:05:170
56472570,2sleep00-21swapper/011:32:100
73252560,2sleep00-21swapper/009:10:110
35272530,2chrt0-21swapper/311:55:003
89242500,1sleep10-21swapper/111:58:391
96192470,1sleep20-21swapper/211:35:172
235212460,1sleep10-21swapper/112:09:271
160722430,2sleep10-21swapper/111:40:131
118882430,1sleep10-21swapper/110:01:171
69912420,2chrt0-21swapper/009:58:110
60152420,1sleep20-21swapper/209:57:122
136042400,2chrt0-21swapper/311:14:473
136042400,2chrt0-21swapper/311:14:473
214062390,2chrt0-21swapper/011:20:140
2015099381,16cyclictest15082-21awk09:40:200
2015099381,16cyclictest15082-21awk09:40:200
2015099381,14cyclictest21042-21cpuspeed_turbos09:45:140
262392360,9sleep122-21ksoftirqd/112:35:111
35702350,1sleep20-21swapper/209:31:482
2015099359,6cyclictest28029-21awk11:00:250
20150993510,6cyclictest16167-21ls08:15:140
20150993510,5cyclictest1131-21cat09:30:170
20150993411,3cyclictest4291-21unixbench-2d09:00:240
20150993410,6cyclictest29147-21date11:50:000
160682340,2sleep30-21swapper/310:05:123
20150993311,6cyclictest12-21ksoftirqd/010:20:120
325562320,2chrt0-21swapper/111:28:251
20150993213,3cyclictest28865-21/usr/sbin/munin08:45:250
20150993211,6cyclictest12-21ksoftirqd/012:20:240
20150993211,6cyclictest12-21ksoftirqd/009:25:120
20150993211,3cyclictest16583-21sh10:52:300
163082320,2sleep00-21swapper/009:15:530
227022310,2sleep20-21swapper/208:30:122
2015099312,10cyclictest30183-21hddtemp_smartct11:50:160
20150993111,3cyclictest6939-21diskmemload10:27:350
20150993111,3cyclictest11374-21sh09:37:470
102272310,1chrt0-21swapper/008:00:160
39332300,3sleep20-21swapper/211:06:552
221502300,3sleep21354-21cron11:45:002
2015099302,8cyclictest9920-21ssh10:00:160
2015099302,7cyclictest5045-21apt-get09:05:120
2015099301,10cyclictest640-21snmpd07:50:130
289762290,1sleep30-21swapper/310:37:593
20151992910,4cyclictest22-21ksoftirqd/111:51:121
2015099291,7cyclictest27982-21ssh12:12:090
2015099291,7cyclictest24300-21ssh10:58:260
20152992819,7cyclictest27380-21ntp_states11:00:202
2015199288,10cyclictest22-21ksoftirqd/111:13:361
2015199288,10cyclictest22-21ksoftirqd/111:13:361
2015199281,13cyclictest22-21ksoftirqd/111:48:451
2015099281,9cyclictest52950irq/141-enp3s0-12:35:100
127692280,3sleep228-21ksoftirqd/210:25:302
2015399271,24cyclictest0-21swapper/307:19:083
2015399271,24cyclictest0-21swapper/307:19:083
2015199275,11cyclictest22-21ksoftirqd/109:50:241
2015199274,12cyclictest22-21ksoftirqd/111:19:371
2015199274,10cyclictest22-21ksoftirqd/110:27:531
20151992711,5cyclictest22-21ksoftirqd/112:02:091
20151992711,4cyclictest22-21ksoftirqd/111:33:571
260482260,1sleep126051-21latency_hist11:00:001
253412260,3sleep225339-21ssh10:35:172
2015299260,23cyclictest23259-21ntp_states07:15:212
2015299260,23cyclictest23259-21ntp_states07:15:212
2015199268,4cyclictest22-21ksoftirqd/110:10:201
2015199261,24cyclictest0-21swapper/107:18:381
2015199261,24cyclictest0-21swapper/107:18:381
20150992620,5cyclictest0-21swapper/007:23:140
2015099261,2cyclictest0-21swapper/007:17:480
2015099261,2cyclictest0-21swapper/007:17:480
2015099261,10cyclictest13866-21apt-get08:10:100
133262260,2sleep30-21swapper/309:14:403
2015399251,1cyclictest0-21swapper/307:23:163
2015299257,12cyclictest141rcu_preempt12:26:522
2015299251,23cyclictest0-21swapper/207:20:102
2015199259,4cyclictest22-21ksoftirqd/111:06:041
2015199257,2cyclictest141rcu_preempt10:35:351
20151992523,1cyclictest0-21swapper/107:22:581
20151992510,4cyclictest22-21ksoftirqd/110:47:381
20151992510,4cyclictest22-21ksoftirqd/109:49:481
288092240,2sleep30-21swapper/312:12:583
2015299243,15cyclictest16097-21ssh10:05:132
2015299242,15cyclictest10792-21apt-get12:00:032
2015199248,9cyclictest22-21ksoftirqd/110:15:411
2015199248,2cyclictest141rcu_preempt09:40:351
2015199248,2cyclictest141rcu_preempt09:40:351
2015199246,3cyclictest141rcu_preempt10:34:491
2015199245,6cyclictest22-21ksoftirqd/109:30:121
2015199245,11cyclictest22-21ksoftirqd/112:33:481
2015199244,2cyclictest141rcu_preempt10:40:091
2015199242,10cyclictest0-21swapper/112:20:451
20151992418,4cyclictest22-21ksoftirqd/110:00:001
20151992414,3cyclictest141rcu_preempt09:23:211
20151992411,4cyclictest22-21ksoftirqd/109:16:021
2015199240,11cyclictest22-21ksoftirqd/110:07:511
20150992414,7cyclictest560-21dbus-daemon08:40:000
20153992314,7cyclictest16117-21df_inode11:40:143
2015299232,14cyclictest12776-21sh10:50:132
20152992313,7cyclictest278-21systemd-journal07:30:002
2015199239,8cyclictest22-21ksoftirqd/112:12:001
2015199239,4cyclictest22-21ksoftirqd/112:17:251
2015199239,2cyclictest141rcu_preempt11:35:531
2015199238,4cyclictest22-21ksoftirqd/110:23:381
2015199238,2cyclictest141rcu_preempt10:51:461
2015199232,9cyclictest22-21ksoftirqd/109:39:121
20151992314,7cyclictest5376-21df_inode09:05:141
281922220,1sleep30-21swapper/311:49:063
2015399229,11cyclictest16696-21ssh09:16:163
20153992215,4cyclictest10324-21idleruntime-cro10:25:003
2015399221,12cyclictest11294-21expr12:00:133
2015299226,4cyclictest28-21ksoftirqd/212:15:202
2015299222,13cyclictest17767-21ssh10:30:032
20152992215,5cyclictest28-21ksoftirqd/211:35:002
20152992214,5cyclictest6939-21diskmemload09:51:182
20152992214,5cyclictest19626-21grep08:20:252
2015199229,5cyclictest22-21ksoftirqd/108:45:191
2015199226,6cyclictest22-21ksoftirqd/112:25:421
2015199225,3cyclictest141rcu_preempt09:11:131
2015199224,10cyclictest22-21ksoftirqd/111:01:141
20151992215,5cyclictest22-21ksoftirqd/109:30:001
2015099222,14cyclictest17766-21sh10:30:030
326292210,11sleep334-21ksoftirqd/312:15:233
2015399217,12cyclictest0-21swapper/310:00:163
20153992113,6cyclictest29163-21diskstats07:30:143
20153992111,8cyclictest9372-21tune2fs11:35:153
20153992111,7cyclictest4310-21sensors_temp07:45:233
2015399210,6cyclictest512-21perf07:40:013
2015399210,5cyclictest6939-21diskmemload10:10:143
2015299213,4cyclictest0-21swapper/211:56:402
2015299211,7cyclictest0-21swapper/212:11:392
2015299211,2cyclictest141rcu_preempt07:55:212
20152992112,7cyclictest20966-21sed12:30:252
2015199217,8cyclictest22-21ksoftirqd/108:10:121
2015199216,5cyclictest17847-21chrt08:19:361
20151992114,5cyclictest27190-21perl08:40:151
20151992114,4cyclictest22505-21cpuspeed_turbos08:30:121
2015199211,2cyclictest141rcu_preempt11:21:291
2015099219,7cyclictest52950irq/141-enp3s0-10:10:130
2015099211,8cyclictest193342sh11:18:440
20150992114,5cyclictest13847-21basename11:15:000
20150992114,5cyclictest13847-21basename11:15:000
20150992113,5cyclictest636-21runrttasks10:41:540
2015399203,4cyclictest141rcu_preempt11:25:073
20153992012,6cyclictest444-21cat10:40:223
20153992012,6cyclictest32403-21ls08:50:253
20153992012,6cyclictest28117-21cat08:40:243
20153992012,6cyclictest26621-21cat07:25:113
20153992012,6cyclictest22337-21cat09:45:253
20153992012,5cyclictest12626-21cat12:25:113
2015399201,16cyclictest21365-21ntpq09:20:193
20153992011,6cyclictest19924-21cat10:55:143
20153992011,6cyclictest15897-21cat08:15:003
20153992010,7cyclictest3892-21pmu-power09:00:213
20153992010,7cyclictest17909-21apt-get10:30:123
2015299205,13cyclictest141rcu_preempt09:43:132
2015299205,13cyclictest141rcu_preempt09:43:132
2015299203,5cyclictest14106-21apt-get09:15:122
20152992015,3cyclictest20961-21interrupts08:25:172
20152992013,4cyclictest4779-21cron09:05:002
20152992013,4cyclictest3511-21latency_hist10:20:002
20152992012,6cyclictest7876-21sed09:35:162
20152992012,6cyclictest15761-21pluginstate08:15:002
20152992012,6cyclictest12835-21tr08:05:212
20152992012,6cyclictest12835-21tr08:05:212
20152992012,5cyclictest6939-21diskmemload12:21:082
20152992011,7cyclictest20148-21cyclictest10:45:502
20152992011,5cyclictest8264-21missed_timers09:10:202
2015299201,12cyclictest7481-21chrt11:10:122
2015299201,12cyclictest7481-21chrt11:10:122
20152992010,8cyclictest25984-21cut11:00:002
2015199208,3cyclictest141rcu_preempt07:55:121
2015199204,7cyclictest5269-21chrt07:50:111
20151992015,3cyclictest211rcuc/107:36:091
20151992014,4cyclictest22-21ksoftirqd/108:55:001
20151992012,6cyclictest13583-21seq08:09:481
20151992012,6cyclictest13583-21seq08:09:481
20151992011,6cyclictest1512-21sed07:40:171
20151992011,3cyclictest141rcu_preempt07:29:511
2015199201,11cyclictest22-21ksoftirqd/108:27:381
2015099205,5cyclictest12-21ksoftirqd/008:55:260
2015099202,12cyclictest19486-21ssh12:30:130
2015099202,11cyclictest515-21irqbalance08:50:110
20150992011,7cyclictest8274-21grep07:55:200
20150992010,7cyclictest24677-21apt-get10:35:120
44642190,2sleep00-21swapper/007:45:240
2015399199,6cyclictest52750irq/139-enp3s0-11:00:113
2015399191,6cyclictest0-21swapper/309:34:283
20153991914,3cyclictest5559-21grep09:05:163
20153991913,4cyclictest9032-21sh11:58:463
20153991913,4cyclictest3482-21sh10:19:583
20153991912,5cyclictest21073-21ntp_kernel_err07:10:193
20153991911,6cyclictest5757-21fw_packets07:50:153
20153991911,6cyclictest18166-21seq10:54:053
20153991911,6cyclictest11396-21grep10:25:153
20153991911,6cyclictest11263-21seq08:02:263
20153991910,7cyclictest560-21dbus-daemon08:00:003
20153991910,7cyclictest2482-21apt-get09:55:113
20153991910,7cyclictest2245-21cat11:30:133
20153991910,6cyclictest17122-21mailstats08:15:243
20153991910,6cyclictest1201-21tune2fs07:40:143
2015399190,17cyclictest31010-21rm12:38:263
2015399190,17cyclictest1981-21run-parts11:05:223
2015299191,3cyclictest6939-21diskmemload11:53:002
20152991913,4cyclictest32578-21cron12:40:002
20152991911,6cyclictest397-21ls10:40:222
20152991911,6cyclictest28869-21ntpq11:25:202
20152991911,6cyclictest27551-21wc09:25:242
20152991911,6cyclictest25508-21ntp_states08:35:202
20152991911,6cyclictest2539-21seq08:59:002
20152991911,6cyclictest23676-21sed11:45:192
20152991911,6cyclictest18264-21perl12:05:152
2015199199,4cyclictest22-21ksoftirqd/107:33:131
2015199198,3cyclictest141rcu_preempt08:00:171
2015199197,2cyclictest141rcu_preempt09:00:151
2015199193,14cyclictest25150-21smartctl08:35:151
2015199192,9cyclictest22-21ksoftirqd/108:57:031
2015199191,6cyclictest22-21ksoftirqd/107:10:111
2015099199,6cyclictest52950irq/141-enp3s0-12:15:180
2015099191,7cyclictest4743-21ssh11:55:190
20150991917,1cyclictest0-21swapper/007:25:060
20150991912,5cyclictest2151-21latency_hist09:55:010
20150991911,6cyclictest28725-21sed08:45:000
20150991911,6cyclictest10783-21seq12:00:030
20150991910,7cyclictest12-21ksoftirqd/009:25:010
2015399188,7cyclictest21052-21find11:20:113
2015399181,7cyclictest14420-21df09:40:143
2015399181,7cyclictest14420-21df09:40:143
20153991813,3cyclictest25659-21processes08:35:213
20153991813,3cyclictest19583-21sh11:18:593
20153991812,4cyclictest6122-21tr10:45:133
20153991812,4cyclictest17703-21/usr/sbin/munin12:05:143
2015399181,15cyclictest31116-21ssh09:51:533
2015399181,13cyclictest18812-21apt-get12:30:033
20153991810,6cyclictest5984-21expr12:20:143
20153991810,6cyclictest22394-21cut08:30:013
20153991810,6cyclictest12158-21tune2fs08:05:143
20153991810,6cyclictest12158-21tune2fs08:05:143
2015399180,16cyclictest27483-21sendmail_mailtr09:25:233
2015299189,6cyclictest278-21systemd-journal07:36:082
2015299185,10cyclictest278-21systemd-journal07:50:082
2015299182,13cyclictest14726-21grep11:15:162
2015299182,11cyclictest20194-21apt-get07:10:112
2015299182,10cyclictest10566-21kworker/u8:210:00:112
2015299181,8cyclictest23241-21ssh09:46:022
2015299181,5cyclictest23035-21cpuspeed_turbos10:10:132
20152991813,3cyclictest22811-21sh09:22:062
20152991811,5cyclictest28-21ksoftirqd/207:50:002
20152991811,4cyclictest6939-21diskmemload10:20:242
20152991810,6cyclictest26740-21find08:40:112
2015199187,4cyclictest22-21ksoftirqd/107:49:181
20151991810,5cyclictest19159-21grep08:20:211
2015099182,10cyclictest515-21irqbalance07:40:120
2015099181,7cyclictest11704-21chrt10:49:240
20150991813,3cyclictest19089-21switchtime12:05:230
20150991812,3cyclictest19930-21sh11:42:510
20150991811,4cyclictest13612-21idleruntime-cro08:10:000
20150991811,4cyclictest13612-21idleruntime-cro08:10:000
20150991811,4cyclictest12395-21/usr/sbin/munin12:25:190
20150991810,6cyclictest9030-21cat11:35:130
20150991810,6cyclictest560-21dbus-daemon07:35:000
20150991810,6cyclictest29950-21df10:15:140
20150991810,6cyclictest1999-21mailstats11:05:220
2015399179,6cyclictest19490-21cat08:20:243
20153991712,3cyclictest1071-21sh08:55:143
20153991711,4cyclictest8248-21ntp_kernel_pll_09:35:193
2015399171,14cyclictest29295-21fschecks_time08:45:153
20153991710,5cyclictest23016-21hostname08:30:163
2015299178,6cyclictest17900-21cat08:20:002
20152991711,4cyclictest1600-21wc07:40:182
20152991710,5cyclictest31256-21expr08:50:132
2015099179,6cyclictest20288-21expr07:10:120
20150991712,3cyclictest19338-21switchtime08:20:220
20150991711,4cyclictest23653-21switchtime08:30:230
20150991711,4cyclictest21212-21ntp_kernel_pll_08:25:200
2015299167,6cyclictest5197-21tr09:05:132
2015299161,1cyclictest141rcu_preempt08:05:002
20152991610,4cyclictest29921-21mailstats07:30:222
2015299151,2cyclictest141rcu_preempt08:45:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional