You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-12 - 16:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue May 12, 2026 12:45:07)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
11980215999,42sleep10-21swapper/107:06:191
1481521480,1sleep30-21swapper/311:22:333
11957214697,41sleep30-21swapper/307:06:013
120182141105,25sleep20-21swapper/207:06:472
122252139103,25sleep00-21swapper/007:09:300
318312640,2sleep10-21swapper/109:33:591
41642600,1chrt0-21swapper/210:01:102
266252600,2sleep10-21swapper/107:40:201
266252600,2sleep10-21swapper/107:40:191
92472590,6sleep3331rcuc/310:05:203
263292590,2sleep20-21swapper/210:18:272
125502560,1sleep20-21swapper/212:10:042
153332550,2sleep10-21swapper/109:20:431
143922530,2sleep10-21swapper/110:33:511
67112520,1sleep20-21swapper/212:05:182
275582520,1sleep30-21swapper/311:07:453
1254199492,33cyclictest0-21swapper/109:55:131
245462480,2sleep10-21swapper/110:16:381
138272480,1sleep213828-21sh11:21:322
299592470,1sleep20-21swapper/209:56:122
12541994611,24cyclictest11064-21sh11:44:481
15702450,2sleep30-21swapper/311:36:493
18902440,2sleep10-21swapper/112:01:111
24442430,1sleep10-21swapper/108:00:191
12541994310,19cyclictest27662-21sh11:07:521
20382410,2sleep00-21swapper/011:37:190
150242410,1sleep00-21swapper/012:11:030
1254199371,5cyclictest12159-21ssh10:55:471
1254199362,26cyclictest17633-21apt-get11:25:131
1254199350,5cyclictest9767-21unixbench_multi08:15:251
1254199350,5cyclictest8035-21sh12:30:211
1254099353,5cyclictest12-21ksoftirqd/011:05:000
12541993412,19cyclictest29272-21cpuspeed_turbos10:45:131
1254199340,5cyclictest31693-21diskmemload10:40:251
1254199340,5cyclictest23585-21chrt11:53:491
1254199340,4cyclictest31472-21grep11:35:181
1254199340,3cyclictest0-21swapper/107:45:131
64032330,10sleep122-21ksoftirqd/110:27:301
1254199339,21cyclictest5438-21sed08:05:231
1254199338,22cyclictest14268-21grep12:35:191
12541993312,18cyclictest15638-21sed10:10:171
12541993311,20cyclictest13673-21awk12:10:181
1254199329,20cyclictest22109-21awk08:45:201
1254199327,22cyclictest24233-21netstat08:50:201
12541993216,13cyclictest15445-21kernelversion08:30:161
12541993216,13cyclictest15445-21kernelversion08:30:151
1254199321,4cyclictest0-21swapper/111:20:131
1254199320,5cyclictest31373-21timerwakeupswit07:50:261
1254199320,5cyclictest11257-21ssh10:06:381
1254199320,4cyclictest2329-21gltestperf10:00:171
127772310,3chrt12778-21apt-config09:20:002
1254199319,19cyclictest9011-21cat09:40:231
1254199313,22cyclictest25907-21apt-get12:20:111
1254199312,24cyclictest0-21swapper/107:15:111
1254199310,5cyclictest11866-21cstates11:45:141
1254199310,3cyclictest17864-21sshd10:35:381
12543993021,7cyclictest3824-21ssh10:00:483
1254199309,18cyclictest17813-21latency07:20:191
1254199308,19cyclictest21126-21sh07:30:001
1254199307,20cyclictest447-21latency07:55:171
1254199303,5cyclictest19262-21apt-get12:15:131
1254199301,7cyclictest22441-21grep09:50:251
1254199301,26cyclictest7596-21grep08:10:251
12541993010,18cyclictest22-21ksoftirqd/110:51:481
1254199300,4cyclictest29345-21latency10:20:171
1254099300,6cyclictest31703-21perf09:09:590
230042290,2chrt0-21swapper/211:53:152
1254199298,18cyclictest11263-21cut08:20:171
1254199295,3cyclictest0-21swapper/109:25:171
1254199294,3cyclictest0-21swapper/111:05:001
1254199294,22cyclictest0-21swapper/109:45:131
1254199291,4cyclictest0-21swapper/111:30:111
12541992911,16cyclictest4430-21sh09:37:191
12541992911,15cyclictest21836-21grep07:30:171
1254099297,10cyclictest12-21ksoftirqd/012:00:330
1254199288,17cyclictest142402chrt08:26:181
1254199287,5cyclictest20205-21tr08:40:221
1254199284,3cyclictest883-21cut12:25:171
1254199281,8cyclictest878-21ssh09:10:231
1254199281,8cyclictest7290-21/usr/sbin/munin12:05:241
1254199281,8cyclictest27440-21wc09:00:001
1254199281,7cyclictest6869-21fschecks_count09:15:161
1254199281,7cyclictest30901-21awk09:05:231
1254199281,7cyclictest25542-21sendmail-msp07:40:001
1254199281,7cyclictest18862-21sendmail-msp08:40:011
1254199281,6cyclictest31201-21rm11:10:201
1254199281,6cyclictest26451-21mailstats11:55:251
12543992716,7cyclictest31693-21diskmemload09:20:033
1254199275,5cyclictest12638-21cpuspeed_turbos07:10:131
1254199271,6cyclictest27757-21cpuspeed_turbos09:00:141
1254199270,8cyclictest5286-21ssh11:15:201
1254099271,24cyclictest0-21swapper/007:13:010
1254099270,7cyclictest10238-21munin-run08:20:000
257822260,2sleep30-21swapper/308:55:133
156142260,9chrt12-21ksoftirqd/010:59:190
1254099266,8cyclictest12-21ksoftirqd/010:09:440
1254099264,6cyclictest12-21ksoftirqd/010:30:200
1254099263,6cyclictest25196-21apt-get11:55:130
12540992618,6cyclictest12-21ksoftirqd/010:15:010
306812250,9sleep228-21ksoftirqd/212:23:332
12542992523,1cyclictest0-21swapper/207:13:002
1254099259,4cyclictest12-21ksoftirqd/011:05:550
1254099258,5cyclictest12-21ksoftirqd/011:50:150
1254099256,10cyclictest12-21ksoftirqd/011:45:340
1254099256,10cyclictest0-21swapper/011:40:280
1254099250,6cyclictest27405-21munin-run08:59:590
73142240,1sleep20-21swapper/211:40:592
12543992416,6cyclictest28294-21unixbench_singl09:30:253
12543992413,7cyclictest4224-21grep10:50:183
1254399240,23cyclictest0-21swapper/307:12:513
1254099247,4cyclictest27499-21apt-get09:55:110
1254099247,4cyclictest12-21ksoftirqd/012:35:230
1254099245,11cyclictest12-21ksoftirqd/012:25:170
1254099242,7cyclictest12-21ksoftirqd/009:10:270
1254099242,6cyclictest12-21ksoftirqd/009:50:210
12540992419,3cyclictest131rcuc/010:29:020
1254099241,7cyclictest12-21ksoftirqd/012:20:190
1254099241,6cyclictest24236-21apt-get11:30:130
12540992413,5cyclictest12-21ksoftirqd/012:30:390
1254099240,6cyclictest560-21dbus-daemon07:50:000
1254099240,6cyclictest31032-21ls11:10:190
188762230,2sleep228-21ksoftirqd/209:24:222
12543992315,6cyclictest32105-21cron12:25:013
1254299232,15cyclictest15054-21ssh10:10:142
12542992315,4cyclictest22929-21latency_hist11:05:012
12542992312,7cyclictest3911-21cut10:50:162
1254099239,4cyclictest12-21ksoftirqd/010:53:110
1254099237,8cyclictest12-21ksoftirqd/010:38:230
1254099236,9cyclictest12-21ksoftirqd/012:15:200
1254099236,10cyclictest12-21ksoftirqd/011:15:450
1254099235,7cyclictest12-21ksoftirqd/011:27:070
1254099232,4cyclictest12-21ksoftirqd/010:00:220
1254099232,14cyclictest26604-21rm09:30:120
12540992318,3cyclictest131rcuc/009:26:000
1254099231,7cyclictest12-21ksoftirqd/009:40:110
1254099231,6cyclictest14213-21apt-get09:45:120
12540992310,3cyclictest12-21ksoftirqd/009:16:180
1254099230,11cyclictest12-21ksoftirqd/009:22:020
6982220,6sleep228-21ksoftirqd/211:35:572
12543992213,6cyclictest15693-21memory07:15:193
12543992212,7cyclictest22135-21memory07:30:203
12543992212,7cyclictest20500-21rt-features12:15:223
1254399221,16cyclictest11681-21apt-get11:45:133
12543992210,8cyclictest19240-21apt-get07:25:153
1254299221,11cyclictest24512-21kworker/u8:211:05:122
1254099224,9cyclictest12-21ksoftirqd/010:48:040
1254099221,6cyclictest12-21ksoftirqd/011:21:110
1254099221,6cyclictest12-21ksoftirqd/010:21:190
1254099221,6cyclictest12-21ksoftirqd/010:17:400
1254099220,6cyclictest16834-21munin-run07:20:010
1254099220,5cyclictest7777-21users12:05:270
277952210,1sleep30-21swapper/310:19:563
1254399211,9cyclictest129782ssh09:44:123
12543992113,6cyclictest736-21ntp_kernel_pll_09:10:223
12543992113,6cyclictest3818-21cron08:05:013
12543992113,6cyclictest14029-21ls08:25:243
12543992113,6cyclictest1165-21grep07:55:233
12543992113,5cyclictest636-21runrttasks10:36:173
12543992112,7cyclictest24676-21ls08:50:253
12543992112,7cyclictest16961-21cat11:00:143
12543992112,6cyclictest278-21systemd-journal09:50:003
12542992115,4cyclictest31386-21munin-plugin-st12:00:002
12542992113,6cyclictest30133-21df_inode09:05:132
12542992113,6cyclictest21090-21awk09:25:212
1254299211,16cyclictest23917-21sed08:50:172
12542992111,6cyclictest10762-21wc10:30:252
12542992110,8cyclictest31693-21diskmemload09:45:222
12542992110,7cyclictest8294-21timerwakeupswit12:30:242
1254099215,3cyclictest141rcu_preempt08:10:010
1254099213,7cyclictest12-21ksoftirqd/008:35:100
12540992115,4cyclictest12-21ksoftirqd/007:45:000
12540992115,4cyclictest12-21ksoftirqd/007:45:000
12540992115,4cyclictest12-21ksoftirqd/007:34:590
12540992112,6cyclictest23975-21perl07:35:160
1254099210,6cyclictest19095-21apt-get08:40:120
77782200,2chrt0-21swapper/312:05:273
288842200,5sleep228-21ksoftirqd/209:31:002
1254399208,8cyclictest16934-21apt-get08:35:113
1254399202,13cyclictest908-21ssh09:35:133
1254399202,12cyclictest29208-21ssh10:45:123
12543992014,4cyclictest23661-21ntp_states10:40:213
12543992014,4cyclictest20794-21sh09:50:093
12543992012,6cyclictest7360-21sensors_temp08:10:223
12543992012,6cyclictest6623-21ls09:15:133
12543992012,6cyclictest5237-21tr11:40:133
12543992012,6cyclictest14875-21ls12:35:243
12543992011,6cyclictest31685-21cat11:10:243
12543992011,6cyclictest25476-21ls11:55:153
1254299202,14cyclictest75-21kswapd012:35:112
1254299202,13cyclictest882-21apt-get09:35:112
1254299202,12cyclictest4421-21apt-get11:15:122
12542992012,5cyclictest19144-21sh11:25:252
12542992011,6cyclictest9016-21cat09:40:242
12542992011,6cyclictest31798-21sed11:10:262
12542992011,5cyclictest52850irq/140-enp3s0-10:45:112
1254099205,8cyclictest0-21swapper/010:40:120
12540992013,5cyclictest30933-21ntp_kernel_err07:50:200
12540992012,6cyclictest29378-21seq09:02:570
12540992012,6cyclictest16294-21users08:30:240
12540992012,6cyclictest16294-21users08:30:230
1254099200,7cyclictest897-21ssh09:35:120
244692190,2sleep30-21swapper/311:54:443
1254399199,7cyclictest32543-21if_err_enp3s012:00:183
12543991914,3cyclictest29039-21sendmail_mailtr07:45:233
12543991911,6cyclictest740-21smartctl12:25:153
12543991911,6cyclictest24072-21smartctl07:35:183
12543991911,6cyclictest22252-21seq11:28:253
12543991911,6cyclictest21344-21expr08:45:123
12543991911,6cyclictest13561-21cat12:10:193
1254399191,15cyclictest25777-21apt-get07:40:133
1254399191,15cyclictest25777-21apt-get07:40:123
12543991911,5cyclictest21396-21cut10:15:003
1254399191,15cyclictest19648-21hddtemp_smartct08:40:163
1254399191,12cyclictest10092-21rm10:55:133
12543991910,7cyclictest560-21dbus-daemon11:20:003
12543991910,7cyclictest29757-21date09:05:013
12543991910,6cyclictest24810-21grep11:30:153
1254299192,12cyclictest640-21snmpd11:48:392
12542991912,4cyclictest31693-21diskmemload10:20:212
12542991911,6cyclictest21326-21sed07:30:002
12542991911,5cyclictest26478-21sed07:40:192
12542991911,5cyclictest26478-21sed07:40:182
12542991911,4cyclictest636-21runrttasks10:43:172
1254299191,14cyclictest21325-21fschecks_time09:50:162
12542991910,6cyclictest52850irq/140-enp3s0-12:00:132
1254099191,5cyclictest0-21swapper/008:10:210
12540991914,3cyclictest24022-21kernelversion08:50:190
12540991912,5cyclictest3386-21unixbench_singl08:00:270
1254399188,8cyclictest29087-21sed09:55:263
12543991813,3cyclictest14766-21/usr/sbin/munin08:30:123
12543991813,3cyclictest14766-21/usr/sbin/munin08:30:123
12543991812,4cyclictest10873-21unixbench-2d10:30:253
12543991811,5cyclictest6732-21find12:30:113
12543991811,5cyclictest52950irq/141-enp3s0-10:25:153
12543991811,4cyclictest31693-21diskmemload10:21:003
1254399180,8cyclictest20955-21ntp_kernel_err09:25:203
1254299188,7cyclictest19230-21/usr/sbin/munin12:15:262
1254299184,2cyclictest141rcu_preempt12:25:342
1254299183,12cyclictest31947-21dump-pmu-power07:55:002
1254299182,3cyclictest141rcu_preempt09:10:132
1254299182,11cyclictest10072-21apt-get10:55:122
12542991813,3cyclictest27641-21/usr/sbin/munin09:00:152
12542991811,5cyclictest6992-21cat08:10:182
12542991811,5cyclictest23204-21sed08:50:002
12542991811,5cyclictest16339-21unixbench_multi07:15:242
12542991810,6cyclictest9016-21wc10:30:002
12542991810,6cyclictest278-21systemd-journal07:34:032
12542991810,5cyclictest29803-21basename07:50:012
1254099181,6cyclictest12-21ksoftirqd/008:25:040
1254099181,6cyclictest12-21ksoftirqd/007:25:240
12540991813,4cyclictest12-21ksoftirqd/007:24:590
12540991813,3cyclictest10481-21/usr/sbin/munin08:20:180
12540991810,5cyclictest32160-21seq07:55:080
1254399178,6cyclictest8777-21tune2fs08:15:163
1254399178,6cyclictest17952-21needreboot07:20:203
1254399176,9cyclictest30184-21cut07:50:153
12543991710,5cyclictest10558-21awk08:20:113
1254299179,6cyclictest546-21sed07:55:192
1254299179,6cyclictest26873-21sed08:55:242
1254299179,6cyclictest15219-21cut08:30:142
1254299179,6cyclictest15219-21cut08:30:132
1254299178,5cyclictest52850irq/140-enp3s0-10:35:132
1254299171,5cyclictest0-21swapper/210:05:182
12542991712,3cyclictest24206-21/usr/sbin/munin11:30:232
12542991711,4cyclictest8896-21cut08:15:172
1254299171,12cyclictest278-21systemd-journal08:45:012
12542991710,5cyclictest14073-21awk08:25:262
12542991710,5cyclictest11269-21wc08:20:172
1254099175,3cyclictest141rcu_preempt08:50:000
1254399169,5cyclictest4821-21load08:05:173
1254399161,11cyclictest29804-21apt-get09:05:123
1254299168,6cyclictest17794-21latency07:20:192
1254299163,8cyclictest4072-21apt-get08:05:132
1254299162,7cyclictest169312sleep208:35:112
1254299151,10cyclictest2858-21ntp_states08:00:232
12542991510,3cyclictest23979-21gltestperf07:35:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional