You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-08 - 14:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Tue Sep 08, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
316002238161,25sleep00-21swapper/007:05:090
9772221192,19sleep10-21swapper/107:07:281
9992208132,25sleep20-21swapper/207:07:452
9322203167,23sleep30-21swapper/307:06:543
1274998576,8cyclictest0-21swapper/212:05:052
1274998178,2cyclictest0-21swapper/211:10:492
1274998178,2cyclictest0-21swapper/210:29:582
24252610,4sleep121-21rcuc/109:35:241
101222450,3sleep310124-21cpuspeed_turbos09:55:133
1262993511,11cyclictest26763-21idleruntime-cro09:20:020
218232330,4sleep011-21rcuc/009:05:250
126299326,12cyclictest7-21ksoftirqd/009:35:010
1267993110,18cyclictest6498-21expr11:00:151
1267993110,18cyclictest6498-21expr11:00:141
126299302,21cyclictest31081-21kworker/0:111:55:570
126299301,10cyclictest32664-21kworker/0:108:22:070
1274992825,2cyclictest0-21swapper/208:31:372
126299280,13cyclictest24565-21awk09:15:010
126799277,12cyclictest12050irq/123-ahci[0010:51:201
126799263,17cyclictest0-21swapper/112:34:511
126299262,16cyclictest8702-21kworker/0:212:24:570
127899251,11cyclictest31528-21sshd10:42:163
127899251,11cyclictest28717-21sshd09:22:153
1267992515,3cyclictest329-21kernelversion10:45:171
126799251,16cyclictest16922-21cstates10:10:171
126299257,15cyclictest6345-21munin-run07:20:010
126299256,16cyclictest4433-21munin-run12:10:010
126299251,12cyclictest5383-21munin-run09:45:010
127899247,3cyclictest39-21rcuc/311:01:253
127899247,3cyclictest39-21rcuc/311:01:253
127899241,16cyclictest0-21swapper/310:09:593
127899241,16cyclictest0-21swapper/308:50:253
127899241,11cyclictest21453-21ntp_states09:05:223
127899241,11cyclictest1273-21sshd10:46:163
1267992414,3cyclictest25156-21timerandwakeup11:40:251
126799241,16cyclictest19769-21entropy11:30:141
126799241,16cyclictest0-21swapper/111:28:071
126799241,16cyclictest0-21swapper/107:20:271
126299242,7cyclictest23392-21kworker/0:012:15:250
126299242,7cyclictest23392-21kworker/0:011:52:170
127899231,7cyclictest23582-21latency_hist11:40:023
127499236,9cyclictest32-21ksoftirqd/209:21:302
127499231,9cyclictest244272grep11:40:162
126799236,4cyclictest0-21swapper/110:06:261
126799231,19cyclictest13497-21df07:35:121
126299232,7cyclictest5667-21kworker/0:011:07:250
126299231,7cyclictest28702-21kworker/0:109:25:190
127899221,6cyclictest0-21swapper/310:30:333
127899221,6cyclictest0-21swapper/309:29:343
127899221,4cyclictest0-21swapper/312:02:073
127899221,13cyclictest0-21swapper/309:00:213
127899221,12cyclictest0-21swapper/311:27:573
127899221,12cyclictest0-21swapper/309:42:073
126799229,3cyclictest27985-21munin-run11:50:011
126799221,8cyclictest0-21swapper/112:17:031
126799221,8cyclictest0-21swapper/109:16:261
126799221,7cyclictest0-21swapper/110:36:561
1267992215,2cyclictest0-21swapper/111:35:041
1267992214,3cyclictest0-21swapper/110:42:461
126799221,3cyclictest0-21swapper/110:19:301
126799221,16cyclictest0-21swapper/108:20:571
126799221,12cyclictest7808-21/usr/sbin/munin09:50:121
126799221,12cyclictest7808-21/usr/sbin/munin09:50:121
126799221,11cyclictest14188-21munin-run10:05:011
126799221,11cyclictest0-21swapper/112:02:571
1267992210,4cyclictest30744-21df11:55:131
126799220,20cyclictest0-21swapper/109:04:131
126799220,15cyclictest0-21swapper/107:55:201
126299229,2cyclictest31729-21kworker/0:210:53:350
126299228,2cyclictest29448-21kworker/0:110:43:030
126299222,6cyclictest15045-21kworker/0:208:08:150
126299221,9cyclictest32664-21kworker/0:108:25:290
126299221,9cyclictest1451-21kworker/0:207:14:230
126299221,7cyclictest21077-21kworker/0:211:41:130
126299221,11cyclictest22351-21diskmemload12:35:020
126299221,10cyclictest30435-21kworker/0:008:17:230
158732210,3sleep00-21swapper/011:20:180
127899217,12cyclictest0-21swapper/309:35:253
127899212,15cyclictest23812-21/usr/sbin/munin11:40:193
127899211,7cyclictest0-21swapper/312:25:263
127899211,6cyclictest0-21swapper/310:35:333
127899211,5cyclictest4654-21/usr/sbin/munin12:10:183
127899211,5cyclictest4654-21/usr/sbin/munin12:10:183
127899211,12cyclictest0-21swapper/311:20:283
127899211,12cyclictest0-21swapper/307:26:213
127499212,9cyclictest0-21swapper/210:10:312
127499211,9cyclictest0-21swapper/211:03:312
127499211,9cyclictest0-21swapper/211:03:312
127499211,6cyclictest0-21swapper/209:25:212
127499211,5cyclictest0-21swapper/210:55:142
127499211,13cyclictest0-21swapper/211:08:242
126799219,3cyclictest12146-21awk11:10:251
126799213,2cyclictest26703-21kworker/u8:009:21:531
126799212,5cyclictest20569-21irqstats07:50:161
126799211,8cyclictest0-21swapper/112:20:321
126799211,6cyclictest0-21swapper/110:56:401
1267992114,2cyclictest0-21swapper/109:41:121
1267992113,3cyclictest0-21swapper/111:51:141
126799211,2cyclictest0-21swapper/112:08:001
126799211,15cyclictest0-21swapper/112:27:341
126799211,15cyclictest0-21swapper/109:46:211
126799211,15cyclictest0-21swapper/109:07:011
126799211,15cyclictest0-21swapper/108:12:471
126799211,15cyclictest0-21swapper/108:05:061
126799211,11cyclictest4055-21smart_sda08:25:241
126799211,11cyclictest25322-21latency_hist10:30:011
126799211,11cyclictest24971-21kernelversion08:00:181
126799211,11cyclictest10048-21gltestperf08:40:181
126799211,11cyclictest10048-21gltestperf08:40:181
126799211,10cyclictest0-21swapper/111:05:301
126799210,16cyclictest0-21swapper/109:30:391
126299219,2cyclictest16308-21kworker/0:010:38:430
126299217,6cyclictest870-21kworker/0:009:50:030
126299217,6cyclictest870-21kworker/0:009:50:030
126299217,3cyclictest11798-21kworker/0:110:04:200
126299216,3cyclictest16308-21kworker/0:010:32:250
126299213,2cyclictest18856-21kworker/0:011:28:130
126299213,2cyclictest0-21swapper/009:21:380
126299211,8cyclictest870-21kworker/0:009:47:010
126299211,8cyclictest193332fschecks_time10:15:170
126299211,7cyclictest870-21kworker/0:009:55:430
126299211,12cyclictest26009-21/usr/sbin/munin11:45:250
126299211,12cyclictest0-21swapper/010:24:340
126299210,9cyclictest870-21kworker/0:009:39:250
126299210,10cyclictest2104-21kworker/0:110:55:190
127899202,5cyclictest2800-21df_inode12:05:133
127899201,9cyclictest87350irq/126-enp1s0-11:18:493
127899201,6cyclictest10093-21awk12:20:183
127899201,6cyclictest0-21swapper/312:15:503
1278992014,4cyclictest41-21ksoftirqd/308:30:013
127899201,2cyclictest87650irq/129-enp1s0-09:10:163
1278992010,8cyclictest16592-21find07:40:243
127499207,10cyclictest28799-21grep11:50:152
127499205,7cyclictest87650irq/129-enp1s0-11:55:152
127499203,2cyclictest89050irq/133-enp2s0-07:47:532
127499201,8cyclictest0-21swapper/210:54:152
127499201,7cyclictest16092-21ntp_states11:20:212
127499201,5cyclictest19953-21pmu-power10:15:242
1274992015,3cyclictest32-21ksoftirqd/207:15:012
127499201,4cyclictest808-21snmpd09:12:522
127499201,3cyclictest0-21swapper/210:30:182
127499201,11cyclictest0-21swapper/212:36:102
126799207,3cyclictest5047-21sshd07:15:171
126799202,10cyclictest11762-21chrt09:57:281
126799201,8cyclictest0-21swapper/110:21:201
1267992013,2cyclictest0-21swapper/112:10:451
1267992013,2cyclictest0-21swapper/112:10:451
1267992013,2cyclictest0-21swapper/107:12:061
1267992012,3cyclictest0-21swapper/107:44:091
126799201,15cyclictest0-21swapper/108:37:041
126799201,15cyclictest0-21swapper/108:32:031
126799201,14cyclictest0-21swapper/108:54:101
126799201,13cyclictest0-21swapper/112:36:221
126799201,13cyclictest0-21swapper/111:15:041
126799201,11cyclictest17902-21diskstats07:45:141
126799201,10cyclictest17033-21ntp_states08:55:191
126799200,5cyclictest0-21swapper/111:22:001
126299209,3cyclictest13773-21/usr/sbin/munin08:50:160
126299208,2cyclictest16308-21kworker/0:010:25:350
126299207,2cyclictest23392-21kworker/0:011:39:350
126299207,2cyclictest15629-21kworker/0:012:39:070
126299202,9cyclictest1451-21kworker/0:207:24:070
1262992015,3cyclictest23423-21grep07:55:260
1262992014,3cyclictest793-21sshd12:00:160
126299201,3cyclictest7602-21timerandwakeup11:00:270
126299201,3cyclictest7602-21timerandwakeup11:00:270
126299201,11cyclictest15968-21/usr/sbin/munin08:55:160
126299201,11cyclictest0-21swapper/010:45:400
83522191,3sleep00-21swapper/008:35:250
58952191,3sleep30-21swapper/308:30:183
31302191,2sleep23141-21/usr/sbin/munin08:25:152
127899199,5cyclictest401ktimersoftd/309:48:193
127899198,3cyclictest12940-21awk07:35:013
127899198,3cyclictest12940-21awk07:35:013
127899197,3cyclictest8345-21cut09:50:163
127899197,3cyclictest8345-21cut09:50:163
127899197,3cyclictest0-21swapper/307:35:163
127899196,4cyclictest0-21swapper/311:05:343
127899195,3cyclictest87350irq/126-enp1s0-10:00:163
127899195,2cyclictest41-21ksoftirqd/307:20:223
127899193,9cyclictest0-21swapper/309:31:213
127899191,6cyclictest827-21runrttasks11:45:173
127899191,6cyclictest0-21swapper/312:37:213
127899191,5cyclictest11299-21hddtemp_smartct11:10:153
127899191,5cyclictest0-21swapper/311:55:363
1278991914,3cyclictest17946-21idleruntime-cro09:00:013
127899191,2cyclictest0-21swapper/311:31:493
127899190,5cyclictest0-21swapper/307:15:133
127899190,10cyclictest0-21swapper/308:35:153
127499193,4cyclictest30-21rcuc/209:35:152
127499193,2cyclictest0-21swapper/209:05:152
127499192,3cyclictest87650irq/129-enp1s0-08:40:232
127499192,3cyclictest87650irq/129-enp1s0-08:40:232
127499191,9cyclictest0-21swapper/209:51:292
127499191,9cyclictest0-21swapper/209:51:292
127499191,8cyclictest0-21swapper/209:00:102
127499191,6cyclictest0-21swapper/211:29:242
127499191,5cyclictest30-21rcuc/210:00:142
127499191,5cyclictest0-21swapper/208:47:152
127499191,4cyclictest21954-21/usr/sbin/munin07:55:262
127499191,4cyclictest0-21swapper/212:25:242
127499191,15cyclictest7304-21chrt09:46:412
127499191,11cyclictest0-21swapper/212:03:492
127499191,11cyclictest0-21swapper/211:48:322
127499191,10cyclictest0-21swapper/210:39:032
127499190,4cyclictest0-21swapper/208:15:152
126799192,3cyclictest27368-21munin-run10:35:011
1267991912,3cyclictest0-21swapper/109:12:301
1267991912,3cyclictest0-21swapper/108:17:471
1267991912,2cyclictest0-21swapper/107:30:441
1267991912,2cyclictest0-21swapper/107:30:441
126799191,10cyclictest30000-21uname09:25:181
1267991910,2cyclictest0-21swapper/108:45:191
126299199,3cyclictest11575-21/usr/sbin/munin08:45:240
126299198,3cyclictest19416-21/usr/sbin/munin11:30:220
126299198,10cyclictest0-21swapper/009:05:010
126299197,3cyclictest16308-21kworker/0:010:13:010
126299197,3cyclictest0-21swapper/011:16:030
126299196,2cyclictest12732-21kworker/0:007:35:160
126299192,9cyclictest15045-21kworker/0:207:48:180
126299192,8cyclictest424450kworker/u8:212:10:410
126299192,8cyclictest424450kworker/u8:212:10:410
126299192,7cyclictest5667-21kworker/0:011:11:530
126299191,9cyclictest28209-21kworker/0:108:10:070
126299191,9cyclictest15045-21kworker/0:208:01:550
126299191,3cyclictest0-21swapper/010:06:070
287992180,4sleep230-21rcuc/208:10:132
250922181,2sleep30-21swapper/310:29:173
128812181,2sleep00-21swapper/007:34:280
128812181,2sleep00-21swapper/007:34:280
127899185,3cyclictest0-21swapper/310:52:593
127899181,5cyclictest0-21swapper/308:20:153
127899181,4cyclictest18804-21/usr/sbin/munin10:15:143
127899181,4cyclictest0-21swapper/310:23:103
127899181,14cyclictest26547-21chrt09:17:363
127499184,3cyclictest87650irq/129-enp1s0-10:05:242
127499183,8cyclictest0-21swapper/208:50:502
127499183,6cyclictest30-21rcuc/210:45:162
127499182,7cyclictest0-21swapper/207:29:222
127499181,5cyclictest10337-21df09:55:142
127499181,5cyclictest0-21swapper/210:40:512
127499181,4cyclictest909-21sssd_nss07:24:162
127499181,4cyclictest0-21swapper/211:37:002
1274991813,3cyclictest32-21ksoftirqd/212:15:242
1274991812,3cyclictest827-21runrttasks09:15:202
1274991812,3cyclictest827-21runrttasks08:20:182
127499181,15cyclictest0-21swapper/210:20:512
127499181,15cyclictest0-21swapper/207:30:372
127499181,15cyclictest0-21swapper/207:30:372
127499181,14cyclictest17530-21grep08:55:222
127499181,10cyclictest0-21swapper/212:33:112
126799186,4cyclictest8949-21cstates07:25:121
126299181,9cyclictest14438-21kworker/u8:112:25:120
213802171,3sleep00-21swapper/007:50:420
127899174,3cyclictest0-21swapper/307:58:163
127899172,4cyclictest22351-21diskmemload10:10:263
127899172,12cyclictest24151-21/usr/sbin/munin08:00:173
127899171,6cyclictest0-21swapper/311:51:343
127899171,6cyclictest0-21swapper/307:47:073
127899171,5cyclictest5260-21sendmail_mailqu10:55:243
127899171,4cyclictest19754-21/usr/sbin/munin07:50:173
127899171,15cyclictest0-21swapper/307:14:073
127899171,13cyclictest26359-21/usr/sbin/munin08:05:183
1278991711,3cyclictest14242-21interrupts12:30:173
127899171,10cyclictest0-21swapper/308:43:493
127899171,10cyclictest0-21swapper/308:43:493
127499171,5cyclictest0-21swapper/211:15:392
127499171,15cyclictest0-21swapper/208:03:182
1274991711,3cyclictest9177-21grep12:20:112
127499171,10cyclictest0-21swapper/211:31:362
127499171,10cyclictest0-21swapper/209:41:322
127499171,10cyclictest0-21swapper/209:31:022
126299172,13cyclictest9725-21sshd08:40:160
126299172,13cyclictest9725-21sshd08:40:160
127899161,2cyclictest0-21swapper/308:45:313
127899161,12cyclictest28884-21cstates08:10:143
127899160,4cyclictest0-21swapper/308:18:493
127499161,3cyclictest87350irq/126-enp1s0-12:10:162
127499161,3cyclictest87350irq/126-enp1s0-12:10:162
1274991612,3cyclictest0-21swapper/208:08:112
1274991612,3cyclictest0-21swapper/207:39:032
127499161,14cyclictest0-21swapper/207:42:012
1274991611,3cyclictest20804-21ntp_kernel_pll_07:50:192
127499160,14cyclictest0-21swapper/208:35:152
126299161,6cyclictest0-21swapper/008:30:110
126299161,6cyclictest0-21swapper/007:40:240
1262991610,3cyclictest10928-21cut07:30:000
127499151,5cyclictest0-21swapper/207:15:232
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional