You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-07 - 14:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Jan 07, 2026 12:44:30)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
221972232161,58sleep00-21swapper/007:06:510
221532226152,24sleep30-21swapper/307:06:193
224102221141,68sleep10-21swapper/107:09:381
224232201163,25sleep20-21swapper/207:09:472
331121680,5sleep02252999cyclictest07:35:250
678821320,2sleep02252999cyclictest10:10:240
22540996859,3cyclictest0-21swapper/207:12:022
22540995855,2cyclictest0-21swapper/209:07:072
22540995855,2cyclictest0-21swapper/209:07:072
23442550,6sleep121-21rcuc/107:35:141
149492550,2sleep30-21swapper/310:30:173
84262490,4sleep339-21rcuc/307:50:013
250052490,3sleep29-21rcu_preempt08:25:192
22540994639,3cyclictest0-21swapper/212:21:572
22529993611,12cyclictest2430-21munin-run08:50:000
211632350,4sleep121-21rcuc/108:15:271
264452340,3sleep02252999cyclictest07:15:190
2252999335,13cyclictest14053-21awk10:30:010
2254699311,11cyclictest30190-21grep12:20:013
2254699307,10cyclictest24744-21kworker/3:107:23:533
2254699301,13cyclictest751-21rngd08:15:593
2254699300,21cyclictest751-21rngd07:57:423
2254699300,21cyclictest751-21rngd07:57:423
2254699291,17cyclictest20246-21users10:40:263
2254699285,11cyclictest751-21rngd11:20:143
2254699282,18cyclictest751-21rngd08:46:453
2254699280,13cyclictest5342-21latency_hist10:10:013
2254699276,2cyclictest0-21swapper/312:06:213
2254699273,11cyclictest751-21rngd09:01:033
2254699270,17cyclictest0-21swapper/307:35:143
2252999271,4cyclictest0-21swapper/011:45:140
2252999270,13cyclictest3681-21perf11:20:000
2254699266,10cyclictest751-21rngd12:13:173
2254699265,8cyclictest0-21swapper/311:30:573
2254699261,10cyclictest0-21swapper/308:42:453
2253499261,16cyclictest673-21gltestperf12:25:161
2252999261,3cyclictest0-21swapper/007:46:320
2252999261,13cyclictest11848-21munin-run10:25:010
2252999261,12cyclictest2086-21munin-run12:30:010
2254699258,2cyclictest0-21swapper/307:32:143
2254699257,1cyclictest0-21swapper/311:50:023
2254699255,8cyclictest0-21swapper/308:03:183
2254699254,8cyclictest751-21rngd07:19:493
2254699253,9cyclictest21358-21fschecks_time10:45:163
2254699253,14cyclictest751-21rngd10:04:533
2254699251,9cyclictest0-21swapper/309:05:053
2254699251,9cyclictest0-21swapper/309:05:053
2254699251,14cyclictest22705-21sed08:20:183
2254699251,11cyclictest2303-21grep11:15:153
2254699251,11cyclictest0-21swapper/309:15:013
2254699251,11cyclictest0-21swapper/308:12:073
2254699251,11cyclictest0-21swapper/307:52:463
2254699251,10cyclictest0-21swapper/308:34:573
2254699250,16cyclictest0-21swapper/308:05:133
2254099255,11cyclictest751-21rngd12:15:252
2253499251,3cyclictest24419-21munin-run09:40:011
22534992511,12cyclictest19918-21kworker/u8:109:55:161
2252999251,2cyclictest86950irq/126-enp1s0-09:37:190
2254699245,5cyclictest7675-21/usr/sbin/munin10:15:253
2254699245,5cyclictest7675-21/usr/sbin/munin10:15:253
2254699242,14cyclictest23915-21sort10:50:183
2254699241,11cyclictest0-21swapper/307:41:263
2254699241,10cyclictest10819-21df11:35:123
2254699241,10cyclictest0-21swapper/308:39:173
2254699240,9cyclictest30689-21fschecks_count12:20:143
2254099241,6cyclictest0-21swapper/210:55:302
22534992418,3cyclictest25316-21hddtemp_smartct09:40:161
22529992410,6cyclictest4855-21sh12:35:130
22529992410,6cyclictest4855-21sh12:35:130
2254699239,11cyclictest15623-21munin-plugin-st09:20:023
2254699237,2cyclictest0-21swapper/311:55:263
2254699233,8cyclictest0-21swapper/308:28:093
2254699233,7cyclictest0-21swapper/311:25:573
2254699233,7cyclictest0-21swapper/311:25:573
2254699232,13cyclictest22439-21/usr/sbin/munin09:35:203
2254699231,9cyclictest0-21swapper/308:55:093
2254699231,11cyclictest0-21swapper/311:12:583
2254699231,10cyclictest16563-21cpuspeed_turbos10:35:123
2254699231,10cyclictest11873-21cat10:25:013
2254699230,13cyclictest20243-21/usr/sbin/munin09:30:153
2254699230,13cyclictest18381-21switchtime11:50:253
2254699230,13cyclictest0-21swapper/312:35:193
2254699230,13cyclictest0-21swapper/312:35:193
2254699230,11cyclictest13355-21wc10:25:253
2254099231,6cyclictest21532-21hddtemp_smartct10:45:182
2254099230,9cyclictest18278-21kworker/2:107:43:212
2253499238,12cyclictest18410-21forks09:25:151
2253499236,9cyclictest0-21swapper/111:11:401
2253499231,19cyclictest0-21swapper/110:05:211
2253499231,17cyclictest0-21swapper/112:19:351
2253499231,17cyclictest0-21swapper/108:21:111
2253499231,16cyclictest0-21swapper/108:45:381
2253499230,18cyclictest0-21swapper/112:02:121
2252999232,10cyclictest18050-21/usr/sbin/munin09:25:210
2252999232,10cyclictest1050-21/usr/sbin/munin10:00:170
2252999231,4cyclictest0-21swapper/012:19:100
2252999231,4cyclictest0-21swapper/011:22:120
2252999231,4cyclictest0-21swapper/010:42:560
2252999231,4cyclictest0-21swapper/010:16:070
2252999231,4cyclictest0-21swapper/010:16:070
2252999231,4cyclictest0-21swapper/009:17:140
2252999231,11cyclictest9257-21/usr/sbin/munin09:05:250
2252999231,11cyclictest9257-21/usr/sbin/munin09:05:250
2252999231,11cyclictest26013-21awk08:30:020
2252999231,11cyclictest20857-21/usr/sbin/munin10:45:170
2252999231,11cyclictest12011-21tune2fs09:10:160
2252999231,10cyclictest21827-21/usr/sbin/munin08:20:250
2254699227,3cyclictest0-21swapper/307:26:423
2254699224,1cyclictest0-21swapper/310:56:173
2254699221,9cyclictest0-21swapper/312:25:153
2254699221,7cyclictest29437-21dump-pmu-power11:05:003
2254699221,7cyclictest0-21swapper/308:54:153
2254699221,6cyclictest0-21swapper/309:40:133
2254699221,5cyclictest30104-21grep11:05:153
22546992213,6cyclictest0-21swapper/311:40:103
2254699221,14cyclictest0-21swapper/310:13:173
2254699221,13cyclictest21472-21/usr/sbin/munin12:00:253
2254699221,10cyclictest23990-21smartctl07:10:253
2254699221,10cyclictest16197-21/usr/sbin/munin09:20:143
2254699220,14cyclictest18233-21cstates09:25:153
2254699220,13cyclictest0-21swapper/309:55:203
2254099221,7cyclictest0-21swapper/207:35:252
2254099221,6cyclictest0-21swapper/211:55:142
22534992217,3cyclictest25666-21fschecks_count10:55:161
22534992212,6cyclictest12202-40dnf11:36:571
22534992212,2cyclictest28479-21/usr/sbin/munin11:00:191
2253499221,18cyclictest17512-21grep10:35:221
2253499221,16cyclictest0-21swapper/109:02:031
2253499221,16cyclictest0-21swapper/108:08:521
2253499221,16cyclictest0-21swapper/107:43:511
2253499221,15cyclictest0-21swapper/111:49:441
2253499221,15cyclictest0-21swapper/111:30:221
2253499220,4cyclictest0-21swapper/112:12:181
2252999221,4cyclictest0-21swapper/012:03:430
2252999221,4cyclictest0-21swapper/011:52:380
2252999221,4cyclictest0-21swapper/011:42:370
2252999221,4cyclictest0-21swapper/011:09:490
2252999221,4cyclictest0-21swapper/009:42:510
2252999221,3cyclictest17818-21cat10:35:250
2252999221,13cyclictest0-21swapper/012:10:120
2252999221,11cyclictest32207-21sshd12:24:410
2252999220,3cyclictest3535-21gzip12:30:220
2254699211,10cyclictest0-21swapper/309:46:063
2254099215,4cyclictest15585-21sshd09:19:402
2254099211,7cyclictest23195-21grep10:50:132
2254099211,5cyclictest0-21swapper/209:42:232
2254099211,2cyclictest0-21swapper/211:44:352
2253499213,14cyclictest26881-21smart_sda07:15:251
22534992115,3cyclictest1271-21cstates10:00:121
22534992114,2cyclictest0-21swapper/107:52:061
2253499211,3cyclictest31344-21ntp_states12:20:221
22534992113,3cyclictest0-21swapper/112:06:001
22534992113,3cyclictest0-21swapper/110:53:001
2253499211,17cyclictest21436-21perl10:45:171
2253499211,15cyclictest0-21swapper/111:08:241
2253499211,14cyclictest0-21swapper/112:30:111
2253499211,14cyclictest0-21swapper/109:33:381
2253499211,11cyclictest1042-21ps07:30:241
2253499211,10cyclictest0-21swapper/110:17:091
2253499211,10cyclictest0-21swapper/110:17:091
2253499210,16cyclictest0-21swapper/108:58:211
2253499210,15cyclictest0-21swapper/107:48:061
2252999211,7cyclictest0-21swapper/009:50:170
2252999211,6cyclictest30023-21awk08:35:270
2252999211,4cyclictest0-21swapper/011:13:300
2254699200,8cyclictest0-21swapper/309:51:133
2254099205,3cyclictest23206-21kworker/2:212:12:452
2254099203,5cyclictest7750-21cpuspeed_turbos10:15:112
2254099203,5cyclictest7750-21cpuspeed_turbos10:15:112
2254099202,5cyclictest0-21swapper/211:25:232
2254099202,5cyclictest0-21swapper/211:25:232
2254099201,9cyclictest0-21swapper/210:09:092
2254099201,6cyclictest907-21systemd-logind09:20:282
2254099201,6cyclictest17973-21sed10:35:262
2254099201,4cyclictest0-21swapper/210:40:562
2254099201,4cyclictest0-21swapper/210:25:122
2254099201,4cyclictest0-21swapper/210:10:232
2254099201,10cyclictest32595-21cpuspeed_turbos12:25:122
2254099200,7cyclictest5268-21timerandwakeup11:20:262
2253499209,3cyclictest27190-21diskstats09:45:121
2253499209,3cyclictest14000-21df_inode09:15:151
2253499201,8cyclictest0-21swapper/111:28:201
2253499201,8cyclictest0-21swapper/111:28:201
22534992016,2cyclictest22738-21acpi07:10:131
22534992012,3cyclictest0-21swapper/109:07:121
22534992012,3cyclictest0-21swapper/109:07:111
2253499201,14cyclictest0-21swapper/111:58:131
2253499201,14cyclictest0-21swapper/111:50:521
2253499201,14cyclictest0-21swapper/108:45:001
2253499201,14cyclictest0-21swapper/108:30:061
2253499201,14cyclictest0-21swapper/108:11:301
2253499201,11cyclictest4858-21diskstats12:35:151
2253499201,11cyclictest4858-21diskstats12:35:151
2253499201,11cyclictest20626-21latency_hist10:45:021
22534992010,3cyclictest30106-21ntp_states09:50:201
2253499200,16cyclictest0-21swapper/109:21:291
2252999205,3cyclictest1565-21kworker/0:107:50:210
2252999203,9cyclictest835-21runrttasks11:36:570
2252999201,4cyclictest4246-21/usr/sbin/munin07:40:240
2252999201,4cyclictest0-21swapper/009:56:410
2252999201,11cyclictest0-21swapper/011:00:400
2252999201,11cyclictest0-21swapper/009:22:240
2254699190,11cyclictest87250irq/129-enp1s0-12:30:113
2254099195,8cyclictest87250irq/129-enp1s0-11:06:342
2254099194,9cyclictest19918-21kworker/u8:109:49:592
2254099194,7cyclictest10946-21kworker/2:208:46:532
2254099193,9cyclictest2164-21/usr/sbin/munin11:15:152
2254099192,5cyclictest5766-21memory08:55:192
2254099191,7cyclictest0-21swapper/209:54:422
2254099191,6cyclictest5336-21kernelversion12:35:182
2254099191,6cyclictest5336-21kernelversion12:35:182
2254099191,6cyclictest32634-21timerwakeupswit09:55:252
2254099191,6cyclictest27455-21/usr/sbin/munin11:00:182
2254099191,6cyclictest17078-21/usr/sbin/munin11:50:252
2254099191,6cyclictest10835-21/usr/sbin/munin07:55:182
2254099191,6cyclictest10835-21/usr/sbin/munin07:55:182
2254099191,6cyclictest0-21swapper/208:40:272
2254099191,5cyclictest8636-21/usr/sbin/munin07:50:222
2254099191,5cyclictest10577-21grep11:35:102
2254099191,4cyclictest0-21swapper/212:32:432
2254099191,11cyclictest0-21swapper/211:11:072
2254099191,10cyclictest904-21sssd_nss08:35:182
2254099191,10cyclictest0-21swapper/210:01:212
2254099190,6cyclictest23286-21kernelversion09:35:182
2253499196,4cyclictest0-21swapper/110:25:251
2253499191,10cyclictest7445-21latency_hist10:15:021
2253499191,10cyclictest3898-21cat11:20:121
2253499191,10cyclictest30359-21diskstats07:25:131
2253499191,10cyclictest28156-21df_inode07:20:141
2253499191,10cyclictest11525-21sed07:55:161
2253499191,10cyclictest11525-21sed07:55:161
22534991910,6cyclictest0-21swapper/108:35:431
2252999197,4cyclictest11-21rcuc/010:57:270
2252999196,2cyclictest0-21swapper/011:59:180
2252999194,8cyclictest0-21swapper/009:49:350
2252999193,7cyclictest17168-21sshd08:09:410
2252999193,5cyclictest10664-1kworker/u9:110:30:020
2252999192,10cyclictest0-21swapper/012:09:260
2252999191,9cyclictest0-21swapper/009:30:230
22529991913,3cyclictest9021-21crond09:05:000
2252999191,15cyclictest14271-21/usr/sbin/munin08:00:250
2252999190,12cyclictest0-21swapper/008:10:170
146182191,3sleep114846-21perl10:30:171
2254099182,4cyclictest11227-21diskmemload12:08:572
2254099181,9cyclictest0-21swapper/210:24:422
2254099181,6cyclictest18258-21kernelversion08:10:182
2254099181,6cyclictest0-21swapper/209:25:192
2254099181,5cyclictest22539-21ntp_states12:00:212
2254099181,5cyclictest21311-21ntp_states09:30:202
2254099181,5cyclictest1778-21sshd07:34:402
2254099181,4cyclictest16245-21perf10:35:012
22540991813,3cyclictest15580-21df_inode08:05:122
2254099181,10cyclictest0-21swapper/211:31:462
2253499186,3cyclictest24631-21tail08:25:171
2253499181,3cyclictest2689-21cpu08:50:121
2253499181,2cyclictest0-21swapper/108:02:571
2253499181,16cyclictest0-21swapper/111:16:151
2253499181,16cyclictest0-21swapper/110:20:351
2253499181,10cyclictest0-21swapper/109:10:161
2253499180,11cyclictest0-21swapper/111:40:201
2252999183,13cyclictest3283-21/usr/sbin/munin10:05:210
2252999182,2cyclictest11441-21gltestperf07:55:160
2252999182,2cyclictest11441-21gltestperf07:55:160
2252999182,13cyclictest1823-21munin-run07:35:000
2252999181,7cyclictest24425-21sshd07:11:410
2252999181,7cyclictest0-21swapper/007:27:440
2252999181,4cyclictest19633-21/usr/sbin/munin08:15:290
2252999181,3cyclictest30623-21/usr/sbin/munin08:40:170
2252999181,15cyclictest3506-21wc08:50:170
2252999180,5cyclictest0-21swapper/011:30:080
2254099174,3cyclictest86950irq/126-enp1s0-08:20:012
2254099173,4cyclictest0-21swapper/209:10:142
2254099172,9cyclictest0-21swapper/211:45:022
2254099172,3cyclictest27816-21/usr/sbin/munin07:20:142
2254099171,7cyclictest26837-21ps07:15:232
2254099171,5cyclictest0-21swapper/208:50:122
22540991711,3cyclictest13100-21wc08:00:132
2252999171,9cyclictest27816-21/usr/sbin/munin07:20:130
2252999171,15cyclictest0-21swapper/008:32:030
2252999171,13cyclictest6090-21/usr/sbin/munin11:25:190
2252999171,13cyclictest6090-21/usr/sbin/munin11:25:190
2254099163,3cyclictest12050irq/123-ahci[0008:25:012
2254099161,9cyclictest0-21swapper/207:45:212
2254099161,14cyclictest0-21swapper/209:03:082
22540991611,3cyclictest29781-21latency_hist07:25:032
22540991610,4cyclictest27167-21missed_timers08:30:182
2252999161,6cyclictest23868-21irqstats10:50:170
22529991610,3cyclictest4854-21/usr/sbin/munin08:55:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional