You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 06:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Wed Mar 11, 2026 00:44:27)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26476218099,24sleep30-21swapper/319:08:223
262962178106,25sleep00-21swapper/019:06:020
265812175104,58sleep10-21swapper/119:09:451
264472172101,58sleep20-21swapper/219:08:012
26704999995,3cyclictest0-21swapper/323:54:153
26704997775,1cyclictest0-21swapper/319:45:383
162592730,1sleep10-21swapper/119:55:231
26704995854,3cyclictest0-21swapper/319:34:413
158872480,2sleep20-21swapper/221:10:132
186682460,4sleep318664-21latency21:15:183
26704994137,3cyclictest0-21swapper/322:03:063
2670499394,23cyclictest88150irq/126-enp1s0-21:55:153
2670499391,5cyclictest0-21swapper/322:55:173
2670499391,29cyclictest823-21snmpd20:39:493
2670499384,29cyclictest5534-40dnf20:45:213
2670499377,21cyclictest14996-21/usr/sbin/munin19:55:113
2670499375,19cyclictest88150irq/126-enp1s0-21:40:133
2670499371,32cyclictest23794-21cut23:55:153
2670499371,27cyclictest1812-21/usr/sbin/munin19:25:243
2670499371,25cyclictest13795-21latency_hist22:20:013
2670499371,23cyclictest32450-21cat21:45:243
2670499361,22cyclictest27844-21/usr/sbin/munin00:05:263
2670499357,14cyclictest30389-21/usr/sbin/munin20:30:213
2670499355,24cyclictest88150irq/126-enp1s0-23:15:243
2670499354,18cyclictest88150irq/126-enp1s0-23:00:143
2670499353,3cyclictest88150irq/126-enp1s0-23:05:133
2670499343,3cyclictest88150irq/126-enp1s0-21:35:153
2670499343,16cyclictest0-21swapper/323:31:593
26704993413,18cyclictest25761-21ls20:20:013
2670499341,24cyclictest823-21snmpd22:05:193
26694993423,6cyclictest5534-40dnf20:45:211
2668799347,12cyclictest1877-21perf00:20:020
2670499337,13cyclictest21593-21/usr/sbin/munin20:10:133
2670499331,6cyclictest0-21swapper/321:50:023
2670499331,5cyclictest0-21swapper/319:13:183
2670499331,23cyclictest25677-21cut22:45:143
2670499331,18cyclictest22813-21/usr/sbin/munin22:40:103
2670499330,3cyclictest88150irq/126-enp1s0-23:35:163
2670499321,5cyclictest18401-21unixbench_singl23:40:263
2670499321,3cyclictest25491-21awk21:30:213
2670499321,3cyclictest20522-21/usr/sbin/munin23:45:253
2670499321,3cyclictest15392-21munin-run21:10:013
2670499321,3cyclictest0-21swapper/300:10:143
2670499321,19cyclictest26771-21kworker/u8:220:20:263
2670499313,3cyclictest14027-21/usr/sbin/munin22:20:153
26704993124,2cyclictest0-21swapper/322:31:353
26704993124,2cyclictest0-21swapper/322:31:353
2670499311,4cyclictest20821-21grep21:20:163
2670499311,3cyclictest31744-21latency_hist19:20:023
2670499311,3cyclictest21863-21/usr/sbin/munin22:35:223
2670499311,21cyclictest2664-21perl00:20:163
2670499311,20cyclictest0-21swapper/323:25:133
2670499303,2cyclictest25710-21cpuspeed_turbos00:00:133
2670499303,16cyclictest0-21swapper/321:13:323
26704993016,12cyclictest7235-21grep20:50:163
2670499301,5cyclictest0-21swapper/322:28:083
26704993013,14cyclictest3652-21/usr/sbin/munin23:10:193
2670499301,16cyclictest11602-21crond22:15:013
2670499300,2cyclictest0-21swapper/319:40:163
2670499299,17cyclictest17336-21grep20:00:133
2670499297,19cyclictest578-21ntpq19:20:213
2670499293,4cyclictest913-21sssd_nss20:25:123
2670499291,3cyclictest6747-21expr00:30:123
2670499291,20cyclictest0-21swapper/300:39:473
2670499291,17cyclictest0-21swapper/300:17:053
2670499290,4cyclictest0-21swapper/321:26:403
2670499288,17cyclictest4002-21unixbench_singl20:40:263
2670499281,24cyclictest19496-21tr20:05:123
2670499281,24cyclictest19496-21tr20:05:123
2670499281,23cyclictest12282-21ntp_states21:00:203
26704992811,14cyclictest9014-21/usr/sbin/munin20:55:163
2670499280,4cyclictest0-21swapper/323:20:173
2668799283,12cyclictest1603-21idleruntime-cro19:25:010
2670499271,3cyclictest4465-21/usr/sbin/munin00:25:123
2670499271,2cyclictest0-21swapper/322:50:143
2670499271,23cyclictest6210-21/usr/sbin/munin19:35:133
2670499271,22cyclictest823-21snmpd19:50:223
2669499271,16cyclictest22540-21df21:25:121
2668799271,13cyclictest27966-21munin-run20:25:010
2669499253,15cyclictest0-21swapper/100:36:221
2669499251,20cyclictest32025-21munin-run00:15:011
26694992510,13cyclictest9169-21kworker/u8:022:11:271
313392240,5sleep22669999cyclictest21:45:122
2669499242,12cyclictest28572-21munin-run21:40:011
2669499241,16cyclictest0-21swapper/123:55:101
2669499241,12cyclictest0-21swapper/100:06:021
2668799247,2cyclictest4464-21kworker/0:322:21:020
2668799241,12cyclictest26371-21munin-run21:35:010
2668799241,12cyclictest18213-21basename22:30:010
2669499239,3cyclictest0-21swapper/100:27:531
2669499237,4cyclictest0-21swapper/123:28:171
2669499231,16cyclictest0-21swapper/123:46:221
2669499231,16cyclictest0-21swapper/122:22:071
2669499231,16cyclictest0-21swapper/121:51:591
2669499230,15cyclictest0-21swapper/121:10:181
171822230,3sleep29-21rcu_preempt23:40:142
2669999221,6cyclictest0-21swapper/200:09:422
2669999221,5cyclictest0-21swapper/221:25:272
2669499229,5cyclictest23-21ksoftirqd/120:35:491
2669499228,3cyclictest5912-21awk23:15:121
2669499221,6cyclictest24405-21users22:40:251
2669499221,12cyclictest24738-21df_inode21:30:131
2669499221,12cyclictest23200-21awk23:55:011
2669499221,10cyclictest15387-21diskmemload23:32:061
2669499221,10cyclictest0-21swapper/122:09:001
2669999214,11cyclictest5691-21ntp_states20:45:212
2669499218,3cyclictest0-21swapper/120:25:121
2669499217,4cyclictest0-21swapper/100:33:301
2669499214,10cyclictest4091-21ntp_offset21:55:201
2669499213,3cyclictest0-21swapper/123:40:191
2669499212,12cyclictest9169-21kworker/u8:022:15:291
2669499211,8cyclictest0-21swapper/121:23:091
2669499211,5cyclictest0-21swapper/100:15:031
2669499211,11cyclictest27968-21perf20:25:011
2669499211,11cyclictest26701-21ntp_states00:00:221
2669499211,11cyclictest2420-21df00:20:141
2669499211,11cyclictest20964-21df_inode22:35:131
2669499211,11cyclictest0-21swapper/123:12:361
2669499211,10cyclictest0-21swapper/122:52:481
2668799217,9cyclictest913-21sssd_nss20:45:200
2668799212,11cyclictest11208-21/usr/sbin/munin21:00:140
2668799211,6cyclictest0-21swapper/021:50:290
2668799211,5cyclictest0-21swapper/021:42:360
2668799211,4cyclictest0-21swapper/021:27:200
2669999204,4cyclictest807-21sssd_be23:30:162
2669999201,12cyclictest0-21swapper/200:00:222
2669999200,4cyclictest28503-21kworker/u8:222:05:282
2669499209,3cyclictest5242-21cpu22:00:111
2669499209,3cyclictest30737-21diskstats20:30:141
2669499209,3cyclictest29662-21ntp_states19:15:041
2669499208,9cyclictest0-21swapper/121:20:011
2669499208,4cyclictest9310-21cstates20:55:131
2669499204,10cyclictest17972-21kworker/u8:222:28:261
2669499201,5cyclictest0-21swapper/123:35:251
2669499201,14cyclictest29617-21expr22:55:141
2669499201,11cyclictest5110-21open_inodes19:30:211
2669499201,11cyclictest1-21systemd19:40:181
2669499201,11cyclictest11543-21df21:00:121
2669499201,10cyclictest0-21swapper/123:23:441
26687992013,4cyclictest17584-21turbostat21:15:010
2668799201,11cyclictest1436-21/usr/sbin/munin23:05:270
218172200,5sleep22669999cyclictest20:10:132
2669999191,7cyclictest29050-21switchtime19:15:022
2669499198,7cyclictest0-21swapper/123:02:271
2669499195,4cyclictest0-21swapper/122:45:461
26694991912,4cyclictest1616-21tr23:05:131
2669499191,11cyclictest0-21swapper/121:47:001
2669499191,10cyclictest6825-21acpi20:50:121
2669499191,10cyclictest3650-21gzip20:40:241
2669499191,10cyclictest24463-21hddtemp_smartct20:15:171
2669499191,10cyclictest19746-21diskstats20:05:141
2669499191,10cyclictest19746-21diskstats20:05:141
2669499191,10cyclictest13481-21grep21:05:121
2669499191,10cyclictest0-21swapper/121:40:501
2668799198,3cyclictest14637-21/usr/sbin/munin23:35:200
2668799198,3cyclictest11833-21/usr/sbin/munin22:15:170
2668799198,3cyclictest0-21swapper/023:00:160
2668799194,3cyclictest5990-21perf19:35:010
2668799192,9cyclictest6723-21kworker/0:223:10:110
26687991915,3cyclictest0-21swapper/023:45:180
2668799191,4cyclictest0-21swapper/022:37:410
2668799191,10cyclictest5224-21/usr/sbin/munin22:00:180
2668799191,10cyclictest4464-21kworker/0:320:00:160
2668799191,10cyclictest3030-21/usr/sbin/munin21:55:260
2668799191,10cyclictest28186-21/usr/sbin/munin20:25:200
2668799191,10cyclictest27217-21/usr/sbin/munin22:50:210
19282190,5sleep230-21rcuc/221:50:222
107542190,4sleep00-21swapper/023:25:150
304242180,6sleep230-21rcuc/219:15:152
267722180,10sleep230-21rcuc/221:35:132
2669999186,4cyclictest2646-21processes23:05:242
2669999182,13cyclictest3415-21grep20:40:202
2669999181,8cyclictest0-21swapper/222:51:402
2669999181,7cyclictest19399-21/usr/sbin/munin20:05:252
2669999181,7cyclictest19399-21/usr/sbin/munin20:05:252
2669999181,7cyclictest14458-21ntp_kernel_pll_21:05:202
2669999181,7cyclictest0-21swapper/200:20:362
26699991813,3cyclictest88150irq/126-enp1s0-23:35:342
26699991813,3cyclictest5616-21sh23:15:012
2669999181,15cyclictest0-21swapper/200:35:582
26699991811,4cyclictest17194-21acpi20:00:122
26699991811,4cyclictest14569-21seq19:52:332
2669999181,11cyclictest0-21swapper/223:55:172
2669999181,10cyclictest0-21swapper/221:15:262
2669499188,2cyclictest3075-21smart_sda19:25:231
2669499185,11cyclictest23143-21unixbench_singl20:10:261
2669499181,9cyclictest12569-21latency_hist19:50:011
2669499181,6cyclictest19026-21tail22:30:151
2669499181,6cyclictest19026-21tail22:30:151
2669499181,6cyclictest13625-21sadc19:50:181
2669499181,4cyclictest26968-21cpuspeed_turbos19:10:131
26694991812,3cyclictest18259-21ntp_states20:00:211
2669499181,10cyclictest0-21swapper/119:20:161
2668799183,12cyclictest25619-21/usr/sbin/munin00:00:180
2668799181,9cyclictest17814-21/usr/sbin/munin21:15:160
2668799181,7cyclictest0-21swapper/023:41:230
2668799181,6cyclictest15387-21diskmemload22:40:130
2668799181,10cyclictest0-21swapper/023:50:160
80322175,2sleep2311ktimersoftd/223:20:042
2669999176,3cyclictest5216-21missed_timers00:25:192
2669999175,4cyclictest31934-21cstates23:00:122
2669999175,2cyclictest20709-21kworker/u8:323:51:212
2669999174,4cyclictest25922-21date20:20:022
2669999172,2cyclictest0-21swapper/200:12:212
2669999171,9cyclictest0-21swapper/222:27:432
2669999171,7cyclictest18436-21basename22:30:112
2669999171,7cyclictest18436-21basename22:30:112
2669999171,7cyclictest0-21swapper/222:56:092
2669999171,7cyclictest0-21swapper/222:47:382
2669999171,6cyclictest0-21swapper/222:20:152
2669999171,6cyclictest0-21swapper/221:41:132
2669999171,4cyclictest2725-21memory19:25:172
26699991713,2cyclictest32276-21expr19:20:132
2669999171,15cyclictest0-21swapper/220:38:012
2669999171,15cyclictest0-21swapper/219:49:392
2669999171,14cyclictest0-21swapper/221:30:042
2669999171,14cyclictest0-21swapper/219:59:072
26699991711,3cyclictest32666-21fschecks_count00:15:162
26699991711,3cyclictest23158-21sh22:40:132
2669499178,3cyclictest0-21swapper/119:35:261
2668799175,3cyclictest0-21swapper/020:54:230
2668799174,3cyclictest31601-21kworker/0:123:57:310
2668799174,3cyclictest31601-21kworker/0:123:30:040
2668799173,8cyclictest0-21swapper/021:20:280
2668799171,9cyclictest0-21swapper/023:15:190
2668799171,9cyclictest0-21swapper/022:06:410
2668799171,9cyclictest0-21swapper/021:35:210
2668799171,8cyclictest27750-21sort19:10:190
2668799171,7cyclictest0-21swapper/000:30:030
2668799171,4cyclictest823-21snmpd22:30:110
2668799171,4cyclictest823-21snmpd22:30:110
26687991714,2cyclictest88350irq/128-enp1s0-19:46:190
2668799171,3cyclictest0-21swapper/000:06:000
104882170,2sleep00-21swapper/000:38:080
2669999165,3cyclictest88150irq/126-enp1s0-20:24:172
2669999164,3cyclictest88450irq/129-enp1s0-22:10:192
2669999161,6cyclictest0-21swapper/223:25:542
2669999161,5cyclictest20681-21hddtemp_smartct21:20:152
2669999161,4cyclictest20168-21pmu-power23:45:222
2669999161,3cyclictest0-21swapper/220:50:122
26699991612,3cyclictest0-21swapper/221:55:482
26699991611,3cyclictest9932-21missed_timers20:55:192
26699991611,3cyclictest8988-21gltestperf19:40:152
26699991611,3cyclictest6591-21if_enp2s023:15:172
26699991611,3cyclictest6538-21cpu00:30:122
26699991611,3cyclictest28374-21seq20:25:102
26699991610,3cyclictest5911-21hwlatdetect22:00:162
26699991610,3cyclictest31084-21idleruntime20:30:172
26699991610,3cyclictest20680-21cpu22:35:112
26699991610,3cyclictest13186-21latency_hist21:05:012
26699991610,3cyclictest12748-21memory22:15:202
2668799161,9cyclictest0-21swapper/021:05:160
2668799161,7cyclictest0-21swapper/019:50:420
2668799161,6cyclictest6210-21/usr/sbin/munin19:35:210
2668799161,6cyclictest30389-21/usr/sbin/munin20:30:250
2668799161,6cyclictest0-21swapper/021:45:270
2668799161,6cyclictest0-21swapper/020:35:140
2668799161,5cyclictest0-21swapper/000:20:080
2668799161,4cyclictest0-21swapper/022:55:190
26687991612,3cyclictest0-21swapper/019:26:200
26687991611,3cyclictest29779-21/usr/sbin/munin19:15:150
2668799161,13cyclictest26079-21ntp_states22:45:200
26687991611,3cyclictest2420-21/usr/sbin/munin20:40:270
2668799161,13cyclictest21068-21sshd20:06:180
2668799161,13cyclictest21068-21sshd20:06:180
26687991611,3cyclictest10331-21sort22:10:160
26687991610,3cyclictest4380-21tr00:25:120
26687991610,3cyclictest30163-21cpuspeed_turbos00:10:120
2668799160,6cyclictest0-21swapper/020:15:150
2668799160,5cyclictest0-21swapper/019:56:490
2668799160,14cyclictest0-21swapper/019:40:060
2669999151,4cyclictest4982-21sadc19:30:182
26699991510,3cyclictest6431-21expr19:35:142
2668799158,4cyclictest22572-21sed20:10:190
26687991510,3cyclictest11030-21cut21:00:010
2668799150,8cyclictest0-21swapper/023:20:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional