You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-17 - 08:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Apr 17, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
292962238162,23sleep10-21swapper/119:09:211
291552232158,24sleep20-21swapper/219:07:292
291732228155,25sleep00-21swapper/019:07:430
293202219144,24sleep30-21swapper/319:09:393
29439991070,105cyclictest0-21swapper/221:22:182
172282750,2sleep00-21swapper/021:05:220
29439995753,3cyclictest0-21swapper/221:59:572
77772550,4sleep230-21rcuc/223:10:252
226102520,1sleep00-21swapper/020:05:140
74132470,3sleep17416-21gltestperf19:30:151
58252470,2sleep10-21swapper/100:20:201
156692450,2sleep339-21rcuc/319:50:123
29439994326,8cyclictest0-21swapper/222:49:282
2942499356,13cyclictest20349-21munin-run21:15:010
2942499356,13cyclictest19716-21sadc20:00:010
2942499316,11cyclictest0-21swapper/019:30:010
2942499316,11cyclictest0-21swapper/019:30:010
2942499315,12cyclictest12829-21perf23:25:010
2943999302,9cyclictest11153-21/usr/sbin/munin19:40:232
2942499305,12cyclictest15339-21awk19:50:010
270302300,3sleep011-21rcuc/023:55:170
2943299291,18cyclictest0-21swapper/121:55:491
29424992911,15cyclictest4019-21perf23:05:010
2943299281,17cyclictest0-21swapper/121:28:021
249362280,4sleep123-21ksoftirqd/123:50:211
160352280,1sleep00-21swapper/021:05:010
2943299272,11cyclictest9082-21munin-run00:30:011
29432992711,7cyclictest0-21swapper/122:35:161
2943299270,18cyclictest0-21swapper/100:35:241
67922260,3sleep06795-21fschecks_count23:10:150
2943299268,9cyclictest0-21swapper/122:30:021
2943299262,17cyclictest5356-21kworker/u8:122:15:441
2943299261,16cyclictest0-21swapper/122:00:051
29432992610,3cyclictest15329-21munin-run19:50:011
2942499262,8cyclictest30226-21kworker/0:200:03:180
2942499261,12cyclictest20977-21awk22:30:010
2944599250,11cyclictest0-21swapper/321:30:573
2943299258,10cyclictest0-21swapper/123:09:441
2943299253,18cyclictest6166-21strings20:40:191
2943299253,18cyclictest6166-21strings20:40:191
2943299251,18cyclictest5356-21kworker/u8:123:04:251
2943299251,17cyclictest0-21swapper/122:52:491
2943299251,17cyclictest0-21swapper/122:48:081
2943299251,17cyclictest0-21swapper/119:40:151
2943299251,16cyclictest0-21swapper/121:18:401
2942499252,9cyclictest9118-21kworker/0:320:53:010
2942499252,9cyclictest6863-21kworker/0:000:28:290
2942499252,8cyclictest4639-21kworker/0:100:24:530
2942499252,8cyclictest25766-21kworker/0:323:54:490
2942499251,12cyclictest1810-21idleruntime-cro23:00:010
2942499251,12cyclictest1810-21idleruntime-cro23:00:010
29445992417,4cyclictest3601-21grep20:35:163
2943299246,2cyclictest87350irq/126-enp1s0-19:20:151
2943299242,8cyclictest808-21snmpd21:00:111
29432992420,3cyclictest0-21swapper/120:23:531
2943299241,17cyclictest0-21swapper/123:39:441
2943299241,17cyclictest0-21swapper/121:52:351
2943299241,16cyclictest0-21swapper/123:40:271
2943299241,16cyclictest0-21swapper/121:47:131
2943299241,15cyclictest827-21runrttasks20:15:501
2942499242,9cyclictest31250-21kworker/0:119:18:250
2942499242,9cyclictest21856-21kworker/0:020:13:210
2942499242,8cyclictest880-21kworker/0:221:46:290
2942499242,8cyclictest23744-21kworker/0:000:14:170
2942499241,11cyclictest10522-21kworker/0:123:28:330
2942499241,10cyclictest2593-21kworker/0:220:55:050
2942499241,10cyclictest19547-21kworker/0:220:00:030
2944599235,10cyclictest11085-21unixbench_multi20:50:263
2944599231,13cyclictest31731-21ntp_states00:05:213
2944599231,13cyclictest31731-21ntp_states00:05:213
2943999239,2cyclictest4013-21awk23:05:012
29439992319,2cyclictest31716-21ntpq00:05:202
29439992319,2cyclictest31716-21ntpq00:05:202
2943999231,7cyclictest0-21swapper/221:14:092
2943299238,3cyclictest26317-21munin-run20:15:011
2943299232,3cyclictest87550irq/128-enp1s0-23:10:181
2943299232,11cyclictest4952-21munin-run20:40:011
29432992319,3cyclictest0-21swapper/121:22:541
29432992319,3cyclictest0-21swapper/120:45:401
29432992319,2cyclictest27261-21cpuspeed_turbos21:30:121
2943299231,19cyclictest21863-21hddtemp_smartct22:30:161
2943299231,19cyclictest0-21swapper/119:55:191
2943299231,16cyclictest0-21swapper/122:44:511
2943299231,16cyclictest0-21swapper/100:04:501
2943299231,15cyclictest827-21runrttasks22:20:551
29432992311,4cyclictest0-21swapper/123:26:431
2943299230,7cyclictest32089-21fschecks_time21:40:151
2943299230,6cyclictest0-21swapper/100:09:591
2943299230,6cyclictest0-21swapper/100:09:591
2943299230,15cyclictest0-21swapper/123:59:441
2943299230,15cyclictest0-21swapper/122:10:501
2943299230,15cyclictest0-21swapper/100:16:461
2942499232,9cyclictest15124-21kworker/0:319:50:350
2942499232,9cyclictest10903-21kworker/0:219:40:430
2942499232,8cyclictest20303-21kworker/0:121:22:490
2942499231,9cyclictest4639-21kworker/0:100:31:370
2942499231,9cyclictest32722-21kworker/0:020:32:250
2942499231,8cyclictest28806-21kworker/0:021:37:450
2942499231,8cyclictest18058-21kworker/0:021:16:490
2944599227,3cyclictest0-21swapper/323:43:053
2944599226,6cyclictest0-21swapper/322:37:183
2944599224,3cyclictest87350irq/126-enp1s0-21:56:123
29445992216,3cyclictest20958-21dump-pmu-power22:30:013
2943999229,2cyclictest87650irq/129-enp1s0-23:15:172
2943999227,3cyclictest27307-40dnf22:42:032
2943999221,5cyclictest0-21swapper/223:30:132
2943299228,8cyclictest0-21swapper/123:33:301
2943299226,4cyclictest19182-21irqstats21:10:171
29432992218,3cyclictest0-21swapper/119:16:191
2943299221,7cyclictest0-21swapper/123:46:321
29432992216,3cyclictest22494-21df_inode20:05:131
2943299221,15cyclictest0-21swapper/123:16:571
2943299221,10cyclictest0-21swapper/120:05:011
2943299220,5cyclictest0-21swapper/121:35:101
2943299220,14cyclictest0-21swapper/122:07:541
2943299220,14cyclictest0-21swapper/100:31:551
2942499222,9cyclictest4836-21kworker/0:120:40:070
2942499222,9cyclictest4836-21kworker/0:120:40:070
2942499222,9cyclictest14011-21kworker/0:322:10:570
2942499222,9cyclictest11893-21kworker/0:222:22:430
2942499222,9cyclictest10622-21kworker/0:019:39:130
2942499221,9cyclictest880-21kworker/0:221:59:290
2942499221,9cyclictest28271-21kworker/0:320:26:510
2942499221,9cyclictest11893-21kworker/0:222:19:590
2942499220,9cyclictest31360-21kworker/0:319:30:170
2944599215,9cyclictest0-21swapper/321:25:063
29445992116,3cyclictest21536-21diskstats22:30:143
2944599211,11cyclictest13375-21df23:25:143
2944599210,7cyclictest0-21swapper/322:13:593
2943999215,5cyclictest26542-21/usr/sbin/munin20:15:242
2943999211,6cyclictest1-21systemd20:40:212
2943999211,6cyclictest1-21systemd20:40:212
2943999211,5cyclictest9057-21ntp_states22:00:212
29439992115,3cyclictest670-21grep19:15:142
29439992110,3cyclictest13381-21df_inode23:25:152
2943299219,3cyclictest0-21swapper/121:05:171
29432992118,2cyclictest0-21swapper/120:57:021
29432992117,3cyclictest0-21swapper/120:50:181
29432992115,3cyclictest17024-21grep19:50:251
2943299211,11cyclictest10924-21latency_hist19:40:011
2943299211,10cyclictest31502-21fschecks_time20:25:141
2943299211,10cyclictest0-21swapper/122:56:491
2943299211,10cyclictest0-21swapper/122:56:491
29432992110,3cyclictest2751-21latency_hist20:35:011
2943299210,5cyclictest0-21swapper/100:14:471
2942499219,3cyclictest2090-21kworker/0:219:22:070
2942499217,3cyclictest17026-21kworker/0:223:42:230
2942499212,9cyclictest32722-21kworker/0:020:37:310
2942499211,8cyclictest9615-21kworker/0:122:02:290
2942499211,8cyclictest30226-21kworker/0:200:06:370
2942499211,8cyclictest30226-21kworker/0:200:06:370
2942499211,8cyclictest26571-21kworker/0:221:31:090
2942499211,8cyclictest20800-21kworker/0:322:48:410
2942499210,10cyclictest20800-21kworker/0:322:42:530
2944599205,5cyclictest0-21swapper/300:27:543
2944599201,9cyclictest0-21swapper/322:21:033
2944599201,6cyclictest12569-21ntp_states00:35:213
29445992013,4cyclictest7754-21grep19:30:193
2944599201,11cyclictest808-21snmpd21:15:143
2944599201,11cyclictest26567-21grep22:40:183
2944599201,10cyclictest23707-21head20:05:253
2944599201,10cyclictest19120-21kworker/u8:223:50:213
29445992010,3cyclictest14177-21kernelversion19:45:193
2944599200,7cyclictest0-21swapper/321:23:133
2944599200,6cyclictest0-21swapper/323:01:213
2943999205,4cyclictest5006-21latency_hist20:40:012
2943999203,5cyclictest7110-21/usr/sbin/munin00:25:232
2943999203,3cyclictest87350irq/126-enp1s0-23:47:552
2943999203,3cyclictest87350irq/126-enp1s0-00:10:252
2943999201,7cyclictest909-21sssd_nss21:53:222
2943999201,7cyclictest27504-21diskstats21:30:132
2943999201,7cyclictest14593-21/usr/sbin/munin22:15:172
2943999201,7cyclictest0-21swapper/222:07:282
2943999201,6cyclictest30-21rcuc/200:35:222
2943999201,6cyclictest19419-21dump-pmu-power23:44:562
2943999201,5cyclictest31801-21cat19:15:012
2943999201,5cyclictest0-21swapper/220:55:222
2943999201,16cyclictest14358-21cut21:00:122
2943999201,11cyclictest0-21swapper/222:37:012
2943999201,10cyclictest0-21swapper/222:51:192
29432992013,4cyclictest4580-21/usr/sbin/munin19:25:101
29432992013,4cyclictest4580-21/usr/sbin/munin19:25:101
2943299201,17cyclictest0-21swapper/123:20:061
2942499208,2cyclictest28271-21kworker/0:320:24:070
2942499207,2cyclictest31250-21kworker/0:119:12:270
2942499206,2cyclictest5838-21kworker/0:223:07:410
2942499203,7cyclictest17026-21kworker/0:223:48:050
2942499202,9cyclictest2593-21kworker/0:220:45:150
2942499201,6cyclictest0-21swapper/022:30:560
2942499201,11cyclictest29983-21/usr/sbin/munin22:50:140
2942499201,10cyclictest14710-21kworker/0:323:36:330
128632200,4sleep341-21ksoftirqd/320:55:213
2944599196,4cyclictest0-21swapper/323:35:193
2944599194,4cyclictest0-21swapper/323:24:353
2944599193,7cyclictest0-21swapper/323:16:233
2944599191,9cyclictest22838-21grep23:45:193
2944599191,7cyclictest0-21swapper/323:10:183
2944599191,5cyclictest0-21swapper/322:50:133
2944599191,11cyclictest839-21lldpd20:42:423
2944599191,11cyclictest839-21lldpd20:42:423
2944599191,10cyclictest31267-21sshd21:39:223
2944599191,10cyclictest18144-21diskmemload22:00:423
2944599191,10cyclictest0-21swapper/322:07:143
2944599190,9cyclictest0-21swapper/323:55:143
2944599190,7cyclictest0-21swapper/321:45:433
2944599190,6cyclictest0-21swapper/323:08:363
2944599190,6cyclictest0-21swapper/321:54:503
2944599190,5cyclictest0-21swapper/320:20:173
2943999196,3cyclictest87650irq/129-enp1s0-23:20:262
2943999195,3cyclictest87350irq/126-enp1s0-00:00:192
2943999192,8cyclictest0-21swapper/222:26:362
2943999192,15cyclictest0-21swapper/219:25:302
2943999192,15cyclictest0-21swapper/219:25:302
2943999191,9cyclictest0-21swapper/222:22:532
2943999191,7cyclictest0-21swapper/223:51:052
2943999191,6cyclictest25677-21basename21:25:142
2943999191,6cyclictest2368-21/usr/sbin/munin19:20:112
2943999191,6cyclictest0-21swapper/221:37:472
2943999191,5cyclictest7832-21ntp_states19:30:212
2943999191,5cyclictest32408-21cat21:40:182
2943999191,5cyclictest20912-21df_inode21:15:132
2943999191,5cyclictest13569-21ps22:10:232
2943999191,4cyclictest0-21swapper/221:45:032
29439991913,3cyclictest4895-21sed23:05:152
2943999191,11cyclictest0-21swapper/219:52:132
2943999190,10cyclictest0-21swapper/223:36:172
29432991915,3cyclictest0-21swapper/119:13:581
2942499199,6cyclictest0-21swapper/021:54:560
2942499198,3cyclictest15255-21/usr/sbin/munin23:30:170
2942499198,2cyclictest31054-21kworker/0:121:41:480
2942499198,2cyclictest28271-21kworker/0:320:19:110
2942499197,2cyclictest11893-21kworker/0:222:07:150
2944599188,3cyclictest819-21lldpd20:46:333
2944599187,3cyclictest87350irq/126-enp1s0-21:11:283
2944599185,3cyclictest87350irq/126-enp1s0-19:55:193
2944599182,13cyclictest2069-21wc20:30:253
2944599181,6cyclictest0-21swapper/323:34:143
2944599181,5cyclictest0-21swapper/321:41:303
2944599181,4cyclictest26542-21/usr/sbin/munin20:15:113
29445991812,3cyclictest808-21snmpd20:04:243
2944599181,12cyclictest0-21swapper/322:57:583
2944599181,12cyclictest0-21swapper/322:57:583
2944599181,11cyclictest0-21swapper/322:45:153
2944599180,5cyclictest0-21swapper/300:20:393
2944599180,5cyclictest0-21swapper/300:11:293
2944599180,11cyclictest0-21swapper/300:00:223
2943999184,3cyclictest87650irq/129-enp1s0-20:10:232
2943999184,3cyclictest87350irq/126-enp1s0-20:00:232
2943999184,3cyclictest11250-21sshd20:51:222
2943999183,4cyclictest1570-21sshd22:57:222
2943999183,4cyclictest1570-21sshd22:57:222
2943999182,4cyclictest28236-40updatedb00:00:012
2943999182,4cyclictest22480-21cat20:05:122
2943999181,6cyclictest9297-21/usr/sbin/munin19:35:122
2943999181,6cyclictest7374-21/usr/sbin/munin20:45:092
2943999181,5cyclictest22257-21ntp_states22:30:222
29439991813,3cyclictest16861-21hwlatdetect21:05:172
2943999181,15cyclictest0-21swapper/200:18:292
2942499188,2cyclictest8622-21/usr/sbin/munin23:15:220
2942499183,5cyclictest2368-21kworker/0:200:18:050
2942499181,3cyclictest11500-21/usr/sbin/munin00:35:220
109602181,2sleep20-21swapper/200:31:082
2944599175,7cyclictest12261-40dnf19:40:243
2944599175,3cyclictest0-21swapper/322:15:453
2944599174,4cyclictest2387-21grep19:20:113
2944599174,3cyclictest1683-21sed19:15:243
2944599171,3cyclictest17850-21sshd21:06:223
2944599171,3cyclictest10371-21ntp_states00:30:223
29445991712,3cyclictest4073-21cat00:15:263
2944599171,13cyclictest31930-21strings20:25:183
29445991711,3cyclictest29741-21grep19:10:113
2943999172,3cyclictest0-21swapper/219:46:112
2943999171,6cyclictest0-21swapper/220:20:092
2943999171,4cyclictest1113-21diskstats20:30:152
29439991713,3cyclictest0-21swapper/220:25:062
2943999171,15cyclictest0-21swapper/200:22:282
2942499175,4cyclictest22999-21kworker/0:122:36:370
265672170,2sleep00-21swapper/021:25:290
29445991612,3cyclictest0-21swapper/319:26:293
29445991612,3cyclictest0-21swapper/319:26:293
29445991611,3cyclictest14785-21cut21:00:153
2944599161,12cyclictest8956-21/usr/sbin/munin19:35:163
2943999160,15cyclictest0-21swapper/219:57:532
29445991510,3cyclictest24343-21/usr/sbin/munin20:10:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional