You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-27 - 05:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Mon Jul 27, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
103402240164,25sleep10-21swapper/119:08:141
101342221192,19sleep20-21swapper/219:05:342
100822207177,20sleep00-21swapper/019:05:280
102292203167,24sleep30-21swapper/319:06:483
854821570,5sleep31058299cyclictest23:55:223
90732600,5sleep01056599cyclictest22:45:130
113182550,2sleep10-21swapper/119:10:151
10575995148,2cyclictest0-21swapper/200:33:002
1057099407,17cyclictest4323-21latency_hist22:35:011
1057099401,5cyclictest18048-21chrt19:25:111
1057099401,28cyclictest22048-21fschecks_time22:00:151
1057099394,30cyclictest4596-40dnf20:05:271
1057099391,3cyclictest25544-21tune2fs00:35:151
1057099391,3cyclictest25544-21tune2fs00:35:141
1057099391,28cyclictest19091-21latency_hist21:55:011
1057099382,33cyclictest17567-21munin-run23:05:001
1057099381,3cyclictest22730-21cat00:30:021
1057099377,16cyclictest3089-21latency_hist20:05:011
1057099372,3cyclictest29478-21munin-run21:05:011
1057099371,3cyclictest8333-21sort23:55:201
1057099371,21cyclictest26905-21cpu19:45:121
1057099369,14cyclictest14865-21sed21:45:021
10570993614,19cyclictest18210-21munin-run00:20:011
1057099361,3cyclictest7187-21munin-run23:55:021
295032350,4sleep231680-21diskmemload23:30:172
1057099356,3cyclictest87550irq/128-enp1s0-22:50:131
1057099352,4cyclictest9688-21munin-run20:20:021
1057099351,31cyclictest24071-21cat20:50:201
1057099345,2cyclictest24451-21latency_hist19:40:001
1057099342,3cyclictest19246-21irqstats00:20:181
1057099341,5cyclictest8985-21/usr/sbin/munin22:45:231
1057099340,9cyclictest0-21swapper/121:07:451
1057099340,9cyclictest0-21swapper/121:07:451
1056599347,13cyclictest25073-21munin-run20:55:010
23712330,4sleep230-21rcuc/220:00:242
1057599331,10cyclictest0-21swapper/219:16:542
1057099334,3cyclictest0-21swapper/100:30:241
1057099332,4cyclictest22869-21sh20:50:011
1057099331,5cyclictest28545-21latency_hist23:30:021
1057099331,5cyclictest13386-21/usr/sbin/munin22:55:121
1057099331,3cyclictest391-21/usr/sbin/munin22:25:141
1057099331,29cyclictest0-21swapper/119:20:191
1057599321,23cyclictest0-21swapper/223:54:182
1057099329,20cyclictest31275-21/usr/sbin/munin19:55:221
1057099329,20cyclictest31275-21/usr/sbin/munin19:55:211
1057099329,20cyclictest11920-21cpuspeed_turbos00:05:141
1057099323,3cyclictest32063-21cpuspeed_turbos21:10:141
1057099322,3cyclictest8106-21munin-run21:30:021
1057099321,6cyclictest12494-21crond21:40:021
1056599321,15cyclictest19766-21munin-run23:10:010
40012310,7sleep230-21rcuc/221:20:152
1057099315,2cyclictest7747-21kworker/u8:121:30:121
1057099314,2cyclictest87650irq/129-enp1s0-23:40:271
1057099311,5cyclictest22187-21/usr/sbin/munin23:15:251
1057099311,4cyclictest0-21swapper/100:00:281
1057099311,3cyclictest493-21munin-run23:40:021
1057099311,3cyclictest3906-21cat21:20:021
1057099311,3cyclictest11242-21sed20:20:251
10570993111,17cyclictest16915-21basename21:50:011
10570993110,18cyclictest3621-21tail23:45:161
207962300,4sleep11057099cyclictest21:55:261
1057599301,8cyclictest10048-21ntp_states22:45:212
1057599301,16cyclictest0-21swapper/223:15:572
1057099308,10cyclictest0-21swapper/122:05:101
1057099303,4cyclictest30979-21gltestperf22:20:151
1057099303,2cyclictest0-21swapper/123:20:211
1057099301,6cyclictest0-21swapper/123:09:551
1057099301,5cyclictest0-21swapper/123:10:241
1057099301,4cyclictest5656-21/usr/sbin/munin22:35:241
1057099301,4cyclictest19876-21/usr/sbin/munin20:40:241
1057099301,3cyclictest7118-21dnf22:40:141
1057099301,3cyclictest4278-21df_inode21:20:161
1057099301,3cyclictest14859-21irqstats00:10:211
10570993011,16cyclictest13352-21latency19:15:041
10570993010,17cyclictest6857-21threads20:10:271
1056599307,10cyclictest4341-21awk22:35:020
1057099298,17cyclictest26316-21tune2fs22:10:141
1057099295,3cyclictest25357-21/usr/sbin/munin19:40:161
1057099291,4cyclictest29932-21/usr/sbin/munin23:30:241
1057099291,4cyclictest17579-21ntp_states20:35:231
1057099291,3cyclictest25637-21df_inode20:55:141
10570992910,15cyclictest827-21runrttasks20:30:261
1056599291,13cyclictest19101-21munin-run21:55:000
1058299286,14cyclictest4912-21users20:05:283
1057599280,9cyclictest0-21swapper/221:28:182
1057599280,12cyclictest0-21swapper/221:42:522
1057599280,12cyclictest0-21swapper/219:50:132
1057099288,17cyclictest20920-21cut19:30:151
1057099282,3cyclictest909-21sssd_nss19:50:151
1057099281,3cyclictest12449-21df20:25:131
1057099281,2cyclictest87350irq/126-enp1s0-22:15:121
119162270,4sleep311918-21cpuspeed_turbos00:05:133
1057599270,12cyclictest0-21swapper/221:37:122
1057599270,12cyclictest0-21swapper/200:20:182
1057599270,11cyclictest0-21swapper/223:05:292
1057599270,10cyclictest0-21swapper/200:08:522
1056599271,12cyclictest16026-21basename00:15:000
10582992613,3cyclictest0-21swapper/300:00:433
1058299261,19cyclictest87350irq/126-enp1s0-23:39:253
1057599261,17cyclictest0-21swapper/223:59:002
1057599261,12cyclictest0-21swapper/222:30:332
1057599260,19cyclictest0-21swapper/223:35:172
1057599260,11cyclictest0-21swapper/222:15:222
1057599260,11cyclictest0-21swapper/222:08:182
1057599260,11cyclictest0-21swapper/221:16:382
1057599260,11cyclictest0-21swapper/219:45:272
1057599260,11cyclictest0-21swapper/200:26:562
1057599260,10cyclictest0-21swapper/220:56:252
1057599260,10cyclictest0-21swapper/219:20:182
1056599269,14cyclictest22616-21basename00:30:000
1056599269,14cyclictest21977-21idleruntime-cro23:15:010
1056599261,12cyclictest20681-21munin-run20:45:010
10582992513,3cyclictest26656-21awk19:45:013
1058299251,17cyclictest0-21swapper/319:25:243
1058299251,11cyclictest2655-21memory21:15:203
1058299251,10cyclictest9280-21kworker/u8:100:11:303
1057599253,5cyclictest0-21swapper/223:40:192
1057599251,8cyclictest0-21swapper/200:16:462
1057599251,7cyclictest0-21swapper/220:21:572
1057599251,10cyclictest0-21swapper/222:53:392
1057599250,11cyclictest0-21swapper/200:14:402
1056599251,9cyclictest32556-21tail21:10:170
1056599251,12cyclictest5299-21munin-run20:10:010
1056599251,12cyclictest2790-21munin-run23:45:020
1058299242,7cyclictest12038-21sed20:25:013
1057599241,9cyclictest0-21swapper/220:35:212
1057599240,17cyclictest0-21swapper/222:25:442
1057599240,12cyclictest0-21swapper/221:55:442
1057599240,10cyclictest0-21swapper/222:59:192
1057599240,10cyclictest0-21swapper/221:54:182
1056599249,9cyclictest0-21swapper/022:21:580
1056599244,13cyclictest7-21ksoftirqd/021:30:040
1056599242,3cyclictest87550irq/128-enp1s0-21:20:180
1056599242,15cyclictest0-21swapper/022:12:330
1056599242,10cyclictest17113-21tail20:35:170
10565992420,3cyclictest0-21swapper/021:05:220
10565992420,3cyclictest0-21swapper/021:05:220
1056599241,7cyclictest8049-21kworker/0:221:46:360
1056599241,17cyclictest0-21swapper/023:55:060
1056599241,17cyclictest0-21swapper/021:29:380
1056599241,16cyclictest0-21swapper/000:36:570
1056599241,16cyclictest0-21swapper/000:36:570
1056599241,11cyclictest31823-21cut23:35:210
1058299239,3cyclictest4826-21expr22:35:153
1058299236,3cyclictest87350irq/126-enp1s0-23:40:133
1058299232,10cyclictest22834-21gltestperf23:15:153
1058299231,20cyclictest0-21swapper/320:50:063
1058299231,16cyclictest835-21basename22:25:183
1058299231,11cyclictest29340-21sshd21:03:253
1058299231,10cyclictest6733-21grep21:25:163
1058299231,10cyclictest3945-21users22:30:283
1058299231,10cyclictest13376-21smart_sda20:25:233
1058299231,10cyclictest0-21swapper/321:20:263
1058299230,21cyclictest0-21swapper/320:46:243
1057599230,8cyclictest0-21swapper/222:39:212
1057599230,11cyclictest0-21swapper/223:01:212
1057599230,10cyclictest0-21swapper/223:13:372
1057599230,10cyclictest0-21swapper/221:33:222
1056599237,9cyclictest0-21swapper/023:21:270
10565992317,4cyclictest0-21swapper/022:59:260
10565992317,3cyclictest26903-21cpu19:45:130
1056599231,5cyclictest808-21snmpd22:15:050
1056599231,17cyclictest0-21swapper/000:03:310
1056599231,16cyclictest0-21swapper/023:33:280
1056599231,16cyclictest0-21swapper/022:03:470
1056599231,16cyclictest0-21swapper/000:15:430
1056599231,15cyclictest0-21swapper/023:15:360
1056599231,15cyclictest0-21swapper/022:50:220
1056599231,13cyclictest19770-21sshd00:20:240
1056599230,16cyclictest0-21swapper/020:10:180
204402220,4sleep011-21rcuc/019:30:120
165352220,2sleep30-21swapper/320:35:123
1058299221,5cyclictest26582-21/usr/sbin/munin23:25:193
1058299221,19cyclictest0-21swapper/319:21:273
1058299221,17cyclictest808-21snmpd23:05:163
1058299221,11cyclictest0-21swapper/321:50:183
1058299221,10cyclictest30319-21/usr/sbin/munin22:20:273
1058299221,10cyclictest16071-21sshd20:32:253
1058299221,10cyclictest14928-21/usr/sbin/munin21:45:273
1057599221,9cyclictest0-21swapper/221:12:222
1057599221,11cyclictest0-21swapper/219:57:532
1057599221,11cyclictest0-21swapper/219:57:532
1057599220,9cyclictest0-21swapper/200:03:002
1057599220,11cyclictest0-21swapper/223:22:092
1056599229,3cyclictest0-21swapper/019:52:410
1056599228,3cyclictest27300-21basename23:25:180
1056599227,3cyclictest26271-21users19:40:260
1056599221,9cyclictest15932-21diskstats23:00:150
1056599221,8cyclictest16923-21open_files19:20:200
1056599221,16cyclictest0-21swapper/021:55:140
1056599221,16cyclictest0-21swapper/021:42:130
1056599221,16cyclictest0-21swapper/021:18:070
1056599221,15cyclictest0-21swapper/023:50:220
1056599221,10cyclictest7942-21ps22:40:240
1056599221,10cyclictest12216-21fschecks_count00:05:160
1056599221,10cyclictest0-21swapper/020:15:270
1058299218,2cyclictest39-21rcuc/300:25:273
1058299217,3cyclictest11187-21/usr/sbin/munin22:50:163
1058299216,5cyclictest839-21lldpd23:54:493
1058299216,4cyclictest39-21rcuc/323:20:263
1058299216,3cyclictest30691-21needreboot21:05:203
1058299216,3cyclictest30691-21needreboot21:05:203
1058299214,4cyclictest579-21systemd-journal21:40:013
1058299211,6cyclictest23051-21unixbench_multi22:00:273
10582992115,4cyclictest31680-21diskmemload21:43:503
1058299211,17cyclictest8329-21/usr/sbin/munin21:30:193
1058299211,12cyclictest87350irq/126-enp1s0-19:46:273
1058299211,10cyclictest0-21swapper/320:15:473
10582992110,2cyclictest0-21swapper/319:35:193
1057599215,3cyclictest0-21swapper/222:12:222
1057599211,11cyclictest0-21swapper/221:06:222
1057599211,11cyclictest0-21swapper/221:06:212
1057599210,10cyclictest0-21swapper/223:45:492
1057599210,10cyclictest0-21swapper/221:48:102
1056599218,2cyclictest0-21swapper/021:00:100
1056599216,10cyclictest0-21swapper/022:25:190
1056599215,10cyclictest0-21swapper/021:37:410
1056599212,14cyclictest0-21swapper/000:30:250
1056599211,10cyclictest0-21swapper/019:29:320
1058299208,2cyclictest0-21swapper/319:30:163
1058299205,3cyclictest24506-21if_err_enp1s022:05:173
1058299203,14cyclictest0-21swapper/300:22:253
1058299201,6cyclictest25998-21sed00:35:193
1058299201,6cyclictest25998-21sed00:35:193
1058299201,4cyclictest0-21swapper/323:45:033
1058299201,4cyclictest0-21swapper/323:10:453
1058299201,16cyclictest32417-21pmu-power19:55:223
1058299201,16cyclictest32417-21pmu-power19:55:223
1058299201,10cyclictest0-21swapper/322:40:273
1058299200,18cyclictest0-21swapper/320:00:223
1057599204,10cyclictest0-21swapper/222:22:082
1057599201,10cyclictest24806-21chrt20:51:452
1057599201,10cyclictest0-21swapper/220:07:292
1057599200,13cyclictest0-21swapper/222:04:062
1057599200,11cyclictest0-21swapper/219:29:582
1056599205,3cyclictest1925-21irqstats20:00:180
1056599201,9cyclictest16481-21cat20:35:020
10565992017,1cyclictest0-21swapper/020:45:230
1056599201,6cyclictest0-21swapper/022:36:010
1056599201,11cyclictest0-21swapper/022:07:380
1056599200,10cyclictest0-21swapper/019:36:260
1058299194,3cyclictest87650irq/129-enp1s0-22:55:193
1058299194,3cyclictest0-21swapper/320:55:433
1058299191,8cyclictest27100-21ps22:10:233
1058299191,5cyclictest808-21snmpd23:00:153
1058299191,5cyclictest32732-21kernelversion21:10:183
1058299191,5cyclictest28781-21/usr/sbin/munin23:30:133
1058299191,5cyclictest28123-21munin-node22:15:133
1058299191,5cyclictest11827-21ntp_states19:10:203
10582991914,3cyclictest18204-21dump-pmu-power00:20:003
10582991913,3cyclictest13412-21processes19:15:033
1058299191,15cyclictest0-21swapper/320:14:433
1057599196,4cyclictest21852-21missed_timers20:45:172
1057599192,14cyclictest28106-21grep23:25:262
1057599190,12cyclictest0-21swapper/220:12:012
1056599191,5cyclictest25296-21/usr/sbin/munin20:55:260
10565991912,4cyclictest9910-21/usr/sbin/munin20:20:130
1056599190,9cyclictest0-21swapper/023:45:130
116072180,4sleep011-21rcuc/019:10:180
1058299186,2cyclictest39-21rcuc/322:45:243
1058299182,5cyclictest20081-21/usr/sbin/munin20:40:253
1058299181,8cyclictest0-21swapper/300:30:273
1058299181,5cyclictest31246-21cat19:55:023
1058299181,14cyclictest808-21snmpd21:55:023
1057599186,3cyclictest0-21swapper/219:31:322
1057599184,3cyclictest0-21swapper/219:40:252
1057599181,7cyclictest7101-21df22:40:142
1056599184,3cyclictest15117-21timerwakeupswit19:15:270
1056599181,5cyclictest32085-21irqstats19:55:170
1056599181,5cyclictest32085-21irqstats19:55:170
1057599178,3cyclictest0-21swapper/220:40:232
1057599171,7cyclictest12730-21latency_hist19:15:012
1057599171,6cyclictest28562-21ntp_states21:00:192
1057599171,3cyclictest25837-21irqstats00:35:182
1057599171,3cyclictest25837-21irqstats00:35:182
10575991712,3cyclictest7855-21cpuspeed_turbos20:15:122
1056599174,11cyclictest0-21swapper/020:26:250
1057599165,3cyclictest87550irq/128-enp1s0-20:25:232
10575991611,3cyclictest23158-21hddtemp_smartct19:35:182
10575991510,3cyclictest15351-21ntp_kernel_pll_20:30:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional