You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-01-03 - 02:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sat Jan 03, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
201772230163,55sleep10-21swapper/119:06:081
203792204135,56sleep20-21swapper/219:08:432
202732203166,25sleep30-21swapper/319:07:233
203282201164,25sleep00-21swapper/019:08:040
3006421270,5sleep12057499cyclictest20:40:231
20580998784,2cyclictest0-21swapper/219:10:162
322232670,4sleep081ktimersoftd/019:35:000
20580996359,3cyclictest0-21swapper/200:24:092
20580994542,2cyclictest0-21swapper/222:02:042
20569994510,4cyclictest21165-21kworker/0:300:01:460
2056999417,3cyclictest15110-21kworker/0:120:09:180
20569994016,11cyclictest87250irq/129-enp1s0-23:40:020
2056999398,3cyclictest2587-21kworker/0:221:14:500
2056999393,3cyclictest87050irq/127-enp1s0-20:46:500
20569993812,13cyclictest32526-21munin-run00:30:000
20569993812,13cyclictest32526-21munin-run00:30:000
2056999372,2cyclictest31073-21kworker/0:222:22:140
2056999368,13cyclictest17458-21awk20:15:020
20569993610,13cyclictest30639-21munin-run20:45:010
2056999335,13cyclictest8327-21basename23:35:010
2058699321,12cyclictest0-21swapper/320:38:203
2058699311,13cyclictest15067-21perl22:35:163
2058699311,10cyclictest0-21swapper/319:10:363
2056999312,10cyclictest3949-21kworker/0:019:43:450
2056999311,14cyclictest28444-21munin-run20:40:010
2056999311,12cyclictest0-21swapper/021:00:040
2056999310,24cyclictest0-21swapper/021:56:430
2058699301,12cyclictest0-21swapper/320:31:563
2056999302,17cyclictest19645-21munin-run20:20:020
2057499291,17cyclictest0-21swapper/123:10:321
2057499291,16cyclictest32246-21expr23:15:141
2057499291,16cyclictest32246-21expr23:15:141
2056999291,8cyclictest0-21swapper/019:18:020
2056999291,11cyclictest0-21swapper/020:58:380
2058699281,12cyclictest28243-21/usr/sbin/munin23:05:163
2058699281,12cyclictest0-21swapper/322:45:003
2058699281,12cyclictest0-21swapper/322:20:043
2058699281,12cyclictest0-21swapper/322:10:203
2058699281,11cyclictest0-21swapper/322:19:483
2058699281,11cyclictest0-21swapper/322:00:203
2058699280,10cyclictest0-21swapper/300:34:363
2057499289,4cyclictest8296-40dnf19:50:421
2056999286,8cyclictest16448-21kworker/0:322:46:540
2056999281,9cyclictest0-21swapper/019:49:540
191102280,3sleep00-21swapper/021:30:190
2058699271,12cyclictest0-21swapper/322:46:363
2058699271,11cyclictest0-21swapper/321:30:273
2058699271,10cyclictest0-21swapper/321:59:003
2058699270,11cyclictest0-21swapper/319:32:123
2057499272,11cyclictest27495-21munin-run23:05:001
2057499272,11cyclictest21115-21/usr/sbin/munin19:10:151
2056999278,3cyclictest0-21swapper/023:29:320
2056999271,11cyclictest0-21swapper/023:20:120
2056999271,11cyclictest0-21swapper/020:27:080
2056999270,12cyclictest31073-21kworker/0:222:18:000
2058699261,12cyclictest0-21swapper/323:45:153
2058699261,11cyclictest0-21swapper/320:27:083
2058699261,11cyclictest0-21swapper/300:22:283
2058699261,10cyclictest0-21swapper/321:29:233
2058699260,13cyclictest0-21swapper/319:25:243
20574992614,3cyclictest0-21swapper/121:35:411
2057499261,12cyclictest0-21swapper/119:21:521
2057499261,11cyclictest24664-21munin-run21:45:011
2057499260,2cyclictest0-21swapper/122:01:111
2056999268,8cyclictest31073-21kworker/0:222:13:440
2056999266,3cyclictest22072-21kworker/0:120:30:080
2056999261,12cyclictest0-21swapper/022:31:480
2056999261,12cyclictest0-21swapper/000:35:070
2056999261,11cyclictest0-21swapper/020:20:120
2056999261,11cyclictest0-21swapper/019:28:510
2056999261,10cyclictest0-21swapper/019:23:400
2056999260,12cyclictest0-21swapper/023:44:200
2058699251,11cyclictest0-21swapper/323:18:283
2058699251,11cyclictest0-21swapper/323:18:283
2058699251,11cyclictest0-21swapper/322:56:363
2058699251,11cyclictest0-21swapper/322:52:123
2058699251,11cyclictest0-21swapper/322:32:523
2058699251,10cyclictest0-21swapper/319:22:123
2058699250,11cyclictest0-21swapper/323:23:563
2057499251,8cyclictest6681-21df_inode23:30:141
2057499251,4cyclictest30330-21munin-run00:25:011
2057499251,2cyclictest0-21swapper/122:16:141
2056999258,3cyclictest3949-21kworker/0:019:54:020
2056999258,15cyclictest18697-21perf22:45:020
2056999257,3cyclictest2587-21kworker/0:221:28:020
2056999256,3cyclictest23378-21kworker/0:000:08:460
2056999256,3cyclictest22867-21kworker/0:221:42:070
2056999251,13cyclictest3302-21munin-run22:10:000
2056999251,13cyclictest3302-21munin-run22:10:000
2056999251,11cyclictest0-21swapper/021:06:280
2056999251,10cyclictest0-21swapper/023:58:280
2056999250,14cyclictest22711-21kworker/0:122:58:540
204582250,3sleep1221ktimersoftd/120:20:171
255142240,4sleep22058099cyclictest20:30:242
2058699241,16cyclictest0-21swapper/319:40:203
2058699241,13cyclictest0-21swapper/300:28:303
2058699241,13cyclictest0-21swapper/300:28:303
2058699241,12cyclictest0-21swapper/323:11:163
2058699241,11cyclictest0-21swapper/321:39:553
2058699241,10cyclictest0-21swapper/321:21:553
2058699241,10cyclictest0-21swapper/300:38:443
2057499249,12cyclictest5866-21systemd-journal23:27:011
2057499241,3cyclictest16848-40dnf23:51:551
2057499240,5cyclictest0-21swapper/123:40:201
2057499240,17cyclictest0-21swapper/123:20:401
2056999248,3cyclictest0-21swapper/019:56:040
2056999242,9cyclictest14146-21kworker/0:122:36:580
2056999241,13cyclictest0-21swapper/023:46:400
2056999241,11cyclictest26867-21awk21:50:000
2056999241,11cyclictest0-21swapper/020:03:400
2056999241,11cyclictest0-21swapper/019:10:280
2056999241,10cyclictest0-21swapper/021:22:460
2056999241,10cyclictest0-21swapper/000:13:000
3342230,4sleep232-21ksoftirqd/219:35:152
3342230,4sleep232-21ksoftirqd/219:35:152
206492230,1sleep30-21swapper/300:00:223
2058699236,9cyclictest25822-21cstates23:00:133
2058699235,4cyclictest15257-21tr23:50:123
2058699231,11cyclictest22050-21cat20:25:023
2058699230,10cyclictest0-21swapper/323:26:443
2057499231,3cyclictest32666-21chrt20:48:021
2057499231,3cyclictest28443-21perf20:40:011
2057499231,16cyclictest0-21swapper/121:19:491
2057499231,16cyclictest0-21swapper/120:34:451
2057499231,16cyclictest0-21swapper/100:31:041
2057499231,12cyclictest0-21swapper/123:46:321
2056999236,3cyclictest26497-21kworker/0:121:51:510
2056999235,2cyclictest21165-21kworker/0:300:16:020
2056999234,7cyclictest31179-21kworker/0:322:01:030
2056999234,4cyclictest2587-21kworker/0:221:16:080
2056999231,9cyclictest0-21swapper/020:52:520
2056999231,9cyclictest0-21swapper/020:52:520
20569992310,2cyclictest18657-21kworker/0:022:50:260
2056999230,6cyclictest32097-21kworker/0:319:35:290
2056999230,6cyclictest32097-21kworker/0:319:35:290
2058699221,13cyclictest0-21swapper/321:54:123
2058699221,11cyclictest8308-21latency_hist23:35:023
2058699221,11cyclictest0-21swapper/323:58:033
2057499221,19cyclictest0-21swapper/122:20:221
2057499221,18cyclictest9278-21diskmemload22:30:531
2057499221,18cyclictest19875-21diskstats00:00:151
2057499221,15cyclictest0-21swapper/123:35:171
2056999227,3cyclictest0-21swapper/023:19:080
2056999227,3cyclictest0-21swapper/023:19:080
2056999226,3cyclictest0-21swapper/000:22:190
2056999225,3cyclictest14620-21kworker/0:223:50:460
2056999221,11cyclictest0-21swapper/023:00:360
2056999220,11cyclictest0-21swapper/023:09:400
123832220,2sleep30-21swapper/321:15:173
2058699215,8cyclictest32249-21timerandwakeup20:45:253
2058699212,2cyclictest23213-21fschecks_time21:40:133
2058699211,2cyclictest0-21swapper/319:15:453
2058699211,11cyclictest3295-21latency_hist22:10:023
2058699210,12cyclictest0-21swapper/300:14:273
2058099216,7cyclictest751-21rngd21:10:012
2058099215,7cyclictest10867-21cpuspeed_turbos23:40:112
2058099211,8cyclictest0-21swapper/222:32:202
2058099211,4cyclictest1587-21kworker/u8:023:19:542
2058099211,4cyclictest1587-21kworker/u8:023:19:542
2058099211,4cyclictest0-21swapper/221:15:382
20580992114,4cyclictest25490-21grep00:10:272
20574992116,3cyclictest31141-21cat19:30:191
20574992115,3cyclictest17386-21hddtemp_smartct22:40:161
2057499211,3cyclictest3928-21chrt00:35:251
2057499211,17cyclictest28883-21ps23:05:231
2057499211,15cyclictest0-21swapper/120:19:061
2057499211,14cyclictest0-21swapper/120:51:121
2057499211,14cyclictest0-21swapper/120:51:121
2057499211,14cyclictest0-21swapper/100:09:491
2057499211,12cyclictest24453-21fschecks_time00:10:171
2057499211,11cyclictest8771-21chrt21:05:281
2057499211,11cyclictest17579-21expr23:55:121
2056999218,3cyclictest27137-21kworker/0:323:10:340
2056999215,5cyclictest9710-21kworker/0:322:26:540
2056999215,3cyclictest20206-21kworker/0:021:37:580
2056999215,2cyclictest32198-21kworker/0:100:33:270
184992210,2sleep318622-21missed_timers20:15:223
2058699202,10cyclictest6673-21/usr/sbin/munin19:50:233
2058699201,5cyclictest26148-21smartctl21:45:233
2058699201,11cyclictest21752-21/usr/sbin/munin00:05:113
2058699201,11cyclictest1308-21/usr/sbin/munin22:05:203
2058699201,10cyclictest10508-21latency_hist23:40:023
2058699200,12cyclictest0-21swapper/321:14:273
2058099203,5cyclictest1-21systemd21:40:282
2058099201,7cyclictest0-21swapper/223:39:332
2058099201,6cyclictest9278-21diskmemload21:20:172
2058099201,4cyclictest0-21swapper/221:14:222
2058099201,10cyclictest0-21swapper/222:40:212
2057499209,3cyclictest30193-21memory21:55:181
2057499208,5cyclictest0-21swapper/122:56:551
2057499204,9cyclictest801-21snmpd22:51:421
20574992015,3cyclictest7091-21awk21:05:011
2057499201,18cyclictest0-21swapper/122:29:531
2057499201,18cyclictest0-21swapper/122:05:351
2057499201,18cyclictest0-21swapper/122:05:341
2057499201,18cyclictest0-21swapper/120:12:471
2057499201,18cyclictest0-21swapper/119:55:041
2057499201,17cyclictest0-21swapper/122:38:461
2057499201,17cyclictest0-21swapper/119:47:071
2057499201,17cyclictest0-21swapper/119:25:081
2057499201,14cyclictest0-21swapper/121:50:401
2057499201,14cyclictest0-21swapper/120:29:201
2057499201,14cyclictest0-21swapper/100:25:461
2057499201,14cyclictest0-21swapper/100:25:461
2057499201,14cyclictest0-21swapper/100:16:561
20574992011,3cyclictest0-21swapper/121:25:261
2057499201,10cyclictest11610-21cat21:15:021
2057499200,18cyclictest0-21swapper/119:15:131
2058699197,9cyclictest401ktimersoftd/320:50:153
2058699197,9cyclictest401ktimersoftd/320:50:153
2058699194,3cyclictest9495-21head19:55:163
2058699191,5cyclictest0-21swapper/321:10:003
2058699191,10cyclictest4481-21/usr/sbin/munin19:45:243
2058099193,4cyclictest9278-21diskmemload22:10:182
2058099191,6cyclictest0-21swapper/223:30:112
2058099191,6cyclictest0-21swapper/222:51:332
20580991913,4cyclictest29947-21grep21:55:152
2058099191,10cyclictest0-21swapper/221:34:412
2058099190,11cyclictest0-21swapper/219:51:482
2057499197,3cyclictest14960-21ntp_states21:20:211
2057499197,10cyclictest0-21swapper/120:00:581
20574991912,4cyclictest14866-21users20:05:251
2057499191,16cyclictest0-21swapper/121:46:081
2057499191,16cyclictest0-21swapper/119:42:591
2057499191,10cyclictest19082-21/usr/sbin/munin21:30:181
2057499190,14cyclictest0-21swapper/120:55:071
2058699186,3cyclictest801-21snmpd19:35:123
2058699186,3cyclictest801-21snmpd19:35:123
2058699181,9cyclictest16158-21smartctl20:10:183
2058699181,5cyclictest28682-21grep20:40:113
2058699181,5cyclictest11153-21fschecks_count23:40:143
2058699181,4cyclictest577-21systemd-journal21:00:003
2058699181,10cyclictest801-21snmpd22:27:463
2058699181,10cyclictest0-21swapper/300:17:313
2058099187,3cyclictest86950irq/126-enp1s0-23:25:122
2058099185,5cyclictest0-21swapper/222:08:552
2058099185,5cyclictest0-21swapper/222:08:552
2058099182,6cyclictest22199-21grep20:25:122
2058099181,7cyclictest25862-21kworker/u8:122:49:052
2058099181,7cyclictest22002-21tail21:35:262
2058099181,7cyclictest0-21swapper/222:39:522
2058099181,7cyclictest0-21swapper/221:49:372
2058099181,6cyclictest0-21swapper/222:26:052
2058099181,5cyclictest30421-1kworker/u9:222:20:212
2058099181,5cyclictest15468-21df23:50:142
2058099181,3cyclictest3246-21df_inode20:55:132
20580991812,3cyclictest29909-21/usr/sbin/munin23:10:232
2058099181,15cyclictest0-21swapper/222:16:442
2058099181,15cyclictest0-21swapper/219:31:052
20574991813,3cyclictest5120-21vmstat22:10:231
20574991812,3cyclictest20514-21seq22:45:311
2057499181,15cyclictest0-21swapper/119:35:071
2057499181,15cyclictest0-21swapper/119:35:061
20586991710,4cyclictest5564-21tune2fs21:00:163
2058099179,3cyclictest0-21swapper/220:53:532
2058099179,3cyclictest0-21swapper/220:53:532
2058099176,4cyclictest25248-21taskset22:59:452
2058099176,2cyclictest14763-21kworker/u8:200:02:192
2058099175,3cyclictest21103-21gzip20:20:232
2058099173,6cyclictest0-21swapper/200:05:462
2058099171,7cyclictest27423-21df_inode21:50:142
2058099171,7cyclictest13061-21munin-run20:05:012
2058099171,6cyclictest3401-21irqstats00:35:182
2058099171,5cyclictest0-21swapper/221:25:212
2058099171,5cyclictest0-21swapper/200:25:152
2058099171,5cyclictest0-21swapper/200:25:152
20580991713,3cyclictest0-21swapper/223:47:402
20580991713,3cyclictest0-21swapper/223:23:422
20580991713,3cyclictest0-21swapper/219:25:372
20580991712,3cyclictest28358-21head23:05:152
2058699163,3cyclictest86950irq/126-enp1s0-20:00:173
2058699161,12cyclictest15080-21NetworkManager20:07:593
2058099165,3cyclictest18845-21ps20:15:252
2058099165,3cyclictest12050irq/123-ahci[0019:20:162
2058099163,4cyclictest87250irq/129-enp1s0-19:45:142
2058099162,7cyclictest0-21swapper/200:17:182
2058099161,6cyclictest577-21systemd-journal20:11:342
2058099161,6cyclictest30863-21/usr/sbin/munin20:45:132
2058099161,6cyclictest29585-21missed_timers20:40:172
2058099161,6cyclictest2343-21grep19:40:122
20580991612,3cyclictest0-21swapper/221:02:562
20580991611,3cyclictest9078-21expr19:55:132
20580991611,3cyclictest26470-21/usr/sbin/munin20:35:222
20580991611,3cyclictest23658-21/usr/sbin/munin19:15:112
20580991611,3cyclictest18356-21ntp_kernel_err23:55:192
20580991611,3cyclictest14308-21nfs4_client20:05:202
20580991610,3cyclictest25799-21expr23:00:122
20580991610,3cyclictest1050-21sh00:30:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional