You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-22 - 08:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat Aug 22, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
183912238163,24sleep20-21swapper/219:08:052
184142236158,26sleep00-21swapper/019:08:240
183432217165,40sleep10-21swapper/119:07:291
183202202166,24sleep30-21swapper/319:07:103
1629521500,3sleep01862899cyclictest23:55:180
1423521370,4sleep01862899cyclictest23:50:200
18640991103,105cyclictest0-21swapper/221:01:282
18640998986,2cyclictest0-21swapper/221:11:272
18640998885,2cyclictest0-21swapper/222:39:422
18640997471,2cyclictest0-21swapper/200:39:162
110892480,2sleep121-21rcuc/123:45:111
1862899367,13cyclictest2328-21idleruntime-cro19:45:010
1864599343,20cyclictest10516-21grep22:30:113
1862899334,14cyclictest5753-21munin-run22:20:010
1863399311,12cyclictest0-21swapper/120:20:391
1863399311,10cyclictest0-21swapper/123:03:441
1862899313,13cyclictest17713-21awk20:20:010
1863399300,11cyclictest0-21swapper/122:19:481
1863399300,10cyclictest0-21swapper/121:30:271
1863399291,13cyclictest0-21swapper/119:43:311
1863399290,5cyclictest0-21swapper/100:38:401
1863399290,23cyclictest0-21swapper/121:45:111
1862899294,12cyclictest26192-21awk00:20:020
1863399281,13cyclictest0-21swapper/123:25:031
1864599272,9cyclictest29948-21perf23:15:013
1864599271,5cyclictest0-21swapper/322:40:213
18645992715,8cyclictest0-21swapper/321:32:593
1863399271,12cyclictest0-21swapper/100:32:041
1862899272,14cyclictest28084-21munin-run19:30:020
1862899272,13cyclictest21151-21munin-run22:55:020
1864599262,13cyclictest0-21swapper/300:34:243
1864599261,7cyclictest26585-21irqstats23:05:193
1863399262,5cyclictest26182-21munin-run00:20:021
1863399261,17cyclictest32032-21memory20:50:201
1863399261,12cyclictest0-21swapper/122:31:241
1863399261,12cyclictest0-21swapper/121:56:151
1863399261,11cyclictest0-21swapper/119:28:561
1863399261,10cyclictest0-21swapper/120:43:321
1863399260,10cyclictest0-21swapper/120:36:391
1863399260,10cyclictest0-21swapper/119:21:161
44712250,3sleep31864599cyclictest23:30:133
1864599257,9cyclictest29538-21/usr/sbin/munin22:00:183
1864599254,2cyclictest24778-21/usr/sbin/munin23:00:223
1864599251,12cyclictest0-21swapper/300:17:543
1864599251,11cyclictest0-21swapper/322:50:333
1864099256,3cyclictest7473-21latency_hist21:10:012
1863399259,7cyclictest0-21swapper/120:28:211
1863399254,2cyclictest0-21swapper/121:20:251
1863399251,9cyclictest0-21swapper/123:43:551
1863399251,9cyclictest0-21swapper/123:43:551
1863399251,9cyclictest0-21swapper/119:54:251
1863399251,8cyclictest0-21swapper/120:34:051
1863399251,12cyclictest0-21swapper/100:25:161
1863399251,11cyclictest0-21swapper/119:59:351
1863399251,10cyclictest0-21swapper/120:57:241
1863399251,10cyclictest0-21swapper/119:37:021
1863399251,10cyclictest0-21swapper/119:31:141
1863399251,10cyclictest0-21swapper/100:13:361
1864599246,6cyclictest7001-21systemd-journal21:05:573
1864599245,8cyclictest24116-21cat00:15:023
1864599245,6cyclictest21308-21fschecks_time21:40:143
1864599241,9cyclictest0-21swapper/323:39:023
1864599241,9cyclictest0-21swapper/321:26:163
18645992417,4cyclictest25786-21gltestperf21:50:163
1864599241,10cyclictest32362-21/usr/sbin/munin23:20:263
1864099244,6cyclictest32706-40dnf00:34:252
1863399248,3cyclictest0-21swapper/120:16:071
1863399248,3cyclictest0-21swapper/119:16:391
1863399241,9cyclictest0-21swapper/123:55:551
1863399241,9cyclictest0-21swapper/123:17:431
1863399241,9cyclictest0-21swapper/119:14:571
1863399241,8cyclictest0-21swapper/122:05:321
1863399241,7cyclictest0-21swapper/123:11:341
1863399241,12cyclictest0-21swapper/122:45:121
1863399241,11cyclictest909-21sssd_nss21:10:171
1863399241,11cyclictest0-21swapper/122:55:121
1863399241,11cyclictest0-21swapper/122:27:321
1863399241,11cyclictest0-21swapper/122:00:151
1863399241,11cyclictest0-21swapper/120:47:191
1863399241,10cyclictest0-21swapper/123:39:431
1863399241,10cyclictest0-21swapper/123:24:391
1863399241,10cyclictest0-21swapper/122:43:281
1863399241,10cyclictest0-21swapper/121:50:111
1863399241,10cyclictest0-21swapper/121:37:391
1863399240,10cyclictest0-21swapper/100:20:161
1862899241,10cyclictest27521-21taskset23:07:170
1864599238,3cyclictest3797-21vmstat23:25:253
1864599236,6cyclictest0-21swapper/300:05:023
1864599235,7cyclictest18824-21/usr/sbin/munin19:10:143
1864599233,10cyclictest6934-21/usr/sbin/munin19:55:183
1864599231,6cyclictest0-21swapper/322:48:463
1864099233,5cyclictest18147-40dnf21:32:582
1864099231,5cyclictest0-21swapper/223:00:212
1863399237,8cyclictest0-21swapper/120:10:121
1863399237,3cyclictest0-21swapper/120:00:471
1863399231,9cyclictest0-21swapper/122:11:321
1863399231,11cyclictest0-21swapper/121:07:161
1863399231,11cyclictest0-21swapper/119:48:051
1862899230,11cyclictest12363-21perf22:35:010
1864599228,3cyclictest87350irq/126-enp1s0-21:20:163
1864599225,8cyclictest26109-21/usr/sbin/munin19:25:223
1864599225,3cyclictest20929-21grep00:05:233
1864599224,8cyclictest29565-21mailstats19:30:233
1864599224,8cyclictest17832-21latency_hist20:20:013
1864599224,3cyclictest0-21swapper/321:16:423
1864599224,3cyclictest0-21swapper/321:16:423
1864599222,14cyclictest0-21swapper/321:48:093
1864599221,6cyclictest0-21swapper/323:17:433
18645992214,2cyclictest0-21swapper/322:20:013
1864599221,11cyclictest12524-21unixbench_singl23:45:253
1864599221,10cyclictest0-21swapper/323:40:383
1864599221,10cyclictest0-21swapper/323:40:383
1864099229,6cyclictest755-21rngd22:48:472
1864099228,3cyclictest20717-21memory00:05:192
1864099225,7cyclictest755-21rngd23:05:192
1864099221,8cyclictest9533-21kworker/u8:020:43:022
1863399229,1cyclictest0-21swapper/122:20:161
1863399227,7cyclictest0-21swapper/121:28:031
1863399227,3cyclictest0-21swapper/121:00:091
1863399221,9cyclictest0-21swapper/122:38:201
1863399221,9cyclictest0-21swapper/100:04:001
18633992214,3cyclictest0-21swapper/121:17:291
18633992214,3cyclictest0-21swapper/121:17:291
1863399221,16cyclictest0-21swapper/100:09:081
1863399221,10cyclictest0-21swapper/120:08:071
1863399221,10cyclictest0-21swapper/120:08:071
1862899229,2cyclictest13562-21strings22:35:200
1862899224,10cyclictest7-21ksoftirqd/021:30:290
1862899222,11cyclictest28602-21/usr/sbin/munin00:25:110
1862899221,10cyclictest31538-21perf22:05:010
61242210,2sleep30-21swapper/319:50:243
1864599216,2cyclictest11107-21sh20:05:013
1864599214,9cyclictest24524-21/usr/sbin/munin20:35:093
1864599213,6cyclictest0-21swapper/300:28:493
1864599212,3cyclictest22672-21df_inode20:30:143
1864599210,9cyclictest0-21swapper/323:52:063
1864599210,8cyclictest0-21swapper/321:55:373
1864599210,10cyclictest0-21swapper/322:55:533
1864099217,3cyclictest169822vmstat23:55:262
1864099212,2cyclictest87550irq/128-enp1s0-23:11:312
1864099211,6cyclictest27341-21/usr/sbin/munin21:55:242
1863399218,3cyclictest0-21swapper/123:08:121
1863399217,5cyclictest4394-21/usr/sbin/munin23:30:271
1863399214,7cyclictest0-21swapper/121:45:011
1863399211,12cyclictest14098-21memory23:50:181
1862899211,12cyclictest24524-21/usr/sbin/munin20:35:210
1862899211,11cyclictest13542-21users21:20:250
1862899211,11cyclictest0-21swapper/023:40:110
1862899211,11cyclictest0-21swapper/023:40:110
1862899211,10cyclictest18856-21smart_sda00:00:240
1864599206,3cyclictest6239-21cstates22:20:133
1864599204,8cyclictest401ktimersoftd/320:46:553
1864599203,6cyclictest0-21swapper/322:10:173
1864599203,2cyclictest4796-21sshd21:00:573
1864599203,13cyclictest12737-21tr22:35:123
18645992012,2cyclictest87350irq/126-enp1s0-21:10:203
1864599201,10cyclictest0-21swapper/300:24:343
1864599200,7cyclictest2459-21sort19:45:013
1864599200,11cyclictest0-21swapper/323:55:373
1864099208,2cyclictest0-21swapper/223:21:582
1864099206,3cyclictest0-21swapper/219:48:362
1864099201,7cyclictest0-21swapper/223:40:082
1864099201,7cyclictest0-21swapper/223:40:082
1864099201,6cyclictest0-21swapper/222:55:192
1864099201,4cyclictest30-21rcuc/222:25:142
1863399201,10cyclictest0-21swapper/122:53:161
1862899205,12cyclictest6587-21grep21:05:220
1862899204,14cyclictest1989-21perf23:25:000
1862899204,10cyclictest11-21rcuc/019:45:140
1862899202,9cyclictest24827-21kworker/0:121:58:240
1862899201,11cyclictest20129-21/usr/sbin/munin20:25:190
1862899201,10cyclictest8057-21unixbench-2d23:35:250
1862899201,10cyclictest30505-21users20:45:250
1862899201,10cyclictest0-21swapper/023:10:150
29072191,3sleep00-21swapper/020:59:560
1864599191,7cyclictest12751-21kworker/u8:200:35:233
1864599191,7cyclictest0-21swapper/320:25:513
1864599191,11cyclictest87350irq/126-enp1s0-20:55:013
1864099195,4cyclictest19401-21sshd00:02:562
1864099192,9cyclictest19321-21/usr/sbin/munin21:35:172
1864099192,7cyclictest3174-21users22:10:262
18640991915,2cyclictest30-21rcuc/219:15:022
1864099190,2cyclictest89250irq/135-enp2s0-20:14:482
1862899194,6cyclictest11-21rcuc/021:45:020
1862899191,8cyclictest21160-21cat00:05:250
1862899191,8cyclictest0-21swapper/019:23:200
1862899191,7cyclictest8564-1kworker/u9:122:11:010
1862899191,3cyclictest27124-21basename00:20:170
1862899191,2cyclictest24553-21needreboot23:00:180
1862899191,10cyclictest6934-21/usr/sbin/munin19:55:190
1862899191,10cyclictest30168-21/usr/sbin/munin23:15:130
38742181,2sleep00-21swapper/023:26:250
195582181,2sleep00-21swapper/020:20:440
1864599187,3cyclictest32379-21sshd19:38:563
1864599182,6cyclictest0-21swapper/322:05:023
1864599181,6cyclictest0-21swapper/321:39:133
18645991813,3cyclictest0-21swapper/319:20:163
18645991812,4cyclictest21393-21munin-node19:15:043
18645991812,3cyclictest18517-21gltestperf20:20:153
1864099187,2cyclictest32-21ksoftirqd/200:25:152
1864099185,7cyclictest87350irq/126-enp1s0-23:31:372
1864099185,4cyclictest32-21ksoftirqd/222:19:192
1864099183,7cyclictest26528-21cut00:20:112
1864099182,5cyclictest24923-21munin-run21:50:022
1864099182,3cyclictest0-21swapper/221:19:182
1864099182,3cyclictest0-21swapper/221:19:182
1864099181,9cyclictest30505-21/usr/sbin/munin19:35:152
1864099181,8cyclictest32-21ksoftirqd/222:41:452
1864099181,6cyclictest30276-21tr23:15:142
1864099181,6cyclictest0-21swapper/219:12:392
1864099181,4cyclictest30-21rcuc/221:00:012
1864099181,4cyclictest0-21swapper/223:50:212
1864099181,4cyclictest0-21swapper/222:00:192
1864099181,3cyclictest0-21swapper/220:53:352
18640991813,3cyclictest30-21rcuc/221:45:022
18640991811,4cyclictest23981-21dump-pmu-power00:15:012
1864099181,10cyclictest0-21swapper/200:18:522
1862899183,9cyclictest11-21rcuc/019:10:150
1862899183,6cyclictest11-21rcuc/020:40:260
1862899182,6cyclictest18443-21unixbench-2d22:45:250
1862899182,14cyclictest13321-21perf20:10:010
1862899182,14cyclictest13321-21perf20:10:010
1862899181,9cyclictest808-21snmpd21:54:450
1862899181,8cyclictest0-21swapper/020:50:130
1862899181,5cyclictest0-21swapper/022:21:060
1862899181,3cyclictest32265-21runrttasks19:37:280
1862899181,3cyclictest11316-21sed21:15:250
1862899181,3cyclictest11316-21sed21:15:250
1862899181,15cyclictest0-21swapper/022:25:370
1862899181,10cyclictest0-21swapper/000:35:230
159692181,2sleep00-21swapper/021:28:140
1864599176,4cyclictest87550irq/128-enp1s0-20:10:173
1864599174,3cyclictest87350irq/126-enp1s0-20:09:063
1864599174,3cyclictest87350irq/126-enp1s0-20:09:063
1864599173,5cyclictest8514-21df22:25:123
18645991713,3cyclictest0-21swapper/320:40:113
18645991711,3cyclictest31398-21ls20:50:123
1864099176,5cyclictest87650irq/129-enp1s0-19:55:122
1864099176,3cyclictest87350irq/126-enp1s0-23:46:052
1864099175,3cyclictest3174-21/usr/sbin/munin23:25:182
1864099173,9cyclictest32-21ksoftirqd/222:08:312
1864099173,8cyclictest87350irq/126-enp1s0-19:28:022
1864099173,3cyclictest87350irq/126-enp1s0-20:46:402
1864099172,7cyclictest808-21snmpd19:54:212
1864099172,7cyclictest22618-21expr20:30:142
1864099172,5cyclictest19847-21hddtemp_smartct22:50:162
1864099171,9cyclictest32-21ksoftirqd/223:37:332
1864099171,8cyclictest0-21swapper/221:20:142
1864099171,6cyclictest7029-21tr22:20:202
1864099171,5cyclictest87350irq/126-enp1s0-22:30:052
1864099171,5cyclictest14152-21/usr/sbin/munin21:25:182
18640991712,3cyclictest29131-21kernelversion19:30:172
18640991712,3cyclictest25502-21users19:20:252
18640991711,3cyclictest9131-21/usr/sbin/munin20:00:112
18640991711,3cyclictest26186-21ntp_kernel_pll_21:50:212
1864099171,12cyclictest988-21gltestperf19:40:142
1862899174,5cyclictest5942-21processes19:50:220
1862899174,11cyclictest29480-21ps19:30:220
1862899174,11cyclictest11-21rcuc/020:30:160
1862899172,6cyclictest32083-21df22:05:140
1862899172,6cyclictest20041-21/usr/sbin/munin21:35:250
1862899171,8cyclictest32310-21sed00:30:250
1862899171,7cyclictest22947-21/usr/sbin/munin21:45:230
1862899171,7cyclictest14768-21/usr/sbin/munin22:40:140
1862899171,7cyclictest0-21swapper/023:48:510
1862899171,7cyclictest0-21swapper/023:30:270
1862899171,7cyclictest0-21swapper/020:03:340
1862899171,15cyclictest0-21swapper/000:12:150
1862899171,14cyclictest0-21swapper/019:15:340
1862899171,10cyclictest0-21swapper/021:14:460
1862899170,7cyclictest0-21swapper/020:10:120
157232170,2sleep20-21swapper/220:15:082
1864599161,4cyclictest839-21lldpd19:45:073
1864099166,3cyclictest87350irq/126-enp1s0-20:20:252
18640991611,3cyclictest25169-21grep20:35:132
18640991610,3cyclictest827-21runrttasks20:27:222
18640991610,3cyclictest13133-21sshd20:07:552
18640991610,3cyclictest13133-21sshd20:07:552
1862899162,6cyclictest21367-21/usr/sbin/munin22:55:120
18628991510,3cyclictest3495-21sh21:00:110
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional