You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-18 - 05:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Sat Apr 18, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
113122240163,25sleep00-21swapper/019:05:420
113292236164,22sleep20-21swapper/219:05:562
116232202167,23sleep10-21swapper/119:09:451
114332201165,24sleep30-21swapper/319:07:193
2699621400,5sleep21174199cyclictest23:20:252
191982690,1sleep10-21swapper/119:25:021
11741995148,2cyclictest0-21swapper/221:31:492
26522500,4sleep011-21rcuc/022:25:250
145802500,2sleep314456-21/usr/sbin/munin22:55:133
231952490,4sleep223196-21grep00:25:262
11741994642,2cyclictest0-21swapper/221:48:492
1173599390,34cyclictest0-21swapper/121:41:241
1174699348,3cyclictest32086-21cpu23:35:123
11735993413,12cyclictest0-21swapper/121:57:171
11741993128,2cyclictest0-21swapper/221:26:532
1173599301,11cyclictest579-21systemd-journal21:15:011
11735993010,3cyclictest579-21systemd-journal23:30:011
11735993010,3cyclictest579-21systemd-journal23:30:011
1173599300,21cyclictest0-21swapper/123:30:221
1172799302,17cyclictest8921-21kworker/0:022:41:230
1174699296,10cyclictest0-21swapper/319:55:233
1174699296,10cyclictest0-21swapper/319:55:233
1174699291,6cyclictest0-21swapper/323:25:413
1174699291,6cyclictest0-21swapper/323:25:413
1174199292,14cyclictest1082-40dnf19:55:232
1174199292,14cyclictest1082-40dnf19:55:232
1173599292,8cyclictest13902-21munin-run19:15:021
11735992925,3cyclictest0-21swapper/123:17:441
1173599291,18cyclictest23438-21irqstats22:00:141
1172799292,7cyclictest654-21kworker/0:022:23:210
1174199283,9cyclictest21223-21taskset00:22:032
11735992823,3cyclictest0-21swapper/119:28:391
11735992817,9cyclictest6854-21fw_packets23:50:151
1173599281,3cyclictest0-21swapper/123:45:231
1173599281,10cyclictest13169-21memory22:50:181
1172799286,5cyclictest11603-21kworker/0:120:03:260
1172799281,21cyclictest0-21swapper/022:45:190
1174699271,11cyclictest27994-21/usr/sbin/munin19:45:263
11735992724,2cyclictest0-21swapper/119:17:251
11735992723,3cyclictest0-21swapper/122:46:271
11735992723,3cyclictest0-21swapper/122:43:591
11735992723,3cyclictest0-21swapper/121:50:071
11735992723,3cyclictest0-21swapper/120:46:591
11735992723,2cyclictest0-21swapper/121:45:121
1173599272,12cyclictest20565-21ntp_states00:20:211
1173599271,23cyclictest436-21unixbench-2d22:20:251
1174699261,7cyclictest0-21swapper/323:11:523
1174699261,7cyclictest0-21swapper/322:54:503
1174699261,13cyclictest0-21swapper/321:40:243
1174699261,13cyclictest0-21swapper/321:32:203
1174699261,10cyclictest23237-21sshd19:33:213
1173599266,17cyclictest5676-21cpu22:35:121
11735992623,2cyclictest0-21swapper/123:09:211
11735992623,2cyclictest0-21swapper/121:03:561
11735992623,2cyclictest0-21swapper/119:56:401
11735992623,2cyclictest0-21swapper/119:56:401
11735992623,2cyclictest0-21swapper/100:02:221
11735992622,3cyclictest0-21swapper/123:42:111
11735992622,3cyclictest0-21swapper/121:31:541
11735992622,3cyclictest0-21swapper/100:30:451
1173599261,8cyclictest0-21swapper/100:06:221
1173599261,19cyclictest0-21swapper/122:15:581
1173599261,19cyclictest0-21swapper/122:15:581
1172799268,4cyclictest26630-21kworker/0:222:35:550
1172799261,7cyclictest87350irq/126-enp1s0-22:11:400
11727992613,8cyclictest0-21swapper/023:05:540
11727992613,8cyclictest0-21swapper/000:17:410
1172799261,20cyclictest0-21swapper/023:55:220
1174699254,4cyclictest0-21swapper/321:35:333
1174699253,6cyclictest0-21swapper/319:25:363
11746992520,4cyclictest87350irq/126-enp1s0-23:00:153
1174699251,7cyclictest0-21swapper/322:44:553
1174699251,6cyclictest19494-21sed23:05:163
1174699251,6cyclictest0-21swapper/321:14:163
1174699251,5cyclictest0-21swapper/320:00:223
11746992515,1cyclictest0-21swapper/321:55:173
1174699251,3cyclictest26477-21/usr/sbin/munin23:20:183
1173599259,2cyclictest579-21systemd-journal20:20:011
1173599258,4cyclictest15566-21sed22:55:231
11735992522,2cyclictest0-21swapper/100:17:311
1173599252,20cyclictest0-21swapper/121:37:211
1173599252,11cyclictest0-21swapper/100:37:041
1173599252,11cyclictest0-21swapper/100:37:041
1173599251,11cyclictest4279-21irqstats22:30:181
1173599251,11cyclictest24674-21ntp_states19:35:211
1173599251,11cyclictest1-21systemd00:00:011
1173599250,23cyclictest0-21swapper/120:35:001
1172799255,4cyclictest20032-21kworker/0:022:07:310
1172799254,6cyclictest755-21rngd21:45:210
1172799252,17cyclictest1661-21perf23:40:000
11727992512,7cyclictest0-21swapper/021:11:110
1172799251,18cyclictest0-21swapper/022:15:220
1172799251,18cyclictest0-21swapper/022:15:220
1172799251,17cyclictest0-21swapper/023:01:240
1174699246,9cyclictest0-21swapper/322:45:123
1174699246,4cyclictest316-21diskmemload21:21:133
1174699244,9cyclictest27833-21sshd00:37:223
1174699244,9cyclictest27833-21sshd00:37:223
1174699241,7cyclictest20253-21cat21:55:013
1174699241,7cyclictest10463-40updatedb00:00:023
1174699241,5cyclictest22070-21kworker/u8:122:05:573
1174199244,4cyclictest0-21swapper/222:25:102
1174199241,5cyclictest11115-21ntp_states22:45:192
1173599247,2cyclictest27766-21latency_hist19:45:001
1173599242,20cyclictest0-21swapper/123:37:001
1173599242,12cyclictest17077-21munin-run00:15:001
11735992421,2cyclictest0-21swapper/122:13:551
1173599241,9cyclictest808-21snmpd23:00:151
1173599241,8cyclictest4253-21sed21:15:261
1173599241,21cyclictest0-21swapper/122:05:241
1173599241,21cyclictest0-21swapper/121:05:251
1173599241,21cyclictest0-21swapper/121:05:251
1173599241,21cyclictest0-21swapper/120:57:121
1173599241,21cyclictest0-21swapper/120:39:531
1173599241,20cyclictest3275-21tr20:00:191
1173599241,20cyclictest0-21swapper/120:10:201
1173599241,20cyclictest0-21swapper/100:25:251
1173599241,11cyclictest9000-21chrt21:27:231
1173599241,11cyclictest28906-21memory19:45:171
1173599241,11cyclictest0-21swapper/122:25:161
1173599240,9cyclictest0-21swapper/119:50:141
1172799247,12cyclictest87350irq/126-enp1s0-00:39:060
1172799247,12cyclictest87350irq/126-enp1s0-00:39:060
1172799241,19cyclictest0-21swapper/021:23:150
1172799241,19cyclictest0-21swapper/000:07:340
1172799241,17cyclictest0-21swapper/021:15:420
11727992411,6cyclictest0-21swapper/000:28:510
1172799241,12cyclictest0-21swapper/023:20:040
1172799241,12cyclictest0-21swapper/000:20:140
1172799241,11cyclictest30192-21/usr/sbin/munin19:50:110
1172799241,11cyclictest0-21swapper/023:26:430
1172799241,11cyclictest0-21swapper/023:26:430
1172799240,20cyclictest0-21swapper/023:11:190
1172799240,19cyclictest0-21swapper/021:53:540
1172799240,18cyclictest0-21swapper/021:42:530
1172799240,17cyclictest0-21swapper/022:04:250
199752230,4sleep339-21rcuc/320:40:123
1174699235,8cyclictest87350irq/126-enp1s0-22:00:143
1174699235,7cyclictest87650irq/129-enp1s0-22:21:093
1174699234,8cyclictest0-21swapper/300:04:383
1174699232,6cyclictest0-21swapper/321:48:103
1174699231,6cyclictest0-21swapper/321:05:023
1174699231,6cyclictest0-21swapper/321:05:023
1174199234,10cyclictest15937-21unixbench-2d22:55:252
1174199232,12cyclictest29927-21sed21:00:242
1174199231,8cyclictest0-21swapper/223:48:002
1174199231,8cyclictest0-21swapper/221:51:412
1174199231,7cyclictest0-21swapper/221:18:082
1174199231,6cyclictest0-21swapper/221:43:082
1174199231,15cyclictest0-21swapper/221:10:472
1174199231,13cyclictest0-21swapper/200:32:002
1173599232,19cyclictest0-21swapper/119:30:361
1173599232,10cyclictest0-21swapper/120:24:371
1173599231,20cyclictest0-21swapper/123:10:131
1173599231,20cyclictest0-21swapper/120:50:121
1173599231,20cyclictest0-21swapper/120:42:421
1173599231,20cyclictest0-21swapper/120:25:191
1173599230,21cyclictest0-21swapper/121:20:391
1172799237,11cyclictest87350irq/126-enp1s0-23:45:280
1172799236,8cyclictest11027-21/usr/sbin/munin20:20:140
1172799234,11cyclictest3578-21kworker/u8:000:11:270
1172799234,10cyclictest10094-21sed21:30:140
1172799233,10cyclictest1880-21/usr/sbin/munin23:40:200
1172799232,15cyclictest0-21swapper/021:39:440
1172799231,16cyclictest6575-21cat23:50:110
1172799231,16cyclictest0-21swapper/022:54:010
1172799231,16cyclictest0-21swapper/021:55:050
1172799231,11cyclictest32390-21/usr/sbin/munin19:55:210
1172799231,11cyclictest32390-21/usr/sbin/munin19:55:210
1172799230,18cyclictest0-21swapper/023:34:040
1174699228,2cyclictest0-21swapper/320:35:143
1174699227,3cyclictest25798-21/usr/sbin/munin19:40:173
1174699225,8cyclictest316-21diskmemload22:39:253
1174699225,8cyclictest1344-21kworker/u8:123:53:473
1174699225,8cyclictest12050irq/123-ahci[0019:20:013
1174699225,8cyclictest0-21swapper/320:50:083
1174699224,9cyclictest316-21diskmemload00:26:353
1174699223,9cyclictest316-21diskmemload23:48:343
1174699221,9cyclictest3835-21fschecks_count22:30:143
1174699221,9cyclictest12890-21/usr/sbin/munin00:05:253
1174699221,9cyclictest0-21swapper/321:19:043
1174699221,6cyclictest18327-21ntp_kernel_err00:15:223
1174699221,6cyclictest0-21swapper/320:20:153
1174699221,6cyclictest0-21swapper/300:10:213
1174699221,5cyclictest0-21swapper/322:16:283
1174699221,5cyclictest0-21swapper/322:16:283
1174699221,5cyclictest0-21swapper/300:33:113
1174199225,9cyclictest5592-21awk20:05:222
1174199225,8cyclictest1615-1kworker/u9:123:04:042
1174199225,6cyclictest0-21swapper/222:40:212
1174199224,10cyclictest28058-21runrttasks22:10:192
1174199223,10cyclictest22187-40dnf21:57:222
1174199222,11cyclictest30143-21memory22:15:172
1174199222,11cyclictest30143-21memory22:15:172
1174199221,4cyclictest22070-21kworker/u8:122:00:372
1174199221,12cyclictest25392-21perl22:05:152
1174199221,12cyclictest15157-21df19:15:142
1174199221,12cyclictest12604-21hddtemp_smartct19:10:152
1174199221,11cyclictest0-21swapper/223:42:452
1174199221,11cyclictest0-21swapper/200:36:222
1174199221,11cyclictest0-21swapper/200:36:222
1173599221,19cyclictest0-21swapper/123:22:111
1173599221,19cyclictest0-21swapper/120:05:111
1172799228,3cyclictest808-21snmpd21:09:540
1172799228,3cyclictest808-21snmpd21:09:540
1172799226,8cyclictest755-21rngd19:45:400
1172799225,8cyclictest28609-21/usr/sbin/munin21:00:230
1172799224,9cyclictest25798-21/usr/sbin/munin19:40:110
1172799223,8cyclictest7592-21sh21:25:130
1172799221,9cyclictest0-21swapper/000:32:060
11727992215,4cyclictest10795-21idleruntime-cro20:20:020
1172799221,14cyclictest0-21swapper/022:56:280
1172799221,11cyclictest0-21swapper/022:30:100
1172799221,10cyclictest23254-21/usr/sbin/munin23:15:230
1172799221,10cyclictest22018-21/usr/sbin/munin20:45:220
1172799220,16cyclictest0-21swapper/000:01:200
1174699214,3cyclictest87350irq/126-enp1s0-20:05:123
1174699213,3cyclictest87350irq/126-enp1s0-20:30:253
1174699213,15cyclictest7247-21/usr/sbin/munin21:25:133
1174699211,7cyclictest0-21swapper/323:30:033
1174699211,7cyclictest0-21swapper/322:11:053
1174699211,7cyclictest0-21swapper/300:21:483
11746992117,3cyclictest0-21swapper/321:00:253
1174699211,6cyclictest9179-21diskstats20:15:133
1174699211,6cyclictest7624-21strings20:10:203
1174699211,10cyclictest30192-21/usr/sbin/munin19:50:123
1174699210,6cyclictest4217-21kworker/3:219:10:073
1174199219,3cyclictest19603-21awk20:40:012
1174199216,4cyclictest316-21diskmemload23:16:042
1174199216,4cyclictest31100-21runrttasks23:30:242
1174199215,8cyclictest26474-21/usr/sbin/munin20:55:102
1174199212,9cyclictest10772-21cpuspeed_turbos00:00:132
1174199211,9cyclictest0-21swapper/200:10:162
1174199211,8cyclictest0-21swapper/223:05:262
1174199211,4cyclictest0-21swapper/200:19:042
11741992111,7cyclictest21394-21grep20:40:252
1174199211,12cyclictest22627-21unixbench_singl23:10:252
1174199211,11cyclictest8823-21/usr/sbin/munin23:55:152
1174199211,11cyclictest6366-21idleruntime22:35:152
1174199211,11cyclictest3026-21if_err_enp2s020:00:172
1174199211,11cyclictest23696-21tr19:35:122
1174199211,10cyclictest32415-21vmstat21:05:262
1174199211,10cyclictest32415-21vmstat21:05:262
1174199211,10cyclictest25441-21/usr/sbin/munin20:50:202
1174199211,10cyclictest1649-21latency_hist23:40:022
1172799211,6cyclictest15882-21ntp_states19:15:210
11727992114,4cyclictest24658-21ntpq19:35:200
1174699202,9cyclictest22936-21kworker/u8:223:16:013
1174699201,8cyclictest0-21swapper/322:25:123
1174699201,6cyclictest0-21swapper/323:42:533
1174699201,6cyclictest0-21swapper/320:45:173
1174699201,2cyclictest0-21swapper/320:55:213
1174199206,7cyclictest19213-21/usr/sbin/munin19:25:262
1174199202,8cyclictest827-21runrttasks19:30:352
1174199202,12cyclictest1615-1kworker/u9:122:52:002
1174199201,9cyclictest29610-21latency_hist23:30:022
1174199201,9cyclictest29610-21latency_hist23:30:022
1174199201,9cyclictest22039-21grep20:45:112
1174199201,6cyclictest12030-21uname20:20:192
1174199201,5cyclictest316-21diskmemload22:23:502
1174199201,3cyclictest0-21swapper/221:35:232
1174199201,10cyclictest8772-21cat20:15:022
1174199201,10cyclictest6620-21diskstats23:50:132
1174199201,10cyclictest17179-21cstates19:20:112
1174199201,10cyclictest16459-21ntp_kernel_err20:30:192
1174199201,10cyclictest14477-21/usr/sbin/munin00:05:252
1174199201,10cyclictest0-21swapper/222:30:032
1172799205,7cyclictest17012-21/usr/sbin/munin19:20:130
1172799204,7cyclictest14041-21irqstats20:25:170
1172799203,8cyclictest19821-21/usr/sbin/munin20:40:130
1172799202,7cyclictest495-21jbd2/dm-0-820:34:280
1172799201,8cyclictest808-21snmpd20:39:480
1172799201,10cyclictest19213-21/usr/sbin/munin19:25:200
1174699194,3cyclictest87350irq/126-enp1s0-20:25:123
1174699191,6cyclictest0-21swapper/319:20:053
1174699191,3cyclictest25568-21crond19:40:013
1174199191,9cyclictest909-21sssd_nss19:50:222
1174199191,9cyclictest5050-21/usr/sbin/munin21:20:232
1174199191,10cyclictest9578-21if_enp2s020:15:152
1174199191,10cyclictest29668-21sshd19:46:212
1174199191,10cyclictest25857-21/usr/sbin/munin19:40:122
1174199191,10cyclictest13915-21taskset20:25:142
1172799195,3cyclictest26410-21/usr/sbin/munin20:55:180
1172799191,9cyclictest25205-21cut20:50:170
1172799191,15cyclictest7990-21sort20:10:250
1172799191,15cyclictest11929-21/usr/sbin/munin19:10:190
11727991813,3cyclictest5508-21ntp_states20:05:210
1172799181,10cyclictest0-21swapper/019:30:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional