You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 10:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sat Dec 27, 2025 00:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
48282211158,40sleep20-21swapper/219:09:432
46092201164,25sleep00-21swapper/019:06:520
47252200163,25sleep30-21swapper/319:08:233
46402200164,24sleep10-21swapper/119:07:171
1964821420,2sleep1493899cyclictest00:35:131
178102600,1sleep00-21swapper/023:15:250
194222590,5sleep011-21rcuc/000:35:110
20142510,1sleep00-21swapper/023:55:150
211322500,2sleep121-21rcuc/123:25:131
255842360,1sleep30-21swapper/322:20:223
4929993510,12cyclictest14057-21munin-run23:10:010
191082340,2sleep0492999cyclictest22:05:220
492999321,15cyclictest25379-21munin-run19:55:020
492999298,10cyclictest0-21swapper/021:30:020
63012280,2sleep00-21swapper/021:35:290
4938992814,6cyclictest29473-21systemd-journal20:01:131
492999284,11cyclictest16902-21basename00:30:010
4938992711,7cyclictest0-21swapper/123:20:241
492999264,13cyclictest31243-21sed23:45:280
492999262,12cyclictest7112-21munin-run19:15:010
492999261,12cyclictest6216-21munin-run20:25:010
492999261,12cyclictest21607-21awk21:00:010
495099251,11cyclictest0-21swapper/323:50:203
4944992522,2cyclictest0-21swapper/220:24:072
493899256,8cyclictest0-21swapper/122:29:091
494499244,8cyclictest21183-40dnf00:38:572
494499243,2cyclictest88750irq/134-enp2s0-00:30:232
493899246,8cyclictest0-21swapper/123:39:211
492999244,13cyclictest0-21swapper/021:12:520
80182230,5sleep19-21rcu_preempt20:25:271
80182230,5sleep19-21rcu_preempt20:25:261
495099239,3cyclictest27476-21mailstats21:10:263
495099231,7cyclictest0-21swapper/323:10:453
495099231,14cyclictest0-21swapper/320:54:193
4950992310,3cyclictest9090-21ntp_states19:15:203
494499231,10cyclictest0-21swapper/222:45:442
494499231,10cyclictest0-21swapper/222:45:432
493899239,3cyclictest23-21ksoftirqd/122:21:331
493899238,3cyclictest30400-21munin-run21:20:001
493899231,19cyclictest13152-21ntp_states23:05:201
493899231,11cyclictest2344-21grep23:55:161
493899231,11cyclictest12816-21dump-pmu-power20:40:011
493899230,4cyclictest0-21swapper/100:32:411
492999239,3cyclictest244772cut22:20:010
492999235,11cyclictest14024-21kworker/u8:123:10:320
492999234,12cyclictest0-21swapper/021:47:540
492999233,13cyclictest0-21swapper/000:32:180
492999230,10cyclictest0-21swapper/019:40:020
495099228,4cyclictest11231-21awk21:50:013
495099224,6cyclictest2508-21interrupts23:55:183
495099221,9cyclictest0-21swapper/319:30:253
494499221,10cyclictest0-21swapper/222:50:372
494499220,6cyclictest0-21swapper/223:49:432
493899228,3cyclictest19190-21ls22:05:251
493899226,4cyclictest0-21swapper/122:40:411
493899225,9cyclictest0-21swapper/100:05:171
493899221,3cyclictest2720-21hddtemp_smartct20:15:171
4938992213,3cyclictest0-21swapper/122:15:481
493899221,2cyclictest87150irq/128-enp1s0-22:37:531
493899221,19cyclictest0-21swapper/120:20:531
493899221,15cyclictest0-21swapper/121:55:421
493899221,12cyclictest13789-21ntp_states00:20:211
493899221,11cyclictest16487-21cut22:00:161
492999228,13cyclictest0-21swapper/022:05:010
492999223,12cyclictest0-21swapper/023:31:170
492999223,12cyclictest0-21swapper/022:26:200
492999221,15cyclictest0-21swapper/000:09:110
492999221,14cyclictest0-21swapper/021:19:050
230362220,9sleep341-21ksoftirqd/322:15:163
495099213,10cyclictest39-21rcuc/320:55:263
495099212,16cyclictest27985-21head22:25:233
495099211,7cyclictest0-21swapper/300:36:043
495099211,6cyclictest0-21swapper/323:07:533
495099211,4cyclictest0-21swapper/322:06:443
495099210,10cyclictest1585-21pmu-power21:25:213
494499211,6cyclictest32-21ksoftirqd/200:11:462
494499211,12cyclictest0-21swapper/223:12:072
494499211,12cyclictest0-21swapper/221:17:302
493899219,3cyclictest18064-21users23:15:261
493899214,5cyclictest25999-21diskmemload23:50:271
493899211,8cyclictest0-21swapper/121:45:191
4938992116,3cyclictest11689-21grep19:20:251
4938992113,6cyclictest28344-21kworker/u8:119:33:151
493899211,18cyclictest0-21swapper/120:55:131
493899211,18cyclictest0-21swapper/119:36:271
493899211,11cyclictest27816-21diskstats23:40:131
493899211,11cyclictest20358-21tr22:10:131
493899211,11cyclictest0-21swapper/120:10:191
493899211,11cyclictest0-21swapper/100:00:241
493899211,11cyclictest0-21swapper/100:00:241
493899211,10cyclictest0-21swapper/123:04:591
4938992110,3cyclictest8022-21df_inode22:55:141
4938992110,3cyclictest63022wc21:35:281
493899210,16cyclictest0-21swapper/121:34:391
492999214,11cyclictest31428-21kworker/u8:100:01:570
492999214,11cyclictest31428-21kworker/u8:100:01:570
492999213,5cyclictest5830-21diskstats22:50:140
492999212,12cyclictest0-21swapper/021:50:480
492999212,12cyclictest0-21swapper/000:24:400
492999212,11cyclictest24851-21kworker/u8:223:40:040
4929992115,3cyclictest18769-21tr23:20:110
495099209,3cyclictest7060-21/usr/sbin/munin21:40:153
495099207,4cyclictest2594-21sed22:40:253
495099206,4cyclictest6514-21fschecks_count00:05:163
495099202,10cyclictest16476-21/usr/sbin/munin23:15:203
495099201,7cyclictest0-21swapper/322:34:103
495099201,6cyclictest27817-21diskstats23:40:153
495099201,6cyclictest0-21swapper/300:30:153
495099201,5cyclictest0-21swapper/322:11:203
495099201,5cyclictest0-21swapper/321:18:283
495099201,12cyclictest0-21swapper/322:45:193
495099201,12cyclictest0-21swapper/322:45:193
495099201,11cyclictest801-21snmpd21:30:573
495099201,10cyclictest7435-21grep20:25:193
495099201,10cyclictest7435-21grep20:25:193
495099201,10cyclictest11497-21sshd23:00:413
494499208,3cyclictest10556-21sed21:45:242
494499207,2cyclictest30600-21kworker/2:120:07:332
494499207,2cyclictest18270-21kworker/2:023:58:052
494499207,2cyclictest18270-21kworker/2:023:07:192
494499206,5cyclictest86950irq/126-enp1s0-00:00:152
494499206,5cyclictest86950irq/126-enp1s0-00:00:152
494499204,13cyclictest801-21snmpd19:10:262
494499203,8cyclictest30-21rcuc/222:10:202
494499201,9cyclictest0-21swapper/223:04:252
494499201,8cyclictest0-21swapper/222:30:202
4944992014,3cyclictest6600-21tr20:25:142
4944992014,3cyclictest6600-21tr20:25:142
4944992012,5cyclictest14683-21crond00:25:012
493899202,7cyclictest17425-21/usr/sbin/munin20:50:261
493899202,4cyclictest5825-21df_inode22:50:131
493899201,8cyclictest13034-21/usr/sbin/munin20:40:241
493899201,8cyclictest0-21swapper/121:10:101
493899201,6cyclictest0-21swapper/123:45:231
493899201,3cyclictest0-21swapper/121:50:141
4938992013,4cyclictest2036-21grep21:25:241
493899201,2cyclictest25999-21diskmemload23:10:421
493899201,18cyclictest0-21swapper/119:48:541
493899201,18cyclictest0-21swapper/119:44:561
493899201,17cyclictest0-21swapper/120:45:151
493899201,12cyclictest0-21swapper/119:10:141
493899201,11cyclictest8028-21/usr/sbin/munin19:15:121
493899200,15cyclictest0-21swapper/121:23:241
492999209,3cyclictest9878-21/usr/sbin/munin23:00:230
492999203,11cyclictest24851-21kworker/u8:223:39:040
492999202,12cyclictest0-21swapper/021:43:510
492999202,11cyclictest0-21swapper/022:35:400
492999201,16cyclictest10003-21munin-run19:20:010
492999201,12cyclictest0-21swapper/023:27:450
492999201,12cyclictest0-21swapper/022:40:380
492999201,12cyclictest0-21swapper/021:33:380
225412201,3sleep20-21swapper/223:26:192
202492200,1sleep20-21swapper/220:55:172
110532201,2sleep00-21swapper/000:15:170
92342190,4sleep121-21rcuc/100:10:211
495099194,6cyclictest41-21ksoftirqd/319:45:143
495099194,12cyclictest39-21rcuc/320:05:103
495099193,3cyclictest177512missed_timers19:35:183
495099191,9cyclictest0-21swapper/319:10:183
495099191,7cyclictest10148-21grep20:30:273
495099191,2cyclictest0-21swapper/300:10:203
495099191,10cyclictest9-21rcu_preempt22:00:253
495099191,10cyclictest32200-21/usr/sbin/munin20:10:163
495099191,10cyclictest10527-21/usr/sbin/munin00:15:203
495099191,10cyclictest0-21swapper/323:26:103
495099190,9cyclictest4840-21gltestperf20:20:153
494499198,2cyclictest18270-21kworker/2:023:37:252
494499197,2cyclictest18270-21kworker/2:021:22:232
494499194,6cyclictest751-21rngd22:39:572
494499193,3cyclictest18270-21kworker/2:022:07:352
494499192,6cyclictest32-21ksoftirqd/219:30:112
494499192,4cyclictest32-21ksoftirqd/221:40:172
494499191,8cyclictest0-21swapper/200:17:052
494499191,7cyclictest32-21ksoftirqd/223:50:172
4944991914,3cyclictest9985-21grep20:30:272
494499191,3cyclictest87250irq/129-enp1s0-23:20:012
494499191,12cyclictest0-21swapper/221:25:182
494499191,11cyclictest0-21swapper/222:41:072
493899191,8cyclictest0-21swapper/100:15:111
493899191,6cyclictest31560-21head20:05:251
4938991915,3cyclictest0-21swapper/122:31:471
4938991915,3cyclictest0-21swapper/121:42:071
493899191,3cyclictest15527-21gltestperf00:25:171
4938991912,4cyclictest24441-21cat23:30:251
493899191,17cyclictest0-21swapper/122:47:391
493899191,17cyclictest0-21swapper/122:47:391
493899191,16cyclictest0-21swapper/121:01:001
493899191,16cyclictest0-21swapper/120:32:571
493899190,18cyclictest0-21swapper/119:55:231
493899190,17cyclictest0-21swapper/119:50:221
492999196,3cyclictest25999-21diskmemload23:54:150
492999193,11cyclictest0-21swapper/023:00:000
492999192,12cyclictest0-21swapper/022:33:250
492999192,11cyclictest0-21swapper/022:46:200
492999192,11cyclictest0-21swapper/022:46:200
492999191,6cyclictest0-21swapper/021:20:230
492999191,12cyclictest24117-21kworker/u8:122:20:090
492999191,12cyclictest0-21swapper/022:10:590
492999191,10cyclictest12421-21/usr/sbin/munin19:25:190
288082190,1sleep20-21swapper/222:30:022
495099187,3cyclictest23069-21/usr/sbin/munin23:30:193
495099187,3cyclictest2043-21/usr/sbin/munin20:15:173
495099186,3cyclictest23809-21fschecks_count19:50:143
495099182,3cyclictest87250irq/129-enp1s0-00:24:503
495099181,9cyclictest0-21swapper/321:55:203
495099181,7cyclictest0-21swapper/321:50:023
495099181,6cyclictest5884-21crond00:05:013
495099181,6cyclictest5884-21crond00:05:013
495099181,6cyclictest0-21swapper/300:25:163
495099181,3cyclictest27860-21cpu20:00:123
495099181,10cyclictest14177-21pmu-power20:40:223
494499186,3cyclictest86950irq/126-enp1s0-00:05:212
494499186,3cyclictest0-21swapper/223:42:092
494499186,2cyclictest30600-21kworker/2:119:38:372
494499184,8cyclictest32-21ksoftirqd/219:20:122
494499181,7cyclictest8941-21memory19:15:192
494499181,7cyclictest88750irq/134-enp2s0-19:43:412
494499181,7cyclictest0-21swapper/221:53:552
4944991812,3cyclictest24768-21cpuspeed_turbos22:20:162
4944991812,3cyclictest22490-21gltestperf21:00:162
494499181,15cyclictest0-21swapper/223:30:522
494499181,15cyclictest0-21swapper/221:31:492
4944991811,4cyclictest835-21runrttasks21:05:222
494499181,11cyclictest0-21swapper/223:22:352
494499181,11cyclictest0-21swapper/221:10:072
4944991810,5cyclictest19391-21idleruntime-cro20:55:022
493899181,16cyclictest0-21swapper/121:05:201
493899181,16cyclictest0-21swapper/119:26:201
492999188,8cyclictest4028-21awk20:20:010
492999186,3cyclictest18186-21if_enp2s020:50:170
492999185,4cyclictest15124-21grep21:55:250
492999182,7cyclictest0-21swapper/000:10:140
492999181,9cyclictest14999-21latency_hist20:45:010
492999181,8cyclictest20565-21grep19:40:270
492999181,5cyclictest11436-21grep20:35:150
492999181,4cyclictest24030-21/usr/sbin/munin21:05:210
168302181,2sleep30-21swapper/320:45:273
495099175,3cyclictest26770-21ps19:55:223
495099172,5cyclictest8354-21hddtemp_smartct22:55:173
495099172,12cyclictest4861-21/usr/sbin/munin21:35:213
495099171,6cyclictest25408-21grep23:35:113
495099171,2cyclictest0-21swapper/323:47:223
4950991712,3cyclictest21886-21cpuspeed_turbos21:00:143
4950991711,3cyclictest6553-21ntpq22:50:223
4950991711,3cyclictest19612-21grep19:40:153
494499175,9cyclictest0-21swapper/220:45:162
494499175,2cyclictest88750irq/134-enp2s0-19:53:132
494499175,2cyclictest12050irq/123-ahci[0022:20:032
494499173,3cyclictest30600-21kworker/2:120:36:122
494499171,6cyclictest14326-21hddtemp_smartct21:55:152
494499171,6cyclictest0-21swapper/200:27:152
4944991713,3cyclictest0-21swapper/222:03:282
4944991713,3cyclictest0-21swapper/219:47:152
494499171,15cyclictest0-21swapper/221:39:572
494499171,10cyclictest0-21swapper/219:58:392
492999171,8cyclictest27683-21cat20:00:020
492999171,4cyclictest16235-21sshd19:30:410
4929991711,4cyclictest8886-21expr20:30:140
4929991710,6cyclictest28344-21kworker/u8:119:49:460
141202170,2sleep30-21swapper/319:26:433
495099165,3cyclictest18671-21/usr/sbin/munin23:20:153
495099161,7cyclictest30620-21/usr/sbin/munin21:20:103
495099161,6cyclictest10836-21/usr/sbin/munin20:35:153
495099161,6cyclictest0-21swapper/322:35:023
4950991611,3cyclictest25087-21ntp_offset21:05:233
4950991611,3cyclictest11495-21switchtime19:20:253
494499164,7cyclictest2043-21/usr/sbin/munin20:15:292
494499161,7cyclictest0-21swapper/219:25:122
494499161,6cyclictest0-21swapper/222:55:132
494499161,14cyclictest0-21swapper/220:01:362
492999165,3cyclictest15231-21/usr/sbin/munin20:45:240
492999162,4cyclictest10897-21hddtemp_smartct19:20:160
492999161,9cyclictest0-21swapper/020:10:230
492999161,5cyclictest6791-21diskstats20:25:140
492999161,5cyclictest6791-21diskstats20:25:140
4929991611,3cyclictest28001-21expr20:00:140
316132161,2sleep00-21swapper/020:05:310
494499151,5cyclictest13034-21/usr/sbin/munin20:40:132
4944991512,2cyclictest3122sleep220:10:152
492999151,3cyclictest21820-21/usr/sbin/munin21:00:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional