You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-27 - 21:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat Jun 27, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
222282238162,25sleep00-21swapper/007:09:440
220422238163,25sleep20-21swapper/207:07:212
219012233162,58sleep30-21swapper/307:05:293
220062202166,24sleep10-21swapper/107:06:511
2164621070,2sleep02232599cyclictest12:00:140
22342999693,2cyclictest0-21swapper/208:48:142
170122610,3sleep02232599cyclictest11:50:110
254682590,4sleep341-21ksoftirqd/307:15:123
305122520,3sleep130516-21fschecks_count12:20:141
305122520,3sleep130516-21fschecks_count12:20:141
105792510,1sleep30-21swapper/311:35:143
44042460,1sleep10-21swapper/111:20:181
22325993914,12cyclictest14827-21munin-run08:05:010
2232599359,12cyclictest29870-21munin-run12:20:010
2234299341,23cyclictest0-21swapper/211:30:022
2232599327,11cyclictest23635-21perf08:25:020
2232599322,14cyclictest7885-21munin-run11:30:010
2233699310,7cyclictest0-21swapper/110:43:541
2232599308,10cyclictest31471-21perf11:10:000
22325993011,16cyclictest29262-21munin-run11:05:000
22336992918,3cyclictest27628-21diskstats11:00:141
22325992910,16cyclictest2248-21munin-run08:50:010
2233699288,9cyclictest0-21swapper/111:38:211
2233699281,10cyclictest0-21swapper/109:17:321
2232599286,11cyclictest7-21ksoftirqd/009:10:010
2233699271,9cyclictest0-21swapper/109:20:501
22336992716,3cyclictest0-21swapper/110:19:381
2233699271,16cyclictest1537-21crond11:25:011
2233699271,16cyclictest0-21swapper/111:40:251
2232599273,12cyclictest27060-21idleruntime-cro11:00:000
2235099261,22cyclictest11614-21df_inode09:10:153
2235099261,17cyclictest0-21swapper/312:10:173
2233699261,9cyclictest0-21swapper/111:16:191
2233699261,16cyclictest0-21swapper/112:15:131
2233699261,15cyclictest0-21swapper/110:35:381
2233699261,13cyclictest0-21swapper/110:55:331
2233699261,10cyclictest0-21swapper/112:00:281
22350992517,3cyclictest0-21swapper/309:15:113
22350992516,5cyclictest20390-21cat11:55:263
2235099251,21cyclictest32019-21/usr/sbin/munin07:30:243
2235099251,21cyclictest27570-21memory09:45:203
2235099251,21cyclictest1862-21/usr/sbin/munin07:35:193
2235099251,21cyclictest11039-21diskmemload12:30:243
2233699259,7cyclictest0-21swapper/112:39:021
2233699256,2cyclictest13656-21kworker/u8:111:06:071
2233699252,11cyclictest9700-21users11:30:281
2233699251,9cyclictest0-21swapper/109:54:001
22336992517,3cyclictest0-21swapper/110:06:141
2233699251,12cyclictest0-21swapper/111:55:411
2233699251,12cyclictest0-21swapper/110:46:511
2233699251,12cyclictest0-21swapper/110:04:101
2233699251,12cyclictest0-21swapper/110:04:101
2233699251,11cyclictest0-21swapper/110:51:181
2233699251,11cyclictest0-21swapper/109:10:381
2233699250,9cyclictest0-21swapper/110:11:091
2233699250,7cyclictest0-21swapper/110:31:351
2232599256,11cyclictest0-21swapper/010:40:010
2232599251,5cyclictest0-21swapper/009:11:160
2235099241,3cyclictest0-21swapper/310:25:103
2235099241,20cyclictest852-21/usr/sbin/munin10:00:253
2235099241,20cyclictest852-21/usr/sbin/munin10:00:253
2235099241,20cyclictest15665-21/usr/sbin/munin09:20:173
2235099241,15cyclictest0-21swapper/311:02:383
22350992411,3cyclictest30550-21ntp_states11:05:223
22350992411,3cyclictest19549-21ntp_states10:40:213
2235099241,11cyclictest4667-21/usr/sbin/munin08:55:143
2235099241,11cyclictest4667-21/usr/sbin/munin08:55:143
2235099241,10cyclictest0-21swapper/309:55:143
2233699241,5cyclictest18410-21grep11:50:241
2233699241,17cyclictest0-21swapper/110:29:151
2233699241,15cyclictest0-21swapper/112:05:241
2233699241,12cyclictest0-21swapper/109:35:101
2233699240,10cyclictest87550irq/128-enp1s0-09:49:081
2233699240,10cyclictest0-21swapper/111:14:371
2235099239,3cyclictest31682-21/usr/sbin/munin11:10:233
2235099231,6cyclictest0-21swapper/311:54:293
22350992314,3cyclictest0-21swapper/310:46:173
2235099231,15cyclictest0-21swapper/309:26:513
2235099231,11cyclictest14084-21/usr/sbin/munin10:30:213
2235099231,11cyclictest10439-21sshd09:05:253
2235099231,10cyclictest7012-21open_inodes11:25:243
2234299231,5cyclictest0-21swapper/210:23:002
2234299231,4cyclictest0-21swapper/211:14:092
22342992312,4cyclictest0-21swapper/212:08:542
2233699232,15cyclictest26443-21kworker/u8:012:26:071
2233699232,13cyclictest0-21swapper/110:23:521
2233699231,16cyclictest17555-21kworker/u8:209:58:141
2233699231,11cyclictest0-21swapper/111:47:051
2233699231,11cyclictest0-21swapper/107:23:401
2233699230,17cyclictest17555-21kworker/u8:209:42:151
2233699230,12cyclictest0-21swapper/109:26:381
2232599231,5cyclictest0-21swapper/009:33:470
25472220,3sleep29-21rcu_preempt10:01:012
25472220,3sleep29-21rcu_preempt10:01:012
2235099228,7cyclictest0-21swapper/312:18:103
2235099227,8cyclictest0-21swapper/311:23:263
2235099225,4cyclictest0-21swapper/309:35:223
2235099221,2cyclictest0-21swapper/311:30:263
2235099221,14cyclictest0-21swapper/312:22:413
2235099221,14cyclictest0-21swapper/312:22:413
22350992210,2cyclictest17247-21/usr/sbin/munin08:10:133
2234299224,3cyclictest87350irq/126-enp1s0-11:55:252
2233699228,4cyclictest27487-21date07:20:001
2233699226,6cyclictest0-21swapper/108:55:281
2233699226,6cyclictest0-21swapper/108:55:281
2233699225,4cyclictest24657-21cat07:15:011
2233699222,11cyclictest909-21sssd_nss08:25:191
2233699221,18cyclictest7163-21memory07:45:181
2233699221,18cyclictest7163-21memory07:45:181
2233699221,16cyclictest0-21swapper/107:56:091
2233699221,15cyclictest0-21swapper/109:04:001
2233699221,15cyclictest0-21swapper/108:36:121
2233699221,11cyclictest24655-21munin-node08:10:281
2233699220,17cyclictest0-21swapper/112:30:351
2233699220,16cyclictest0-21swapper/108:16:531
2232599224,10cyclictest6866-21/usr/sbin/munin09:00:220
2232599221,5cyclictest28297-21kworker/u8:009:52:410
2232599221,5cyclictest0-21swapper/010:13:430
2232599221,10cyclictest12878-21strings10:25:220
52902210,4sleep20-21swapper/207:40:242
2235099217,4cyclictest26615-21/usr/sbin/munin10:55:263
2235099211,3cyclictest24423-21memory12:05:203
2235099211,10cyclictest8446-21/usr/sbin/munin07:50:163
2235099210,15cyclictest0-21swapper/312:29:153
2234299213,4cyclictest17862-21/usr/sbin/munin09:25:182
2234299212,7cyclictest28854-21/usr/sbin/munin09:50:142
2234299211,9cyclictest0-21swapper/211:18:022
2234299211,6cyclictest21969-21sshd09:34:262
2234299211,6cyclictest0-21swapper/208:05:022
2234299211,5cyclictest28969-21sshd11:01:262
2234299211,5cyclictest2135-21/usr/sbin/munin12:30:132
2233699216,3cyclictest87550irq/128-enp1s0-09:05:271
2233699211,8cyclictest0-21swapper/112:10:021
2233699211,14cyclictest0-21swapper/107:44:451
2233699211,11cyclictest26708-21cut08:30:171
22336992110,3cyclictest20971-21memory09:30:171
2233699210,16cyclictest0-21swapper/108:54:071
2233699210,11cyclictest0-21swapper/107:31:501
2232599213,9cyclictest0-21swapper/009:45:170
2232599211,8cyclictest0-21swapper/010:48:010
2232599211,6cyclictest0-21swapper/010:42:190
2232599211,6cyclictest0-21swapper/009:43:340
2232599211,5cyclictest0-21swapper/010:50:440
2232599211,11cyclictest28232-21gltestperf07:20:160
2235099202,15cyclictest5293-21fgrep07:40:233
2235099202,15cyclictest24456-21/usr/sbin/munin09:40:263
2235099201,9cyclictest4330-21/usr/sbin/munin12:35:243
2235099201,4cyclictest909-21sssd_nss12:01:263
2235099200,14cyclictest0-21swapper/311:40:113
2234299202,15cyclictest11314-21cut07:55:172
2234299201,7cyclictest14700-21/usr/sbin/munin11:45:162
2234299201,6cyclictest0-21swapper/210:11:582
2234299201,5cyclictest7490-21/usr/sbin/munin10:15:122
2234299201,5cyclictest3909-21irqstats10:05:172
2234299201,5cyclictest22883-21/usr/sbin/munin10:50:142
2234299201,4cyclictest0-21swapper/209:35:592
2233699201,14cyclictest0-21swapper/107:39:211
2233699201,13cyclictest0-21swapper/108:48:101
2233699201,10cyclictest13221-21fschecks_count08:00:141
2232599209,3cyclictest11-21rcuc/011:45:160
2232599209,2cyclictest18245-1kworker/u9:210:20:490
2232599205,9cyclictest20168-40dnf11:55:240
2232599204,10cyclictest808-21snmpd08:53:260
2232599202,5cyclictest18245-1kworker/u9:209:19:370
2232599202,3cyclictest0-21swapper/011:30:460
2232599202,11cyclictest15115-21cpuspeed_turbos08:05:120
2232599201,9cyclictest0-21swapper/009:35:120
2232599201,6cyclictest0-21swapper/012:37:170
2232599201,6cyclictest0-21swapper/011:41:180
2232599201,4cyclictest0-21swapper/009:26:480
2232599201,2cyclictest23536-21needreboot07:10:220
22325992011,2cyclictest87550irq/128-enp1s0-11:14:380
2232599201,11cyclictest0-21swapper/012:20:260
2232599201,11cyclictest0-21swapper/012:20:260
2232599201,10cyclictest21033-21users08:15:260
2235099198,3cyclictest0-21swapper/310:35:173
2235099192,3cyclictest0-21swapper/311:48:493
2235099192,14cyclictest6613-21taskset07:45:143
2235099192,14cyclictest6613-21taskset07:45:143
2235099191,9cyclictest0-21swapper/311:18:023
2235099191,3cyclictest3099-21/usr/sbin/munin10:05:143
22350991913,3cyclictest26649-21gltestperf08:30:183
2235099191,10cyclictest21126-21ntp_states09:30:203
22350991910,2cyclictest0-21swapper/309:51:173
2235099190,1cyclictest0-21swapper/308:20:263
2234299196,3cyclictest87650irq/129-enp1s0-10:25:212
2234299194,3cyclictest87650irq/129-enp1s0-09:10:152
2234299193,3cyclictest18644-21kworker/u8:108:53:532
2234299192,3cyclictest2078-21cat12:30:012
2234299191,7cyclictest0-21swapper/210:56:232
2234299191,7cyclictest0-21swapper/210:35:022
2234299191,7cyclictest0-21swapper/209:20:212
2234299191,6cyclictest21670-21cpu08:20:122
2234299191,4cyclictest10653-21diskstats11:35:152
2234299191,4cyclictest0-21swapper/209:43:412
2234299191,3cyclictest5140-21irqstats12:35:192
2234299190,5cyclictest21525-21cstates12:00:132
2233699194,3cyclictest16529-21unixbench-2d08:05:251
2233699191,6cyclictest21640-21/usr/sbin/munin08:20:151
2233699191,17cyclictest0-21swapper/107:29:091
2233699191,10cyclictest9004-21fschecks_time07:50:151
2232599198,2cyclictest31608-21ps08:40:240
2232599194,6cyclictest14339-21expr10:30:130
2232599191,8cyclictest1594750kworker/u8:211:16:270
2232599191,8cyclictest0-21swapper/010:06:370
2232599191,6cyclictest18245-1kworker/u9:211:24:280
2232599191,4cyclictest0-21swapper/009:23:490
2232599191,3cyclictest0-21swapper/011:35:200
2232599191,15cyclictest8835-21cut07:50:140
2232599191,15cyclictest2839-21idleruntime12:30:190
2232599191,12cyclictest0-21swapper/007:45:250
2232599191,12cyclictest0-21swapper/007:45:250
2232599191,10cyclictest7490-21/usr/sbin/munin10:15:240
2232599191,10cyclictest23511-21/usr/sbin/munin12:05:160
2235099184,3cyclictest0-21swapper/310:18:113
2235099183,4cyclictest9455-21latency_hist10:20:013
2235099181,5cyclictest24210-21switchtime10:50:253
2235099181,3cyclictest31577-21sshd07:27:243
2235099181,3cyclictest21122-21sshd08:16:263
22350991813,3cyclictest28241-21/usr/sbin/munin08:35:233
22350991812,3cyclictest3078-21grep08:50:173
22350991812,3cyclictest24984-21head08:25:243
2235099181,14cyclictest0-21swapper/310:10:203
2234299185,3cyclictest87650irq/129-enp1s0-08:15:232
2234299185,3cyclictest12050irq/123-ahci[0012:15:012
2234299184,3cyclictest6422-21sshd08:57:242
2234299184,3cyclictest6422-21sshd08:57:242
2234299184,3cyclictest28665-21sed09:50:002
2234299184,3cyclictest12050irq/123-ahci[0007:50:022
2234299184,3cyclictest12050irq/123-ahci[0007:50:022
2234299182,12cyclictest28241-21/usr/sbin/munin08:35:192
2234299181,7cyclictest0-21swapper/211:32:122
2234299181,7cyclictest0-21swapper/210:47:382
2234299181,7cyclictest0-21swapper/209:17:582
2234299181,6cyclictest0-21swapper/210:42:482
2234299181,5cyclictest6866-21/usr/sbin/munin09:00:242
22342991815,2cyclictest11049-21awk09:10:012
2234299181,4cyclictest30401-21memory11:05:212
2234299181,3cyclictest15051-21/usr/sbin/munin08:05:212
2234299181,3cyclictest0-21swapper/211:51:092
2234299181,3cyclictest0-21swapper/210:31:202
2234299181,15cyclictest103692sleep207:54:332
22342991811,4cyclictest24655-21munin-node07:15:022
2233699181,5cyclictest30435-21/usr/sbin/munin08:40:251
2232599187,3cyclictest0-21swapper/010:02:550
2232599187,3cyclictest0-21swapper/010:02:540
2232599181,8cyclictest0-21swapper/012:29:040
2232599181,6cyclictest18245-1kworker/u9:212:10:120
22325991811,4cyclictest827-21runrttasks09:00:020
22325991811,4cyclictest5580-21grep07:40:250
2232599181,10cyclictest31382-21head07:25:250
277122171,2sleep00-21swapper/008:31:200
2235099172,3cyclictest0-21swapper/309:00:563
2235099171,3cyclictest0-21swapper/308:05:053
2235099171,15cyclictest0-21swapper/307:24:593
2235099171,15cyclictest0-21swapper/307:14:033
22350991710,3cyclictest31033-21gltestperf08:40:163
2234299177,2cyclictest827-21runrttasks07:21:112
2234299173,3cyclictest17247-21/usr/sbin/munin08:10:212
2234299171,4cyclictest31332-21grep07:25:262
2234299171,4cyclictest0-21swapper/211:40:512
22342991712,3cyclictest28018-21cpuspeed_turbos12:15:132
2234299171,15cyclictest0-21swapper/208:43:472
2234299171,14cyclictest0-21swapper/207:32:132
22342991711,3cyclictest30988-21sed12:20:172
22342991711,3cyclictest30988-21sed12:20:172
22342991711,3cyclictest23698-21proc_pri07:10:252
2232599173,6cyclictest11-21rcuc/007:35:150
2232599173,6cyclictest11-21rcuc/007:15:180
2232599171,6cyclictest5838-21ps08:55:230
122792171,2sleep00-21swapper/007:55:510
22350991612,3cyclictest0-21swapper/308:45:093
22350991612,3cyclictest0-21swapper/307:56:393
2234299161,5cyclictest0-21swapper/211:21:202
2234299161,4cyclictest26040-21/usr/sbin/munin08:30:152
22342991610,3cyclictest32205-21head09:55:222
22342991610,3cyclictest23975-21expr08:25:132
2232599163,4cyclictest808-21snmpd08:37:460
2232599161,6cyclictest0-21swapper/009:55:320
2232599161,6cyclictest0-21swapper/008:10:180
22325991612,3cyclictest0-21swapper/008:26:180
2232599160,14cyclictest1140-21grep07:30:250
2235099159,3cyclictest12986-21cut08:00:123
22342991511,3cyclictest0-21swapper/207:35:102
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional