You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-13 - 15:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Mon Apr 13, 2026 12:44:28)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
232432235164,58sleep30-21swapper/307:07:353
231092235162,24sleep00-21swapper/007:05:520
231912214163,38sleep10-21swapper/107:06:541
231392201165,24sleep20-21swapper/207:06:132
36012610,4sleep02351599cyclictest11:15:170
74152470,2sleep30-21swapper/310:10:193
317622470,1sleep10-21swapper/108:40:121
239542450,1sleep30-21swapper/307:10:133
2352199430,35cyclictest24221-21wc10:50:121
2352899411,17cyclictest0-21swapper/210:42:472
2352199401,27cyclictest17227-21latency_hist10:35:021
2352199401,27cyclictest12213-21dump-pmu-power09:10:001
2352199391,26cyclictest10817-21grep10:20:021
2352199381,33cyclictest13066-21tail09:10:151
2352199381,20cyclictest23709-21chrt10:48:421
2352899371,14cyclictest22661-21irqstats10:45:192
2352899371,13cyclictest0-21swapper/210:15:192
2352899371,12cyclictest0-21swapper/208:43:232
2352199371,29cyclictest808-21snmpd11:08:081
2352199370,25cyclictest827-21runrttasks09:45:451
2352199368,19cyclictest24788-21systemd-journal08:25:011
2352199361,32cyclictest7219-21awk07:45:011
2352199361,32cyclictest18808-21awk09:25:001
2352199361,26cyclictest8561-21cat10:15:011
2352199361,26cyclictest29410-21/usr/sbin/munin08:35:151
2352199361,26cyclictest10863-21/usr/sbin/munin10:20:171
2352199361,22cyclictest5505-21/usr/sbin/munin12:35:161
2352199360,1cyclictest0-21swapper/111:35:001
2352199360,19cyclictest21007-21if_enp1s011:55:171
2352899351,13cyclictest0-21swapper/210:53:072
2352199358,13cyclictest27383-21cstates08:30:151
2352199354,26cyclictest3800-40dnf10:02:011
2352199351,31cyclictest5583-21wc10:05:251
2352199351,31cyclictest25270-21grep07:10:261
2352199351,31cyclictest18043-21sed11:50:011
2352199351,26cyclictest87550irq/128-enp1s0-09:15:201
2352199351,26cyclictest87550irq/128-enp1s0-09:15:201
2352199351,26cyclictest7090-21/usr/sbin/munin11:25:201
2352199351,26cyclictest0-21swapper/111:14:561
2352199351,21cyclictest32222-21/usr/sbin/munin09:55:131
2351599357,13cyclictest5012-21munin-run07:40:020
2351599356,13cyclictest28853-21munin-run12:15:020
2352899341,13cyclictest0-21swapper/211:34:472
2352899341,10cyclictest0-21swapper/209:58:582
2352199343,19cyclictest3418-21latency_hist08:50:021
2352199342,24cyclictest22467-21/usr/sbin/munin12:00:161
2352199341,29cyclictest13615-21grep10:25:161
2352199340,26cyclictest0-21swapper/109:45:021
2351599346,13cyclictest18813-21munin-run09:25:010
2352899331,20cyclictest0-21swapper/209:35:312
2352899331,14cyclictest0-21swapper/210:22:472
2352899331,13cyclictest0-21swapper/212:39:512
2352899331,13cyclictest0-21swapper/209:03:072
2352899331,11cyclictest0-21swapper/212:18:462
2352199332,20cyclictest16228-21/usr/sbin/munin08:05:131
2352199331,29cyclictest9414-21awk07:50:011
2352199331,29cyclictest0-21swapper/111:55:011
2352199331,28cyclictest29700-21mailstats11:00:231
2352199331,24cyclictest19653-21/usr/sbin/munin10:40:151
2352199331,23cyclictest17459-21/usr/sbin/munin10:35:151
2352199331,21cyclictest14030-21diskstats11:40:151
2352199331,17cyclictest26880-21/usr/sbin/munin12:10:131
2352199330,25cyclictest22259-21/usr/sbin/munin09:30:201
2351599332,10cyclictest568-21kworker/0:112:34:200
2351599331,13cyclictest32415-21kworker/0:211:25:140
2351599331,10cyclictest30145-21kworker/0:111:01:430
23515993310,11cyclictest15028-21idleruntime-cro10:30:010
2352899321,14cyclictest0-21swapper/207:28:262
2352899321,11cyclictest0-21swapper/212:24:282
2352899320,12cyclictest0-21swapper/211:02:312
2352199328,15cyclictest909-21sssd_nss12:30:241
2352199325,17cyclictest0-21swapper/107:35:131
2352199321,28cyclictest5567-21hddtemp_smartct11:20:131
2352199321,28cyclictest4665-21latency_hist11:20:011
2352199321,28cyclictest32652-21sort12:20:261
2352199321,27cyclictest27179-21ls10:55:211
2352199321,27cyclictest25405-21awk09:40:001
2352199321,20cyclictest21616-21needreboot08:15:201
2352199321,19cyclictest12746-21smartctl11:35:231
2352199321,19cyclictest12746-21smartctl11:35:231
23521993210,19cyclictest30213-21ls09:50:131
2352199320,22cyclictest20145-21grep09:25:231
2351599322,9cyclictest9971-21kworker/0:009:11:270
2351599322,12cyclictest22844-21kworker/0:109:37:430
2352899311,9cyclictest0-21swapper/212:11:582
2352899311,10cyclictest0-21swapper/212:06:112
2352899311,10cyclictest0-21swapper/211:14:552
2352899310,13cyclictest0-21swapper/211:35:512
2352899310,13cyclictest0-21swapper/211:35:512
2352899310,11cyclictest0-21swapper/211:50:552
2352899310,11cyclictest0-21swapper/210:26:192
2352199311,20cyclictest808-21snmpd07:17:231
2352199311,19cyclictest29769-21hwlatdetect12:15:171
2352199311,17cyclictest19413-21strings08:10:191
2352199310,24cyclictest22166-21kworker/u8:212:06:571
2351599316,2cyclictest9971-21kworker/0:009:07:110
2351599312,11cyclictest22409-21kworker/0:108:25:590
2351599311,11cyclictest32616-21kworker/0:311:12:380
2352899301,9cyclictest0-21swapper/207:52:502
2352899301,12cyclictest0-21swapper/210:58:432
2352899301,10cyclictest0-21swapper/211:09:592
2352199301,26cyclictest29220-21/usr/sbin/munin07:20:141
23521993011,16cyclictest10884-21mailstats07:50:231
2351599302,17cyclictest9971-21kworker/0:009:34:310
2351599301,23cyclictest23715-21kworker/0:010:57:270
2351599301,10cyclictest29639-21kworker/0:010:05:310
2352899291,12cyclictest0-21swapper/209:47:142
2352199292,24cyclictest6868-21/usr/sbin/munin08:55:201
2352199291,25cyclictest1811-21/usr/sbin/munin12:25:171
2352199291,25cyclictest14018-21/usr/sbin/munin08:00:131
2352199291,25cyclictest12565-21/usr/sbin/munin07:55:181
2352199290,16cyclictest9114-21open_files09:00:211
2351599292,9cyclictest17683-21kworker/0:012:04:120
2351599291,9cyclictest568-21kworker/0:112:28:550
2352899288,2cyclictest827-21runrttasks09:09:152
2352899280,10cyclictest0-21swapper/211:42:202
2352199281,24cyclictest781-21/usr/sbin/munin07:30:201
2352199281,24cyclictest3649-21/usr/sbin/munin08:50:121
2352199281,24cyclictest32508-21grep07:25:261
2352199281,24cyclictest25359-21/usr/sbin/munin08:25:131
2351599284,2cyclictest32415-21kworker/0:211:33:500
2351599281,5cyclictest29639-21kworker/0:009:54:190
2351599281,10cyclictest24416-21kworker/0:212:18:280
2351599280,10cyclictest22844-21kworker/0:109:47:030
23528992713,7cyclictest0-21swapper/211:24:392
2352899271,16cyclictest0-21swapper/211:55:422
2352899271,14cyclictest0-21swapper/207:47:382
2352899271,13cyclictest0-21swapper/210:38:472
23528992711,3cyclictest0-21swapper/208:55:512
2352899271,10cyclictest0-21swapper/210:06:422
2352899270,12cyclictest0-21swapper/209:27:352
2351599276,3cyclictest17835-21kworker/0:108:08:270
2351599272,4cyclictest29639-21kworker/0:010:18:470
2351599271,9cyclictest29639-21kworker/0:010:10:310
2351599271,10cyclictest9971-21kworker/0:009:18:030
2351599271,10cyclictest9971-21kworker/0:009:18:030
2353299261,10cyclictest9590-21sed07:50:013
2352899269,3cyclictest0-21swapper/209:23:032
2352899269,3cyclictest0-21swapper/208:49:392
2352899269,3cyclictest0-21swapper/208:14:112
2352899269,3cyclictest0-21swapper/207:21:472
2352899269,3cyclictest0-21swapper/207:11:152
2352899261,11cyclictest0-21swapper/211:29:432
2351599267,3cyclictest14793-21kworker/0:210:36:270
2351599263,3cyclictest14793-21kworker/0:210:52:270
2351599261,9cyclictest17683-21kworker/0:012:07:590
2351599261,2cyclictest7498-21kworker/0:108:56:350
2351599261,12cyclictest17683-21kworker/0:011:52:550
2351599261,10cyclictest14793-21kworker/0:210:33:590
2352899259,4cyclictest0-21swapper/210:14:342
2352899258,3cyclictest0-21swapper/208:23:032
2352899258,3cyclictest0-21swapper/208:19:352
2352899258,3cyclictest0-21swapper/208:05:112
2352899258,3cyclictest0-21swapper/208:02:512
2352899250,14cyclictest0-21swapper/209:18:032
2352899250,14cyclictest0-21swapper/209:18:032
2352899250,13cyclictest0-21swapper/212:34:202
2352899250,12cyclictest0-21swapper/209:34:432
2351599251,12cyclictest11555-21kworker/0:008:01:070
23515992510,3cyclictest4783-21kworker/0:107:40:070
2352899248,3cyclictest0-21swapper/210:34:432
2352899248,3cyclictest0-21swapper/209:53:142
2352899248,3cyclictest0-21swapper/207:55:542
2352899247,3cyclictest0-21swapper/209:11:072
2352899247,3cyclictest0-21swapper/207:16:032
23528992413,3cyclictest0-21swapper/208:34:352
2351599247,4cyclictest30145-21kworker/0:111:08:500
2351599245,7cyclictest16360-21kworker/0:309:27:070
2351599244,4cyclictest14793-21kworker/0:210:41:350
2351599242,9cyclictest7109-21kworker/0:207:53:350
2351599242,9cyclictest30731-21kworker/0:007:34:590
2351599242,13cyclictest22844-21kworker/0:109:41:550
2351599241,8cyclictest568-21kworker/0:112:35:480
2351599241,3cyclictest26880-21kworker/0:008:41:460
2351599241,11cyclictest24416-21kworker/0:212:20:150
2351599241,10cyclictest1813-21kworker/0:110:03:270
2353299231,5cyclictest0-21swapper/311:32:233
2352899238,3cyclictest0-21swapper/211:45:402
2352899231,8cyclictest3800-40dnf10:02:002
2351599239,3cyclictest9025-21kworker/0:111:35:430
2351599239,3cyclictest9025-21kworker/0:111:35:430
2351599239,3cyclictest26880-21kworker/0:008:31:420
2351599238,2cyclictest12556-21kworker/0:110:22:150
2351599237,3cyclictest29639-21kworker/0:009:59:310
2351599237,3cyclictest26880-21kworker/0:008:36:020
2351599234,2cyclictest88950irq/132-enp2s0-08:48:140
2351599231,13cyclictest3399-21kworker/0:309:01:430
2351599231,10cyclictest9025-21kworker/0:111:43:190
2353299227,9cyclictest0-21swapper/310:45:193
2353299226,7cyclictest0-21swapper/311:41:253
23532992216,3cyclictest27799-21fw_packets08:30:183
2353299221,10cyclictest0-21swapper/312:15:413
2352899229,3cyclictest0-21swapper/207:37:542
2352899228,3cyclictest0-21swapper/207:31:222
2352899227,3cyclictest0-21swapper/208:27:472
2352899221,9cyclictest0-21swapper/209:42:342
2351599227,3cyclictest15416-21kworker/0:211:45:560
2351599224,2cyclictest87350irq/126-enp1s0-07:29:030
2351599221,5cyclictest7109-21kworker/0:207:55:030
2351599221,4cyclictest0-21swapper/011:23:280
309582210,3sleep32353299cyclictest11:05:143
2353299217,11cyclictest87350irq/126-enp1s0-10:40:173
2353299212,9cyclictest10740-21wc10:20:003
2353299211,8cyclictest294102/usr/sbin/munin08:35:193
23532992117,3cyclictest0-21swapper/312:13:293
2353299211,6cyclictest405-1kworker/u9:011:49:413
2353299211,10cyclictest0-21swapper/309:45:023
2353299210,8cyclictest0-21swapper/309:58:093
2352899217,3cyclictest0-21swapper/211:16:032
2352899211,6cyclictest6043-21interrupts07:40:172
2352899211,11cyclictest0-21swapper/212:26:202
2351599219,2cyclictest23715-21kworker/0:010:49:470
2351599219,2cyclictest23138-21kworker/0:107:10:030
2351599217,8cyclictest0-21swapper/007:47:280
2351599215,5cyclictest11555-21kworker/0:008:18:550
2351599215,4cyclictest22409-21kworker/0:108:22:470
2353299205,5cyclictest0-21swapper/310:28:573
2353299205,2cyclictest87350irq/126-enp1s0-12:27:053
2353299201,8cyclictest0-21swapper/312:23:483
2353299201,2cyclictest0-21swapper/310:20:143
2353299201,16cyclictest401ktimersoftd/309:05:163
2353299201,16cyclictest20720-21cut08:15:123
2353299201,12cyclictest0-21swapper/311:10:163
2353299201,11cyclictest14018-21/usr/sbin/munin08:00:233
2353299201,11cyclictest0-21swapper/311:35:183
2353299201,11cyclictest0-21swapper/311:35:183
2353299201,11cyclictest0-21swapper/311:17:303
2353299201,10cyclictest29798-21awk09:50:023
2353299200,9cyclictest22464-21kworker/u8:209:22:303
2353299200,10cyclictest0-21swapper/310:33:363
2352899208,3cyclictest0-21swapper/208:50:272
2351599207,6cyclictest19966-21kworker/0:111:55:510
2351599201,6cyclictest28325-21kworker/0:207:21:590
23515992012,5cyclictest27025-21fschecks_count07:15:140
2351599201,10cyclictest26880-21kworker/0:008:50:540
2353299198,4cyclictest839-21lldpd09:04:043
2353299194,2cyclictest87350irq/126-enp1s0-09:54:063
2353299194,12cyclictest41-21ksoftirqd/307:35:193
2353299193,3cyclictest0-21swapper/310:36:073
2353299191,9cyclictest0-21swapper/308:05:153
2353299191,8cyclictest0-21swapper/312:31:193
2353299191,5cyclictest4952-21sh10:05:153
23532991913,4cyclictest0-21swapper/309:28:203
2353299191,11cyclictest12214-21diskmemload11:03:323
2353299191,11cyclictest0-21swapper/312:02:003
2353299191,11cyclictest0-21swapper/310:55:053
2353299191,11cyclictest0-21swapper/309:14:523
2353299191,10cyclictest7147-21sshd12:35:573
2353299191,10cyclictest0-21swapper/311:20:253
2352899199,3cyclictest0-21swapper/208:39:472
23528991913,3cyclictest24013-21unixbench_singl12:00:252
121752191,2sleep312379-21fschecks_time07:55:153
2353299188,3cyclictest87350irq/126-enp1s0-11:25:223
2353299187,4cyclictest0-21swapper/307:25:153
2353299187,3cyclictest3654-21users10:00:263
2353299187,3cyclictest0-21swapper/309:17:213
2353299187,3cyclictest0-21swapper/309:17:213
2353299184,3cyclictest0-21swapper/311:51:213
2353299181,5cyclictest0-21swapper/312:08:313
2353299181,5cyclictest0-21swapper/310:50:113
23532991813,3cyclictest31608-21/usr/sbin/munin08:40:223
23532991812,4cyclictest12214-21diskmemload09:30:043
2353299181,15cyclictest0-21swapper/308:45:333
2353299181,10cyclictest12214-21diskmemload09:35:483
2351599182,4cyclictest18675-21cstates08:10:130
2353299176,3cyclictest39-21rcuc/308:20:183
2353299174,4cyclictest5007-21cat08:50:263
2353299171,9cyclictest0-21swapper/311:56:173
2353299171,15cyclictest0-21swapper/307:53:563
2353299171,15cyclictest0-21swapper/307:42:153
23532991710,4cyclictest7082-21fgrep08:55:253
23532991710,4cyclictest1546-21basename07:30:163
2353299169,4cyclictest18564-21tr08:10:113
2353299161,2cyclictest0-21swapper/307:15:293
2353299161,14cyclictest0-21swapper/307:22:083
23532991611,3cyclictest25019-21/usr/sbin/munin08:25:243
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional