You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-24 - 16:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Sun May 24, 2026 12:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
175802236163,22sleep30-21swapper/307:09:253
174432232155,26sleep00-21swapper/007:07:360
174072209158,39sleep20-21swapper/207:07:112
174642202165,24sleep10-21swapper/107:07:531
193472620,4sleep11771999cyclictest12:05:151
111252590,2sleep10-21swapper/111:45:201
46892560,5sleep01771399cyclictest11:30:220
40912530,5sleep01771399cyclictest10:15:220
312612490,2sleep20-21swapper/212:30:242
312612490,2sleep20-21swapper/212:30:242
4462460,3sleep11771999cyclictest10:10:011
277322460,1sleep10-21swapper/107:30:131
1771999351,15cyclictest0-21swapper/109:20:141
1771399356,14cyclictest32183-21munin-run08:55:010
17713993410,11cyclictest2031-21awk09:00:020
17713993410,11cyclictest2031-21awk09:00:020
1771999331,16cyclictest0-21swapper/108:45:141
1771399312,14cyclictest13021-21munin-run09:25:010
57192300,2sleep121-21rcuc/109:05:251
1771999301,13cyclictest0-21swapper/111:21:561
1771999301,13cyclictest0-21swapper/111:21:561
1771399301,7cyclictest19004-21munin-run08:25:020
96322290,4sleep31773099cyclictest09:15:173
1771999291,10cyclictest0-21swapper/109:44:541
17713992911,15cyclictest19610-21idleruntime-cro09:40:010
1771999281,18cyclictest11620-21kworker/u8:008:25:131
1771999281,12cyclictest0-21swapper/112:14:131
1771399285,11cyclictest24989-21perf07:25:000
1771399281,13cyclictest23045-21idleruntime-cro12:15:000
1771999271,9cyclictest0-21swapper/110:15:331
1771999271,9cyclictest0-21swapper/109:27:141
1771999271,11cyclictest0-21swapper/111:17:241
1771999271,10cyclictest0-21swapper/111:28:261
1771399278,8cyclictest15385-21cat09:30:010
1771999263,10cyclictest0-21swapper/110:22:211
1771999261,2cyclictest0-21swapper/110:02:531
17719992612,2cyclictest0-21swapper/112:20:011
1771999261,17cyclictest21305-21latency10:55:191
1771999261,11cyclictest0-21swapper/110:39:331
1771999261,10cyclictest0-21swapper/110:51:221
1771399267,16cyclictest18036-21munin-run10:50:020
1771399261,13cyclictest12067-21awk11:50:010
1771399261,12cyclictest7727-21ntp_states09:10:210
1771399261,12cyclictest6428-21munin-run09:10:020
1773099253,8cyclictest25363-40dnf08:37:193
1771999259,3cyclictest8756-21unixbench_multi10:25:251
1771999251,17cyclictest12613-21df11:50:131
1771999251,16cyclictest18144-21/usr/sbin/munin12:00:281
1771999251,14cyclictest0-21swapper/107:25:221
17719992511,3cyclictest4404-21missed_timers11:30:201
1771999251,12cyclictest0-21swapper/112:21:391
1771999251,12cyclictest0-21swapper/108:21:271
1771999251,11cyclictest0-21swapper/112:37:431
1771999251,11cyclictest0-21swapper/109:20:001
1771999251,11cyclictest0-21swapper/108:36:031
1772599241,8cyclictest0-21swapper/211:08:412
1772599241,17cyclictest0-21swapper/210:53:132
1771999242,12cyclictest9470-21users11:40:251
1771999241,15cyclictest0-21swapper/109:45:251
1771399245,5cyclictest0-21swapper/012:30:220
1771399245,5cyclictest0-21swapper/012:30:220
1771399245,16cyclictest27802-21awk08:45:010
1771399242,11cyclictest17912-21/usr/sbin/munin07:10:140
17713992416,5cyclictest25363-40dnf08:37:180
1771399241,11cyclictest9237-21munin-run10:30:010
1771399241,11cyclictest30840-21/usr/sbin/munin10:05:220
1771399241,11cyclictest29252-21/usr/sbin/munin11:15:120
1773099235,9cyclictest27099-21sshd09:55:183
1772599231,9cyclictest0-21swapper/208:27:052
1771999236,3cyclictest30941-21ntp_states12:30:211
1771999236,3cyclictest30941-21ntp_states12:30:211
1771999235,3cyclictest14194-21df10:40:131
1771999234,14cyclictest0-21swapper/111:38:121
1771999231,3cyclictest0-21swapper/107:48:591
1771999231,3cyclictest0-21swapper/107:48:591
1771999231,16cyclictest0-21swapper/109:13:211
1771999231,16cyclictest0-21swapper/108:32:541
1771999231,16cyclictest0-21swapper/108:03:501
1771999231,14cyclictest0-21swapper/111:14:301
1771999231,11cyclictest0-21swapper/108:55:201
1771999231,11cyclictest0-21swapper/108:55:201
1771999231,10cyclictest31706-21latency_hist07:40:011
1771399239,3cyclictest8851-21/usr/sbin/munin09:15:220
1771399236,13cyclictest21207-21awk08:30:020
1771399235,16cyclictest7-21ksoftirqd/007:45:000
1771399231,11cyclictest577-21/usr/sbin/munin10:10:220
1771399231,11cyclictest577-21/usr/sbin/munin10:10:220
1771399231,11cyclictest24235-21/usr/sbin/munin09:50:150
98122221,3sleep310013-21fschecks_time10:30:153
1773099223,9cyclictest6424-21diskmemload09:50:253
1772599221,9cyclictest0-21swapper/212:35:132
1772599221,7cyclictest0-21swapper/212:25:472
1772599221,6cyclictest0-21swapper/212:01:522
1772599221,11cyclictest0-21swapper/212:18:412
1771999228,1cyclictest0-21swapper/111:01:021
1771999227,3cyclictest0-21swapper/107:42:031
1771999225,8cyclictest0-21swapper/109:58:011
1771999223,10cyclictest87550irq/128-enp1s0-08:51:541
1771999222,8cyclictest87350irq/126-enp1s0-12:29:291
1771999222,3cyclictest17109-21awk10:45:211
1771999221,13cyclictest0-21swapper/109:00:221
1771999221,11cyclictest22338-21grep07:15:241
1771999221,10cyclictest20028-21munin-node07:15:011
1771999221,10cyclictest0-21swapper/110:13:551
1771999221,10cyclictest0-21swapper/110:13:551
1771399221,3cyclictest15826-21fschecks_count09:30:140
129982221,3sleep013178-21sed10:35:260
1773099219,9cyclictest20226-21sh10:55:003
1773099219,2cyclictest22363-21grep07:15:243
1773099215,4cyclictest909-21sssd_nss11:40:143
1773099214,9cyclictest87650irq/129-enp1s0-12:25:253
1773099212,11cyclictest39-21rcuc/310:15:263
1773099211,7cyclictest22243-21taskset10:57:483
1773099211,6cyclictest30933-21sort10:05:123
1773099211,4cyclictest0-21swapper/312:20:213
1773099211,4cyclictest0-21swapper/309:28:383
1773099211,11cyclictest7270-21users11:35:253
1772599217,3cyclictest0-21swapper/211:47:302
1772599217,3cyclictest0-21swapper/211:41:022
1772599216,3cyclictest0-21swapper/208:14:372
1772599215,3cyclictest0-21swapper/208:52:452
1772599212,3cyclictest87650irq/129-enp1s0-10:30:262
1772599211,9cyclictest74702kernelversion09:10:172
1772599211,9cyclictest268642ntp_kernel_pll_08:40:192
1772599211,9cyclictest0-21swapper/209:21:332
1772599210,9cyclictest0-21swapper/209:17:532
1771999216,3cyclictest0-21swapper/107:22:121
1771999215,3cyclictest0-21swapper/110:30:131
1771999211,14cyclictest0-21swapper/111:59:241
1771999211,11cyclictest0-21swapper/108:14:211
1771999210,13cyclictest0-21swapper/109:35:181
1771399214,3cyclictest18813-21kworker/0:108:30:430
1771399211,8cyclictest0-21swapper/011:40:440
1771399211,3cyclictest0-21swapper/012:25:180
17713992111,9cyclictest7-21ksoftirqd/008:04:420
224602201,15sleep222709-21/usr/sbin/munin09:45:162
1773099209,2cyclictest26372-21unixbench_multi11:05:273
1773099206,3cyclictest87650irq/129-enp1s0-12:00:293
1773099204,8cyclictest87350irq/126-enp1s0-12:05:133
1773099202,10cyclictest22655-21/usr/sbin/munin11:00:203
1773099201,8cyclictest0-21swapper/310:36:163
1773099201,7cyclictest0-21swapper/308:10:183
1773099201,7cyclictest0-21swapper/307:40:173
1773099201,12cyclictest0-21swapper/311:26:553
1773099201,11cyclictest5069-21/usr/sbin/munin10:20:233
1773099201,11cyclictest25816-21/usr/sbin/munin08:40:233
1773099201,11cyclictest22038-21/usr/sbin/munin09:45:213
1773099201,10cyclictest10891-21irqstats11:45:183
1773099200,4cyclictest14823-21kworker/u8:012:11:193
1773099200,11cyclictest20604-21cat09:40:163
1772599208,2cyclictest2927-21taskset11:25:542
1772599207,3cyclictest0-21swapper/210:55:332
1772599207,3cyclictest0-21swapper/210:14:012
1772599207,3cyclictest0-21swapper/210:14:012
1772599201,4cyclictest0-21swapper/210:24:402
1772599200,9cyclictest0-21swapper/212:08:222
1772599200,10cyclictest0-21swapper/209:55:462
1771999202,9cyclictest0-21swapper/111:08:581
17719992013,2cyclictest0-21swapper/108:06:401
1771999201,14cyclictest0-21swapper/109:52:101
1771999201,14cyclictest0-21swapper/109:31:321
1771999201,14cyclictest0-21swapper/108:41:491
1771399207,5cyclictest23294-21cpu12:15:120
1771399207,3cyclictest0-21swapper/007:33:160
1771399204,10cyclictest0-21swapper/010:50:170
1771399202,10cyclictest2248-21/usr/sbin/munin09:00:210
1771399201,16cyclictest27450-21munin-run12:25:010
1771399201,13cyclictest0-21swapper/007:25:220
1771399201,11cyclictest0-21swapper/011:26:250
1771399201,10cyclictest26131-21swap11:05:270
1771399201,10cyclictest20458-21/usr/sbin/munin10:55:150
1771399201,10cyclictest16671-21/usr/sbin/munin12:00:230
287062191,3sleep20-21swapper/211:11:032
239472190,3sleep20-21swapper/207:20:192
1773099197,4cyclictest0-21swapper/309:05:173
1773099191,7cyclictest0-21swapper/309:35:123
1773099191,6cyclictest4778-21cat11:30:243
1773099191,6cyclictest32505-21ntpq11:20:203
1773099191,6cyclictest32505-21ntpq11:20:203
17730991911,3cyclictest0-21swapper/308:55:273
17730991911,3cyclictest0-21swapper/308:55:273
1773099191,11cyclictest0-21swapper/310:04:583
1773099191,10cyclictest808-21snmpd08:07:583
1773099191,10cyclictest14475-21/usr/sbin/munin11:55:263
1773099191,10cyclictest0-21swapper/312:35:253
1773099191,10cyclictest0-21swapper/311:51:023
1772599196,3cyclictest0-21swapper/207:16:212
1772599196,2cyclictest0-21swapper/209:00:092
1772599193,3cyclictest87650irq/129-enp1s0-11:22:312
1772599193,3cyclictest87650irq/129-enp1s0-11:22:312
1772599193,3cyclictest87650irq/129-enp1s0-09:51:562
1772599192,14cyclictest5066-21head09:05:152
17725991913,3cyclictest17925-21cat09:35:132
1771999194,3cyclictest5416-21/usr/sbin/munin07:50:271
1771999191,15cyclictest76362chrt07:55:331
1771999190,14cyclictest0-21swapper/108:15:521
1771399197,3cyclictest11343-21memory08:05:180
1771399196,5cyclictest0-21swapper/011:14:120
1771399196,5cyclictest0-21swapper/009:57:200
1771399192,14cyclictest6424-21diskmemload09:45:120
1771399191,3cyclictest31735-21cat07:40:010
1771399191,3cyclictest14271-21grep10:40:150
1771399190,4cyclictest0-21swapper/011:59:000
1771399190,17cyclictest0-21swapper/012:05:180
1773099183,3cyclictest0-21swapper/310:25:243
1773099183,12cyclictest39-21rcuc/308:15:143
1773099182,5cyclictest31228-21munin-run11:20:013
1773099182,5cyclictest29985-21munin-run08:50:003
1773099181,9cyclictest0-21swapper/310:42:363
1773099181,9cyclictest0-21swapper/310:13:383
1773099181,9cyclictest0-21swapper/310:13:383
1773099181,6cyclictest8225-21/usr/sbin/munin08:00:153
1773099181,6cyclictest19816-21sshd07:14:183
1773099181,2cyclictest17047-21strings10:45:203
17730991812,3cyclictest24125-21pmu-power07:20:213
1773099181,15cyclictest0-21swapper/308:51:003
17730991811,4cyclictest29361-21dump-pmu-power07:35:003
1773099181,10cyclictest27420-21kworker/u8:012:30:063
1773099181,10cyclictest27420-21kworker/u8:012:30:063
1773099180,10cyclictest27660-21gltestperf11:10:153
1772599185,3cyclictest0-21swapper/208:02:582
1772599184,2cyclictest0-21swapper/207:51:342
1772599183,5cyclictest87550irq/128-enp1s0-09:40:142
1772599183,3cyclictest87350irq/126-enp1s0-10:05:252
1772599181,7cyclictest0-21swapper/209:30:002
1772599181,7cyclictest0-21swapper/208:47:462
17725991812,3cyclictest21753-21df08:30:112
1772599181,15cyclictest0-21swapper/208:36:252
17725991811,4cyclictest12923-21/usr/sbin/munin10:35:232
1772599181,11cyclictest0-21swapper/210:45:262
1771399185,3cyclictest0-21swapper/010:32:270
1771399183,6cyclictest0-21swapper/009:41:520
1771399181,9cyclictest0-21swapper/010:00:140
1771399181,3cyclictest0-21swapper/012:38:490
1771399181,15cyclictest8012-21perf08:00:010
1771399181,10cyclictest0-21swapper/011:23:270
1771399181,10cyclictest0-21swapper/011:23:270
173002180,2sleep20-21swapper/209:33:482
73182171,2sleep20-21swapper/211:35:522
311062171,2sleep20-21swapper/211:18:412
1773099173,8cyclictest0-21swapper/308:25:193
1773099171,7cyclictest0-21swapper/309:30:183
1773099171,5cyclictest0-21swapper/307:50:193
1773099171,4cyclictest11048-21/usr/sbin/munin09:20:263
17730991712,3cyclictest30448-21latency07:35:183
17730991712,3cyclictest2475-21latency07:45:183
17730991712,3cyclictest2475-21latency07:45:173
1773099171,15cyclictest0-21swapper/307:28:553
1773099171,14cyclictest401ktimersoftd/312:15:193
17730991711,3cyclictest7560-21unixbench_singl07:55:273
1773099171,13cyclictest401ktimersoftd/309:10:163
1772599175,2cyclictest0-21swapper/208:24:532
1772599171,6cyclictest28977-21diskstats10:00:142
1772599171,5cyclictest0-21swapper/212:12:232
1772599171,5cyclictest0-21swapper/210:29:332
17725991712,3cyclictest2730-21open_inodes07:45:222
17725991712,3cyclictest2730-21open_inodes07:45:222
1772599171,15cyclictest0-21swapper/208:09:572
1772599171,14cyclictest0-21swapper/207:55:372
1772599171,10cyclictest0-21swapper/211:58:142
1772599170,4cyclictest24655-1kworker/u9:011:50:102
1771399176,3cyclictest87350irq/126-enp1s0-11:00:270
1771399176,3cyclictest0-21swapper/011:50:250
1771399171,3cyclictest0-21swapper/008:16:340
17713991712,3cyclictest20782-21sed07:15:050
1771399171,15cyclictest0-21swapper/008:48:480
1771399171,13cyclictest3829-21/usr/sbin/munin07:50:170
1771399171,10cyclictest0-21swapper/011:36:010
1771399170,10cyclictest0-21swapper/007:45:110
1771399170,10cyclictest0-21swapper/007:45:110
1773099166,3cyclictest0-21swapper/308:30:153
1773099161,9cyclictest0-21swapper/308:22:373
17730991610,4cyclictest2960-21sort09:00:183
1772599165,3cyclictest12050irq/123-ahci[0007:30:222
1772599163,4cyclictest25656-21/usr/sbin/munin12:20:122
1772599161,7cyclictest29592-21/usr/sbin/munin07:35:202
1772599161,5cyclictest14929-21sh10:40:212
1772599161,5cyclictest0-21swapper/210:17:202
1772599161,2cyclictest0-21swapper/207:12:072
1772599161,14cyclictest0-21swapper/207:28:152
17725991611,3cyclictest3830-21df_inode11:30:142
17725991611,3cyclictest22655-21/usr/sbin/munin11:00:262
17725991610,3cyclictest32744-21sh08:55:142
17725991610,3cyclictest32744-21sh08:55:142
1771399161,6cyclictest0-21swapper/010:20:090
1771399161,14cyclictest0-21swapper/008:10:020
1772599151,3cyclictest32501-21idleruntime07:40:172
1772599140,5cyclictest0-21swapper/208:15:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional