You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-10 - 22:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Fri Apr 10, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
85252238164,24sleep30-21swapper/307:07:443
84942204166,25sleep20-21swapper/207:07:182
85582202165,24sleep10-21swapper/107:08:091
84072201165,24sleep00-21swapper/007:06:110
121132700,1sleep20-21swapper/210:55:162
8800996965,2cyclictest29853-21diskmemload11:33:552
8800996965,2cyclictest29853-21diskmemload11:33:542
283492560,2sleep230-21rcuc/210:20:012
128782490,1sleep012879-21uname07:15:190
80112420,1sleep20-21swapper/212:00:102
879599314,13cyclictest6864-40dnf09:30:171
99422300,3sleep230-21rcuc/209:35:222
878999291,20cyclictest25863-21kworker/0:010:28:350
8795992816,3cyclictest0-21swapper/110:27:311
879599281,15cyclictest0-21swapper/112:31:181
44522280,5sleep34456-21cpuspeed_turbos09:25:123
879599279,7cyclictest10663-21kworker/u8:109:50:021
879599272,21cyclictest16662-21munin-run08:40:011
879599271,11cyclictest5330-21turbostat.cron11:55:011
3992270,3sleep121-21rcuc/109:15:161
880599262,9cyclictest0-21swapper/309:43:413
878999268,15cyclictest14482-21perf08:35:010
878999261,19cyclictest23097-21kworker/0:208:58:260
878999260,19cyclictest6683-21kworker/0:010:43:400
878999260,19cyclictest28120-21kworker/0:110:22:190
878999260,19cyclictest28120-21kworker/0:110:22:190
878999260,19cyclictest13155-21kworker/0:211:04:150
880599250,7cyclictest0-21swapper/309:05:173
879599252,15cyclictest21036-21sshd08:49:561
878999251,17cyclictest19991-21kworker/0:311:27:390
312092250,3sleep00-21swapper/011:40:010
150932250,2sleep339-21rcuc/307:20:193
880599241,4cyclictest0-21swapper/310:18:253
879599241,3cyclictest0-21swapper/110:44:131
879599241,17cyclictest0-21swapper/110:14:331
879599240,2cyclictest87550irq/128-enp1s0-12:35:121
878999242,9cyclictest20683-21kworker/0:312:34:570
878999242,12cyclictest25863-21kworker/0:010:19:430
19512240,4sleep3401ktimersoftd/308:05:153
880599239,3cyclictest301652idleruntime07:55:163
880599231,4cyclictest0-21swapper/312:28:103
880599231,10cyclictest808-21snmpd10:07:383
8795992310,11cyclictest3250-21kworker/u8:207:27:451
8795992310,11cyclictest20382-21kworker/u8:008:52:441
878999237,8cyclictest17950-21kworker/0:207:35:150
39092231,2sleep00-21swapper/008:10:130
153972230,2sleep20-21swapper/209:50:132
15172231,18sleep21589-21awk10:30:172
880599221,12cyclictest4427-21ntp_states11:50:203
880599221,11cyclictest910-21systemd-logind08:40:143
880099227,4cyclictest214112sh11:15:212
880099225,10cyclictest22625-40dnf12:31:192
879599228,9cyclictest0-21swapper/110:01:461
879599226,15cyclictest0-21swapper/109:05:151
879599221,18cyclictest9137-21ls12:00:221
879599221,18cyclictest8675-21perl08:20:161
879599221,16cyclictest0-21swapper/110:18:511
879599221,15cyclictest0-21swapper/112:25:511
879599221,15cyclictest0-21swapper/111:40:451
879599221,13cyclictest0-21swapper/109:10:171
879599221,13cyclictest0-21swapper/109:10:171
879599221,11cyclictest0-21swapper/111:30:301
879599221,11cyclictest0-21swapper/111:30:301
8795992210,4cyclictest16125-21cat07:25:021
878999221,9cyclictest31953-21kworker/0:109:17:390
878999221,10cyclictest20457-21perf07:35:000
878999220,8cyclictest29195-21kworker/0:307:59:150
880599219,3cyclictest14141-21munin-run12:15:013
880599218,4cyclictest0-21swapper/311:30:413
880599218,4cyclictest0-21swapper/311:30:413
880599215,2cyclictest87350irq/126-enp1s0-11:01:433
880599213,8cyclictest0-21swapper/309:21:073
880599212,5cyclictest0-21swapper/310:10:253
880599211,11cyclictest0-21swapper/311:49:273
880599211,10cyclictest0-21swapper/312:15:333
880599210,5cyclictest0-21swapper/310:23:103
880599210,5cyclictest0-21swapper/310:23:103
880099212,5cyclictest20447-21munin-run07:35:012
8800992114,4cyclictest20721-21dump-pmu-power12:30:012
879599218,3cyclictest0-21swapper/112:13:421
879599216,4cyclictest0-21swapper/112:19:151
879599214,7cyclictest13609-21latency_hist11:00:021
879599211,9cyclictest21-21rcuc/109:25:171
879599211,6cyclictest22803-21cat10:05:181
8795992116,3cyclictest3145-21awk11:50:011
879599211,3cyclictest17326-21taskset12:20:151
879599211,17cyclictest29853-21diskmemload11:59:471
879599211,15cyclictest0-21swapper/110:21:571
879599211,15cyclictest0-21swapper/110:21:571
879599211,14cyclictest0-21swapper/110:49:371
879599211,14cyclictest0-21swapper/109:56:381
879599211,12cyclictest0-21swapper/110:52:251
879599211,11cyclictest28664-21/usr/sbin/munin07:50:241
879599211,10cyclictest0-21swapper/112:07:191
878999212,7cyclictest6864-40dnf09:30:160
878999211,8cyclictest14688-21kworker/0:008:49:450
878999211,12cyclictest20827-21kworker/0:108:51:300
878999211,11cyclictest9338-21/usr/sbin/munin10:50:130
301812210,1sleep10-21swapper/107:55:161
880599204,3cyclictest0-21swapper/311:19:133
880599201,6cyclictest0-21swapper/309:58:273
880599201,6cyclictest0-21swapper/309:32:383
880599201,6cyclictest0-21swapper/309:17:003
880599201,12cyclictest0-21swapper/311:38:003
880599201,12cyclictest0-21swapper/310:45:533
880599201,10cyclictest437-21/usr/sbin/munin10:30:113
880599201,10cyclictest0-21swapper/310:26:413
880099206,3cyclictest87350irq/126-enp1s0-07:55:222
880099201,7cyclictest14275-21fschecks_time11:00:142
880099201,7cyclictest0-21swapper/208:40:342
880099201,3cyclictest0-21swapper/209:24:022
879599208,3cyclictest0-21swapper/111:00:061
879599208,2cyclictest0-21swapper/108:30:291
879599207,3cyclictest0-21swapper/111:29:591
8795992017,2cyclictest0-21swapper/107:14:501
8795992016,3cyclictest0-21swapper/110:30:151
879599201,3cyclictest0-21swapper/109:40:101
879599201,3cyclictest0-21swapper/108:01:341
879599201,3cyclictest0-21swapper/108:01:341
879599201,18cyclictest0-21swapper/109:00:461
879599201,17cyclictest0-21swapper/107:36:091
8795992011,3cyclictest0-21swapper/109:21:591
879599201,11cyclictest18556-21grep11:10:151
879599200,4cyclictest0-21swapper/110:35:121
878999209,3cyclictest13115-21/usr/sbin/munin09:45:220
878999205,3cyclictest87550irq/128-enp1s0-09:10:020
878999204,8cyclictest87550irq/128-enp1s0-12:05:120
878999204,8cyclictest87550irq/128-enp1s0-11:55:190
878999204,2cyclictest20114-21kworker/0:307:40:530
878999202,10cyclictest16273-21/usr/sbin/munin07:25:260
878999201,9cyclictest18455-21kworker/0:112:27:110
878999201,7cyclictest25676-21/usr/sbin/munin09:00:190
878999201,5cyclictest909-21sssd_nss12:39:160
878999201,11cyclictest2617-21fschecks_time09:20:130
878999201,11cyclictest14074-21/usr/sbin/munin07:20:170
878999201,10cyclictest0-21swapper/009:25:050
880599197,3cyclictest10297-21/usr/sbin/munin08:25:173
880599196,6cyclictest0-21swapper/311:57:523
880599196,4cyclictest0-21swapper/309:39:553
880599196,3cyclictest0-21swapper/312:09:093
880599191,10cyclictest29853-21diskmemload09:50:253
880599191,10cyclictest0-21swapper/312:35:213
880599191,10cyclictest0-21swapper/310:03:003
880599191,10cyclictest0-21swapper/307:10:103
880599190,7cyclictest0-21swapper/312:34:393
880599190,5cyclictest0-21swapper/312:21:083
880599190,5cyclictest0-21swapper/311:42:023
880599190,13cyclictest0-21swapper/308:45:223
880099197,4cyclictest29853-21diskmemload11:20:142
880099195,5cyclictest0-21swapper/212:05:192
880099191,9cyclictest27030-21sh07:50:012
880099191,7cyclictest0-21swapper/211:35:572
880099191,7cyclictest0-21swapper/210:09:512
880099191,6cyclictest0-21swapper/212:17:252
880099191,5cyclictest7474-21df_inode10:45:132
880099191,5cyclictest13282-21timerwakeupswit07:15:242
879599198,6cyclictest0-21swapper/111:35:331
879599196,3cyclictest5672-21munin-run08:15:011
879599191,5cyclictest13517-21kworker/1:111:20:141
8795991915,3cyclictest0-21swapper/108:19:461
8795991915,3cyclictest0-21swapper/107:33:001
8795991915,2cyclictest29853-21diskmemload09:38:401
879599191,17cyclictest0-21swapper/108:09:561
879599191,16cyclictest0-21swapper/109:50:171
879599191,16cyclictest0-21swapper/108:40:541
879599191,10cyclictest26339-21switchtime07:45:231
879599191,10cyclictest17361-21timerwakeupswit11:05:281
879599190,2cyclictest0-21swapper/111:17:011
878999199,3cyclictest0-21swapper/011:40:210
878999197,3cyclictest1479-21/usr/sbin/munin08:05:240
878999196,2cyclictest26818-21kworker/0:107:51:030
878999195,3cyclictest25790-21NetworkManager10:11:220
878999195,2cyclictest13860-21kworker/0:212:14:280
878999192,9cyclictest9880-21kworker/0:208:36:330
878999192,5cyclictest23598-21ntp_states11:20:190
878999191,9cyclictest20272-21cat11:15:010
878999191,8cyclictest17014-21kworker/0:009:52:390
878999191,6cyclictest30080-21/usr/sbin/munin09:10:130
878999191,6cyclictest30080-21/usr/sbin/munin09:10:130
878999191,6cyclictest18441-21sort09:55:190
878999191,5cyclictest7860-21grep12:00:090
878999191,5cyclictest19707-21/usr/sbin/munin10:00:230
8789991914,3cyclictest8760-21cut08:20:160
878999191,10cyclictest14994-21gltestperf12:15:160
55012191,3sleep3808-21snmpd08:13:023
38752190,5sleep230-21rcuc/210:35:222
880599189,6cyclictest9148-21ntp_kernel_pll_08:20:213
880599186,4cyclictest29853-21diskmemload11:20:493
880599182,13cyclictest11882-21/usr/sbin/munin07:15:173
880599181,9cyclictest0-21swapper/311:05:063
880599181,7cyclictest0-21swapper/311:25:203
8805991812,3cyclictest22863-21/usr/sbin/munin07:40:203
8805991811,4cyclictest21380-21sort07:35:163
880599181,10cyclictest0-21swapper/312:04:403
880599181,10cyclictest0-21swapper/311:13:353
880599181,10cyclictest0-21swapper/310:55:253
880599181,10cyclictest0-21swapper/310:40:213
880599181,10cyclictest0-21swapper/309:49:003
880099186,4cyclictest87350irq/126-enp1s0-11:25:212
880099182,3cyclictest0-21swapper/209:57:102
880099182,15cyclictest0-21swapper/209:34:502
880099182,13cyclictest11766-21grep08:25:262
880099181,9cyclictest0-21swapper/210:23:232
880099181,9cyclictest0-21swapper/210:23:232
880099181,7cyclictest0-21swapper/212:35:222
880099181,15cyclictest0-21swapper/208:35:322
880099181,15cyclictest0-21swapper/207:42:272
8800991811,4cyclictest22395-21ls08:50:222
880099181,10cyclictest0-21swapper/211:05:272
879599181,16cyclictest0-21swapper/108:55:171
879599181,16cyclictest0-21swapper/108:27:061
879599181,16cyclictest0-21swapper/107:41:051
879599181,15cyclictest0-21swapper/107:15:531
878999183,8cyclictest87550irq/128-enp1s0-07:10:160
878999182,3cyclictest87550irq/128-enp1s0-10:48:240
878999181,9cyclictest0-21swapper/011:09:170
878999181,7cyclictest0-21swapper/009:40:030
878999181,6cyclictest0-21swapper/010:09:530
878999181,5cyclictest4458-21taskset10:36:540
878999181,5cyclictest27990-21ntpq11:30:190
878999181,5cyclictest27990-21ntpq11:30:190
7142181,2sleep20-21swapper/211:43:252
880599178,3cyclictest0-21swapper/308:30:163
880599175,9cyclictest0-21swapper/310:35:223
880599171,9cyclictest0-21swapper/309:10:023
880599171,9cyclictest0-21swapper/309:10:023
8805991710,4cyclictest26940-21smart_sda09:00:233
8805991710,4cyclictest0-21swapper/310:53:153
880099174,4cyclictest1695-21sort11:45:142
880099172,2cyclictest87350irq/126-enp1s0-10:50:202
880099171,9cyclictest0-21swapper/207:10:122
880099171,7cyclictest22632-21sh07:40:012
880099171,7cyclictest20387-21smartctl10:00:162
880099171,7cyclictest1850-21diskstats08:05:132
880099171,6cyclictest5677-21perf08:15:012
880099171,6cyclictest4316-21/usr/sbin/munin09:25:112
880099171,6cyclictest0-21swapper/210:26:282
880099171,6cyclictest0-21swapper/208:23:252
880099171,4cyclictest18135-21/usr/sbin/munin11:10:122
8800991713,3cyclictest0-21swapper/211:57:512
880099171,15cyclictest0-21swapper/207:53:332
8800991711,4cyclictest0-21swapper/209:45:032
880099171,14cyclictest0-21swapper/208:48:062
8800991711,3cyclictest30892-21interrupts09:10:162
8800991711,3cyclictest30892-21interrupts09:10:162
878999179,3cyclictest17755-21latency08:40:180
878999176,2cyclictest25863-21kworker/0:010:34:310
878999173,2cyclictest87550irq/128-enp1s0-12:20:070
878999171,6cyclictest3908-21grep11:50:140
878999171,5cyclictest839-21lldpd07:45:120
878999171,5cyclictest5897-21/usr/sbin/munin08:15:150
878999171,4cyclictest20335-21/usr/sbin/munin11:15:170
878999171,4cyclictest0-21swapper/009:35:210
878999171,3cyclictest11539-21/usr/sbin/munin10:55:130
8789991712,3cyclictest827-21runrttasks11:45:170
880599161,7cyclictest808-21snmpd08:02:143
880599161,7cyclictest808-21snmpd08:02:143
880599161,6cyclictest0-21swapper/307:25:153
880599161,2cyclictest0-21swapper/308:54:523
880599161,2cyclictest0-21swapper/308:35:303
8805991610,4cyclictest24474-21head08:55:193
8805991610,3cyclictest827-21runrttasks07:51:453
8805991610,3cyclictest6091-21cat08:15:143
8805991610,3cyclictest26121-21ntp_offset07:45:193
880099169,4cyclictest25582-21unixbench-2d10:10:242
880099163,4cyclictest790-21in:imjournal12:24:162
880099161,9cyclictest0-21swapper/211:53:302
880099161,7cyclictest0-21swapper/208:30:172
8800991612,3cyclictest0-21swapper/208:18:582
8800991611,3cyclictest29219-21threads09:05:232
8800991611,3cyclictest26733-21ntp_offset09:00:202
8800991611,3cyclictest24062-21perl08:55:152
8800991611,3cyclictest16626-21dnf07:25:162
8800991611,3cyclictest16048-21grep07:25:012
878999169,4cyclictest10596-21sshd08:25:130
8789991610,3cyclictest31875-21expr08:00:130
8789991610,3cyclictest31875-21expr08:00:130
137902160,2sleep20-21swapper/212:10:422
880599158,4cyclictest19389-21missed_timers07:30:173
880099158,4cyclictest5471-21uniq10:40:142
880099151,5cyclictest0-21swapper/208:01:462
880099151,5cyclictest0-21swapper/208:01:452
8800991510,3cyclictest32749-21grep09:15:142
8800991510,3cyclictest13115-21/usr/sbin/munin09:45:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional