You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-07 - 01:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack5slot3s.osadl.org (updated Mon Jul 06, 2026 12:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
174622238169,57sleep20-21swapper/207:06:592
174622238169,57sleep20-21swapper/207:06:592
174552236163,23sleep00-21swapper/007:06:530
174552236163,23sleep00-21swapper/007:06:530
175012204167,25sleep30-21swapper/307:07:283
175012204167,25sleep30-21swapper/307:07:283
175552202166,24sleep10-21swapper/107:08:111
175552202166,24sleep10-21swapper/107:08:111
17796991081,106cyclictest0-21swapper/210:52:092
17796999692,3cyclictest0-21swapper/211:45:432
119262950,4sleep21779699cyclictest09:20:182
54152740,2sleep10-21swapper/110:20:131
137752530,1sleep00-21swapper/008:10:210
6722500,4sleep230-21rcuc/210:10:112
217272490,4sleep31780199cyclictest10:55:233
304312470,2sleep20-21swapper/208:50:132
300642470,1sleep30-21swapper/308:50:023
17796994137,3cyclictest0-21swapper/211:54:202
17784993810,13cyclictest8691-21munin-run09:15:010
1778499337,12cyclictest8074-21awk08:00:010
1778499336,12cyclictest1761-21awk12:40:010
1778499327,12cyclictest19960-21munin-run07:15:010
1778499324,13cyclictest10273-21munin-run08:05:010
1778499315,12cyclictest17482-21munin-run09:35:010
1779699301,13cyclictest0-21swapper/211:23:042
17790993015,6cyclictest0-21swapper/107:29:551
300092290,8sleep1221ktimersoftd/112:30:121
21012290,5sleep081ktimersoftd/011:25:180
1778499296,11cyclictest27259-21perf07:30:010
1779699282,5cyclictest25328-21munin-run12:20:022
1778499283,12cyclictest27854-21munin-run08:45:010
17784992810,15cyclictest23461-21munin-run08:35:010
17801992713,7cyclictest10794-21sensors10:30:253
1779099279,5cyclictest21268-40dnf08:30:011
17790992720,4cyclictest28730-21smart_sda07:30:231
1779099261,17cyclictest0-21swapper/112:06:261
1778499262,10cyclictest27100-21kworker/0:307:30:190
1778499261,12cyclictest3673-21munin-run07:50:020
215012251,3sleep121608-21cut12:10:151
1780199251,13cyclictest0-21swapper/312:15:163
1779699253,8cyclictest10620-40dnf10:30:252
17790992521,3cyclictest0-21swapper/112:25:361
17790992517,5cyclictest14374-21ntp_states09:25:201
17790992517,5cyclictest14374-21ntp_states09:25:201
17790992511,9cyclictest0-21swapper/108:01:351
1779099251,11cyclictest12936-21processes10:35:231
1779099251,11cyclictest12936-21processes10:35:231
17784992514,7cyclictest10838-21switchtime10:30:240
17796992417,4cyclictest6480-21sh09:10:002
1779699241,13cyclictest0-21swapper/212:09:592
1779099248,4cyclictest13283-21tr08:10:151
1779099247,6cyclictest10637-21grep08:05:121
17790992420,3cyclictest0-21swapper/111:39:441
1779099241,17cyclictest0-21swapper/107:55:241
1779099241,16cyclictest0-21swapper/110:45:211
1779099241,11cyclictest23658-21unixbench_singl09:45:251
1779099241,11cyclictest19954-21latency_hist07:15:001
1779099241,10cyclictest7897-21proc_pri09:10:241
1779099241,10cyclictest25648-21sh08:40:001
31092231,18sleep03272-21df_inode10:15:120
1780199232,6cyclictest0-21swapper/311:50:543
1779699236,4cyclictest31550-21grep10:05:152
1779699236,4cyclictest28057-21grep07:30:152
1779699231,13cyclictest0-21swapper/210:46:152
1779099239,3cyclictest4776-21rt-features11:30:231
1779099238,4cyclictest9261-21diskstats09:15:141
1779099238,3cyclictest10823-21hddtemp_smartct11:45:151
1779099235,9cyclictest87550irq/128-enp1s0-09:50:251
17790992321,1cyclictest0-21swapper/110:05:031
1779099231,6cyclictest8870-21memory11:40:191
1779099231,20cyclictest0-21swapper/111:50:241
1779099231,16cyclictest0-21swapper/109:21:541
1779099231,16cyclictest0-21swapper/107:20:161
1779099231,14cyclictest0-21swapper/111:15:111
1779099231,10cyclictest15617-21ntp_states11:55:241
1779099230,17cyclictest0-21swapper/107:45:211
1778499231,11cyclictest23019-21memory09:45:190
218602220,1sleep10-21swapper/107:15:201
1780199221,10cyclictest5586-21ntp_states09:05:213
1780199221,10cyclictest0-21swapper/307:45:263
1779699225,9cyclictest6486-21diskmemload12:34:542
1779699225,4cyclictest22494-21turbostat11:00:012
1779099228,3cyclictest18593-21ls10:50:141
1779099228,3cyclictest15186-21smart_sda10:40:221
1779099227,3cyclictest25278-21diskstats11:05:131
1779099225,8cyclictest87550irq/128-enp1s0-08:45:221
1779099223,7cyclictest8029-21grep10:25:181
17790992219,2cyclictest0-21swapper/112:39:461
17790992212,2cyclictest0-21swapper/109:30:111
1779099221,19cyclictest0-21swapper/111:20:261
1779099221,18cyclictest5869-21munin-run07:55:011
1779099221,18cyclictest10620-40dnf10:30:231
1779099221,15cyclictest0-21swapper/110:57:101
1779099221,15cyclictest0-21swapper/108:17:091
1779099221,15cyclictest0-21swapper/108:17:091
1779099221,15cyclictest0-21swapper/107:36:221
1779099221,10cyclictest4002-21ntp_states10:15:191
1779099221,10cyclictest18996-21chrt08:24:151
1779099221,10cyclictest1444-21cpuspeed_turbos11:25:131
1779099221,10cyclictest0-21swapper/109:00:211
17790992210,3cyclictest0-21swapper/109:35:351
1779099220,15cyclictest0-21swapper/109:41:041
1778499226,2cyclictest23871-21kworker/0:209:51:390
17784992215,3cyclictest28365-21kworker/0:210:00:150
1780199219,3cyclictest2932-21/usr/sbin/munin10:15:193
1780199219,3cyclictest17823-21ntp_states12:00:243
1780199219,3cyclictest16203-21wc09:30:163
1780199219,3cyclictest10045-21sshd08:02:153
1780199219,3cyclictest0-21swapper/309:38:143
1780199211,6cyclictest0-21swapper/311:23:293
1780199211,11cyclictest0-21swapper/312:30:153
1780199211,10cyclictest31142-21sed07:35:253
1780199211,10cyclictest22664-21ps08:30:223
17801992110,2cyclictest0-21swapper/307:40:243
1779699215,3cyclictest87650irq/129-enp1s0-11:00:152
1779699212,6cyclictest4893-21grep07:50:202
1779699212,6cyclictest18056-21diskstats09:35:132
1779699211,9cyclictest0-21swapper/209:57:372
1779699211,8cyclictest0-21swapper/209:11:192
1779099215,3cyclictest23335-21grep11:00:161
17790992117,3cyclictest0-21swapper/112:00:091
17790992113,3cyclictest0-21swapper/108:30:211
1779099211,10cyclictest25879-21/usr/sbin/munin08:40:241
1778499217,2cyclictest1937-21kworker/0:109:05:230
1778499211,6cyclictest0-21swapper/009:25:100
1778499211,6cyclictest0-21swapper/009:25:100
17784992115,3cyclictest18736-21munin-plugin-st12:05:010
1778499211,4cyclictest29105-21perf11:15:020
1778499211,11cyclictest6486-21diskmemload10:55:160
1778499211,11cyclictest2515-21kworker/0:110:23:380
1780199209,2cyclictest27310-21irqstats09:55:173
1780199204,10cyclictest272202sleep307:29:553
1780199203,7cyclictest0-21swapper/312:35:193
1780199202,8cyclictest16121-21/usr/sbin/munin10:45:233
1780199201,9cyclictest495-21jbd2/dm-0-809:04:203
1780199201,7cyclictest0-21swapper/309:12:343
1780199201,6cyclictest0-21swapper/309:22:443
1780199201,11cyclictest0-21swapper/308:20:253
1780199201,10cyclictest23685-21/usr/sbin/munin08:35:193
1780199201,10cyclictest0-21swapper/311:03:383
1779699206,3cyclictest87650irq/129-enp1s0-09:45:262
1779699205,8cyclictest87650irq/129-enp1s0-09:40:152
1779699205,4cyclictest6486-21diskmemload12:01:152
1779699205,3cyclictest28891-21head12:25:232
1779699204,4cyclictest23811-21cpuspeed_turbos08:35:132
1779699203,4cyclictest11859-21grep10:35:122
1779699203,4cyclictest11859-21grep10:35:122
1779699202,4cyclictest26095-21fschecks_time12:20:152
1779699201,7cyclictest2310-21/usr/sbin/munin09:00:142
1779699201,7cyclictest0-21swapper/211:58:162
17796992017,2cyclictest0-21swapper/207:25:462
1779699201,6cyclictest1944-21cat12:40:012
1779699201,6cyclictest15850-21diskstats09:30:132
1779699201,6cyclictest14298-21needreboot09:25:182
1779699201,6cyclictest14298-21needreboot09:25:172
1779699201,6cyclictest0-21swapper/209:50:122
17796992016,3cyclictest0-21swapper/211:31:192
17796992012,5cyclictest1-21systemd07:15:022
1779699200,14cyclictest0-21swapper/212:13:392
1779099205,3cyclictest21-21rcuc/112:15:261
1779099201,7cyclictest645-21/usr/sbin/munin10:10:131
17790992017,2cyclictest0-21swapper/108:50:411
1779099201,18cyclictest0-21swapper/110:00:131
1779099201,17cyclictest0-21swapper/112:20:191
1779099201,11cyclictest27079-21perl09:55:151
1779099201,10cyclictest0-21swapper/111:10:091
1779099201,10cyclictest0-21swapper/109:05:381
1779099201,10cyclictest0-21swapper/107:40:161
1778499209,8cyclictest19457-21sshd09:37:160
1778499206,2cyclictest21107-21kworker/0:010:05:230
1778499205,3cyclictest22141-21kworker/0:011:00:590
1778499204,3cyclictest87350irq/126-enp1s0-12:27:060
1778499203,5cyclictest24424-21kworker/0:111:05:270
1778499202,10cyclictest18960-21/usr/sbin/munin12:05:180
1778499201,6cyclictest87650irq/129-enp1s0-12:17:460
1778499201,11cyclictest4627-21ntp_states11:30:220
1778499200,14cyclictest22708-21kworker/0:207:23:210
42492191,2sleep04481-21perl07:50:160
254112191,3sleep00-21swapper/008:37:030
1780199199,2cyclictest0-21swapper/307:10:273
1780199191,7cyclictest0-21swapper/311:58:363
1780199191,6cyclictest28090-21df_inode12:25:153
1780199191,5cyclictest0-21swapper/311:10:023
17801991913,3cyclictest14814-21unixbench_multi09:25:243
17801991913,3cyclictest14814-21unixbench_multi09:25:243
17801991912,4cyclictest14738-21irqstats10:40:173
1780199191,15cyclictest0-21swapper/312:09:213
1780199191,12cyclictest0-21swapper/308:10:223
1780199191,11cyclictest87350irq/126-enp1s0-11:05:213
1780199191,10cyclictest27468-21/usr/sbin/munin07:30:203
1780199191,10cyclictest0-21swapper/309:50:163
1779699195,3cyclictest87650irq/129-enp1s0-08:25:262
1779699194,4cyclictest29487-21cstates11:15:092
1779699194,3cyclictest32-21ksoftirqd/209:15:242
1779699192,5cyclictest31633-21munin-run07:40:012
1779699191,8cyclictest14578-21gltestperf10:40:152
1779699191,6cyclictest6160-21nfs4_client10:20:202
1779699191,6cyclictest21491-21/usr/sbin/munin08:30:162
1779699191,6cyclictest1337-21/usr/sbin/munin11:25:132
1779699191,5cyclictest29111-21ntpq08:45:212
1778499198,3cyclictest19905-21/usr/sbin/munin09:40:250
1778499198,3cyclictest11110-21/usr/sbin/munin09:20:180
1778499196,2cyclictest7028-21kworker/0:010:44:310
1778499194,8cyclictest0-21swapper/007:20:010
1778499194,4cyclictest6268-21kworker/0:009:17:510
1778499194,3cyclictest31438-21kworker/0:007:41:310
1778499194,3cyclictest0-21swapper/011:48:390
1778499193,2cyclictest87350irq/126-enp1s0-10:53:440
1778499191,5cyclictest0-21swapper/011:51:120
1778499191,3cyclictest87550irq/128-enp1s0-08:50:550
1778499191,12cyclictest0-21swapper/008:25:230
1778499191,10cyclictest29671-21kworker/0:112:30:190
1778499191,10cyclictest28862-21kworker/0:111:15:030
1778499191,10cyclictest25546-21/usr/sbin/munin12:20:190
1778499191,10cyclictest14545-21/usr/sbin/munin11:55:220
1778499191,10cyclictest0-21swapper/012:11:540
1778499191,10cyclictest0-21swapper/011:41:510
1780199186,3cyclictest8397-21ntp_states10:25:233
1780199185,4cyclictest29281-21grep10:00:143
1780199185,11cyclictest41-21ksoftirqd/311:45:133
1780199184,4cyclictest8164-21grep08:00:013
1780199182,6cyclictest5128-21/usr/sbin/munin10:20:223
1780199182,13cyclictest21642-21if_enp2s007:15:183
1780199181,7cyclictest0-21swapper/310:10:373
17801991814,3cyclictest0-21swapper/308:18:393
17801991814,3cyclictest0-21swapper/308:18:393
17801991813,3cyclictest10505-21users09:15:273
17801991812,3cyclictest4030-21sort11:30:153
17801991812,3cyclictest30830-21fschecks_time08:50:163
17801991812,3cyclictest25914-21fschecks_count12:20:153
17801991812,3cyclictest22300-21pmu-power12:10:223
17801991812,3cyclictest21124-21processes09:40:213
1780199181,15cyclictest0-21swapper/307:52:363
17801991811,4cyclictest19400-21ntp_states10:50:223
1780199181,10cyclictest0-21swapper/309:45:313
1780199180,8cyclictest0-21swapper/310:35:113
1780199180,8cyclictest0-21swapper/310:35:113
1779699185,3cyclictest87650irq/129-enp1s0-08:57:452
1779699185,3cyclictest0-21swapper/208:18:512
1779699185,3cyclictest0-21swapper/208:18:512
1779699184,3cyclictest13843-21pmu-power08:10:222
1779699183,8cyclictest8118-21date08:00:012
1779699182,8cyclictest17087-21/usr/sbin/munin08:20:132
1779699181,7cyclictest0-21swapper/211:40:242
1779699181,5cyclictest23836-21if_enp2s007:20:162
1779699181,5cyclictest2049-21diskstats07:45:142
1779699181,4cyclictest0-21swapper/210:25:122
17796991813,3cyclictest28188-21ntp_offset11:10:202
1779699181,2cyclictest87650irq/129-enp1s0-11:35:292
1779699181,10cyclictest0-21swapper/210:00:152
1779099181,3cyclictest0-21swapper/108:55:111
1778499184,7cyclictest3305-21kworker/0:311:35:590
1778499184,3cyclictest87550irq/128-enp1s0-08:20:390
1778499181,9cyclictest0-21swapper/010:39:410
1778499181,9cyclictest0-21swapper/010:39:400
1778499181,9cyclictest0-21swapper/009:55:370
1778499181,2cyclictest87650irq/129-enp1s0-11:21:230
1778499181,11cyclictest0-21swapper/010:46:100
1778499181,11cyclictest0-21swapper/008:19:150
1778499181,11cyclictest0-21swapper/008:19:150
1778499180,5cyclictest0-21swapper/010:27:360
1780199177,2cyclictest8016-21cpuspeed_turbos11:40:123
1780199172,5cyclictest29658-21df11:15:113
1780199171,7cyclictest0-21swapper/310:05:103
1780199171,5cyclictest6095-21df_inode11:35:123
17801991713,3cyclictest0-21swapper/308:55:273
1780199171,13cyclictest401ktimersoftd/308:05:143
17801991710,4cyclictest27123-21fgrep08:40:233
1779699173,3cyclictest2951-21egrep10:15:102
1779699172,13cyclictest975-21grep07:40:252
1779699171,4cyclictest12457-21latency_hist08:10:012
1779699171,13cyclictest26494-21grep08:40:162
1778499174,3cyclictest8000-21kworker/0:208:07:240
1778499171,5cyclictest0-21swapper/010:10:340
17784991711,3cyclictest32472-21/usr/sbin/munin08:55:250
17784991711,3cyclictest28448-21uniq08:45:150
1780199161,6cyclictest0-21swapper/311:25:203
17801991611,3cyclictest24319-21sendmail_mailst07:20:223
17801991611,3cyclictest20655-21sed08:25:263
1779699161,6cyclictest0-21swapper/211:05:112
1779699161,4cyclictest20880-21/usr/sbin/munin07:15:132
1779699161,14cyclictest0-21swapper/208:01:432
1778499167,7cyclictest29828-21cpuspeed_turbos07:35:130
17784991612,3cyclictest0-21swapper/009:03:460
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional