You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-28 - 06:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu May 28, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
254162237162,24sleep20-21swapper/219:08:462
251792236163,24sleep10-21swapper/119:05:431
251832225194,20sleep30-21swapper/319:05:453
252932202164,26sleep00-21swapper/019:07:110
1491321790,4sleep12560499cyclictest22:25:011
2808621200,1sleep10-21swapper/120:25:161
301132510,3sleep30-21swapper/322:57:373
237892490,4sleep339-21rcuc/322:45:013
237892490,4sleep339-21rcuc/322:45:013
294292440,1sleep00-21swapper/019:15:170
2559799377,14cyclictest8959-21awk23:25:010
2559799369,13cyclictest1100-21munin-run20:40:010
25610993531,2cyclictest311ktimersoftd/219:20:182
2560499326,14cyclictest22855-21kworker/u8:121:50:531
2560499299,9cyclictest25209-21kworker/u8:019:18:531
236492290,2sleep011-21rcuc/022:44:220
236492290,2sleep011-21rcuc/022:44:220
185662291,4sleep10-21swapper/121:18:241
2560499281,17cyclictest0-21swapper/121:45:181
25614992715,3cyclictest10752-21/usr/sbin/munin22:15:153
2561499271,12cyclictest0-21swapper/321:02:133
2560499270,20cyclictest0-21swapper/121:58:221
2560499270,18cyclictest0-21swapper/123:56:281
25597992711,13cyclictest26551-21munin-run00:05:010
327262260,1sleep30-21swapper/320:35:233
2560499269,5cyclictest0-21swapper/123:26:021
2559799263,12cyclictest16392-21sendmail_mailst22:25:230
2559799262,18cyclictest998-21/usr/sbin/munin00:20:240
2559799261,12cyclictest31895-21munin-run21:50:010
2561499257,2cyclictest1318-21/usr/sbin/munin20:40:233
2561499251,11cyclictest0-21swapper/323:45:233
2561099250,12cyclictest0-21swapper/200:17:372
2560499251,5cyclictest31859-21hddtemp_smartct00:15:161
25604992513,6cyclictest2348-21switchtime00:20:241
2560499251,18cyclictest0-21swapper/123:09:381
2560499251,16cyclictest0-21swapper/122:45:361
2560499251,16cyclictest0-21swapper/121:35:561
2560499251,11cyclictest16674-21memory23:40:191
2560499250,5cyclictest0-21swapper/121:27:221
2560499250,17cyclictest0-21swapper/123:22:381
2559799250,11cyclictest7661-21kworker/0:021:00:250
2561499246,8cyclictest30367-40dnf19:16:233
2561499246,2cyclictest21704-21perl21:25:133
2560499241,4cyclictest6145-21awk22:05:011
2560499241,16cyclictest0-21swapper/122:33:511
2560499240,12cyclictest0-21swapper/122:10:131
2561499239,5cyclictest87650irq/129-enp1s0-21:35:163
2561499231,5cyclictest0-21swapper/300:07:093
2561499231,14cyclictest0-21swapper/323:25:003
2561499231,13cyclictest6132-21sh22:05:013
2561499231,12cyclictest0-21swapper/322:49:173
2560499239,3cyclictest0-21swapper/100:30:391
2560499234,13cyclictest87650irq/129-enp1s0-20:03:221
2560499231,3cyclictest15692-21latency_hist23:40:011
2560499231,19cyclictest7988-21hddtemp_smartct19:40:151
2560499231,19cyclictest5512-21fschecks_count19:35:151
2560499230,16cyclictest0-21swapper/122:15:531
2559799236,3cyclictest28416-21kworker/0:100:07:010
2559799232,12cyclictest0-21swapper/023:40:190
2559799230,11cyclictest6763-21idleruntime-cro23:20:020
2561499229,5cyclictest87350irq/126-enp1s0-21:20:203
2561499226,8cyclictest18588-21sendmail_mailst22:30:223
2561499225,8cyclictest12050irq/123-ahci[0020:55:023
2561499224,7cyclictest15120-21cat22:25:013
2561499224,6cyclictest20853-21diskstats20:10:143
2561499223,3cyclictest13912-21diskstats23:35:143
2561499221,6cyclictest0-21swapper/322:50:503
2561499221,13cyclictest0-21swapper/321:10:183
2561099220,10cyclictest0-21swapper/221:53:332
2560499222,5cyclictest26883-21munin-run20:25:011
2560499222,5cyclictest26883-21munin-run20:25:001
25604992216,3cyclictest23656-21df21:30:121
25604992216,3cyclictest23656-21df21:30:121
2560499221,3cyclictest10073-21sed21:00:021
2560499221,16cyclictest0-21swapper/100:11:421
2560499221,16cyclictest0-21swapper/100:01:031
2560499221,14cyclictest0-21swapper/123:10:591
2560499221,12cyclictest0-21swapper/120:30:081
2560499220,3cyclictest0-21swapper/123:30:101
2560499220,16cyclictest0-21swapper/122:56:091
2560499220,10cyclictest0-21swapper/121:24:191
2559799224,5cyclictest18563-40dnf21:18:240
2559799223,10cyclictest0-21swapper/023:25:220
2559799222,13cyclictest0-21swapper/022:16:560
2559799221,9cyclictest30061-21runrttasks22:56:580
2559799221,12cyclictest2074-21tr21:55:120
2561499217,7cyclictest6048-21sed00:30:153
2561499216,7cyclictest2582-21/usr/sbin/munin23:10:143
2561499214,7cyclictest24537-21cat00:00:023
2561499213,9cyclictest0-21swapper/323:00:123
2561499212,6cyclictest41-21ksoftirqd/321:45:063
2561499212,16cyclictest29099-21grep00:10:113
2561499212,10cyclictest11368-21/usr/sbin/munin23:30:183
2561499211,9cyclictest0-21swapper/323:28:473
2561499211,5cyclictest0-21swapper/300:16:593
2561499211,11cyclictest2178-40dnf00:20:223
2561499210,8cyclictest22828-21grep20:15:113
25610992118,2cyclictest0-21swapper/223:22:582
2561099211,2cyclictest6706-21kworker/u8:122:55:132
2561099210,9cyclictest0-21swapper/223:38:072
2561099210,8cyclictest0-21swapper/222:28:312
2560499218,3cyclictest4606-21ntp_states20:45:201
2560499216,5cyclictest0-21swapper/123:04:401
2560499212,10cyclictest16392-21sendmail_mailst22:25:231
2560499211,3cyclictest1930-21gltestperf20:40:151
2560499211,3cyclictest14715-21cstates21:10:121
2560499211,18cyclictest0-21swapper/119:50:061
2560499211,17cyclictest808-21snmpd20:11:141
2560499211,11cyclictest7306-21grep20:50:261
2560499211,11cyclictest18431-21sort23:45:151
2560499211,11cyclictest0-21swapper/119:20:241
2560499210,15cyclictest0-21swapper/123:16:091
2559799216,6cyclictest0-21swapper/023:04:170
25597992116,3cyclictest29707-21awk21:45:010
25597992116,3cyclictest11487-21munin-run19:50:010
2561499206,7cyclictest87350irq/126-enp1s0-19:10:213
2561499206,2cyclictest0-21swapper/323:40:243
2561499205,8cyclictest87350irq/126-enp1s0-20:45:213
2561499205,6cyclictest6266-21unixbench_multi23:15:253
2561499203,6cyclictest87350irq/126-enp1s0-21:16:013
2561499201,9cyclictest0-21swapper/322:29:543
2561499201,7cyclictest5187-21latency_hist00:30:023
2561499201,7cyclictest20354-21irqstats22:35:173
2561499201,7cyclictest0-21swapper/320:05:183
2561499201,10cyclictest87350irq/126-enp1s0-19:45:153
2561499201,10cyclictest0-21swapper/300:36:033
2561499201,10cyclictest0-21swapper/300:00:183
2561099206,6cyclictest0-21swapper/200:14:562
2561099203,11cyclictest2554-21awk00:20:252
2561099202,5cyclictest18245-1kworker/u9:222:48:192
2561099201,6cyclictest0-21swapper/223:15:262
2561099201,6cyclictest0-21swapper/219:10:242
2561099201,5cyclictest0-21swapper/222:18:072
2561099201,4cyclictest21992-21cstates22:40:132
2561099201,4cyclictest21992-21cstates22:40:132
2561099201,3cyclictest3737-21taskset21:57:412
25610992012,2cyclictest0-21swapper/223:41:332
2561099201,12cyclictest0-21swapper/223:08:192
2560499209,3cyclictest3967-21ntpq19:30:211
25604992017,2cyclictest0-21swapper/119:10:141
25604992016,2cyclictest21145-21strings23:50:191
25604992015,2cyclictest20222-21smartctl22:35:161
2560499201,4cyclictest8526-21memory00:35:191
2560499201,17cyclictest0-21swapper/120:35:331
2560499201,17cyclictest0-21swapper/120:06:521
2560499201,14cyclictest0-21swapper/100:09:261
2560499201,10cyclictest22623-21latency22:40:181
2560499201,10cyclictest22623-21latency22:40:181
2560499201,10cyclictest0-21swapper/121:00:171
2559799209,3cyclictest25071-21pmu-power22:45:220
2559799204,8cyclictest3220-21/usr/sbin/munin00:25:130
2559799202,15cyclictest30224-21find00:10:230
2559799201,4cyclictest5806-21grep00:30:140
2559799201,4cyclictest13757-21awk22:20:170
2559799201,12cyclictest19543-21/usr/sbin/munin22:35:250
2561499198,4cyclictest401ktimersoftd/323:50:113
2561499195,6cyclictest808-21snmpd22:05:243
2561499195,3cyclictest27722-21/usr/sbin/munin21:40:153
2561499193,5cyclictest32744-21/usr/sbin/munin23:05:253
25614991913,3cyclictest12468-21cpuspeed_turbos21:05:123
25614991912,4cyclictest8694-21grep22:10:113
2561499191,11cyclictest839-21lldpd19:37:453
2561499191,10cyclictest24900-21/usr/sbin/munin20:20:113
2561499191,10cyclictest24900-21/usr/sbin/munin20:20:113
2561099194,3cyclictest87650irq/129-enp1s0-23:50:242
2561099192,4cyclictest27211-21ntp_states22:50:212
2561099192,14cyclictest21710-21grep21:25:142
2561099191,7cyclictest0-21swapper/222:14:072
2561099191,6cyclictest19493-21unixbench_singl23:45:252
2561099191,5cyclictest10476-21diskstats21:00:132
2561099191,10cyclictest15872-21dump-pmu-power20:04:562
2561099190,9cyclictest0-21swapper/222:24:312
2560499196,5cyclictest0-21swapper/100:28:561
2560499192,12cyclictest87650irq/129-enp1s0-21:40:181
2560499191,6cyclictest0-21swapper/122:07:031
2560499191,16cyclictest0-21swapper/121:06:001
2560499191,16cyclictest0-21swapper/120:16:281
2560499191,16cyclictest0-21swapper/119:55:441
25604991910,3cyclictest0-21swapper/119:48:181
2560499190,13cyclictest87650irq/129-enp1s0-22:50:341
2559799197,3cyclictest20491-21df23:50:120
2559799197,3cyclictest0-21swapper/022:52:520
2559799197,3cyclictest0-21swapper/000:17:370
2559799196,2cyclictest87350irq/126-enp1s0-21:10:010
2559799195,5cyclictest755-21rngd21:35:340
2559799195,4cyclictest18928-21/usr/sbin/munin21:20:180
2559799194,8cyclictest2928-21/usr/sbin/munin19:30:120
2559799193,6cyclictest755-21rngd23:55:220
2559799192,14cyclictest18614-21head23:45:160
2559799191,8cyclictest0-21swapper/021:30:130
2559799191,8cyclictest0-21swapper/021:30:130
2559799191,10cyclictest0-21swapper/023:35:250
2559799190,8cyclictest0-21swapper/021:50:200
2559799190,13cyclictest20596-21kworker/u8:221:26:100
102762191,2sleep00-21swapper/022:11:550
2561499188,6cyclictest14253-21diskstats19:55:133
2561499182,4cyclictest0-21swapper/319:50:243
2561499182,3cyclictest401ktimersoftd/320:30:173
2561499181,7cyclictest0-21swapper/321:55:133
2561499181,11cyclictest0-21swapper/320:55:173
2561499181,10cyclictest808-21snmpd19:34:053
2561499181,10cyclictest0-21swapper/320:00:273
2561499180,10cyclictest0-21swapper/319:44:043
2561099188,3cyclictest87650irq/129-enp1s0-21:23:362
2561099188,2cyclictest9589-21awk00:40:012
2561099183,4cyclictest12050irq/123-ahci[0021:00:002
2561099182,6cyclictest20279-21ls20:10:012
2561099181,8cyclictest7593-21expr19:40:122
2561099181,7cyclictest28368-21users00:05:272
2561099181,7cyclictest0-21swapper/222:03:022
2561099181,5cyclictest24082-21kworker/u8:200:03:172
2561099181,3cyclictest3650-21ntp_states23:10:212
2561099181,2cyclictest0-21swapper/200:31:292
2561099181,2cyclictest0-21swapper/200:26:462
2561099181,15cyclictest0-21swapper/221:41:462
2561099181,15cyclictest0-21swapper/220:18:472
2560499181,16cyclictest0-21swapper/119:27:121
2559799188,4cyclictest0-21swapper/020:40:170
2559799186,4cyclictest0-21swapper/022:05:240
2559799182,3cyclictest0-21swapper/021:11:170
2559799181,4cyclictest0-21swapper/022:04:440
2559799181,4cyclictest0-21swapper/019:10:130
25597991811,4cyclictest5463-21df_inode19:35:140
2559799181,14cyclictest17612-21unixbench_multi20:00:260
2559799181,10cyclictest30295-21kworker/u8:223:10:110
2559799180,5cyclictest0-21swapper/023:05:120
2559799180,5cyclictest0-21swapper/022:31:200
2559799180,3cyclictest0-21swapper/023:32:160
2561499171,9cyclictest0-21swapper/321:30:163
2561499171,9cyclictest0-21swapper/321:30:153
2561499170,4cyclictest0-21swapper/321:50:173
2561099176,3cyclictest87350irq/126-enp1s0-22:30:212
2561099172,12cyclictest9652-21cpuspeed_turbos19:45:122
2561099171,9cyclictest0-21swapper/220:30:182
2561099171,8cyclictest87350irq/126-enp1s0-21:15:212
2561099171,7cyclictest7199-21cut22:05:192
2561099171,7cyclictest32521-21/usr/sbin/munin23:05:012
2561099171,7cyclictest0-21swapper/223:29:492
2561099171,7cyclictest0-21swapper/219:16:232
25610991713,3cyclictest0-21swapper/221:35:422
25610991712,3cyclictest24099-21if_err_enp1s021:30:162
25610991712,3cyclictest24099-21if_err_enp1s021:30:162
25610991712,3cyclictest14533-21/usr/sbin/munin21:10:172
25610991712,3cyclictest12677-21df_inode21:05:132
2561099171,15cyclictest0-21swapper/220:43:562
25610991711,3cyclictest31177-21sensors_temp21:45:232
25610991710,4cyclictest20847-21sh20:10:142
2559799172,12cyclictest20506-21/usr/sbin/munin20:10:120
2559799171,8cyclictest0-21swapper/019:21:040
2559799171,5cyclictest8287-21hddtemp_smartct00:35:160
25597991712,3cyclictest9072-21seq19:42:220
2559799171,13cyclictest7325-21users20:50:260
2559799171,13cyclictest4298-21ethtool20:45:160
25597991711,3cyclictest27105-21/usr/sbin/munin20:25:140
2559799171,10cyclictest0-21swapper/019:56:450
206122171,3sleep20-21swapper/222:35:222
2561499163,6cyclictest635-21/usr/sbin/munin19:25:143
2561499162,3cyclictest39-21rcuc/320:25:113
2561499161,9cyclictest0-21swapper/319:20:243
2561099161,7cyclictest0-21swapper/219:50:152
2561099161,6cyclictest25026-21cpuspeed_turbos20:20:112
2561099161,6cyclictest25026-21cpuspeed_turbos20:20:112
2561099161,6cyclictest0-21swapper/223:57:382
2561099161,2cyclictest0-21swapper/223:30:182
2561099161,14cyclictest0-21swapper/220:51:452
2561099161,14cyclictest0-21swapper/220:45:422
2561099161,14cyclictest0-21swapper/219:57:562
25610991611,3cyclictest635-21/usr/sbin/munin19:25:142
25610991611,3cyclictest6028-21load19:35:192
25610991611,3cyclictest3228-21expr19:30:132
25610991611,3cyclictest27443-21sh20:25:112
2559799163,11cyclictest11-21rcuc/020:15:150
2559799161,8cyclictest0-21swapper/019:50:160
2559799161,5cyclictest0-21swapper/020:30:220
2559799161,3cyclictest0-21swapper/020:20:520
2559799161,3cyclictest0-21swapper/020:20:520
25597991612,3cyclictest0-21swapper/020:07:320
25597991611,3cyclictest8617-21hwlatdetect20:55:160
25597991611,3cyclictest635-21/usr/sbin/munin19:25:170
2561099154,3cyclictest87650irq/129-enp1s0-20:35:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional