You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-11 - 13:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Fri Sep 11, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
135592240165,23sleep10-21swapper/119:07:551
137002236162,24sleep20-21swapper/219:09:452
1332322120,10sleep339-21rcuc/319:05:383
124432200162,26sleep00-21swapper/019:05:200
13810998582,2cyclictest0-21swapper/221:46:312
13810997971,7cyclictest0-21swapper/220:23:532
13810997971,7cyclictest0-21swapper/220:23:532
15442540,4sleep230-21rcuc/222:20:142
258872530,3sleep230-21rcuc/219:35:122
149592530,3sleep214961-21missed_timers19:10:202
80562490,4sleep232-21ksoftirqd/220:05:262
1379499499,14cyclictest10353-21munin-run23:55:010
1379499482,13cyclictest23282-21idleruntime-cro19:30:010
1379499390,12cyclictest14782-21idleruntime-cro00:05:010
1379499380,12cyclictest19187-21perf00:15:010
13802993312,8cyclictest0-21swapper/121:39:161
25762320,4sleep081ktimersoftd/022:20:250
13810993229,2cyclictest0-21swapper/223:02:212
1379499312,13cyclictest19490-21munin-run20:35:010
1379499307,11cyclictest20747-21perf23:05:010
13814992918,2cyclictest827-21runrttasks00:29:263
316212280,16sleep10-21swapper/123:28:561
1381499281,18cyclictest0-21swapper/322:23:153
1380299281,19cyclictest4082-21latency_hist20:00:011
1379499280,13cyclictest10709-21perf20:15:010
1381499274,17cyclictest41-21ksoftirqd/321:48:103
1381499271,18cyclictest0-21swapper/322:49:503
1380299271,7cyclictest6911-21grep22:30:241
13802992712,13cyclictest12405-21kworker/u8:000:30:071
1380299271,17cyclictest10500-21fschecks_time22:40:151
1380299271,17cyclictest10500-21fschecks_time22:40:151
1380299271,17cyclictest0-21swapper/123:36:151
1380299271,16cyclictest0-21swapper/122:55:231
1379499274,11cyclictest1905-21perf19:55:000
13794992713,8cyclictest32375-21ntp_kernel_pll_22:15:190
1381499269,2cyclictest0-21swapper/323:02:223
1381499268,10cyclictest0-21swapper/321:31:513
1381499261,18cyclictest0-21swapper/322:02:453
13802992614,3cyclictest0-21swapper/120:12:461
1380299261,3cyclictest1897-21perf19:55:001
1380299261,17cyclictest0-21swapper/122:06:211
1380299261,11cyclictest20107-21munin-run21:50:011
1379499269,3cyclictest4995-21kworker/0:222:30:250
1379499261,13cyclictest4090-21munin-run20:00:010
13794992610,9cyclictest7301-21kworker/0:022:53:490
1381499252,15cyclictest0-21swapper/323:35:483
13814992514,3cyclictest0-21swapper/323:48:293
13814992514,3cyclictest0-21swapper/323:48:293
13814992514,3cyclictest0-21swapper/300:20:303
1381499251,18cyclictest0-21swapper/322:40:303
1381499251,18cyclictest0-21swapper/322:40:303
1381499251,18cyclictest0-21swapper/321:22:503
1381499251,17cyclictest0-21swapper/323:34:293
1381499251,16cyclictest0-21swapper/300:37:353
1381499250,16cyclictest0-21swapper/323:15:153
1380299251,6cyclictest0-21swapper/100:08:321
1380299251,5cyclictest0-21swapper/100:15:201
1380299251,21cyclictest9501-21expr21:25:121
1380299251,21cyclictest7094-21memory23:45:181
1380299251,21cyclictest7094-21memory23:45:181
1380299251,21cyclictest29261-21ntp_states00:35:231
1380299251,21cyclictest24473-21/usr/sbin/munin19:30:191
1380299251,21cyclictest24223-21ntp_states23:10:211
1380299251,16cyclictest0-21swapper/120:51:421
1380299251,15cyclictest0-21swapper/100:25:041
1380299251,12cyclictest18075-21fschecks_time20:30:151
1380299251,11cyclictest4798-21cat21:15:011
1380299251,11cyclictest30098-21cpu19:45:111
1380299251,11cyclictest12545-21munin-run00:00:011
1379499251,12cyclictest28293-21perf20:55:010
1381499241,17cyclictest0-21swapper/323:20:303
1381499241,17cyclictest0-21swapper/322:33:573
1381499241,16cyclictest0-21swapper/300:31:343
1380299247,10cyclictest0-21swapper/122:37:171
13802992417,3cyclictest14603-21grep19:10:161
1380299241,14cyclictest0-21swapper/123:54:171
1380299241,14cyclictest0-21swapper/123:17:091
1380299241,14cyclictest0-21swapper/120:19:401
1380299241,12cyclictest20777-21ntp_states20:35:211
1380299241,11cyclictest901-21latency_hist22:20:011
1380299241,11cyclictest4327-21/usr/sbin/munin23:40:131
1380299241,11cyclictest23472-21/usr/sbin/munin21:55:181
1380299241,11cyclictest16127-21irqstats20:25:181
1380299241,11cyclictest1206-21grep22:20:121
1380299241,11cyclictest0-21swapper/123:32:541
1380299240,14cyclictest0-21swapper/120:20:081
1380299240,14cyclictest0-21swapper/120:20:081
1379499246,15cyclictest21086-21awk19:25:010
1379499242,9cyclictest18385-21kworker/0:223:32:010
1379499242,10cyclictest31753-21kworker/0:019:49:010
1381499231,16cyclictest0-21swapper/321:40:023
1381499231,13cyclictest0-21swapper/300:09:083
1381499231,10cyclictest0-21swapper/323:25:253
1381099239,3cyclictest0-21swapper/200:27:152
13810992316,4cyclictest27956-21sh00:35:012
13810992312,3cyclictest87350irq/126-enp1s0-21:35:132
1380299239,3cyclictest6913-21latency_hist21:20:011
13802992313,2cyclictest0-21swapper/121:02:401
13802992312,3cyclictest0-21swapper/121:41:561
1380299231,15cyclictest0-21swapper/121:09:321
1380299231,15cyclictest0-21swapper/100:22:161
1380299231,14cyclictest0-21swapper/120:44:441
1380299231,14cyclictest0-21swapper/119:38:211
1380299231,14cyclictest0-21swapper/100:01:361
1380299231,11cyclictest24916-21ls22:00:131
1380299231,10cyclictest29691-21cat23:25:011
1380299231,10cyclictest22470-21unixbench_multi23:05:271
1380299231,10cyclictest0-21swapper/119:17:411
1380299230,15cyclictest0-21swapper/121:34:401
1380299230,14cyclictest0-21swapper/120:46:541
1379499239,3cyclictest15731-21kworker/0:219:14:380
1379499238,3cyclictest8358-21kworker/0:120:15:130
1379499236,2cyclictest27875-21kworker/0:000:39:050
1379499232,10cyclictest11214-21kworker/0:121:38:330
52862220,4sleep3401ktimersoftd/321:15:143
222552221,2sleep20-21swapper/220:40:152
1381499228,8cyclictest0-21swapper/322:59:293
1381499227,9cyclictest0-21swapper/300:10:413
1381499226,4cyclictest15111-21latency_hist20:25:013
1381499226,4cyclictest15111-21latency_hist20:25:013
1381499221,16cyclictest0-21swapper/322:29:223
1381499221,15cyclictest0-21swapper/322:39:413
13814992210,10cyclictest87350irq/126-enp1s0-21:35:143
1381499220,5cyclictest0-21swapper/323:52:543
1381499220,16cyclictest0-21swapper/321:51:233
1381499220,10cyclictest0-21swapper/322:52:243
13810992216,3cyclictest17904-21dump-pmu-power21:45:012
1380299229,11cyclictest6896-21kworker/u8:019:29:481
1380299222,11cyclictest763-21dbus-daemon21:50:171
1380299221,9cyclictest6578-21cpu20:05:131
13802992212,2cyclictest0-21swapper/123:01:371
13802992212,2cyclictest0-21swapper/122:12:371
13802992212,2cyclictest0-21swapper/119:23:001
1380299221,11cyclictest3383-21cpu22:25:131
1380299221,11cyclictest0-21swapper/120:00:251
1380299221,10cyclictest28509-21acpi20:55:121
1380299221,10cyclictest18849-21chrt00:11:021
1380299221,10cyclictest12461-21expr22:45:141
1379499229,2cyclictest11214-21kworker/0:121:41:050
1379499229,2cyclictest10102-21kworker/0:123:55:210
1379499228,2cyclictest2218-21kworker/0:221:14:410
1379499227,4cyclictest18385-21kworker/0:223:25:090
1379499227,2cyclictest19286-21kworker/0:221:50:290
1379499226,14cyclictest0-21swapper/022:05:010
1379499222,9cyclictest25950-21kworker/0:021:00:570
1379499222,9cyclictest19012-21kworker/0:300:16:250
1379499221,9cyclictest21014-21kworker/0:019:32:370
1379499221,10cyclictest3535-21kworker/0:023:41:250
1379499221,10cyclictest21989-21kworker/0:022:07:490
1381499219,2cyclictest20581-21unixbench-2d19:20:243
1381499218,4cyclictest0-21swapper/321:10:113
1381499218,3cyclictest0-21swapper/322:05:233
1381499217,8cyclictest0-21swapper/322:15:173
1381499211,7cyclictest12476-21head20:15:273
13814992117,3cyclictest0-21swapper/320:10:153
1381499211,5cyclictest13461-21smartctl00:00:173
1381499211,4cyclictest0-21swapper/323:08:323
1381499211,18cyclictest0-21swapper/319:30:123
1381499211,15cyclictest0-21swapper/321:59:343
1381499211,12cyclictest0-21swapper/323:10:183
1381499211,11cyclictest19744-21diskstats00:15:133
1381499210,11cyclictest2532-21basename21:10:013
1381099218,3cyclictest87350irq/126-enp1s0-21:20:212
1381099216,3cyclictest96402missed_timers20:10:192
1381099215,3cyclictest0-21swapper/220:37:512
1381099212,15cyclictest30473-21sa121:00:012
1381099211,6cyclictest0-21swapper/222:01:112
13810992111,2cyclictest87350irq/126-enp1s0-22:50:192
1381099210,5cyclictest32-21ksoftirqd/223:54:572
1380299211,14cyclictest0-21swapper/119:42:051
1380299210,13cyclictest0-21swapper/122:50:201
1379499218,5cyclictest0-21swapper/021:15:280
1379499218,2cyclictest31753-21kworker/0:020:06:220
1379499217,2cyclictest7301-21kworker/0:022:55:090
1379499217,2cyclictest21273-21kworker/0:000:21:010
1379499216,3cyclictest21989-21kworker/0:021:55:490
1379499211,9cyclictest18385-21kworker/0:223:06:490
1379499211,10cyclictest7890-21kworker/0:200:06:210
1379499211,10cyclictest15731-21kworker/0:219:16:570
181282200,4sleep339-21rcuc/320:30:153
1381499209,3cyclictest6525-21/usr/sbin/munin20:05:153
1381499201,7cyclictest41-21ksoftirqd/319:35:223
1381499201,6cyclictest0-21swapper/323:42:123
13814992015,3cyclictest0-21swapper/319:50:123
13814992013,4cyclictest6300-21munin-run20:05:013
1381499201,14cyclictest87350irq/126-enp1s0-21:25:183
1381499201,14cyclictest0-21swapper/319:13:473
1381499200,15cyclictest0-21swapper/322:12:463
1381099207,5cyclictest808-21snmpd21:25:262
1381099204,8cyclictest311ktimersoftd/221:50:272
1381099201,5cyclictest0-21swapper/222:05:122
1379499209,3cyclictest29129-21/usr/sbin/munin22:10:150
1379499207,2cyclictest7890-21kworker/0:223:47:290
1379499207,2cyclictest7890-21kworker/0:223:47:290
1379499201,9cyclictest19012-21kworker/0:300:27:290
1379499201,11cyclictest25347-21/usr/sbin/munin23:15:140
13794992010,3cyclictest20115-21awk21:50:010
241952190,3sleep324199-21cpuspeed_turbos20:45:123
1381499197,3cyclictest0-21swapper/320:55:123
1381499193,13cyclictest26304-21/usr/sbin/munin20:50:153
1381499191,9cyclictest31342-21smart_sda19:45:233
1381499191,7cyclictest15342-21cpu20:25:123
1381499191,6cyclictest827-21runrttasks20:44:483
13814991915,2cyclictest0-21swapper/319:55:133
1381499191,3cyclictest0-21swapper/319:17:413
1381499191,10cyclictest808-21snmpd19:40:053
1381499191,10cyclictest39-21rcuc/319:25:193
1381099197,3cyclictest87550irq/128-enp1s0-21:31:572
1381099193,4cyclictest30-21rcuc/223:15:122
1381099192,8cyclictest11927-21munin-run22:45:012
1381099192,8cyclictest11927-21munin-run22:45:012
1381099191,7cyclictest12900-21idleruntime-cro20:20:012
13810991913,5cyclictest6896-21kworker/u8:021:07:152
13810991912,4cyclictest3876-21ls21:10:212
1381099191,11cyclictest0-21swapper/223:08:272
1380299191,4cyclictest0-21swapper/121:22:061
1379499197,2cyclictest3564-21kworker/0:123:38:290
1379499197,2cyclictest13818-21kworker/0:222:46:570
1379499196,2cyclictest31079-21kworker/0:122:25:450
1379499195,3cyclictest7301-21kworker/0:022:36:010
1379499195,2cyclictest11214-21kworker/0:121:34:010
13794991914,3cyclictest28165-21grep23:20:150
266102180,4sleep07-21ksoftirqd/000:30:150
149012181,3sleep00-21swapper/020:22:430
149012181,3sleep00-21swapper/020:22:430
1381499189,2cyclictest0-21swapper/323:55:143
1381499188,1cyclictest0-21swapper/320:35:203
1381499181,9cyclictest31756-21ntpq21:00:193
1381099188,4cyclictest311ktimersoftd/222:30:232
1381099188,3cyclictest0-21swapper/223:35:182
1381099187,3cyclictest0-21swapper/219:50:232
1381099183,4cyclictest30-21rcuc/222:40:012
1381099183,3cyclictest0-21swapper/223:20:292
1381099181,7cyclictest26917-21grep20:50:172
1381099181,7cyclictest0-21swapper/200:37:232
1381099181,6cyclictest0-21swapper/223:32:562
1381099181,5cyclictest0-21swapper/222:29:182
13810991812,3cyclictest5105-21cpuspeed_turbos21:15:142
13810991811,4cyclictest28628-21if_enp2s019:40:172
1381099180,10cyclictest0-21swapper/222:58:412
1379499186,4cyclictest7146-21/usr/sbin/munin21:20:270
1379499185,2cyclictest9691-21kworker/0:122:41:100
1379499185,2cyclictest9691-21kworker/0:122:41:090
1379499185,2cyclictest25950-21kworker/0:020:55:530
1379499184,2cyclictest26755-21kworker/0:119:36:090
1379499182,7cyclictest4317-21/usr/sbin/munin20:00:180
1379499181,4cyclictest1-21systemd23:10:170
13794991813,3cyclictest23893-21perf20:45:010
13794991811,4cyclictest16024-21sort20:25:160
1381099177,3cyclictest12050irq/123-ahci[0000:25:022
1381099175,3cyclictest87350irq/126-enp1s0-19:30:142
1381099175,2cyclictest0-21swapper/220:30:232
1381099174,3cyclictest30-21rcuc/223:15:012
1381099173,7cyclictest0-21swapper/223:48:322
1381099173,7cyclictest0-21swapper/223:48:312
1381099173,7cyclictest0-21swapper/200:13:282
1381099173,6cyclictest12779-21/usr/sbin/munin00:00:272
1381099173,2cyclictest87350irq/126-enp1s0-20:45:172
1381099173,2cyclictest0-21swapper/219:45:252
1381099171,6cyclictest0-21swapper/219:15:072
1381099171,6cyclictest0-21swapper/200:15:222
1381099171,5cyclictest0-21swapper/223:40:132
1381099171,5cyclictest0-21swapper/221:58:372
1381099171,3cyclictest0-21swapper/223:28:062
13810991712,3cyclictest10575-21/usr/sbin/munin23:55:182
13810991711,4cyclictest21898-21grep19:25:142
1381099171,10cyclictest0-21swapper/222:13:402
13810991710,4cyclictest0-21swapper/222:45:202
1379499172,12cyclictest9005-21kworker/0:221:25:210
1379499171,7cyclictest24105-21/usr/sbin/munin20:45:230
1379499171,10cyclictest0-21swapper/019:40:170
1381099166,7cyclictest4444-21cpuspeed_turbos20:00:122
1381099161,9cyclictest0-21swapper/219:20:162
1381099161,8cyclictest0-21swapper/220:25:122
1381099161,7cyclictest87650irq/129-enp1s0-21:00:162
1381099161,6cyclictest32215-40dnf22:15:172
1381099161,6cyclictest2116-21/usr/sbin/munin19:55:242
1379499165,3cyclictest808-21snmpd21:05:220
1379499163,4cyclictest20366-21gltestperf20:35:160
1381099155,7cyclictest15647-21gltestperf00:05:162
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional