You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-28 - 02:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Mon Sep 28, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2521322680,3sleep325219-21needreboot19:05:203
259452237163,23sleep20-21swapper/219:07:022
260312203165,25sleep10-21swapper/119:08:091
260102200162,26sleep00-21swapper/019:07:520
2371121720,3sleep12626199cyclictest23:55:151
584921590,5sleep32627499cyclictest19:35:113
2626799103100,2cyclictest0-21swapper/223:03:302
26267997369,3cyclictest0-21swapper/222:27:182
167202730,5sleep32627499cyclictest21:10:283
168732670,1sleep00-21swapper/022:25:200
36362620,1sleep20-21swapper/221:55:202
229982390,5sleep323002-21gltestperf22:40:163
2625799358,13cyclictest27555-21munin-run20:25:010
2626199348,16cyclictest0-21swapper/122:15:141
2626199342,15cyclictest1078-40dnf19:24:221
2625799338,12cyclictest12783-21munin-run21:05:010
2626199321,22cyclictest0-21swapper/121:19:371
2625799327,12cyclictest11200-21munin-run22:15:010
2627499311,22cyclictest0-21swapper/322:51:313
2626199302,17cyclictest28582-21munin-node21:35:271
2626199301,23cyclictest0-21swapper/123:22:161
2626199301,23cyclictest0-21swapper/123:22:161
2626199301,18cyclictest23723-21diskstats20:15:131
2627499291,20cyclictest0-21swapper/300:28:433
26261992914,8cyclictest17096-21switchtime22:25:251
2626199291,18cyclictest0-21swapper/122:36:131
2625799291,13cyclictest20003-21awk22:35:010
2627499281,20cyclictest0-21swapper/322:45:423
2626199281,15cyclictest0-21swapper/122:32:551
2627499271,19cyclictest0-21swapper/320:39:173
2626199271,19cyclictest0-21swapper/121:26:261
2625799271,7cyclictest0-21swapper/000:00:220
2627499261,9cyclictest15582-21crond22:25:013
2627499261,17cyclictest0-21swapper/321:32:553
2627499261,12cyclictest0-21swapper/321:40:013
2627499261,12cyclictest0-21swapper/321:16:433
2627499261,11cyclictest7618-21gltestperf22:05:163
2627499261,11cyclictest7618-21gltestperf22:05:163
26261992611,9cyclictest0-21swapper/122:53:401
2626199261,16cyclictest13352-21diskstats21:05:131
275672250,2sleep20-21swapper/219:10:222
2627499252,15cyclictest87350irq/126-enp1s0-23:35:213
26274992521,3cyclictest0-21swapper/320:10:143
2627499251,17cyclictest0-21swapper/321:50:223
2627499251,16cyclictest0-21swapper/323:54:103
2627499251,16cyclictest0-21swapper/300:17:463
2627499251,11cyclictest0-21swapper/321:45:273
2627499250,10cyclictest0-21swapper/319:26:413
2626799251,13cyclictest0-21swapper/222:05:302
2626799251,13cyclictest0-21swapper/222:05:302
2626199258,10cyclictest0-21swapper/121:51:531
2626199252,17cyclictest0-21swapper/123:18:311
2626199252,17cyclictest0-21swapper/122:58:081
2626199252,17cyclictest0-21swapper/100:24:371
2626199252,11cyclictest3364-21munin-run19:30:011
26261992514,2cyclictest0-21swapper/119:50:171
26261992511,9cyclictest0-21swapper/120:41:261
2626199251,17cyclictest0-21swapper/122:20:121
2626199251,17cyclictest0-21swapper/100:02:571
2626199251,15cyclictest0-21swapper/122:07:121
2626199251,15cyclictest0-21swapper/122:07:121
2626199251,12cyclictest0-21swapper/122:48:301
2626199251,11cyclictest28603-21df_inode19:15:021
2626199251,11cyclictest0-21swapper/121:57:551
26261992510,9cyclictest0-21swapper/119:55:391
2626199250,19cyclictest0-21swapper/121:41:291
2627499246,9cyclictest26314-21kworker/u8:122:57:173
2627499245,5cyclictest8688-21idleruntime19:40:173
2627499245,4cyclictest21793-21/usr/sbin/munin21:25:183
2627499241,9cyclictest0-21swapper/323:40:273
2627499241,9cyclictest0-21swapper/321:05:173
2627499241,5cyclictest0-21swapper/323:58:433
2627499241,13cyclictest0-21swapper/322:03:243
2627499241,12cyclictest0-21swapper/300:37:003
2627499241,12cyclictest0-21swapper/300:14:583
2627499241,11cyclictest0-21swapper/300:05:443
2627499241,10cyclictest0-21swapper/322:29:213
2627499241,10cyclictest0-21swapper/300:22:503
2627499240,6cyclictest0-21swapper/322:38:223
2627499240,10cyclictest0-21swapper/323:15:473
2626199249,9cyclictest0-21swapper/123:05:111
2626199242,16cyclictest0-21swapper/122:13:501
2626199242,16cyclictest0-21swapper/119:18:241
26261992415,3cyclictest0-21swapper/120:05:231
2626199241,4cyclictest32021-21sshd23:00:181
2626199241,15cyclictest0-21swapper/122:40:211
2626199241,14cyclictest808-21snmpd21:34:151
2626199241,11cyclictest6300-21unixbench-2d22:00:271
2626199241,11cyclictest30575-21ls00:10:181
2626199241,11cyclictest22733-21sshd23:54:181
26261992410,9cyclictest0-21swapper/100:35:281
26261992410,5cyclictest0-21swapper/123:44:381
2625799242,9cyclictest316-21kworker/0:221:56:350
2625799242,8cyclictest20341-21kworker/0:200:07:470
2625799241,11cyclictest4348-21kworker/0:122:04:190
2627499236,3cyclictest7640-21/usr/sbin/munin23:20:243
2627499236,3cyclictest7640-21/usr/sbin/munin23:20:243
2627499236,3cyclictest27776-21/usr/sbin/munin20:25:203
2627499235,3cyclictest29201-21irqstats21:40:183
26274992319,3cyclictest0-21swapper/319:32:113
2627499231,16cyclictest0-21swapper/320:00:273
2627499231,15cyclictest0-21swapper/323:05:163
2627499231,15cyclictest0-21swapper/321:20:463
2627499231,11cyclictest0-21swapper/323:30:323
2627499231,10cyclictest0-21swapper/323:25:453
2627499231,10cyclictest0-21swapper/321:58:483
2627499231,10cyclictest0-21swapper/300:03:433
26267992313,2cyclictest24930-21grep20:15:252
2626199238,3cyclictest12371-21df23:30:131
2626199232,2cyclictest87550irq/128-enp1s0-00:15:441
2626199232,14cyclictest87550irq/128-enp1s0-23:25:141
2626199231,11cyclictest7539-21pmu-power20:50:241
2626199231,10cyclictest29558-21cat00:10:011
26261992310,8cyclictest0-21swapper/120:57:451
2626199230,21cyclictest0-21swapper/119:30:251
2625799231,7cyclictest0-21swapper/021:50:130
2627499225,3cyclictest87350irq/126-enp1s0-19:20:133
2627499225,3cyclictest29430-21cpuspeed_turbos19:15:113
2627499223,10cyclictest0-21swapper/323:00:223
2627499221,7cyclictest4206-21/usr/sbin/munin20:45:163
2627499221,5cyclictest0-21swapper/323:10:333
2627499221,10cyclictest19579-21sed22:30:273
2627499221,10cyclictest18837-21ls23:45:133
2627499220,16cyclictest0-21swapper/300:33:223
2627499220,10cyclictest0-21swapper/322:12:383
2626799227,4cyclictest23461-21ntp_kernel_pll_22:40:222
2626799226,8cyclictest0-21swapper/222:50:432
2626199229,3cyclictest5887-21tr19:35:111
2626199228,4cyclictest16132-21grep21:10:201
2626199228,2cyclictest32400-21kworker/u8:221:49:401
2626199227,3cyclictest4462-21fw_packets00:25:141
2626199222,15cyclictest0-21swapper/120:04:491
2626199221,9cyclictest5469-21smartctl20:45:231
26261992219,1cyclictest0-21swapper/120:20:231
2626199221,19cyclictest4317-21ntp_states23:10:221
2626199221,17cyclictest7359-21/usr/sbin/munin00:30:241
2626199221,15cyclictest0-21swapper/120:10:191
2626199221,15cyclictest0-21swapper/119:45:051
2626199221,15cyclictest0-21swapper/119:45:051
2626199221,11cyclictest0-21swapper/123:46:121
2626199221,10cyclictest0-21swapper/121:22:141
2625799229,3cyclictest25296-21kworker/0:220:31:360
2625799226,14cyclictest16222-21perf23:40:010
2625799222,8cyclictest28696-21kworker/0:223:21:180
2625799222,8cyclictest28696-21kworker/0:223:21:180
2625799222,11cyclictest18078-21kworker/0:023:46:310
2625799221,9cyclictest28696-21kworker/0:223:13:220
2625799221,4cyclictest0-21swapper/023:52:540
2627499218,2cyclictest16323-21sshd19:57:193
2627499215,3cyclictest87350irq/126-enp1s0-19:10:133
2627499214,4cyclictest13453-21ntp_states19:50:223
2627499212,5cyclictest87350irq/126-enp1s0-20:30:253
2627499212,5cyclictest26646-21ntp_states20:20:223
2627499212,3cyclictest11421-21/usr/sbin/munin22:15:113
2627499211,9cyclictest11662-21unixbench-2d19:45:253
2627499211,9cyclictest11662-21unixbench-2d19:45:253
2627499211,6cyclictest19002-21egrep20:05:133
2626799216,8cyclictest3734-21sshd20:41:582
2626799215,3cyclictest22178-21latency_hist22:40:022
2626799211,11cyclictest31259-21hddtemp_smartct21:45:152
2626199219,7cyclictest0-21swapper/120:30:341
2626199212,18cyclictest0-21swapper/119:40:471
2626199212,17cyclictest0-21swapper/120:35:231
2626199211,18cyclictest0-21swapper/123:35:171
2626199211,14cyclictest0-21swapper/120:27:161
2625799211,7cyclictest0-21swapper/023:56:450
2625799211,5cyclictest0-21swapper/022:06:350
2625799211,5cyclictest0-21swapper/022:06:350
2625799211,12cyclictest2008-21/usr/sbin/munin20:40:150
2625799211,11cyclictest6405-21/usr/sbin/munin20:50:160
2625799211,11cyclictest26188-21/usr/sbin/munin21:35:170
65682200,4sleep011-21rcuc/019:35:160
38902200,4sleep02625799cyclictest19:30:150
307232201,2sleep00-21swapper/022:56:560
2627499203,3cyclictest2008-21/usr/sbin/munin20:40:223
2627499201,7cyclictest0-21swapper/320:15:263
2627499201,6cyclictest7110-21idleruntime20:50:183
2627499201,5cyclictest11779-21needreboot21:00:213
2626799205,4cyclictest16654-21cat23:40:132
2626799205,3cyclictest87350irq/126-enp1s0-19:40:012
2626799204,9cyclictest23606-21/usr/sbin/munin23:55:142
2626799204,3cyclictest8564-1kworker/u9:121:33:242
2626799202,9cyclictest28738-21diskstats21:40:142
2626799201,5cyclictest28513-21ntp_states00:05:212
26267992014,3cyclictest9267-21gltestperf20:55:142
2626799201,11cyclictest15827-21users23:35:272
2626799201,10cyclictest26938-21timerandwakeup20:20:252
2626799201,10cyclictest19097-21open_files22:30:232
2626199200,10cyclictest0-21swapper/121:00:151
2625799209,3cyclictest31208-21/usr/sbin/munin23:00:200
2625799209,3cyclictest23376-21/usr/sbin/munin20:15:120
2625799209,2cyclictest16820-21kworker/0:221:12:400
2625799207,5cyclictest27816-21netstat22:50:190
2625799202,10cyclictest5444-21/usr/sbin/munin23:15:140
2625799201,9cyclictest8004-21kworker/0:000:38:480
2625799201,8cyclictest90602cut20:55:140
2625799201,8cyclictest13378-21kworker/0:122:24:580
2625799201,16cyclictest23154-21munin-run20:15:010
2625799201,11cyclictest4206-21/usr/sbin/munin20:45:130
2625799201,11cyclictest29649-21/usr/sbin/munin00:10:160
2625799201,11cyclictest11743-21sshd23:29:180
2625799200,8cyclictest8564-1kworker/u9:123:35:010
2626799194,8cyclictest4354-21open_inodes23:10:242
2626799194,6cyclictest7983-21/usr/sbin/munin19:40:182
2626799193,4cyclictest18819-21ls23:45:132
2626799193,3cyclictest16455-21sendmail_mailst21:10:242
2626799192,9cyclictest30595-21users22:55:272
2626799192,9cyclictest2604-21memory00:20:182
2626799191,8cyclictest22616-21kernelversion21:25:172
2626799191,6cyclictest5161-21df22:00:142
2626799191,6cyclictest0-21swapper/200:25:222
2626799191,5cyclictest20273-21hddtemp_smartct21:20:162
2626799191,11cyclictest0-21swapper/219:21:532
2626799191,10cyclictest1372-21df_inode23:05:132
2625799198,2cyclictest9135-21kworker/0:019:19:400
2625799197,3cyclictest8564-1kworker/u9:100:25:110
2625799196,2cyclictest25296-21kworker/0:220:27:320
2625799194,3cyclictest87350irq/126-enp1s0-22:35:150
2625799192,11cyclictest16477-21sshd19:59:180
2625799191,9cyclictest21967-21kworker/0:222:44:280
2625799191,8cyclictest21967-21kworker/0:222:46:160
2625799191,6cyclictest0-21swapper/021:31:110
2625799191,10cyclictest31548-21/usr/sbin/munin19:20:200
2625799191,10cyclictest12379-21/usr/sbin/munin19:50:190
2626799186,3cyclictest30707-21ntpq00:10:182
2626799185,4cyclictest17401-21/usr/sbin/munin21:15:162
2626799185,3cyclictest87350irq/126-enp1s0-23:50:232
2626799185,3cyclictest10802-21/usr/sbin/munin21:00:182
2626799184,7cyclictest8799-21tune2fs00:35:142
2626799183,7cyclictest8706-21kworker/u8:022:13:522
2626799183,3cyclictest87650irq/129-enp1s0-21:35:182
2626799183,3cyclictest1922-21latency_hist20:40:012
2626799183,3cyclictest0-21swapper/221:06:282
2626799182,9cyclictest2020-21users21:50:252
2626799181,8cyclictest26341-21/usr/sbin/munin00:00:212
2626799181,5cyclictest25596-21grep22:45:202
2626799181,5cyclictest10908-21ntp_states23:25:212
2626799181,4cyclictest0-21swapper/222:20:452
2626799181,4cyclictest0-21swapper/200:17:062
26267991813,3cyclictest3354-21idleruntime-cro19:30:002
26267991812,3cyclictest29352-21/usr/sbin/munin19:15:212
2625799189,2cyclictest11421-21/usr/sbin/munin22:15:110
2625799187,3cyclictest27156-21kworker/0:100:18:360
2625799185,2cyclictest28696-21kworker/0:223:06:240
2625799183,12cyclictest2896-21grep19:25:240
2625799182,7cyclictest6083-21/usr/sbin/munin00:30:160
2625799181,8cyclictest0-21swapper/021:46:130
2625799181,5cyclictest8564-1kworker/u9:121:28:120
2625799181,14cyclictest8651-21cut19:40:160
2625799181,13cyclictest18466-21ntp_states21:15:220
26257991810,5cyclictest20217-21fgrep20:05:230
183172181,2sleep20-21swapper/220:00:272
147842181,2sleep00-21swapper/021:07:440
143062181,2sleep20-21swapper/219:54:392
26274991712,3cyclictest9277-21hddtemp_smartct20:55:163
2626799177,2cyclictest87550irq/128-enp1s0-22:18:182
2626799174,5cyclictest0-21swapper/220:30:142
2626799174,4cyclictest774-21avahi-daemon20:50:122
2626799173,3cyclictest9-21rcu_preempt23:20:192
2626799173,3cyclictest9-21rcu_preempt23:20:192
2626799172,3cyclictest19623-21gltestperf20:05:162
2626799171,8cyclictest21178-21/usr/sbin/munin20:10:192
2626799171,4cyclictest4649-21grep19:30:232
2626799171,2cyclictest87550irq/128-enp1s0-00:30:112
26267991712,2cyclictest4206-21/usr/sbin/munin20:45:162
26267991711,4cyclictest11134-21grep19:45:202
26267991711,4cyclictest11134-21grep19:45:202
26267991711,3cyclictest297352sleep220:30:012
2625799177,2cyclictest27156-21kworker/0:100:20:480
2625799171,14cyclictest0-21swapper/023:40:350
2626799161,7cyclictest12624-21grep23:30:152
2626799161,2cyclictest0-21swapper/223:15:102
26267991611,3cyclictest15750-21ps19:55:232
2625799162,5cyclictest17421-21tail20:00:150
26257991611,3cyclictest19597-21/usr/sbin/munin21:20:160
26257991611,3cyclictest11262-21awk19:45:210
26257991611,3cyclictest11262-21awk19:45:210
2625799161,12cyclictest32170-21/usr/sbin/munin20:35:220
2625799155,3cyclictest87550irq/128-enp1s0-19:10:120
2625799151,5cyclictest0-21swapper/021:40:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional