You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-25 - 12:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack5slot3s.osadl.org (updated Sat Apr 25, 2026 00:44:29)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
235782227158,56sleep30-21swapper/319:09:513
235472226197,19sleep20-21swapper/219:09:252
234772204166,25sleep00-21swapper/019:08:310
232842202166,24sleep10-21swapper/119:06:031
2368899106103,2cyclictest0-21swapper/221:07:522
23688998581,3cyclictest0-21swapper/200:33:522
157712730,1sleep30-21swapper/321:15:193
23688996965,3cyclictest0-21swapper/223:20:022
236952550,3sleep20-21swapper/221:35:122
25772510,4sleep02367699cyclictest22:00:140
2368199471,36cyclictest19811-21/usr/sbin/munin22:40:271
2368199451,28cyclictest18354-21latency_hist20:10:011
293262440,1sleep00-21swapper/019:20:150
2368199441,25cyclictest533-21tail21:55:151
2368199421,31cyclictest26187-21perf22:55:011
2368199421,28cyclictest23702-21tr21:35:121
2368199411,29cyclictest16174-21awk20:05:011
2368199411,26cyclictest3219-21cut22:00:191
2368199401,32cyclictest17048-21netstat23:45:211
23681994010,19cyclictest30582-21dump-pmu-power23:05:011
2368199391,30cyclictest12846-21cat21:10:111
2368199381,29cyclictest0-21swapper/123:25:141
2368199381,25cyclictest4745-21missed_timers20:50:181
2368199381,24cyclictest31540-21latency_hist20:40:011
2368199381,24cyclictest27759-21latency_hist21:45:011
2368199371,25cyclictest5293-21cat19:40:021
2368199371,24cyclictest5110-21cpuspeed_turbos23:20:121
2368199371,23cyclictest1438-21kworker/u8:100:00:381
2368199371,21cyclictest1-21systemd23:40:201
2368199371,20cyclictest7970-21idleruntime-cro21:00:011
2368199363,18cyclictest7605-21acpi19:45:131
2368199361,32cyclictest22739-21taskset21:30:251
2368199361,32cyclictest10811-21awk22:20:011
2368199361,31cyclictest790-21in:imjournal00:30:011
2368199361,31cyclictest3566-21cat20:49:561
2368199361,31cyclictest21102-21hddtemp_smartct23:55:181
2368199361,25cyclictest19170-21cat21:25:011
2368199361,24cyclictest12540-21cat21:10:011
2368199361,23cyclictest28735-21latency_hist19:20:011
2368199361,22cyclictest11531-21cat23:35:021
2368199361,22cyclictest0-21swapper/123:05:231
2368199361,16cyclictest29251-21/usr/sbin/munin00:15:141
2368199360,29cyclictest28834-21kworker/u8:000:35:141
2368199360,26cyclictest0-21swapper/119:35:021
2368199360,26cyclictest0-21swapper/119:35:021
2367699367,14cyclictest13954-21sadc20:00:010
2367699366,14cyclictest16168-21munin-run20:05:010
2368199357,20cyclictest29429-21cat20:35:011
2368199355,20cyclictest0-21swapper/119:30:021
2368199355,15cyclictest12383-21diskmemload23:15:131
2368199353,24cyclictest1438-21kworker/u8:123:37:351
2368199352,27cyclictest87550irq/128-enp1s0-22:13:551
2368199351,31cyclictest0-21swapper/120:40:171
2368199351,26cyclictest11465-21chrt19:51:221
2368199351,22cyclictest808-21snmpd19:24:281
2368199351,22cyclictest6495-21cat22:10:011
2368199351,20cyclictest17614-21/usr/sbin/munin22:35:251
2367699357,13cyclictest5463-21munin-run00:35:010
2368199345,17cyclictest12383-21diskmemload21:25:141
2368199343,23cyclictest31844-21taskset21:51:081
2368199343,23cyclictest31844-21taskset21:51:081
2368199343,17cyclictest2399-21taskset23:12:161
2368199342,17cyclictest87550irq/128-enp1s0-00:10:211
2368199342,17cyclictest87550irq/128-enp1s0-00:10:211
2368199341,29cyclictest27351-21missed_timers22:55:181
2368199341,29cyclictest27351-21missed_timers22:55:181
2368199341,24cyclictest13223-21/usr/sbin/munin22:25:161
2368199341,23cyclictest15421-21/usr/sbin/munin22:30:221
2368199341,23cyclictest1161-21cat00:25:011
2368199341,20cyclictest0-21swapper/100:05:251
2368199341,17cyclictest29472-21unixbench-2d21:45:271
2367699346,13cyclictest2976-21munin-run19:35:010
2367699346,13cyclictest2976-21munin-run19:35:010
2368199331,29cyclictest21146-21forks20:15:151
2368199331,29cyclictest12693-21sort19:55:161
2368199331,20cyclictest22010-21/usr/sbin/munin22:45:191
2368199331,19cyclictest0-21swapper/120:21:231
2368199331,17cyclictest15875-21ntp_states21:15:221
2367699338,12cyclictest25568-21munin-run21:40:010
22022330,5sleep32203-21grep20:45:173
2368199322,19cyclictest87650irq/129-enp1s0-21:00:221
2368199321,28cyclictest7363-21latency_hist19:45:021
2368199321,28cyclictest18678-21cpuspeed_turbos20:10:121
2368199321,22cyclictest87350irq/126-enp1s0-20:25:121
2368199321,17cyclictest0-21swapper/100:32:311
2368199320,18cyclictest19681-1kworker/1:1H19:10:231
2367699321,16cyclictest17388-21munin-run22:35:000
2368199311,27cyclictest20329-21cat23:55:021
2368199311,21cyclictest6222-21kworker/u8:222:20:231
23676993011,3cyclictest27636-21kworker/0:021:45:170
2367699294,12cyclictest11761-21munin-run19:55:010
2367699292,12cyclictest2636-21awk23:15:020
23676992910,16cyclictest5782-21munin-run20:55:020
2367699289,16cyclictest7018-21munin-run23:25:020
2367699281,6cyclictest24624-21fschecks_count22:50:140
2367699281,13cyclictest1372-21munin-run20:45:010
2367699271,14cyclictest27768-21munin-run21:45:010
23676992710,2cyclictest3856-21kworker/0:222:05:210
23676992610,3cyclictest26582-21kworker/0:100:13:250
23676992610,3cyclictest26582-21kworker/0:100:13:250
2368899251,6cyclictest0-21swapper/222:26:432
2368899251,5cyclictest0-21swapper/223:52:052
2368899251,3cyclictest0-21swapper/222:30:352
2367699252,12cyclictest9994-21perl23:30:140
2367699251,11cyclictest30875-21kworker/0:000:21:410
23688992421,2cyclictest0-21swapper/219:39:142
2368899241,4cyclictest0-21swapper/200:35:042
2367699241,21cyclictest14492-21kworker/0:121:28:540
2369299235,2cyclictest0-21swapper/321:51:493
2369299235,2cyclictest0-21swapper/321:51:493
2369299233,2cyclictest0-21swapper/323:53:063
2369299231,7cyclictest0-21swapper/322:03:053
2369299231,6cyclictest0-21swapper/321:44:333
2369299231,11cyclictest0-21swapper/321:11:273
2367699239,6cyclictest0-21swapper/021:50:150
2367699239,6cyclictest0-21swapper/021:50:150
2367699239,3cyclictest3198-21/usr/sbin/munin19:35:230
2367699236,3cyclictest19849-21smartctl20:10:230
2367699236,2cyclictest10708-21kworker/0:222:35:130
2367699231,6cyclictest0-21swapper/022:59:080
2367699231,6cyclictest0-21swapper/022:59:080
2367699231,6cyclictest0-21swapper/022:13:080
2367699231,11cyclictest29357-21munin-run20:35:010
2367699230,11cyclictest7380-21perf19:45:010
2369299229,8cyclictest0-21swapper/322:45:273
2369299229,6cyclictest0-21swapper/323:21:413
2369299226,3cyclictest0-21swapper/322:07:063
2369299220,14cyclictest0-21swapper/300:05:443
2368899221,6cyclictest0-21swapper/200:05:292
2368899221,5cyclictest0-21swapper/221:33:322
23688992215,3cyclictest29074-21tr19:20:122
2368899221,3cyclictest0-21swapper/221:14:102
2368899221,11cyclictest26743-21df22:55:122
2368899221,11cyclictest26743-21df22:55:122
2367699229,3cyclictest10708-21kworker/0:222:25:250
2367699222,11cyclictest5537-21kworker/0:021:09:540
2367699221,5cyclictest0-21swapper/000:05:140
23676992214,4cyclictest30948-21grep23:05:120
2369299215,3cyclictest0-21swapper/300:27:183
2369299214,11cyclictest0-21swapper/300:02:083
2369299211,5cyclictest0-21swapper/323:39:483
2369299211,3cyclictest87350irq/126-enp1s0-22:25:133
23692992113,3cyclictest0-21swapper/323:56:023
2369299211,13cyclictest27215-21irqstats22:55:173
2369299211,13cyclictest27215-21irqstats22:55:163
2369299211,12cyclictest0-21swapper/323:30:223
2369299211,11cyclictest30314-21ntp_states00:15:223
23692992110,3cyclictest0-21swapper/320:55:193
2369299210,13cyclictest18696-21kworker/u8:021:24:373
2368899211,6cyclictest0-21swapper/221:57:422
2368899211,5cyclictest0-21swapper/223:09:272
2367699218,2cyclictest19218-21kworker/0:122:40:130
2367699217,2cyclictest14492-21kworker/0:121:19:130
2367699211,5cyclictest0-21swapper/023:27:110
2367699211,5cyclictest0-21swapper/023:02:290
2367699211,5cyclictest0-21swapper/000:00:560
2367699211,4cyclictest0-21swapper/022:20:450
2367699211,4cyclictest0-21swapper/021:30:050
2367699211,3cyclictest0-21swapper/022:16:480
23676992110,2cyclictest3953-21kworker/0:023:18:260
2369299208,3cyclictest0-21swapper/323:02:323
2369299205,8cyclictest0-21swapper/300:37:453
2369299204,10cyclictest0-21swapper/321:34:163
2369299203,13cyclictest17380-21grep20:05:193
2369299202,10cyclictest12383-21diskmemload21:36:243
2369299201,5cyclictest0-21swapper/323:07:253
23692992014,3cyclictest28048-21gltestperf20:30:163
23692992013,4cyclictest25289-21wc19:10:243
2369299201,2cyclictest0-21swapper/300:20:153
2369299201,11cyclictest12383-21diskmemload22:50:473
23692992010,2cyclictest19222-21kworker/3:023:10:233
2369299200,12cyclictest0-21swapper/322:34:143
2368899207,4cyclictest87350irq/126-enp1s0-00:20:232
2368899206,4cyclictest23072-21fschecks_count00:00:132
2368899201,9cyclictest0-21swapper/223:18:142
2368899201,6cyclictest0-21swapper/221:53:372
2368899201,6cyclictest0-21swapper/221:53:372
2368899201,5cyclictest24556-21diskstats22:50:132
2368899201,5cyclictest0-21swapper/222:22:002
2368899201,5cyclictest0-21swapper/222:04:112
2368899201,5cyclictest0-21swapper/219:11:152
2368899201,4cyclictest0-21swapper/223:58:592
2368899201,4cyclictest0-21swapper/223:44:302
2368899201,4cyclictest0-21swapper/223:36:042
23688992014,4cyclictest29702-21grep20:35:122
2368899200,4cyclictest0-21swapper/221:43:122
2367699209,3cyclictest26771-21/usr/sbin/munin19:15:150
2367699208,2cyclictest20420-21kworker/0:023:59:010
2367699205,2cyclictest18619-21kworker/0:221:23:420
2367699201,6cyclictest0-21swapper/020:25:260
2367699201,5cyclictest0-21swapper/000:16:420
2367699201,4cyclictest0-21swapper/021:57:250
2367699201,4cyclictest0-21swapper/000:35:560
2367699201,11cyclictest13834-21/usr/sbin/munin23:40:180
2367699201,10cyclictest808-21snmpd23:35:130
2369299197,7cyclictest0-21swapper/300:34:113
2369299196,3cyclictest0-21swapper/319:25:113
2369299196,2cyclictest0-21swapper/322:19:293
2369299195,4cyclictest0-21swapper/323:19:133
2369299195,2cyclictest26174-21strings20:25:213
2369299191,6cyclictest0-21swapper/322:35:173
2369299191,4cyclictest29371-21ls21:45:263
2369299191,3cyclictest0-21swapper/323:40:463
2369299191,3cyclictest0-21swapper/323:29:243
2369299191,11cyclictest0-21swapper/320:50:193
2369299191,10cyclictest12383-21diskmemload22:20:513
23692991910,2cyclictest0-21swapper/321:56:183
2368899195,9cyclictest31001-40dnf00:17:252
2368899193,4cyclictest5775-21latency_hist20:55:012
2368899192,14cyclictest27910-21grep20:30:152
2368899191,6cyclictest9714-21sed22:15:202
2368899191,6cyclictest7363-21latency_hist19:45:012
2368899191,6cyclictest0-21swapper/223:11:312
2368899191,5cyclictest0-21swapper/223:33:052
2368899191,4cyclictest13040-21ntpq19:55:212
2368899191,4cyclictest0-21swapper/222:45:132
2368899191,3cyclictest0-21swapper/200:13:212
2368899191,3cyclictest0-21swapper/200:13:212
2367699198,8cyclictest7683-21sshd20:56:250
2367699195,3cyclictest10708-21kworker/0:222:46:490
2367699191,4cyclictest0-21swapper/023:46:500
2367699190,9cyclictest5537-21kworker/0:021:12:540
2369299183,7cyclictest6628-21/usr/sbin/munin22:10:203
23692991812,3cyclictest21866-40dnf20:15:243
23692991811,2cyclictest0-21swapper/319:33:243
23692991811,2cyclictest0-21swapper/319:33:243
2369299181,10cyclictest12383-21diskmemload00:12:383
2369299181,10cyclictest12383-21diskmemload00:12:383
2369299181,10cyclictest0-21swapper/321:06:263
2369299180,6cyclictest10199-21awk21:05:013
2368899184,9cyclictest16058-40dnf21:15:282
2368899182,6cyclictest24543-21grep20:20:242
2368899181,9cyclictest0-21swapper/221:00:162
2368899181,6cyclictest0-21swapper/223:49:492
2368899181,6cyclictest0-21swapper/222:05:032
2368899181,5cyclictest3433-21cat00:30:022
2368899181,5cyclictest31163-21/usr/sbin/munin19:25:232
2368899181,5cyclictest0-21swapper/222:14:372
2368899181,4cyclictest0-21swapper/221:46:282
2368899181,11cyclictest0-21swapper/223:25:402
2367699186,2cyclictest23871-21kworker/0:319:11:010
2367699183,2cyclictest30600-21kworker/0:119:25:050
2367699181,7cyclictest5537-21kworker/0:021:01:140
2367699181,7cyclictest0-21swapper/023:50:130
2369299179,3cyclictest0-21swapper/321:25:503
2369299177,3cyclictest12383-21diskmemload22:42:493
2369299174,3cyclictest0-21swapper/319:38:243
2369299174,2cyclictest0-21swapper/320:10:183
2369299173,11cyclictest14183-21/usr/sbin/munin20:00:153
2369299172,12cyclictest31768-21/usr/sbin/munin20:40:163
2369299171,8cyclictest0-21swapper/323:46:173
2369299171,11cyclictest0-21swapper/319:51:353
23692991710,4cyclictest7609-21cpu19:45:123
2369299170,11cyclictest0-21swapper/319:55:133
2368899171,6cyclictest0-21swapper/221:26:282
2368899171,5cyclictest0-21swapper/220:45:252
2368899171,4cyclictest6006-21/usr/sbin/munin20:55:152
2368899171,4cyclictest21719-21sshd22:44:242
2368899171,3cyclictest0-21swapper/223:02:542
2368899171,2cyclictest0-21swapper/220:03:232
23688991712,3cyclictest19361-21if_err_enp1s020:10:172
2368899171,15cyclictest0-21swapper/219:47:502
2367699171,7cyclictest0-21swapper/000:25:140
23676991712,3cyclictest18360-21munin-run20:10:010
23676991711,3cyclictest30838-21smartctl20:35:240
23676991710,4cyclictest0-21swapper/020:20:200
2367699170,10cyclictest0-21swapper/020:15:270
2369299168,2cyclictest0-21swapper/319:20:233
23692991611,3cyclictest6154-21awk19:40:163
23692991610,3cyclictest29576-21/usr/sbin/munin20:35:213
2368899161,6cyclictest0-21swapper/220:05:202
2368899161,4cyclictest20783-21/usr/sbin/munin20:15:182
2368899161,3cyclictest0-21swapper/222:36:192
23688991612,3cyclictest0-21swapper/220:25:242
23688991610,4cyclictest26203-21df19:15:022
23688991610,3cyclictest1525-21cat20:45:012
23688991610,3cyclictest10125-21df19:50:122
2367699166,2cyclictest827-21runrttasks19:45:020
23676991610,3cyclictest827-21runrttasks20:46:050
188452161,2sleep20-21swapper/221:23:292
2369299151,6cyclictest0-21swapper/320:20:133
2369299150,2cyclictest0-21swapper/319:16:103
23688991510,3cyclictest827-21runrttasks19:33:062
23688991510,3cyclictest827-21runrttasks19:33:062
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional