You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-21 - 04:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot3s.osadl.org (updated Sun Jun 21, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
215982230160,56sleep10-21swapper/119:06:411
218392205169,24sleep30-21swapper/319:09:483
216992204166,25sleep20-21swapper/219:07:582
198952201164,25sleep00-21swapper/019:05:060
2418221430,3sleep02193399cyclictest00:05:210
2418221430,3sleep02193399cyclictest00:05:210
21947991070,105cyclictest0-21swapper/222:23:312
21947998481,2cyclictest0-21swapper/223:52:242
320562490,5sleep132057-21cpuspeed_turbos00:25:131
283972440,1sleep20-21swapper/219:20:212
2195199441,36cyclictest5615-21runrttasks00:36:173
2195199411,27cyclictest9545-21sshd23:33:243
2195199401,26cyclictest31393-21/usr/sbin/munin21:55:173
21933994010,14cyclictest26660-21munin-run23:00:010
21933994010,14cyclictest26660-21munin-run23:00:010
2195199396,22cyclictest31279-21/usr/sbin/munin23:10:103
2195199392,29cyclictest4228-21memory23:20:183
2195199391,26cyclictest19666-21/usr/sbin/munin21:30:133
2193399398,14cyclictest10042-21awk19:55:010
2195199383,20cyclictest827-21runrttasks22:34:313
2195199381,23cyclictest1521-21awk00:30:013
2195199381,22cyclictest20902-21/usr/sbin/munin00:00:123
2193399388,14cyclictest5302-21awk23:25:020
2195199373,24cyclictest29719-21/usr/sbin/munin00:20:213
2195199371,29cyclictest808-21snmpd22:06:243
2195199366,21cyclictest87650irq/129-enp1s0-21:50:193
2195199363,22cyclictest17470-21/usr/sbin/munin21:25:213
2195199362,26cyclictest13652-21perl21:15:153
2195199361,27cyclictest10648-21diskmemload22:35:183
2195199361,24cyclictest25755-21ntp_states22:55:233
2195199361,24cyclictest25755-21ntp_states22:55:233
2195199361,23cyclictest19918-21sshd22:43:243
2195199360,18cyclictest0-21swapper/321:10:153
2194799361,11cyclictest0-21swapper/221:23:352
2195199356,19cyclictest23449-21/usr/sbin/munin20:25:243
2195199352,22cyclictest29488-21latency_hist00:20:013
2195199351,24cyclictest9906-21/usr/sbin/munin23:35:223
2195199351,24cyclictest20686-21fschecks_count22:45:153
2195199351,23cyclictest6663-21head23:25:233
2195199351,23cyclictest26260-21/usr/sbin/munin21:45:233
2195199351,23cyclictest26260-21/usr/sbin/munin21:45:233
2195199351,22cyclictest21858-21/usr/sbin/munin21:35:233
2195199350,21cyclictest1081-21rt-features20:45:233
2195199343,27cyclictest2383-21smart_sda23:15:253
2195199343,23cyclictest15361-21/usr/sbin/munin23:45:203
2195199341,23cyclictest29088-21/usr/sbin/munin23:05:133
2195199341,23cyclictest2076-21/usr/sbin/munin20:50:203
2195199341,21cyclictest18448-21taskset23:54:513
2195199341,18cyclictest10648-21diskmemload22:11:173
2195199340,19cyclictest0-21swapper/323:05:023
2195199335,18cyclictest10648-21diskmemload21:22:503
2195199333,27cyclictest28907-21ntp_states20:35:213
2195199333,27cyclictest28907-21ntp_states20:35:213
2195199332,27cyclictest12586-21cpuspeed_turbos20:00:113
2195199331,21cyclictest9293-21/usr/sbin/munin22:20:203
2195199331,16cyclictest20691-21kworker/u8:100:12:013
2195199331,11cyclictest87350irq/126-enp1s0-00:31:573
2195199325,12cyclictest0-21swapper/323:40:213
2195199321,27cyclictest19958-21smart_sda23:55:223
2195199321,20cyclictest2187-21runrttasks22:01:293
2195199321,18cyclictest29427-21/usr/sbin/munin19:25:233
2195199321,18cyclictest29427-21/usr/sbin/munin19:25:223
21951993210,19cyclictest16853-21acpi20:10:123
2195199320,23cyclictest87650irq/129-enp1s0-19:30:233
2195199320,14cyclictest87350irq/126-enp1s0-22:18:393
2195199314,19cyclictest87350irq/126-enp1s0-00:05:253
2195199314,19cyclictest87350irq/126-enp1s0-00:05:253
2195199313,13cyclictest25645-21/usr/sbin/munin20:30:183
2195199311,27cyclictest6596-21/usr/sbin/munin19:45:163
2195199311,26cyclictest5337-40dnf19:41:283
2195199311,26cyclictest25814-40dnf21:42:233
2195199311,16cyclictest0-21swapper/319:10:243
21951993110,18cyclictest9212-21proc_pri19:50:233
2194199311,10cyclictest0-21swapper/122:10:291
2194199310,12cyclictest0-21swapper/123:21:041
2194199310,10cyclictest0-21swapper/123:57:211
2193399318,11cyclictest28876-21perf23:05:010
101392310,4sleep2311ktimersoftd/221:05:232
2195199302,25cyclictest9938-21smart_sda21:05:223
2195199301,26cyclictest31778-40dnf20:42:123
2195199301,26cyclictest22489-21/usr/sbin/munin22:50:213
2195199301,26cyclictest19051-21/usr/sbin/munin20:15:223
2195199301,26cyclictest11492-21/usr/sbin/munin22:25:133
21951993011,16cyclictest2530-21ntpq19:35:223
21947993026,3cyclictest0-21swapper/221:50:562
2194199301,11cyclictest0-21swapper/120:02:321
21941993011,10cyclictest27091-21date19:20:001
2194199300,5cyclictest0-21swapper/120:37:491
2194199300,5cyclictest0-21swapper/120:37:481
2194199300,10cyclictest0-21swapper/122:19:331
2195199291,25cyclictest6471-21/usr/sbin/munin21:00:163
2195199291,25cyclictest27412-21/usr/sbin/munin19:20:113
2195199291,25cyclictest11510-21/usr/sbin/munin19:55:243
2194199291,11cyclictest0-21swapper/119:53:201
2194199290,16cyclictest0-21swapper/100:15:571
2194199290,11cyclictest0-21swapper/122:24:211
2195199282,23cyclictest5344-21ntp_states20:55:203
2195199282,23cyclictest15718-21ntp_states20:05:213
2195199281,24cyclictest25848-21/usr/sbin/munin19:15:183
2195199281,24cyclictest21245-21/usr/sbin/munin20:20:203
2194199284,10cyclictest0-21swapper/123:45:161
2194199281,9cyclictest0-21swapper/122:34:191
2194199281,14cyclictest0-21swapper/100:13:551
2194199280,9cyclictest0-21swapper/122:04:211
2194199280,12cyclictest0-21swapper/121:17:491
2194199280,12cyclictest0-21swapper/119:34:001
2194199273,16cyclictest790-21in:imjournal21:46:411
2194199273,16cyclictest790-21in:imjournal21:46:411
2194199271,9cyclictest0-21swapper/120:12:121
2194199271,8cyclictest0-21swapper/122:42:001
2194199271,11cyclictest0-21swapper/120:49:091
2194199271,10cyclictest0-21swapper/121:41:211
2194199271,10cyclictest0-21swapper/120:23:401
2194199270,5cyclictest0-21swapper/122:49:201
2194199270,25cyclictest0-21swapper/100:37:331
2194199270,12cyclictest0-21swapper/123:33:081
2193399272,12cyclictest32627-21munin-run22:00:010
169822270,1sleep00-21swapper/023:50:160
2194799268,8cyclictest0-21swapper/200:20:012
2194799266,9cyclictest0-21swapper/223:20:212
2194199261,9cyclictest10648-21diskmemload00:03:091
2194199261,9cyclictest0-21swapper/121:34:211
2194199261,9cyclictest0-21swapper/119:42:081
2194199261,5cyclictest0-21swapper/119:10:401
2194199261,19cyclictest0-21swapper/123:28:541
2194199261,17cyclictest16290-21chrt20:05:511
2194199261,10cyclictest0-21swapper/121:38:171
2194199261,10cyclictest0-21swapper/120:18:321
2194199260,5cyclictest0-21swapper/123:43:561
2194199260,11cyclictest0-21swapper/122:05:291
2194199260,11cyclictest0-21swapper/119:46:101
2194199260,10cyclictest0-21swapper/123:50:371
2194199260,10cyclictest0-21swapper/119:38:561
2193399261,6cyclictest27432-21ps21:45:220
2193399261,6cyclictest27432-21ps21:45:220
2194799253,13cyclictest22738-21taskset00:03:102
2194799250,15cyclictest7742-21cpu23:30:102
2194199251,9cyclictest0-21swapper/123:01:361
2194199251,8cyclictest0-21swapper/120:28:321
21941992514,3cyclictest0-21swapper/100:21:161
2194199251,11cyclictest0-21swapper/121:06:151
2194199251,10cyclictest0-21swapper/120:44:511
2194199251,10cyclictest0-21swapper/120:32:521
2194199251,10cyclictest0-21swapper/119:24:051
2194199250,9cyclictest0-21swapper/121:22:011
2194199250,8cyclictest0-21swapper/122:51:041
2194199250,8cyclictest0-21swapper/121:02:371
2194199250,11cyclictest0-21swapper/122:55:441
2194199250,11cyclictest0-21swapper/122:55:441
2194199250,11cyclictest0-21swapper/119:55:241
2194199250,10cyclictest0-21swapper/121:26:051
2193399256,16cyclictest21636-21munin-run21:35:010
2193399251,13cyclictest8455-21munin-run21:05:020
2194799247,8cyclictest12082-21sshd19:58:232
2194799244,13cyclictest4782-21sed22:10:022
2194799244,11cyclictest25326-21/usr/sbin/munin00:10:172
21947992411,6cyclictest808-21snmpd19:39:502
2194199241,9cyclictest0-21swapper/100:07:371
2194199241,9cyclictest0-21swapper/100:07:371
2194199241,16cyclictest0-21swapper/122:25:141
2194199241,10cyclictest0-21swapper/123:19:541
2194199241,10cyclictest0-21swapper/121:50:291
2194199240,9cyclictest0-21swapper/123:37:121
2194199240,8cyclictest0-21swapper/122:36:161
2194199240,10cyclictest0-21swapper/121:58:531
2194199240,10cyclictest0-21swapper/121:12:291
2194799231,6cyclictest0-21swapper/222:30:212
2194199239,3cyclictest0-21swapper/100:30:111
2194199236,7cyclictest0-21swapper/119:26:391
2194199236,7cyclictest0-21swapper/119:26:391
2194199230,8cyclictest0-21swapper/120:57:291
2194199230,7cyclictest0-21swapper/123:06:321
2194199230,6cyclictest0-21swapper/120:51:571
2193399237,14cyclictest0-21swapper/000:05:010
2193399235,15cyclictest23829-21idleruntime-cro21:40:000
2194799225,11cyclictest6059-21proc_pri22:10:242
2194799224,6cyclictest1768-21sshd22:00:252
2194799223,8cyclictest11531-21cpu22:25:122
2194799223,11cyclictest27229-21diskstats23:00:132
2194799221,11cyclictest15377-21/usr/sbin/munin23:45:202
2194199220,10cyclictest0-21swapper/123:11:541
2193399227,14cyclictest0-21swapper/022:20:010
2193399223,7cyclictest19838-40dnf22:42:260
2194799217,7cyclictest30066-21diskstats00:20:152
2194799217,5cyclictest755-21rngd21:10:082
2194799215,7cyclictest15882-21/usr/sbin/munin22:35:232
2194799215,6cyclictest755-21rngd23:40:142
2194799214,9cyclictest0-21swapper/221:56:472
2194799214,10cyclictest0-21swapper/221:26:432
2194799211,13cyclictest0-21swapper/221:39:352
2194799210,18cyclictest4980-21ntp_kernel_err00:35:192
2194799210,14cyclictest8564-1kworker/u9:122:51:222
2194799210,12cyclictest13411-21df21:15:132
2193399211,5cyclictest0-21swapper/021:23:380
127172211,3sleep0808-21snmpd21:13:300
12532210,4sleep21256-21timerandwakeup20:45:252
2194799206,7cyclictest839-21lldpd20:39:412
2194799206,7cyclictest839-21lldpd20:39:412
2194799205,2cyclictest24062-21/usr/sbin/munin21:40:182
2194799204,10cyclictest0-21swapper/223:07:402
2194799203,9cyclictest755-21rngd23:12:532
2194799203,8cyclictest25730-21ntp_kernel_pll_22:55:212
2194799203,8cyclictest25730-21ntp_kernel_pll_22:55:212
2194799203,3cyclictest87650irq/129-enp1s0-20:20:272
2194799201,5cyclictest0-21swapper/200:05:082
2194799201,5cyclictest0-21swapper/200:05:082
21947992011,3cyclictest8564-1kworker/u9:121:31:372
2194799201,12cyclictest0-21swapper/223:28:402
2194799201,12cyclictest0-21swapper/220:10:172
2194799201,11cyclictest32-21ksoftirqd/200:26:382
21947992010,7cyclictest30035-21gltestperf19:25:162
21947992010,7cyclictest30035-21gltestperf19:25:162
2194799200,14cyclictest11531-21chrt23:35:412
2193399209,3cyclictest9906-21/usr/sbin/munin23:35:250
2193399209,3cyclictest25326-21/usr/sbin/munin00:10:210
2193399203,9cyclictest0-21swapper/021:08:250
2193399201,7cyclictest0-21swapper/023:28:030
2193399201,4cyclictest12781-21cat22:25:230
2193399201,10cyclictest3850-21grep22:05:240
2194799196,9cyclictest5346-21runrttasks19:41:302
2194799195,2cyclictest28017-21sshd21:47:242
2194799195,2cyclictest28017-21sshd21:47:242
2194799191,9cyclictest8564-1kworker/u9:122:19:102
2194799191,8cyclictest32-21ksoftirqd/200:31:372
2194799191,11cyclictest20602-21taskset23:59:192
2193399199,2cyclictest17470-21/usr/sbin/munin21:25:150
2193399198,3cyclictest808-21snmpd22:10:180
2193399195,3cyclictest12552timerandwakeup20:45:260
2193399193,10cyclictest808-21snmpd19:41:260
2193399191,5cyclictest0-21swapper/023:55:130
21933991913,4cyclictest827-21runrttasks21:15:170
21933991913,3cyclictest28591-21ntp_states00:15:220
21933991912,4cyclictest30677-21tail20:40:150
21933991911,4cyclictest24396-21df21:40:130
2193399191,11cyclictest6029-21sshd20:57:240
2193399191,10cyclictest808-21snmpd19:59:530
2193399191,10cyclictest29088-21/usr/sbin/munin23:05:220
2193399191,10cyclictest27229-21/usr/sbin/munin19:20:240
2193399191,10cyclictest22142-21/usr/sbin/munin19:10:170
2194799188,7cyclictest26571-21sort20:30:162
2194799187,3cyclictest0-21swapper/220:25:582
2194799185,2cyclictest87650irq/129-enp1s0-20:40:202
2194799185,10cyclictest20374-21cpuspeed_turbos22:45:122
2194799184,11cyclictest950-21runrttasks19:31:512
2194799183,2cyclictest0-21swapper/223:15:132
2194799183,11cyclictest24986-21grep19:15:042
2194799181,9cyclictest6471-21/usr/sbin/munin21:00:102
2194799181,9cyclictest4276-21/usr/sbin/munin20:55:182
2194799181,9cyclictest0-21swapper/222:41:432
2194799181,8cyclictest87650irq/129-enp1s0-20:09:552
2193399188,2cyclictest1439-21/usr/sbin/munin19:35:280
2193399186,3cyclictest8564-1kworker/u9:123:42:080
2193399186,3cyclictest0-21swapper/000:28:590
2193399182,14cyclictest21034-21idleruntime-cro20:20:010
2193399181,9cyclictest0-21swapper/020:30:240
2193399181,5cyclictest0-21swapper/023:18:500
2193399181,5cyclictest0-21swapper/000:39:550
2193399181,3cyclictest23740-21awk22:50:250
21933991812,3cyclictest31684-21dump-pmu-power00:25:020
2194799178,7cyclictest14051-21vmstat20:00:252
2194799178,6cyclictest827-21runrttasks19:53:032
2194799177,7cyclictest22274-21grep19:10:132
2194799172,12cyclictest3126-21awk20:50:192
2193399177,3cyclictest87550irq/128-enp1s0-23:31:170
2193399177,2cyclictest866-21awk23:15:020
2193399175,3cyclictest25429-21perf20:30:010
2193399172,6cyclictest6561-21kworker/u8:222:35:140
2193399172,12cyclictest32043-21sort19:30:140
2193399172,12cyclictest21245-21/usr/sbin/munin20:20:150
2193399171,7cyclictest3310-21unixbench_singl00:30:290
2193399171,6cyclictest0-21swapper/022:46:320
2193399171,6cyclictest0-21swapper/020:00:220
2193399171,4cyclictest0-21swapper/021:53:000
2193399171,15cyclictest0-21swapper/022:03:080
21933991711,3cyclictest16822-21cut20:10:020
2194799167,6cyclictest19212-21grep20:15:122
2194799166,7cyclictest6386-21sort19:45:142
2193399166,8cyclictest87350irq/126-enp1s0-19:45:110
2193399165,3cyclictest87550irq/128-enp1s0-19:25:260
2193399165,3cyclictest87550irq/128-enp1s0-19:25:250
2193399161,6cyclictest17189-21df20:10:130
2193399161,5cyclictest0-21swapper/023:46:540
21933991612,3cyclictest0-21swapper/020:36:030
21933991612,3cyclictest0-21swapper/020:36:030
21933991611,3cyclictest2836-21awk20:50:170
2193399159,4cyclictest0-21swapper/019:15:250
2193399154,3cyclictest87350irq/126-enp1s0-22:30:130
2193399151,5cyclictest9635-21df22:20:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional