You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-26 - 18:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System (updated Thu Mar 26, 2026 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2136209994239,2cyclictest2359167-21kworker/u8:2+events_unbound10:50:153
2136209993532,2cyclictest2416302-21kworker/u8:1+flush-259:011:50:233
2136209993532,2cyclictest2416302-21kworker/u8:1+flush-259:011:50:233
2136209993532,2cyclictest2255355-21kworker/u8:4+events_unbound10:40:143
2136205993428,4cyclictest252-21systemd-journal10:04:292
2136205993329,3cyclictest3282653-21kworker/u8:1+flush-259:007:45:012
2136209993128,2cyclictest2229545-21kworker/u8:1+events_unbound08:50:243
2136199993128,2cyclictest2328220-21kworker/u8:1+events_unbound10:25:150
2136209993027,2cyclictest3282653-21kworker/u8:1+events_unbound07:25:203
2136209993027,2cyclictest2245024-21kworker/u8:0+events_unbound09:25:193
2136209993026,3cyclictest2131239-21kworker/u8:0+events_unbound07:35:213
2136209992925,2cyclictest2198633-21kworker/u8:3+events_unbound09:05:173
2136209992825,2cyclictest2287044-21kworker/u8:3+events_unbound09:40:223
2136209992825,2cyclictest2198633-21kworker/u8:3+flush-259:008:20:233
2136205992823,3cyclictest2229758-21/usr/sbin/munin08:40:132
2136201992822,4cyclictest2281639-21latency_hist09:30:001
2136205992723,3cyclictest2287044-21kworker/u8:3+flush-259:009:45:172
2136201992723,3cyclictest2375319-21apt-get11:00:111
2136209992624,2cyclictest2296724-21kworker/u8:0+flush-259:010:15:243
2136209992623,2cyclictest2436789-21kworker/u8:0+events_unbound12:35:153
2136209992623,2cyclictest2401065-21kworker/u8:0+events_unbound11:45:153
2136209992623,2cyclictest2385393-21kworker/u8:3+flush-259:011:40:243
2136209992622,3cyclictest2426611-21kworker/u8:2+events_unbound12:25:223
2136209992622,3cyclictest2131239-21kworker/u8:0+flush-259:007:55:143
2136209992622,2cyclictest2436789-21kworker/u8:0+events_unbound12:05:143
2136199992623,2cyclictest2131239-21kworker/u8:0+events_unbound08:04:280
2136199992622,3cyclictest2136404-21apt-get07:10:110
2136199992620,5cyclictest2337573-21ntpq10:20:210
2136209992522,2cyclictest2255355-21kworker/u8:4+events_unbound09:20:223
2136205992522,2cyclictest2328220-21kworker/u8:1+flush-259:011:10:132
2136205992521,3cyclictest2255354-21kworker/u8:2+flush-259:009:05:002
2136205992518,5cyclictest2349612-21diskstats10:35:132
2136199992522,2cyclictest2198633-21kworker/u8:3+flush-259:008:35:240
2136205992418,4cyclictest2323045-21latency_hist10:10:012
2136201992419,4cyclictest2265712-21apt-get09:15:001
2136199992421,2cyclictest2287044-21kworker/u8:3+flush-259:010:00:150
2136209992320,2cyclictest2436789-21kworker/u8:0+flush-259:012:25:003
2136209992320,2cyclictest2401065-21kworker/u8:0+events_unbound11:27:073
2136209992320,2cyclictest2395676-21kworker/u8:2+events_unbound11:20:213
2136209992320,2cyclictest2328220-21kworker/u8:1+events_unbound10:30:283
2136209992320,2cyclictest2296724-21kworker/u8:0+events_unbound09:55:153
2136205992319,3cyclictest2198633-21kworker/u8:3+flush-259:008:55:182
2136201992317,4cyclictest2426587-21latency_hist11:50:001
2136199992319,3cyclictest2229545-21kworker/u8:1+events_unbound09:00:150
2136209992219,2cyclictest2426611-21kworker/u8:2+events_unbound12:10:193
2136209992219,2cyclictest2198633-21kworker/u8:3+events_unbound08:25:163
2136209992219,2cyclictest1415264-21kworker/u8:2+events_unbound07:15:153
2136209992218,3cyclictest2281656-21kworker/u8:1+events_unbound09:30:153
2136209992218,3cyclictest2281656-21kworker/u8:1+events_unbound09:30:153
2136201992218,2cyclictest0-21swapper/109:10:001
2136209992120,1cyclictest2229545-21kworker/u8:1+flush-259:008:45:243
2136209992118,2cyclictest2376072-21kworker/u8:0+events_unbound11:05:243
2136209992118,2cyclictest1415264-21kworker/u8:2+events_unbound08:15:193
2136209992117,3cyclictest2255355-21kworker/u8:4+flush-259:009:50:203
2136205992118,2cyclictest2385393-21kworker/u8:3+flush-259:011:20:162
2136205992118,2cyclictest2359167-21kworker/u8:2+flush-259:011:00:122
2136205992118,2cyclictest2245024-21kworker/u8:0+flush-259:009:25:142
2136205992117,3cyclictest2447861-21apt-get12:10:122
2136205992117,3cyclictest2255355-21kworker/u8:4+events_unbound09:10:172
2136205992114,5cyclictest2255757-21diskstats09:05:132
2136201992115,4cyclictest252-21systemd-journal12:37:081
2136201992115,4cyclictest2390499-21latency_hist11:15:001
2136199992118,2cyclictest3282653-21kworker/u8:1+flush-259:007:30:130
2136199992117,3cyclictest2287044-21kworker/u8:3+flush-259:009:45:250
2136209992017,2cyclictest2328220-21kworker/u8:1+events_unbound11:10:123
2136209992017,2cyclictest2131239-21kworker/u8:0+events_unbound08:05:133
2136209992014,4cyclictest252-21systemd-journal07:47:423
2136205992017,2cyclictest3282653-21kworker/u8:1+flush-259:008:10:152
2136205992017,2cyclictest2245024-21kworker/u8:0+flush-259:009:17:002
2136205992017,2cyclictest1415264-21kworker/u8:2+flush-259:007:15:172
2136205992014,4cyclictest2437381-21cat12:00:012
2136201992015,4cyclictest2214254-21/usr/sbin/munin08:25:151
2136199992017,2cyclictest1415264-21kworker/u8:2+events_unbound08:15:130
2136205991912,5cyclictest2456999-21ntpq12:15:212
2136201991916,2cyclictest2436912-21systemd-run12:00:011
2136199991916,2cyclictest2255355-21kworker/u8:4+events_unbound10:40:120
2136199991912,5cyclictest2368956-21ntpq10:50:210
2136209991815,2cyclictest2359717-21kworker/u8:3+flush-259:010:45:173
2136209991815,2cyclictest2359717-21kworker/u8:3+events_unbound10:55:233
2136209991815,2cyclictest2255354-21kworker/u8:2+events_unbound09:15:203
2136209991814,3cyclictest2219231-21kworker/u8:0+flush-259:008:40:253
2136209991813,3cyclictest252-21systemd-journal12:35:013
2136209991811,4cyclictest252-21systemd-journal09:50:003
2136205991815,2cyclictest2436789-21kworker/u8:0+events_unbound12:35:232
2136205991815,2cyclictest2255355-21kworker/u8:4+events_unbound10:40:202
2136205991815,2cyclictest2131239-21kworker/u8:0+flush-259:008:00:162
2136205991813,3cyclictest252-21systemd-journal10:20:012
213620199185,2cyclictest301rcuc/108:10:111
2136201991815,2cyclictest2325594-21kthreadcore10:10:171
2136201991814,3cyclictest2229545-21kworker/u8:1+flush-259:008:45:001
2136201991812,4cyclictest2442434-21latency_hist12:05:001
2136199991816,1cyclictest2359167-21kworker/u8:2+events_unbound11:00:240
2136199991815,2cyclictest2323302-21apt-get10:10:110
2136199991814,3cyclictest2426611-21kworker/u8:2+events_unbound12:20:200
2136209991714,2cyclictest3282653-21kworker/u8:1+events_unbound07:34:523
2136209991714,2cyclictest2426611-21kworker/u8:2+flush-259:012:00:273
2136209991714,2cyclictest2296724-21kworker/u8:0+events_unbound10:23:583
2136209991714,2cyclictest2131239-21kworker/u8:0+flush-259:008:00:063
2136209991714,2cyclictest2131239-21kworker/u8:0+events_unbound07:44:573
2136205991715,1cyclictest0-21swapper/208:40:002
2136205991714,2cyclictest2401065-21kworker/u8:0+events_unbound11:30:012
2136205991714,2cyclictest2234881-21kworker/u8:2+flush-259:008:45:202
2136205991713,3cyclictest1415264-21kworker/u8:2+flush-259:007:10:242
2136201991713,3cyclictest2458165-21/usr/sbin/munin12:20:121
2136201991713,2cyclictest2302353-21sh09:50:001
2136199991714,2cyclictest2296724-21kworker/u8:0+flush-259:010:15:160
2136199991714,2cyclictest2234881-21kworker/u8:2+flush-259:008:45:490
2136199991714,2cyclictest2198633-21kworker/u8:3+flush-259:008:25:140
2136199991713,3cyclictest2171675-21needreboot07:40:180
2136199991712,4cyclictest2202698-21ntpq08:10:190
2136199991712,3cyclictest2323046-21turbostat10:10:000
213620999163,6cyclictest50-21ksoftirqd/308:35:133
2136209991613,2cyclictest2296724-21kworker/u8:0+events_unbound10:29:573
2136209991613,2cyclictest2264755-21needreboot09:10:193
2136209991613,2cyclictest1415264-21kworker/u8:2+events_unbound07:10:263
2136205991613,2cyclictest2281656-21kworker/u8:1+flush-259:009:30:202
2136205991613,2cyclictest2281656-21kworker/u8:1+flush-259:009:30:202
2136205991613,2cyclictest2131239-21kworker/u8:0+events_unbound07:20:182
2136205991612,3cyclictest252-21systemd-journal10:29:552
2136205991612,3cyclictest2416302-21kworker/u8:1+flush-259:011:50:172
2136205991612,3cyclictest2416302-21kworker/u8:1+flush-259:011:50:162
2136205991612,3cyclictest2376072-21kworker/u8:0+flush-259:011:15:162
213620199169,1cyclictest0-21swapper/108:30:121
213620199163,3cyclictest301rcuc/110:05:111
213620199162,4cyclictest2212415-21cut08:20:181
213620199161,3cyclictest2182040-21ld07:50:201
2136201991613,2cyclictest2136184-21cyclictest09:20:001
2136201991613,1cyclictest191rcu_preempt07:49:451
2136199991614,1cyclictest60350irq/132-enp0s29f1:rx-012:30:130
2136199991613,2cyclictest2442454-21kworker/u8:1+flush-259:012:05:180
2136199991613,2cyclictest2219231-21kworker/u8:0+events_unbound08:40:150
2136199991613,2cyclictest1415264-21kworker/u8:2+flush-259:007:15:160
2136199991612,3cyclictest2359167-21kworker/u8:2+flush-259:010:45:210
2136199991610,4cyclictest2437719-21diskstats12:00:130
2136209991512,2cyclictest3282653-21kworker/u8:1+events_unbound07:53:423
2136209991512,2cyclictest2401065-21kworker/u8:0+events_unbound11:35:203
2136209991512,2cyclictest2376072-21kworker/u8:0+flush-259:011:15:343
2136209991512,1cyclictest0-21swapper/310:40:003
2136209991511,3cyclictest2287044-21kworker/u8:3+flush-259:010:04:433
2136205991512,2cyclictest2349095-21apt-key10:35:012
2136205991512,2cyclictest2255355-21kworker/u8:4+flush-259:010:20:182
2136205991511,3cyclictest2359167-21kworker/u8:2+events_unbound10:45:222
2136205991511,3cyclictest2255355-21kworker/u8:4+flush-259:009:40:002
213620199158,4cyclictest2328418-21/usr/sbin/munin10:15:201
213620199158,1cyclictest0-21swapper/109:05:001
213620199155,7cyclictest0-21swapper/107:40:211
213620199153,1cyclictest2384542-21ntpq11:05:281
213620199151,4cyclictest2368992-21ntpq10:50:211
213620199151,4cyclictest2275513-21chrt09:20:201
2136201991511,3cyclictest2302611-21apt-get09:50:121
213620199150,5cyclictest2472448-21ld12:30:211
2136199991514,1cyclictest60350irq/132-enp0s29f1:rx-012:12:440
2136199991514,1cyclictest60350irq/132-enp0s29f1:rx-007:45:140
2136199991513,1cyclictest60350irq/132-enp0s29f1:rx-011:35:150
2136199991513,1cyclictest60350irq/132-enp0s29f1:rx-011:05:150
2136199991513,1cyclictest60350irq/132-enp0s29f1:rx-010:55:160
2136199991513,1cyclictest60350irq/132-enp0s29f1:rx-009:25:120
2136199991513,1cyclictest60350irq/132-enp0s29f1:rx-009:15:260
2136199991513,1cyclictest60350irq/132-enp0s29f1:rx-007:20:120
2136199991513,1cyclictest2436889-21http12:00:010
2136199991512,2cyclictest60350irq/132-enp0s29f1:rx-012:39:100
2136199991512,2cyclictest60350irq/132-enp0s29f1:rx-011:10:200
2136199991512,2cyclictest60350irq/132-enp0s29f1:rx-009:30:150
2136199991512,2cyclictest60350irq/132-enp0s29f1:rx-009:30:150
2136199991512,2cyclictest60350irq/132-enp0s29f1:rx-009:05:110
2136199991512,2cyclictest60350irq/132-enp0s29f1:rx-008:30:220
2136199991512,2cyclictest60350irq/132-enp0s29f1:rx-008:05:310
2136199991512,2cyclictest60350irq/132-enp0s29f1:rx-007:50:250
2136199991512,2cyclictest2436789-21kworker/u8:0+events_unbound12:25:140
2136199991512,2cyclictest2198633-21kworker/u8:3+flush-259:008:20:150
2136199991511,3cyclictest60350irq/132-enp0s29f1:rx-011:20:220
2136199991511,3cyclictest2328220-21kworker/u8:1+flush-259:011:18:070
2136199991511,3cyclictest2307542-21kworker/u8:1+flush-259:009:55:220
2136199991510,3cyclictest2349424-21munin-node10:35:100
2136209991412,2cyclictest3282653-21kworker/u8:1+events_unbound08:10:233
2136209991411,2cyclictest2401065-21kworker/u8:0+events_unbound11:33:073
2136209991411,2cyclictest2287044-21kworker/u8:3+flush-259:010:05:053
2136209991411,2cyclictest2250157-21sh09:00:003
2136209991410,3cyclictest2431994-21apt-get11:55:103
2136209991410,3cyclictest2151126-21ntp_kernel_pll_07:20:203
2136209991410,2cyclictest2255355-21kworker/u8:4+events_unbound09:38:413
2136205991411,2cyclictest2401065-21kworker/u8:0+flush-259:011:45:162
2136205991410,3cyclictest2229545-21kworker/u8:1+flush-259:008:50:502
213620199148,4cyclictest2452756-21latency_hist12:15:011
213620199146,3cyclictest0-21swapper/111:29:141
213620199143,3cyclictest32-21ksoftirqd/110:30:021
213620199142,2cyclictest32-21ksoftirqd/109:55:261
213620199142,2cyclictest0-21swapper/108:05:191
213620199141,4cyclictest2421459-21idleruntime-cro11:45:001
213620199141,2cyclictest2358628-21ntpq10:40:221
213620199141,2cyclictest2337578-21ld10:20:211
2136201991412,1cyclictest3282653-21kworker/u8:1+events_unbound07:12:171
2136201991412,1cyclictest2193171-21fwupd08:04:281
2136201991412,1cyclictest191rcu_preempt11:54:221
2136201991412,1cyclictest191rcu_preempt11:54:211
2136201991410,1cyclictest191rcu_preempt07:29:551
213620199140,3cyclictest2363799-21gcc10:45:221
213619999148,5cyclictest2416554-21apt-get11:40:100
2136199991412,1cyclictest60350irq/132-enp0s29f1:rx-008:56:560
2136199991412,1cyclictest60350irq/132-enp0s29f1:rx-008:50:450
2136199991411,2cyclictest60350irq/132-enp0s29f1:rx-011:30:130
2136199991411,2cyclictest60350irq/132-enp0s29f1:rx-011:26:160
2136199991411,2cyclictest60350irq/132-enp0s29f1:rx-009:35:050
2136199991411,2cyclictest2427055-21fschecks_count11:50:130
2136199991411,2cyclictest2427055-21fschecks_count11:50:130
2136199991411,2cyclictest2328220-21kworker/u8:1+flush-259:010:30:210
2136199991410,3cyclictest3282653-21kworker/u8:1+events_unbound07:25:180
2136199991410,3cyclictest2167518-21idleruntime-cro07:40:000
2136209991310,2cyclictest2376072-21kworker/u8:0+events_unbound11:01:453
2136209991310,2cyclictest2229545-21kworker/u8:1+flush-259:009:02:003
2136209991310,2cyclictest0-21swapper/310:10:203
213620599139,3cyclictest2406278-21apt-get11:30:112
2136205991311,1cyclictest2209516-21irqcore08:20:162
2136205991310,2cyclictest2183422-21fschecks_count07:55:132
2136205991310,2cyclictest2166514-21needreboot07:35:192
2136205991310,2cyclictest2131239-21kworker/u8:0+events_unbound07:25:162
213620199135,1cyclictest0-21swapper/110:30:161
213620199132,3cyclictest32-21ksoftirqd/112:25:191
213620199132,3cyclictest32-21ksoftirqd/108:35:211
213620199132,2cyclictest32-21ksoftirqd/112:05:191
213620199132,2cyclictest32-21ksoftirqd/111:20:181
213620199131,3cyclictest2457102-21ntpq12:15:211
213620199131,2cyclictest2353497-21ntpq10:35:211
2136201991311,1cyclictest301rcuc/107:36:491
213620199130,3cyclictest2161430-21ntpq07:30:201
213619999139,3cyclictest2255355-21kworker/u8:4+flush-259:009:44:020
2136199991311,1cyclictest60350irq/132-enp0s29f1:rx-012:15:180
2136199991311,1cyclictest60350irq/132-enp0s29f1:rx-009:53:130
2136199991310,2cyclictest60350irq/132-enp0s29f1:rx-011:45:080
2136199991310,2cyclictest60350irq/132-enp0s29f1:rx-009:20:250
2136199991310,2cyclictest60350irq/132-enp0s29f1:rx-007:57:480
213620999129,2cyclictest2426611-21kworker/u8:2+events_unbound12:16:333
213620999123,1cyclictest191rcu_preempt08:30:113
213620599129,2cyclictest3282653-21kworker/u8:1+events_unbound07:30:242
213620599129,2cyclictest2327168-21needreboot10:10:182
213620599129,2cyclictest2219591-21cpuspeed_turbos08:30:132
213620599129,2cyclictest2219197-21idleruntime-cro08:30:012
213620599128,3cyclictest2255355-21kworker/u8:4+events_unbound09:50:222
213620199129,2cyclictest2203965-21cpuspeed_turbos08:15:131
213620199129,1cyclictest191rcu_preempt11:17:401
213620199129,1cyclictest0-21swapper/107:20:231
213620199128,3cyclictest2401065-21kworker/u8:0+flush-259:011:34:591
213620199124,1cyclictest0-21swapper/108:55:131
213620199123,2cyclictest32-21ksoftirqd/109:30:171
213620199123,2cyclictest32-21ksoftirqd/109:30:161
213620199122,3cyclictest32-21ksoftirqd/109:35:191
213620199122,2cyclictest32-21ksoftirqd/108:45:111
213620199121,2cyclictest32-21ksoftirqd/109:40:221
213620199121,2cyclictest32-21ksoftirqd/108:50:121
213620199121,2cyclictest301rcuc/111:35:211
2136199991210,1cyclictest60350irq/132-enp0s29f1:rx-009:10:230
213620599118,2cyclictest2468870-21irqcore12:30:172
213620599118,2cyclictest2426611-21kworker/u8:2+flush-259:012:20:172
213620599117,3cyclictest2385393-21kworker/u8:3+flush-259:011:40:202
213620599117,3cyclictest2359717-21kworker/u8:3+flush-259:010:56:132
213620599117,2cyclictest2385358-21sh11:09:592
213620599105,3cyclictest2275687-21cat09:20:222
21362059997,1cyclictest0-21swapper/208:15:122
21362059995,2cyclictest2178282-21fschecks_count07:50:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional