You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-22 - 03:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot4s.osadl.org (updated Sun Mar 22, 2026 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2505794994643,2cyclictest2818034-21kworker/u8:3+events_unbound00:35:141
2505794994643,2cyclictest2604939-21kworker/u8:2+events_unbound22:00:161
2505794994440,3cyclictest2687397-21kworker/u8:0+flush-259:023:10:211
2505792994337,4cyclictest2786665-21diskstats23:40:130
2505798994238,3cyclictest451428-21kworker/u8:0+events_unbound19:15:142
2505794994239,2cyclictest2797173-21kworker/u8:3+events_unbound23:55:121
2505794994137,3cyclictest2691593-21kworker/u8:1+events_unbound22:10:181
2505798994036,3cyclictest2691593-21kworker/u8:1+events_unbound22:20:132
2505794994037,2cyclictest2594589-21kworker/u8:3+events_unbound20:50:141
2505794993936,2cyclictest2521833-21kworker/u8:1+events_unbound19:35:251
2505794993834,3cyclictest2547417-21kworker/u8:2+events_unbound20:30:131
2505794993834,3cyclictest2547417-21kworker/u8:2+events_unbound20:30:131
2505794993734,2cyclictest2776150-21kworker/u8:1+events_unbound23:35:171
2505794993734,2cyclictest2594589-21kworker/u8:3+flush-259:020:45:201
2505794993734,2cyclictest2348667-21kworker/u8:1+flush-259:019:10:251
2505794993633,2cyclictest2646120-21kworker/u8:0+events_unbound21:30:171
2505794993633,2cyclictest2578969-21kworker/u8:3+flush-259:020:20:241
2505794993531,3cyclictest2594589-21kworker/u8:3+flush-259:020:55:211
2505794993431,2cyclictest2818034-21kworker/u8:3+flush-259:000:15:211
2505794993431,2cyclictest2776150-21kworker/u8:1+flush-259:000:00:211
2505794993431,2cyclictest2760693-21kworker/u8:2+events_unbound23:25:201
2505794993431,2cyclictest2719440-21kworker/u8:1+flush-259:022:50:251
2505794993431,2cyclictest2547417-21kworker/u8:2+flush-259:019:55:251
2505792993428,4cyclictest2822712-21ntpq00:10:230
2505798993329,3cyclictest2594589-21kworker/u8:3+flush-259:020:45:232
2505794993330,2cyclictest2776150-21kworker/u8:1+flush-259:023:40:191
2505794993330,2cyclictest2734695-21kworker/u8:3+events_unbound23:30:291
2505794993330,2cyclictest2604939-21kworker/u8:2+flush-259:021:55:151
2505794993330,2cyclictest2547417-21kworker/u8:2+events_unbound20:00:141
2505794993329,3cyclictest2776150-21kworker/u8:1+events_unbound23:50:221
2505794993329,3cyclictest2691593-21kworker/u8:1+flush-259:022:20:241
2505794993329,3cyclictest2666673-21kworker/u8:4+events_unbound21:50:131
2505794993326,2cyclictest2687397-21kworker/u8:0+flush-259:022:30:211
2505792993327,4cyclictest2818205-21cp00:10:010
2505798993229,2cyclictest2760693-21kworker/u8:2+flush-259:023:25:242
2505794993229,2cyclictest2687397-21kworker/u8:0+flush-259:022:40:211
2505794993229,2cyclictest2521833-21kworker/u8:1+events_unbound20:15:241
2505798993127,2cyclictest2594589-21kworker/u8:3+events_unbound21:25:152
2505794993129,2cyclictest2687397-21kworker/u8:0+flush-259:022:05:201
2505794993127,3cyclictest2818034-21kworker/u8:3+events_unbound00:10:191
2505794993124,3cyclictest2594589-21kworker/u8:3+flush-259:021:15:011
2505798993027,2cyclictest2666673-21kworker/u8:4+flush-259:021:50:212
2505798993025,3cyclictest2604939-21kworker/u8:2+events_unbound21:00:132
2505794993027,2cyclictest2661510-21kworker/u8:1+flush-259:021:45:161
2505794993025,2cyclictest2724610-21kworker/u8:2+flush-259:022:45:211
2505794993025,2cyclictest2724610-21kworker/u8:2+flush-259:022:45:211
2505792993025,4cyclictest2797182-21/usr/sbin/munin23:50:150
2505798992926,2cyclictest2348667-21kworker/u8:1+events_unbound19:10:182
2505798992924,3cyclictest2797173-21kworker/u8:3+flush-259:023:55:242
2505794992926,2cyclictest2766188-21kworker/u8:0+events_unbound23:20:261
2505794992926,2cyclictest2661510-21kworker/u8:1+flush-259:021:40:241
2505794992926,2cyclictest2547417-21kworker/u8:2+events_unbound20:25:181
2505794992926,2cyclictest2521833-21kworker/u8:1+flush-259:020:05:241
2505794992922,2cyclictest2516691-21kworker/u8:3+events_unbound19:20:131
2505792992925,3cyclictest2666673-21kworker/u8:4+flush-259:021:50:160
2505792992915,13cyclictest2770995-21/usr/sbin/munin23:25:130
2505798992825,2cyclictest2766188-21kworker/u8:0+events_unbound23:20:282
2505798992825,2cyclictest2521833-21kworker/u8:1+flush-259:020:10:202
2505798992824,3cyclictest2818034-21kworker/u8:3+flush-259:000:10:252
2505798992824,3cyclictest2760693-21kworker/u8:2+flush-259:023:15:202
2505794992825,2cyclictest2537340-21kworker/u8:0+events_unbound19:50:181
2505794992821,2cyclictest2594589-21kworker/u8:3+events_unbound20:40:161
2505794992819,3cyclictest2687397-21kworker/u8:0+events_unbound22:35:221
2505798992724,2cyclictest2588537-21kworker/u8:0+flush-259:020:30:242
2505798992724,2cyclictest2588537-21kworker/u8:0+flush-259:020:30:232
2505798992724,2cyclictest2578969-21kworker/u8:3+flush-259:020:20:212
2505794992724,2cyclictest2691593-21kworker/u8:1+events_unbound22:15:231
2505794992721,2cyclictest2760693-21kworker/u8:2+events_unbound23:15:151
2505798992622,3cyclictest2766188-21kworker/u8:0+events_unbound23:40:152
2505798992622,3cyclictest2594589-21kworker/u8:3+flush-259:020:50:222
2505798992622,3cyclictest2537340-21kworker/u8:0+events_unbound19:50:132
2505798992621,3cyclictest2776160-21/usr/sbin/munin23:30:142
2505794992624,1cyclictest2646120-21kworker/u8:0+events_unbound21:35:181
2505794992623,2cyclictest2521833-21kworker/u8:1+flush-259:019:30:231
2505792992619,5cyclictest2588553-21ntpq20:25:200
250579299261,24cyclictest1-21systemd21:35:560
2505792992611,13cyclictest2745208-21/usr/sbin/munin23:00:200
2505798992522,2cyclictest2724610-21kworker/u8:2+flush-259:022:45:222
2505798992522,2cyclictest2724610-21kworker/u8:2+flush-259:022:45:222
2505798992521,3cyclictest2604939-21kworker/u8:2+events_unbound21:05:192
2505794992523,2cyclictest2687397-21kworker/u8:0+events_unbound22:25:231
2505794992522,2cyclictest2818034-21kworker/u8:3+events_unbound00:30:171
2505794992521,3cyclictest2594589-21kworker/u8:3+flush-259:021:25:241
2505792992519,4cyclictest252-21systemd-journal00:25:000
2505802992420,3cyclictest2563539-21apt-get20:05:123
2505798992421,2cyclictest2604939-21kworker/u8:2+events_unbound21:55:232
2505798992420,3cyclictest2661510-21kworker/u8:1+events_unbound21:41:572
2505794992423,1cyclictest2521833-21kworker/u8:1+events_unbound19:40:111
2505794992421,2cyclictest2766188-21kworker/u8:0+flush-259:000:20:151
2505794992421,2cyclictest2521833-21kworker/u8:1+flush-259:020:40:001
2505794992415,3cyclictest2719440-21kworker/u8:1+events_unbound22:55:221
2505802992319,3cyclictest2521833-21kworker/u8:1+flush-259:020:20:163
2505798992320,2cyclictest2532147-21kworker/u8:4+events_unbound19:40:252
2505798992320,2cyclictest2521833-21kworker/u8:1+flush-259:019:25:252
2505798992319,3cyclictest2766188-21kworker/u8:0+events_unbound00:30:292
2505798992319,3cyclictest2687397-21kworker/u8:0+events_unbound23:00:122
2505798992319,3cyclictest2604939-21kworker/u8:2+flush-259:021:20:242
2505798992319,3cyclictest2521833-21kworker/u8:1+events_unbound19:30:122
2505794992320,2cyclictest2807683-21kworker/u8:2+flush-259:000:05:161
2505794992320,2cyclictest2594589-21kworker/u8:3+events_unbound21:20:171
2505794992320,2cyclictest2521833-21kworker/u8:1+events_unbound19:25:121
2505802992218,3cyclictest2818705-21apt-get00:10:123
2505798992220,1cyclictest2687397-21kworker/u8:0+events_unbound22:25:132
2505798992219,2cyclictest2776150-21kworker/u8:1+events_unbound00:05:212
2505798992219,2cyclictest2594589-21kworker/u8:3+events_unbound20:55:252
2505798992218,3cyclictest2766188-21kworker/u8:0+events_unbound00:29:572
2505798992218,3cyclictest2646120-21kworker/u8:0+events_unbound21:46:562
2505798992218,3cyclictest2521833-21kworker/u8:1+flush-259:019:55:162
2505792992216,4cyclictest252-21systemd-journal21:45:570
2505802992117,3cyclictest2656631-21apt-get21:35:093
2505802992115,4cyclictest2516226-21cp19:20:013
2505798992119,1cyclictest2776150-21kworker/u8:1+events_unbound23:50:122
2505798992119,1cyclictest2604939-21kworker/u8:2+events_unbound21:15:092
2505798992118,2cyclictest490-21dbus-daemon22:35:002
2505798992118,2cyclictest2604939-21kworker/u8:2+events_unbound21:10:142
2505798992118,2cyclictest2532147-21kworker/u8:4+events_unbound19:45:132
2505798992117,3cyclictest2594589-21kworker/u8:3+events_unbound20:35:182
2505798992116,3cyclictest2776150-21kworker/u8:1+events_unbound23:35:192
2505794992117,3cyclictest2532147-21kworker/u8:4+flush-259:019:45:261
250579299215,2cyclictest2755969-21if_err_enp0s30f23:10:150
250579299211,6cyclictest2666648-21latency_hist21:45:000
250579299211,6cyclictest2506003-21taskset19:10:130
250579299211,5cyclictest2552544-21idleruntime-cro19:55:000
2505792992115,4cyclictest2577881-21ntpq20:15:210
2505792992114,6cyclictest2702438-21ntpq22:15:230
250579299211,18cyclictest1-21systemd22:26:560
2505802992016,3cyclictest2521833-21kworker/u8:1+flush-259:019:55:163
2505798992018,2cyclictest2521833-21kworker/u8:1+events_unbound20:40:122
2505798992017,2cyclictest2734695-21kworker/u8:3+flush-259:023:05:212
2505798992017,2cyclictest2547417-21kworker/u8:2+flush-259:020:00:152
2505798992016,3cyclictest2521833-21kworker/u8:1+events_unbound20:05:582
2505794992017,2cyclictest2604939-21kworker/u8:2+events_unbound21:05:261
250579299206,13cyclictest2650323-21ntpq21:25:190
250579299204,2cyclictest2537078-21sh19:40:010
250579299202,4cyclictest2584292-21grep20:25:000
250579299201,6cyclictest2640935-21sh21:20:000
250579299201,5cyclictest2839000-21apt00:30:000
250579299201,5cyclictest2553223-21tr19:55:150
250579299201,5cyclictest2527446-21irqcore19:30:150
250579299201,5cyclictest252-21systemd-journal21:59:590
250579299201,4cyclictest252-21systemd-journal23:25:000
250579299201,4cyclictest2109693-21snmpd20:10:000
250579299201,3cyclictest490-21dbus-daemon00:40:000
250579299201,3cyclictest2739832-21apt22:55:000
250579299201,3cyclictest252-21systemd-journal20:05:000
2505792992013,5cyclictest2733779-21ntpq22:45:180
2505792992013,5cyclictest2733779-21ntpq22:45:170
2505792992013,5cyclictest2691631-21ntpq22:05:210
250579299201,2cyclictest2806944-21/usr/sbin/munin23:55:230
2505792992012,7cyclictest60350irq/132-enp0s29f1:rx-019:15:210
250579299200,5cyclictest2776597-21irqcore23:30:170
2505802991913,5cyclictest252-21systemd-journal20:27:453
2505798991916,2cyclictest2818034-21kworker/u8:3+events_unbound00:16:572
2505798991916,2cyclictest2687397-21kworker/u8:0+events_unbound23:10:152
2505798991916,2cyclictest2604939-21kworker/u8:2+flush-259:022:00:252
2505798991916,2cyclictest2594589-21kworker/u8:3+events_unbound22:08:032
2505794991916,2cyclictest2734695-21kworker/u8:3+flush-259:023:00:571
2505794991916,2cyclictest2604939-21kworker/u8:2+events_unbound21:00:061
250579299196,2cyclictest0-21swapper/021:10:190
250579299195,3cyclictest507-21in:imuxsock23:35:010
250579299194,2cyclictest2692898-21cpuspeed_turbos22:10:130
250579299193,2cyclictest2624659-21nvmesmart_nvme021:00:210
250579299192,2cyclictest26514992chrt21:30:110
250579299191,4cyclictest490-21dbus-daemon22:35:000
250579299191,4cyclictest2703614-21cpuspeed_turbos22:20:120
250579299191,4cyclictest2605259-21grep20:45:110
250579299191,4cyclictest2598687-21/usr/sbin/munin20:35:190
250579299191,4cyclictest2594610-21idleruntime-cro20:35:000
250579299191,4cyclictest2594610-21idleruntime-cro20:34:590
250579299191,4cyclictest2547403-21turbostat19:49:590
250579299191,3cyclictest2828720-21awk00:20:000
250579299191,3cyclictest2796512-21idleruntime-cro23:50:000
250579299191,3cyclictest2729510-21apt22:45:000
250579299191,3cyclictest2686408-21strings22:00:200
250579299191,3cyclictest252-21systemd-journal21:10:010
250579299191,2cyclictest2843274-21ntpq00:30:220
250579299190,5cyclictest2811931-21/usr/sbin/munin00:00:200
2505802991816,1cyclictest2687385-21idleruntime-cro22:04:593
2505802991814,3cyclictest252-21systemd-journal20:50:013
2505802991812,4cyclictest509-21rs:main3
2505802991811,5cyclictest252-21systemd-journal23:15:003
2505798991817,1cyclictest2473242-21kworker/u8:2+events_unbound19:40:002
2505798991815,2cyclictest2719440-21kworker/u8:1+flush-259:022:40:002
2505798991814,3cyclictest2687397-21kworker/u8:0+events_unbound22:58:582
2505794991815,2cyclictest451428-21kworker/u8:0+events_unbound19:15:151
2505794991814,3cyclictest2734695-21kworker/u8:3+flush-259:023:05:231
250579299183,3cyclictest2740388-21grep22:55:140
250579299181,4cyclictest2761127-21irqcore23:15:160
250579299181,4cyclictest2719597-21tr22:35:140
250579299181,3cyclictest2614348-21ntpq20:50:210
250579299181,3cyclictest252-21systemd-journal20:45:000
2505792991813,4cyclictest60350irq/132-enp0s29f1:rx-019:25:390
2505792991813,4cyclictest2537342-21apt-get19:40:100
250579299181,2cyclictest252-21systemd-journal23:10:000
250579299181,2cyclictest2521600-21awk19:25:010
250579299180,3cyclictest2615460-21chrt20:55:130
250580299173,2cyclictest2764766-21ntpq23:15:223
2505798991714,2cyclictest2818034-21kworker/u8:3+events_unbound00:35:212
2505798991713,3cyclictest2521833-21kworker/u8:1+events_unbound20:15:192
2505798991712,3cyclictest2651469-21/usr/sbin/munin21:30:172
2505794991714,2cyclictest2776150-21kworker/u8:1+events_unbound23:45:181
2505794991714,2cyclictest2625511-21kworker/u8:1+events_unbound21:17:571
250579299171,3cyclictest2568799-21tr20:10:130
2505792991710,5cyclictest2641198-21apt-get21:20:120
250580299169,4cyclictest0-21swapper/319:13:113
250580299166,6cyclictest2600181-21diskstats20:40:103
250580299163,8cyclictest50-21ksoftirqd/323:00:113
2505802991613,2cyclictest2551623-21ntp_kernel_err19:50:193
2505802991613,2cyclictest0-21swapper/322:10:003
2505802991612,3cyclictest2724610-21kworker/u8:2+flush-259:022:50:183
2505798991613,2cyclictest2687397-21kworker/u8:0+events_unbound22:40:142
2505798991613,2cyclictest2516691-21kworker/u8:3+events_unbound19:20:192
250580299158,4cyclictest0-21swapper/323:30:013
250580299157,4cyclictest0-21swapper/323:46:233
250580299156,4cyclictest0-21swapper/300:35:583
2505802991512,2cyclictest2818034-21kworker/u8:3+events_unbound00:26:473
2505802991511,1cyclictest191rcu_preempt23:40:213
2505802991510,1cyclictest191rcu_preempt20:50:203
2505798991512,2cyclictest2697921-21kworker/u8:2+events_unbound22:16:582
250580299147,5cyclictest2531921-21latency_hist19:35:023
250580299147,4cyclictest0-21swapper/322:22:183
250580299147,4cyclictest0-21swapper/321:28:133
250580299147,4cyclictest0-21swapper/300:05:083
250580299146,4cyclictest0-21swapper/321:09:383
250580299146,2cyclictest191rcu_preempt20:10:243
250580299143,7cyclictest50-21ksoftirqd/321:10:213
250580299142,2cyclictest50-21ksoftirqd/319:20:223
250580299141,3cyclictest50-21ksoftirqd/300:20:213
2505802991411,2cyclictest2828294-21unixbench_singl00:15:253
2505802991411,2cyclictest2797173-21kworker/u8:3+flush-259:000:00:003
2505802991411,1cyclictest191rcu_preempt22:14:453
2505798991412,2cyclictest2766188-21kworker/u8:0+events_unbound00:23:202
2505798991412,1cyclictest0-21swapper/222:15:002
2505798991411,2cyclictest2807683-21kworker/u8:2+flush-259:000:00:112
2505798991411,2cyclictest2547417-21kworker/u8:2+events_unbound20:25:282
2505798991410,3cyclictest2604939-21kworker/u8:2+flush-259:021:37:572
250580299135,1cyclictest191rcu_preempt22:25:163
250580299132,4cyclictest50-21ksoftirqd/322:35:143
250580299132,3cyclictest50-21ksoftirqd/320:55:213
250580299131,4cyclictest50-21ksoftirqd/321:30:203
250580299131,3cyclictest2725252-21sed22:40:153
250580299131,2cyclictest50-21ksoftirqd/321:42:533
2505802991311,1cyclictest491ktimers/323:35:213
2505802991311,1cyclictest491ktimers/322:30:143
2505802991311,1cyclictest0-21swapper/300:05:013
250580299130,3cyclictest2681383-21ntpq21:55:223
250580299130,3cyclictest2562450-21ntpq20:00:233
250580299130,2cyclictest2744080-21ntpq22:55:203
250579499139,3cyclictest2766188-21kworker/u8:0+events_unbound00:25:191
250580299129,2cyclictest2536641-21unixbench_singl19:35:253
250580299129,1cyclictest491ktimers/319:45:143
250580299122,3cyclictest50-21ksoftirqd/323:20:123
250580299121,4cyclictest50-21ksoftirqd/321:15:113
250580299121,3cyclictest50-21ksoftirqd/323:05:193
250580299121,3cyclictest50-21ksoftirqd/322:45:183
250580299121,3cyclictest50-21ksoftirqd/322:45:183
250580299121,3cyclictest50-21ksoftirqd/319:25:123
250580299121,2cyclictest50-21ksoftirqd/321:45:243
250580299121,2cyclictest50-21ksoftirqd/321:20:143
250580299120,8cyclictest50-21ksoftirqd/322:15:013
250580299120,3cyclictest50-21ksoftirqd/300:30:113
250580299120,2cyclictest2801174-21ntp_23:50:203
250580299120,2cyclictest2780231-21ntpq23:30:223
250580299120,2cyclictest2624577-21ntpq21:00:203
250580299120,2cyclictest2573864-21/usr/sbin/munin20:15:123
250579499128,3cyclictest2547417-21kworker/u8:2+events_unbound20:13:281
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional