You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-10 - 19:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Mon Nov 10, 2025 12:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3219550993532,2cyclictest3224900-21kworker/u8:2+events_unbound07:30:140
3219559993429,3cyclictest3224900-21kworker/u8:2+flush-259:007:20:232
3219550993330,2cyclictest3468163-21kworker/u8:1+flush-259:011:45:190
3219550993330,2cyclictest3421556-21kworker/u8:4+events_unbound10:35:250
3219566993227,3cyclictest3468163-21kworker/u8:1+flush-259:011:25:143
3219550993229,2cyclictest3181330-21kworker/u8:0+events_unbound07:10:240
3219550993228,3cyclictest3313683-21kworker/u8:1+events_unbound08:55:200
3219550993129,2cyclictest3277405-21kworker/u8:2+events_unbound10:00:250
3219550993027,2cyclictest3277405-21kworker/u8:2+flush-259:009:15:400
3219559992915,9cyclictest3530293-21idleruntime-cro12:20:002
3219550992927,2cyclictest3468163-21kworker/u8:1+events_unbound11:35:140
3219550992927,2cyclictest3410849-21kworker/u8:0+events_unbound10:40:180
3219550992926,2cyclictest3277405-21kworker/u8:2+flush-259:009:55:280
3219550992926,2cyclictest3261740-21kworker/u8:0+events_unbound07:50:000
3219559992824,3cyclictest3277405-21kworker/u8:2+flush-259:008:35:192
3219550992824,3cyclictest3261963-21kworker/u8:4+events_unbound07:50:290
3219552992716,10cyclictest3379756-21cat09:55:011
3219550992724,2cyclictest3431672-21kworker/u8:2+events_unbound11:00:230
3219550992724,2cyclictest3277405-21kworker/u8:2+events_unbound09:20:120
3219550992723,3cyclictest3416202-21kworker/u8:1+events_unbound10:30:010
3219550992722,3cyclictest2698871-21systemd-journal10:30:560
3219550992719,5cyclictest3000063-21rs:main0
3219566992622,3cyclictest3468163-21kworker/u8:1+flush-259:011:45:303
3219566992619,5cyclictest3342649-21latency_hist09:20:013
3219559992623,2cyclictest3245875-21kworker/u8:3+flush-259:008:05:252
3219559992621,3cyclictest3530054-21kworker/u8:2+flush-259:012:35:202
3219559992620,5cyclictest3426510-21latency_hist10:40:012
3219552992620,5cyclictest3219273-21latency_hist07:10:011
3219552992620,5cyclictest3219273-21latency_hist07:10:001
3219552992620,4cyclictest2698871-21systemd-journal12:01:381
3219550992623,2cyclictest3245875-21kworker/u8:3+events_unbound08:15:250
3219566992519,4cyclictest2698871-21systemd-journal09:10:003
3219559992521,3cyclictest3327340-21kworker/u8:4+flush-259:010:05:212
3219550992522,2cyclictest3504165-21kworker/u8:0+events_unbound11:55:260
3219550992522,2cyclictest3504165-21kworker/u8:0+events_unbound11:55:250
3219550992522,2cyclictest3327340-21kworker/u8:4+events_unbound10:20:110
3219559992421,2cyclictest3468163-21kworker/u8:1+events_unbound11:20:142
3219559992421,2cyclictest3458044-21kworker/u8:4+flush-259:011:30:262
3219559992419,3cyclictest3358281-21gpgv09:35:002
3219550992422,1cyclictest3261740-21kworker/u8:0+events_unbound08:20:170
3219550992421,2cyclictest3530054-21kworker/u8:2+events_unbound12:35:240
3219550992421,2cyclictest3494101-21kworker/u8:2+events_unbound12:01:120
3219550992421,2cyclictest3327340-21kworker/u8:4+events_unbound09:05:220
3219559992319,3cyclictest3327340-21kworker/u8:4+flush-259:009:50:262
3219559992319,3cyclictest3277405-21kworker/u8:2+flush-259:009:20:152
3219559992316,5cyclictest3358170-21apt-get09:35:022
3219559992316,5cyclictest3358170-21apt-get09:35:012
3219550992321,2cyclictest3530054-21kworker/u8:2+events_unbound12:19:390
3219550992319,3cyclictest3468163-21kworker/u8:1+events_unbound11:44:400
3219550992319,3cyclictest3245875-21kworker/u8:3+events_unbound08:05:280
3219550992318,4cyclictest2698871-21systemd-journal11:20:380
3219566992219,2cyclictest3358190-21kworker/u8:3+flush-259:009:45:173
3219566992215,5cyclictest3337487-21latency_hist09:15:003
3219559992219,2cyclictest3326041-21sh08:59:592
3219559992218,3cyclictest3235600-21kworker/u8:0+events_unbound07:40:022
3219550992219,2cyclictest3504165-21kworker/u8:0+events_unbound12:11:150
3219550992219,2cyclictest3468163-21kworker/u8:1+events_unbound11:29:390
3219550992219,2cyclictest3458044-21kworker/u8:4+events_unbound11:10:120
3219550992219,2cyclictest3245875-21kworker/u8:3+flush-259:008:10:210
3219566992117,2cyclictest3327340-21kworker/u8:4+flush-259:010:05:193
3219566992115,4cyclictest3519574-21apt-get12:10:003
3219566992115,4cyclictest3468163-21kworker/u8:1+flush-259:011:35:173
3219559992118,2cyclictest3458044-21kworker/u8:4+events_unbound11:47:382
3219559992118,1cyclictest0-21swapper/207:40:002
3219552992116,4cyclictest3447124-21apt-get11:00:001
3219550992118,2cyclictest3277405-21kworker/u8:2+events_unbound09:30:240
3219550992118,2cyclictest3277405-21kworker/u8:2+events_unbound08:25:270
3219550992118,2cyclictest3261740-21kworker/u8:0+events_unbound08:32:410
3219550992117,3cyclictest3272266-21kworker/u8:1+events_unbound08:02:570
3219550992116,3cyclictest3348080-21apt-get09:25:120
3219566992017,2cyclictest0-21swapper/311:15:003
3219566992013,5cyclictest3342886-21/usr/sbin/munin09:20:123
3219559992017,2cyclictest3224900-21kworker/u8:2+events_unbound07:25:292
3219559992016,3cyclictest3458044-21kworker/u8:4+events_unbound11:40:142
3219559992016,3cyclictest3416202-21kworker/u8:1+events_unbound11:05:162
3219559992016,3cyclictest3327340-21kworker/u8:4+events_unbound09:40:192
3219559992016,3cyclictest3277405-21kworker/u8:2+flush-259:008:35:002
3219559992016,3cyclictest3277405-21kworker/u8:2+events_unbound10:10:292
3219552992014,4cyclictest3261740-21kworker/u8:0+flush-259:007:55:201
3219552992011,8cyclictest3385278-21fschecks_time10:00:151
3219550992017,2cyclictest3458044-21kworker/u8:4+events_unbound11:50:210
3219566991916,2cyclictest3413599-21kthreadcore10:25:183
3219566991915,3cyclictest3353265-21apt-get09:30:123
3219559991916,2cyclictest3458044-21kworker/u8:4+events_unbound11:54:552
3219559991915,3cyclictest3514457-21kworker/u8:1+events_unbound12:24:392
3219559991915,3cyclictest3358190-21kworker/u8:3+events_unbound09:45:122
3219559991915,3cyclictest3277405-21kworker/u8:2+events_unbound08:25:502
3219550991916,2cyclictest3441958-21sh10:55:000
3219550991916,2cyclictest3431672-21kworker/u8:2+events_unbound10:55:210
3219550991916,2cyclictest3319886-21munin-plugin-st08:50:010
3219550991916,2cyclictest3255673-21kworker/u8:1+events_unbound07:44:400
3219550991915,3cyclictest3530054-21kworker/u8:2+events_unbound12:33:340
3219550991914,3cyclictest3400966-21/usr/sbin/munin10:15:160
3219550991913,4cyclictest3431896-21/usr/sbin/munin10:45:170
3219550991913,4cyclictest3378375-21ntpq09:50:240
3219566991812,4cyclictest3320512-21apt-get08:55:013
3219566991812,4cyclictest3308515-21latency_hist08:35:003
3219566991811,5cyclictest3441092-21ntpq10:50:243
3219559991814,3cyclictest3416202-21kworker/u8:1+events_unbound10:50:182
3219559991814,3cyclictest3327340-21kworker/u8:4+events_unbound09:55:242
3219559991814,3cyclictest3313683-21kworker/u8:1+events_unbound08:45:172
3219559991813,4cyclictest3277405-21kworker/u8:2+flush-259:010:20:242
321955299187,9cyclictest0-21swapper/111:40:261
3219552991816,1cyclictest0-21swapper/109:10:001
321955299180,4cyclictest3405618-21seq10:16:481
3219550991812,4cyclictest2698871-21systemd-journal09:38:400
3219550991812,4cyclictest2698871-21systemd-journal09:38:400
3219566991714,2cyclictest3447243-21latency_hist11:00:013
3219559991714,2cyclictest3514457-21kworker/u8:1+events_unbound12:05:222
3219559991714,2cyclictest3410849-21kworker/u8:0+flush-259:010:45:162
3219559991714,2cyclictest3313683-21kworker/u8:1+events_unbound09:25:142
3219559991713,3cyclictest3494101-21kworker/u8:2+events_unbound11:55:132
3219559991713,3cyclictest3494101-21kworker/u8:2+events_unbound11:55:132
3219559991713,3cyclictest3437025-21kworker/u8:3+events_unbound10:59:572
3219559991713,3cyclictest3410849-21kworker/u8:0+events_unbound10:40:582
3219559991713,3cyclictest3327340-21kworker/u8:4+events_unbound10:00:132
3219559991713,3cyclictest3320636-21gpgv08:55:002
3219559991711,4cyclictest3287533-21latency_hist08:15:012
321955299178,8cyclictest3520145-21gpgv12:10:061
321955299173,12cyclictest3277178-21turbostat08:05:001
3219552991714,2cyclictest2698871-21systemd-journal11:06:381
3219552991714,2cyclictest0-21swapper/109:20:001
321955299171,3cyclictest3509301-21cron12:00:001
321955299171,3cyclictest3509301-21cron12:00:001
321955299171,3cyclictest3320511-21dump-pmu-power08:55:001
321955299170,16cyclictest3352560-21sed09:25:251
321955299170,15cyclictest3379241-21latency_hist09:55:011
3219550991714,2cyclictest3458044-21kworker/u8:4+events_unbound11:30:270
3219550991711,4cyclictest3320517-21latency_hist08:54:590
321956699169,5cyclictest3529451-21ntpq12:15:233
321956699169,5cyclictest3251881-21diskstats07:40:183
3219566991612,3cyclictest3255673-21kworker/u8:1+flush-259:007:45:183
3219566991612,2cyclictest0-21swapper/312:00:003
3219566991612,2cyclictest0-21swapper/312:00:003
3219559991612,3cyclictest3327340-21kworker/u8:4+events_unbound09:05:122
321955299169,6cyclictest3518563-21needreboot12:05:211
321955299164,10cyclictest0-21swapper/107:39:591
321955299161,4cyclictest3332817-21fschecks_time09:10:171
3219552991614,1cyclictest3463668-21irqcore11:15:181
321955299161,3cyclictest3530989-21ps12:20:201
321955299161,3cyclictest3358170-21apt-get09:35:001
3219552991613,2cyclictest3309188-21irqcore08:35:181
3219552991612,3cyclictest3530054-21kworker/u8:2+events_unbound12:35:141
3219552991610,5cyclictest0-21swapper/111:10:251
3219552991610,4cyclictest3256595-21awk07:45:001
321955299160,5cyclictest3224026-21sshd07:10:221
321955299160,4cyclictest3503743-21unixbench_singl11:50:251
321955299160,4cyclictest3498997-21cron11:50:001
321955299160,4cyclictest3374102-21turbostat.cron09:50:001
321955299160,4cyclictest3364150-21cpuspeed_turbos09:40:161
321955299160,4cyclictest3363780-21turbostat.cron09:40:001
321955299160,4cyclictest3363780-21turbostat.cron09:40:001
321955299160,4cyclictest3326134-21latency_hist08:59:591
3219550991614,2cyclictest3458044-21kworker/u8:4+events_unbound11:15:200
3219550991613,2cyclictest60650irq/133-enp0s29f1:rx-011:06:530
3219550991613,2cyclictest3327340-21kworker/u8:4+events_unbound10:05:070
3219550991613,2cyclictest3188797-21kworker/u8:1+flush-259:007:20:390
3219550991612,3cyclictest3514457-21kworker/u8:1+events_unbound12:05:180
3219550991612,3cyclictest3327340-21kworker/u8:4+events_unbound09:14:470
321956699159,4cyclictest3245875-21kworker/u8:3+flush-259:008:15:183
3219566991510,3cyclictest3379273-21gpgv09:55:003
3219559991512,2cyclictest3277405-21kworker/u8:2+events_unbound10:25:142
3219559991512,2cyclictest3245875-21kworker/u8:3+events_unbound08:15:142
3219559991512,2cyclictest3181330-21kworker/u8:0+flush-259:007:10:202
3219559991511,3cyclictest3313683-21kworker/u8:1+events_unbound08:40:412
3219559991511,3cyclictest3245875-21kworker/u8:3+events_unbound08:24:412
3219559991511,3cyclictest3235600-21kworker/u8:0+events_unbound07:32:412
3219559991511,3cyclictest3224900-21kworker/u8:2+flush-259:007:15:252
321955299158,5cyclictest3447476-21cat11:00:011
321955299153,3cyclictest3435905-21ld10:45:231
321955299152,2cyclictest3287935-21cpuspeed_turbos08:15:131
321955299151,4cyclictest3277562-21cpuspeed_turbos08:05:141
321955299151,4cyclictest3261727-21cron07:49:591
321955299151,3cyclictest516-21gdbus07:55:011
321955299151,2cyclictest3286676-21ntpq08:10:241
321955299150,6cyclictest3541054-21uniq12:30:161
321955299150,5cyclictest3234670-21ntpq07:20:201
321955299150,4cyclictest3539792-21awk12:25:231
321955299150,4cyclictest3525840-21sort12:15:181
321955299150,4cyclictest3473958-21grep11:25:141
321955299150,4cyclictest3390800-21cpuspeed_turbos10:05:161
321955299150,4cyclictest3347826-21munin-plugin-st09:24:591
321955299150,4cyclictest3303450-21ls08:30:001
321955299150,4cyclictest3297816-21cron08:25:011
321955299150,4cyclictest3240982-21apt-get07:30:111
321955299150,4cyclictest3237751-21kthreadcore07:25:201
321955299150,3cyclictest3484099-21tune2fs11:35:141
321955299150,3cyclictest3478828-21expr11:30:161
321955299150,3cyclictest3406568-21irqcore10:20:151
321955299150,3cyclictest3230444-21basename07:20:001
3219550991513,1cyclictest0-21swapper/009:50:000
3219550991511,3cyclictest3261740-21kworker/u8:0+events_unbound07:56:130
3219550991510,3cyclictest3308941-21diskstats08:35:150
321956699149,3cyclictest3229643-21cut07:15:233
321956699148,4cyclictest3535682-21/usr/sbin/munin12:25:153
321956699148,4cyclictest3447554-21base6411:00:013
321956699146,6cyclictest2698871-21systemd-journal11:20:003
3219566991411,2cyclictest3320516-21sh08:55:003
3219566991411,2cyclictest3319860-21cron08:50:003
3219566991410,3cyclictest3494101-21kworker/u8:2+flush-259:012:00:173
3219566991410,3cyclictest2698871-21systemd-journal09:45:003
3219559991411,2cyclictest3504165-21kworker/u8:0+flush-259:012:10:202
3219559991411,2cyclictest3431672-21kworker/u8:2+events_unbound11:04:572
3219559991411,2cyclictest3416202-21kworker/u8:1+flush-259:010:30:262
3219559991411,2cyclictest3261963-21kworker/u8:4+events_unbound08:02:402
3219559991410,3cyclictest3535707-21apt-get12:25:122
3219559991410,3cyclictest3468163-21kworker/u8:1+flush-259:011:25:102
321955299141,3cyclictest3425644-21ntpq10:35:231
321955299141,3cyclictest3315078-21cpuspeed_turbos08:45:091
321955299141,2cyclictest0-21swapper/110:10:181
321955299140,4cyclictest3437173-21cpuspeed_turbos10:50:161
321955299140,4cyclictest3427193-21irqcore10:40:191
321955299140,4cyclictest3421277-21modprobe10:34:381
321955299140,4cyclictest3303647-21grep08:30:151
321955299140,3cyclictest516-21gdbus09:05:001
321955299140,3cyclictest3468396-21acpi11:20:121
321955299140,3cyclictest3411513-21fschecks_count10:25:141
321955299140,1cyclictest3314718-21awk08:45:001
321955099146,2cyclictest61-21kcompactd009:00:010
321955099144,1cyclictest191rcu_preempt09:40:200
3219550991412,2cyclictest3514457-21kworker/u8:1+events_unbound12:23:020
3219550991411,3cyclictest3277405-21kworker/u8:2+events_unbound08:44:250
321956699139,3cyclictest3530054-21kworker/u8:2+flush-259:012:30:273
321956699139,2cyclictest3314720-21cron08:45:013
321956699138,4cyclictest3478717-21apt-get11:30:143
321956699138,4cyclictest3277405-21kworker/u8:2+flush-259:010:15:003
321956699135,5cyclictest2698871-21systemd-journal08:00:013
321956699131,4cyclictest3301793-21kthreadcore08:25:193
3219566991310,2cyclictest0-21swapper/307:20:203
321955999139,3cyclictest3277405-21kworker/u8:2+events_unbound09:15:272
321955999138,3cyclictest3463220-21/usr/sbin/munin11:15:162
3219559991310,2cyclictest3458044-21kworker/u8:4+events_unbound11:10:182
321955099137,1cyclictest191rcu_preempt10:10:260
3219550991310,2cyclictest3514457-21kworker/u8:1+events_unbound12:25:010
321956699127,3cyclictest3416202-21kworker/u8:1+flush-259:010:45:003
321956699126,4cyclictest2698871-21systemd-journal10:50:013
321956699126,3cyclictest3426515-21awk10:40:003
321956699121,3cyclictest3457816-21latency_hist11:10:003
3219566991210,1cyclictest0-21swapper/309:25:243
3219566991210,1cyclictest0-21swapper/308:25:013
321955999129,2cyclictest3468163-21kworker/u8:1+flush-259:011:35:252
321955999129,2cyclictest3277405-21kworker/u8:2+events_unbound10:15:022
321955999129,2cyclictest3255673-21kworker/u8:1+events_unbound07:45:412
321955999128,3cyclictest3313683-21kworker/u8:1+flush-259:009:00:052
321956699118,2cyclictest3546120-21cpuspeed_turbos12:35:143
321956699118,2cyclictest3277509-21cpuspeed_turbos08:05:133
321956699118,1cyclictest0-21swapper/307:50:163
321956699115,4cyclictest3406129-21/usr/sbin/munin10:20:133
321955999119,1cyclictest0-21swapper/209:14:592
321955999118,2cyclictest3261740-21kworker/u8:0+events_unbound07:50:022
321956699108,1cyclictest0-21swapper/309:55:143
321956699108,1cyclictest0-21swapper/308:05:013
321956699107,2cyclictest3468477-21cpuspeed_turbos11:20:143
321956699107,2cyclictest0-21swapper/309:35:173
321956699107,2cyclictest0-21swapper/309:35:173
321956699107,2cyclictest0-21swapper/307:10:173
321956699101,1cyclictest0-21swapper/310:00:213
321955999104,5cyclictest3509728-21df12:00:142
32195669997,1cyclictest0-21swapper/312:10:153
32195669994,4cyclictest3402053-21cut10:15:183
32195669991,6cyclictest3287529-21sh08:15:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional