You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-24 - 09:06

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Fri Apr 24, 2026 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3327556994540,3cyclictest3587190-21kworker/u8:1+flush-259:023:30:003
3327556994437,5cyclictest3363121-21ntpq19:40:223
3327556994035,3cyclictest3525397-21/usr/sbin/munin22:20:153
3327556993933,4cyclictest3571732-21/usr/sbin/munin23:05:203
3327548993933,4cyclictest3556076-21latency_hist22:50:001
3327551993730,6cyclictest3396052-21kworker/u8:4+flush-259:020:25:012
3327551993730,6cyclictest3396052-21kworker/u8:4+flush-259:020:25:012
3327551993729,6cyclictest3545782-21awk22:40:002
3327551993729,6cyclictest3545782-21awk22:40:002
3327556993632,3cyclictest3587190-21kworker/u8:1+flush-259:023:35:003
3327551993633,2cyclictest3412092-21kworker/u8:0+events_unbound22:10:002
3327548993631,4cyclictest3432039-21/usr/sbin/munin20:50:131
3327546993630,4cyclictest3581793-21latency_hist23:15:010
3327546993629,5cyclictest3420645-21ntpq20:35:200
3327556993428,4cyclictest3623264-21cp23:55:013
3327548993428,4cyclictest3623924-21diskstats23:55:141
3327556993227,4cyclictest3358728-21kworker/u8:1+flush-259:019:50:003
3327551993226,4cyclictest3525202-21kworker/u8:3+flush-259:022:30:002
3327548993226,5cyclictest3343125-21base6419:25:011
3327551993127,3cyclictest3374286-21kworker/u8:3+events_unbound20:10:152
3327551993126,3cyclictest3412092-21kworker/u8:0+flush-259:021:00:162
3327551993124,5cyclictest3342948-21latency_hist19:25:002
3327548993126,4cyclictest252-21systemd-journal19:25:261
3327556993025,4cyclictest3431824-21kworker/u8:1+flush-259:021:05:223
3327556993024,4cyclictest3540627-21latency_hist22:35:013
3327556993024,4cyclictest3525155-21latency_hist22:20:003
3327551993026,3cyclictest3525202-21kworker/u8:3+flush-259:022:55:152
3327551993026,3cyclictest3374286-21kworker/u8:3+flush-259:020:35:162
3327546993023,5cyclictest3427110-21diskstats20:45:140
3327546993023,5cyclictest3369130-21latency_hist19:50:010
3327551992925,3cyclictest3587190-21kworker/u8:1+flush-259:023:40:162
3327551992925,3cyclictest3358728-21kworker/u8:1+flush-259:019:50:162
3327548992924,3cyclictest3353933-21apt-get19:35:121
3327556992821,5cyclictest3504992-21diskstats22:00:133
3327551992824,3cyclictest3431824-21kworker/u8:1+events_unbound21:05:142
3327551992823,4cyclictest3332875-21kworker/u8:0+events_unbound19:45:252
3327551992822,4cyclictest3654637-21kworker/u8:2+flush-259:000:25:172
3327551992822,4cyclictest3431824-21kworker/u8:1+events_unbound22:20:272
3327551992822,4cyclictest3431805-21latency_hist20:50:012
3327556992721,5cyclictest3504345-21gpgv22:00:003
3327551992720,4cyclictest3406327-21/usr/sbin/munin20:25:232
3327548992723,3cyclictest3395366-21apt-get20:15:001
3327548992721,4cyclictest3566351-21latency_hist23:00:011
3327546992724,2cyclictest3493443-21kworker/u8:2+flush-259:022:05:000
3327556992622,3cyclictest3431824-21kworker/u8:1+events_unbound21:35:163
3327556992621,3cyclictest509-21rs:main3
3327556992621,3cyclictest3498583-21apt-get21:55:003
3327551992623,2cyclictest3374286-21kworker/u8:3+flush-259:020:00:232
3327551992622,3cyclictest3566378-21kworker/u8:4+flush-259:023:20:212
3327551992622,3cyclictest3525202-21kworker/u8:3+flush-259:023:00:152
3327551992621,4cyclictest3493443-21kworker/u8:2+events_unbound22:45:172
3327551992621,3cyclictest3412092-21kworker/u8:0+flush-259:021:15:012
3327548992619,5cyclictest3498583-21apt-get21:55:001
3327546992621,4cyclictest3493443-21kworker/u8:2+flush-259:023:05:010
3327546992621,3cyclictest252-21systemd-journal20:55:000
3327556992521,3cyclictest252-21systemd-journal21:00:013
3327556992518,5cyclictest3415529-21ntpq20:30:203
3327556992518,5cyclictest3332648-21latency_hist19:15:013
3327551992521,3cyclictest3374286-21kworker/u8:3+flush-259:020:30:212
3327548992521,3cyclictest3431824-21kworker/u8:1+flush-259:021:20:241
3327548992518,5cyclictest3374243-21latency_hist19:55:011
3327546992521,3cyclictest3587190-21kworker/u8:1+flush-259:000:00:250
3327546992510,2cyclictest3654395-21idleruntime-cro00:25:000
3327556992420,3cyclictest3396052-21kworker/u8:4+events_unbound20:25:223
3327551992420,3cyclictest3493443-21kworker/u8:2+flush-259:023:25:232
3327551992420,3cyclictest3431824-21kworker/u8:1+flush-259:021:20:212
3327551992420,3cyclictest3374286-21kworker/u8:3+events_unbound20:17:062
3327548992419,4cyclictest3587197-21/usr/sbin/munin23:20:131
3327548992419,4cyclictest3483384-21/usr/sbin/munin21:40:231
3327548992418,4cyclictest3592432-21apt-get23:25:111
3327546992419,4cyclictest252-21systemd-journal19:40:000
3327546992418,4cyclictest3353665-21latency_hist19:34:590
3327546992412,11cyclictest60350irq/132-enp0s29f1:rx-022:15:240
3327556992319,3cyclictest3633449-21kworker/u8:0+flush-259:000:10:213
3327556992319,3cyclictest3633449-21kworker/u8:0+events_unbound00:05:153
3327556992318,4cyclictest3396052-21kworker/u8:4+flush-259:020:50:013
3327551992319,3cyclictest507-21in:imuxsock19:15:012
3327551992319,3cyclictest3633449-21kworker/u8:0+events_unbound00:05:132
3327551992319,3cyclictest3602469-21idleruntime-cro23:35:012
3327551992317,4cyclictest3344024-21kworker/u8:3+events_unbound19:35:142
332754699239,2cyclictest3332161-21unixbench_multi19:10:250
3327546992319,3cyclictest3515156-21apt-get22:10:110
3327556992218,3cyclictest3358728-21kworker/u8:1+flush-259:019:50:213
3327556992217,3cyclictest3669886-21apt-get00:39:593
3327551992218,3cyclictest3643846-21kworker/u8:1+events_unbound00:15:202
3327551992217,4cyclictest3587190-21kworker/u8:1+flush-259:023:50:192
3327548992218,3cyclictest507-21in:imuxsock21:49:591
3327548992217,4cyclictest3493443-21kworker/u8:2+flush-259:023:40:001
332754699228,4cyclictest3338022-21/usr/sbin/munin19:20:180
3327546992217,3cyclictest3540884-21apt-get22:35:130
3327546992217,3cyclictest3540884-21apt-get22:35:130
3327546992216,4cyclictest3560317-21ntpq22:50:190
3327556992114,5cyclictest3551322-21diskstats22:45:153
3327551992117,3cyclictest3426676-21idleruntime-cro20:45:002
3327551992117,3cyclictest3412092-21kworker/u8:0+flush-259:021:25:262
3327551992117,3cyclictest3358728-21kworker/u8:1+flush-259:019:40:252
3327551992117,2cyclictest0-21swapper/200:05:002
3327548992116,4cyclictest3633449-21kworker/u8:0+flush-259:000:10:271
3327548992116,4cyclictest3396052-21kworker/u8:4+events_unbound20:15:181
3327548992115,4cyclictest252-21systemd-journal22:53:261
3327556992017,2cyclictest3566378-21kworker/u8:4+events_unbound23:20:203
3327556992017,2cyclictest3406084-21sh20:25:003
3327556992017,2cyclictest3406084-21sh20:25:003
3327556992016,3cyclictest3643846-21kworker/u8:1+flush-259:000:15:153
3327551992017,2cyclictest3374286-21kworker/u8:3+flush-259:019:55:242
3327551992016,3cyclictest3525202-21kworker/u8:3+events_unbound22:40:162
3327551992016,3cyclictest3332875-21kworker/u8:0+events_unbound19:15:172
3327548992016,3cyclictest252-21systemd-journal21:07:241
3327548992015,3cyclictest252-21systemd-journal23:07:561
3327546992016,3cyclictest3654637-21kworker/u8:2+flush-259:000:35:280
3327546992014,4cyclictest3390160-21latency_hist20:10:010
3327546992013,5cyclictest3611899-21ntpq23:40:220
3327546992010,2cyclictest60350irq/132-enp0s29f1:rx-022:30:110
3327556991916,2cyclictest3525202-21kworker/u8:3+events_unbound22:35:243
3327556991916,2cyclictest3525202-21kworker/u8:3+events_unbound22:35:243
3327556991914,4cyclictest3623731-21kworker/u8:2+events_unbound00:00:153
3327551991916,2cyclictest3617961-21kworker/u8:3+events_unbound23:55:202
3327551991916,2cyclictest3412092-21kworker/u8:0+flush-259:020:50:162
3327551991915,3cyclictest3493443-21kworker/u8:2+flush-259:023:20:002
3327551991915,2cyclictest252-21systemd-journal00:24:592
3327551991914,3cyclictest3493443-21kworker/u8:2+events_unbound23:10:012
3327551991914,3cyclictest3431824-21kworker/u8:1+events_unbound22:00:262
3327551991913,4cyclictest3669885-21latency_hist00:40:002
3327548991915,3cyclictest3442341-21apt-get21:00:111
3327548991913,4cyclictest252-21systemd-journal00:31:171
3327548991912,5cyclictest3457488-21latency_hist21:15:011
3327546991916,2cyclictest3617961-21kworker/u8:3+flush-259:023:55:250
3327546991915,3cyclictest3412092-21kworker/u8:0+flush-259:021:25:270
3327556991814,3cyclictest3525202-21kworker/u8:3+flush-259:022:55:253
3327556991814,3cyclictest3431824-21kworker/u8:1+events_unbound21:25:013
3327556991814,3cyclictest3332875-21kworker/u8:0+events_unbound19:25:183
3327556991813,4cyclictest252-21systemd-journal20:17:003
3327556991810,7cyclictest3332875-21kworker/u8:0+flush-259:019:30:023
3327551991815,2cyclictest3498777-21latency_hist21:55:002
3327551991815,2cyclictest3431824-21kworker/u8:1+flush-259:022:20:002
3327551991815,2cyclictest3431824-21kworker/u8:1+flush-259:020:55:222
3327548991815,2cyclictest3332635-21idleruntime-cro19:15:011
3327548991814,3cyclictest3525202-21kworker/u8:3+flush-259:022:30:241
3327548991813,4cyclictest252-21systemd-journal20:04:591
3327546991814,3cyclictest3412092-21kworker/u8:0+flush-259:020:40:250
3327546991813,4cyclictest60350irq/132-enp0s29f1:rx-021:11:550
3327546991813,4cyclictest3396052-21kworker/u8:4+flush-259:020:20:210
3327546991813,4cyclictest3396052-21kworker/u8:4+flush-259:020:20:200
3327546991812,4cyclictest3478305-21apt-get21:35:120
3327556991713,3cyclictest3525202-21kworker/u8:3+events_unbound22:25:213
3327556991713,3cyclictest3384767-21gpgv20:05:013
3327556991713,3cyclictest3344024-21kworker/u8:3+flush-259:019:35:313
3327556991712,3cyclictest3576917-21/usr/sbin/munin23:10:153
3327551991714,2cyclictest3431824-21kworker/u8:1+events_unbound21:40:182
3327551991713,3cyclictest3643846-21kworker/u8:1+flush-259:000:30:202
3327551991713,3cyclictest3412092-21kworker/u8:0+flush-259:021:45:162
3327551991713,3cyclictest3332875-21kworker/u8:0+events_unbound19:26:272
3327551991713,2cyclictest3504141-21sh22:00:002
3327548991713,3cyclictest3412092-21kworker/u8:0+flush-259:021:15:271
3327548991711,4cyclictest3612755-21latency_hist23:45:011
3327546991713,2cyclictest0-21swapper/022:30:000
3327546991712,4cyclictest60350irq/132-enp0s29f1:rx-022:05:150
3327556991613,2cyclictest3655093-21irqcore00:25:163
3327556991613,2cyclictest3515578-21irqcore22:10:173
3327556991611,4cyclictest3412092-21kworker/u8:0+flush-259:021:20:003
3327556991611,3cyclictest252-21systemd-journal21:48:253
3327551991613,2cyclictest3525202-21kworker/u8:3+events_unbound22:35:002
3327551991612,3cyclictest3431824-21kworker/u8:1+events_unbound22:13:282
332754899169,5cyclictest3406078-21awk20:25:001
332754899169,5cyclictest3406078-21awk20:24:591
3327548991614,1cyclictest0-21swapper/120:00:001
3327548991612,3cyclictest507-21in:imuxsock19:49:591
3327548991612,2cyclictest3541490-21irqcore22:35:171
3327548991612,2cyclictest3541490-21irqcore22:35:171
3327548991610,4cyclictest3664964-21/usr/sbin/munin00:35:231
3327546991614,1cyclictest60350irq/132-enp0s29f1:rx-023:45:220
3327546991614,1cyclictest60350irq/132-enp0s29f1:rx-021:40:210
3327546991614,1cyclictest60350irq/132-enp0s29f1:rx-019:50:240
3327546991613,2cyclictest3504167-21systemd-run22:00:000
3327546991611,1cyclictest60350irq/132-enp0s29f1:rx-021:30:060
3327556991513,1cyclictest3617946-21idleruntime-cro23:50:003
3327556991513,1cyclictest0-21swapper/320:15:003
3327556991512,2cyclictest3472883-21sh21:30:013
3327556991511,3cyclictest3431824-21kworker/u8:1+events_unbound21:40:163
3327551991511,3cyclictest3633449-21kworker/u8:0+flush-259:000:10:172
3327551991511,3cyclictest3412092-21kworker/u8:0+events_unbound21:30:152
3327551991510,4cyclictest3431824-21kworker/u8:1+events_unbound21:35:182
332754899159,4cyclictest3535455-21latency_hist22:30:001
3327548991512,2cyclictest3587190-21kworker/u8:1+flush-259:023:30:171
3327548991512,2cyclictest3406774-21irqcore20:25:171
3327548991512,2cyclictest0-21swapper/121:40:011
3327548991511,3cyclictest3643846-21kworker/u8:1+flush-259:000:15:161
332754699159,4cyclictest3327483-21latency_hist19:10:010
3327546991513,2cyclictest60350irq/132-enp0s29f1:rx-020:55:460
3327546991513,1cyclictest60350irq/132-enp0s29f1:rx-023:15:200
3327546991513,1cyclictest60350irq/132-enp0s29f1:rx-022:40:180
3327546991513,1cyclictest60350irq/132-enp0s29f1:rx-021:18:400
3327546991513,1cyclictest60350irq/132-enp0s29f1:rx-020:27:380
3327546991512,2cyclictest60350irq/132-enp0s29f1:rx-021:50:270
3327546991512,2cyclictest60350irq/132-enp0s29f1:rx-021:20:130
3327546991512,2cyclictest60350irq/132-enp0s29f1:rx-020:00:230
3327546991512,2cyclictest60350irq/132-enp0s29f1:rx-019:55:140
3327546991512,2cyclictest60350irq/132-enp0s29f1:rx-019:15:110
3327546991510,3cyclictest252-21systemd-journal20:20:000
3327551991410,3cyclictest3412092-21kworker/u8:0+events_unbound21:15:222
3327551991410,3cyclictest3344024-21kworker/u8:3+events_unbound19:32:272
332754899149,3cyclictest3654671-21apt-get00:25:121
3327548991411,2cyclictest3478037-21idleruntime-cro21:35:011
3327548991411,2cyclictest0-21swapper/122:25:001
3327546991412,2cyclictest60350irq/132-enp0s29f1:rx-019:26:180
3327546991412,1cyclictest60350irq/132-enp0s29f1:rx-023:37:250
3327546991412,1cyclictest60350irq/132-enp0s29f1:rx-021:45:110
3327546991412,1cyclictest60350irq/132-enp0s29f1:rx-021:00:150
3327546991412,1cyclictest60350irq/132-enp0s29f1:rx-020:33:490
3327546991412,1cyclictest60350irq/132-enp0s29f1:rx-000:05:260
3327546991412,1cyclictest0-21swapper/023:00:000
3327546991411,2cyclictest60350irq/132-enp0s29f1:rx-000:15:180
3327546991411,2cyclictest60350irq/132-enp0s29f1:rx-000:10:190
3327546991411,2cyclictest3395350-21idleruntime-cro20:14:590
3327546991410,3cyclictest3587190-21kworker/u8:1+flush-259:023:30:170
3327546991410,2cyclictest3659575-21turbostat00:29:590
3327546991410,1cyclictest60350irq/132-enp0s29f1:rx-019:40:220
332755699138,4cyclictest3374286-21kworker/u8:3+flush-259:020:10:003
3327556991310,2cyclictest3608109-21fschecks_time23:40:143
332755199139,3cyclictest3616922-21needreboot23:45:182
3327551991310,2cyclictest3558624-21kthreadcore22:50:162
332754899139,3cyclictest3431824-21kworker/u8:1+flush-259:022:05:071
332754899139,3cyclictest3412092-21kworker/u8:0+flush-259:020:45:271
3327548991311,1cyclictest0-21swapper/120:30:151
3327546991312,1cyclictest60350irq/132-enp0s29f1:rx-000:30:250
3327546991311,1cyclictest60350irq/132-enp0s29f1:rx-023:25:110
3327546991311,1cyclictest60350irq/132-enp0s29f1:rx-022:49:590
3327546991310,2cyclictest60350irq/132-enp0s29f1:rx-023:21:100
3327546991310,2cyclictest60350irq/132-enp0s29f1:rx-022:20:250
332755699129,2cyclictest3510143-21fschecks_count22:05:133
332755699129,2cyclictest3493443-21kworker/u8:2+flush-259:022:50:153
332755199120,10cyclictest3581804-21awk23:15:002
332754899129,2cyclictest3359456-21irqcore19:40:161
332754899127,4cyclictest3363989-21kworker/u8:2+flush-259:020:05:161
332754899121,2cyclictest3623073-21awk23:55:001
332755699118,2cyclictest3566716-21cpuspeed_turbos23:00:123
332755699118,2cyclictest3374576-21cpuspeed_turbos19:55:133
332755699118,2cyclictest3332974-21cpuspeed_turbos19:15:133
332755699114,5cyclictest252-21systemd-journal23:55:023
332754899119,1cyclictest0-21swapper/119:34:591
332754899118,2cyclictest3582122-21cpuspeed_turbos23:15:131
332754899118,2cyclictest3493752-21cpuspeed_turbos21:50:141
332755699108,1cyclictest0-21swapper/320:44:593
332754899108,1cyclictest0-21swapper/120:35:151
332754899107,2cyclictest3515577-21irqcore22:10:171
332754899107,2cyclictest3468106-21cpuspeed_turbos21:25:141
33275569990,2cyclictest3664723-21latency_hist00:35:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional