You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-28 - 11:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Sun Apr 28, 2024 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3801668994336,5cyclictest253-21systemd-journal19:26:402
3801662994341,1cyclictest3822309-21lspci19:25:470
3801662994237,3cyclictest253-21systemd-journal21:55:010
3801668994036,3cyclictest4082037-21kworker/u8:3+events_unbound23:10:002
3801672993932,5cyclictest253-21systemd-journal19:25:003
3801665993935,3cyclictest3902569-21kworker/u8:1+flush-259:022:00:011
3801672993630,4cyclictest253-21systemd-journal20:06:363
3801668993630,4cyclictest4029889-21latency_hist22:25:002
3801665993628,5cyclictest4141382-21latency_hist23:40:001
3801662993529,4cyclictest4186281-21diskstats00:10:130
380166299350,32cyclictest1-21systemd19:25:010
3801668993430,3cyclictest3970732-21kworker/u8:2+flush-259:022:15:262
3801662993431,1cyclictest131-1kworker/0:1H+kblockd20:28:010
3801672993330,2cyclictest14302-21gpgv00:25:003
3801668993331,1cyclictest4074631-21kworker/u8:0+flush-259:023:00:002
3801668993226,4cyclictest4074350-21latency_hist22:55:002
3801662993226,4cyclictest4015017-21latency_hist22:15:000
3801665993125,4cyclictest4170972-21latency_hist00:00:011
3801668993026,3cyclictest3860742-21kworker/u8:4+events_unbound20:55:042
3801665993025,3cyclictest4007581-21latency_hist22:10:011
3801665992923,4cyclictest4044732-21latency_hist22:35:001
3801665992823,4cyclictest3925754-21gpgv21:15:001
3801665992823,4cyclictest253-21systemd-journal22:15:001
3801672992721,4cyclictest3926163-21cat21:15:003
3801668992723,3cyclictest74-21kswapd000:10:002
3801668992723,3cyclictest4074631-21kworker/u8:0+flush-259:023:05:002
3801665992622,3cyclictest74-21kswapd022:25:121
3801672992421,2cyclictest3941121-21kworker/u8:0+events_unbound21:30:013
3801668992420,3cyclictest3954601-21kworker/u8:3+flush-259:021:50:252
3801665992419,3cyclictest4111285-21apt-get23:20:021
3801665992418,4cyclictest4111290-21latency_hist23:20:001
3801665992414,8cyclictest3823875-21irqcore19:30:201
3801672992319,3cyclictest4141737-21apt-get23:40:123
3801672992318,3cyclictest516-21rs:main3
3801672992317,4cyclictest253-21systemd-journal23:47:203
3801668992317,4cyclictest3970984-21ssh21:45:132
3801668992316,5cyclictest253-21systemd-journal23:40:002
3801665992319,3cyclictest74-21kswapd023:05:001
3801665992319,3cyclictest3902569-21kworker/u8:1+flush-259:021:05:001
3801665992319,3cyclictest253-21systemd-journal21:21:351
3801665992319,3cyclictest14218-21gpgv00:25:001
3801662992317,4cyclictest3891630-21latency_hist20:45:000
3801668992219,2cyclictest4149073-21kworker/u8:0+events_unbound00:00:142
3801668992218,3cyclictest3860742-21kworker/u8:4+flush-259:020:35:312
3801665992211,9cyclictest0-21swapper/120:00:001
380166599220,20cyclictest3855013-21systemd20:06:371
3801662992218,3cyclictest3902569-21kworker/u8:1+flush-259:021:40:140
3801662992217,3cyclictest4037342-21latency_hist22:30:000
3801672992118,2cyclictest3875871-21fwupd20:28:013
3801668992118,2cyclictest74-21kswapd021:45:012
3801668992118,2cyclictest3860742-21kworker/u8:4+events_unbound20:28:042
3801668992117,3cyclictest74-21kswapd023:55:002
3801668992117,3cyclictest3764094-21kworker/u8:4+events_unbound19:15:132
3801668992117,2cyclictest0-21swapper/222:45:002
3801665992116,3cyclictest253-21systemd-journal19:25:561
3801665992110,9cyclictest0-21swapper/121:05:211
3801662992117,2cyclictest3955704-21munin-run21:35:010
3801672992017,2cyclictest74-21kswapd022:00:113
3801672992017,2cyclictest4096398-21ssh23:09:483
3801672992016,3cyclictest508-21systemd-logind23:20:003
3801668992017,2cyclictest4103968-21kworker/u8:2+flush-259:023:22:052
3801668992017,2cyclictest3860742-21kworker/u8:4+events_unbound20:19:102
3801668992016,3cyclictest3791265-21kworker/u8:1+events_unbound19:15:012
3801668992015,3cyclictest3901939-21gpgv20:55:002
380166599201,5cyclictest253-21systemd-journal23:00:001
3801665992015,3cyclictest3956002-21/usr/sbin/munin21:35:301
380166599200,1cyclictest3881326-21turbostat.cron20:39:551
3801662992017,2cyclictest2904-21cron20:14:590
3801662992016,3cyclictest3933766-21apt-get21:20:090
380166299201,17cyclictest516-21rs:main0
380166299201,17cyclictest253-21systemd-journal19:15:590
3801672991916,2cyclictest3861187-21irqcore20:15:183
3801668991913,4cyclictest3955671-21latency_hist21:35:012
3801668991912,5cyclictest3828382-21latency_hist19:35:012
380166899190,13cyclictest3940856-21idleruntime-cro21:25:002
380166899190,12cyclictest29349-21cron00:35:002
380166599199,8cyclictest0-21swapper/120:00:291
3801662991914,4cyclictest4178733-21apt-get00:05:120
380166299191,3cyclictest4119132-21cron23:25:000
3801672991815,2cyclictest74-21kswapd022:00:013
3801672991815,2cyclictest3977823-21ssh21:49:563
3801672991814,3cyclictest74-21kswapd021:35:003
3801672991813,3cyclictest4089165-21latency_hist23:05:003
3801668991815,2cyclictest3902569-21kworker/u8:1+flush-259:022:00:182
3801668991814,3cyclictest0-21swapper/223:20:012
3801668991813,4cyclictest3876118-21kworker/u8:5+flush-259:020:45:102
3801668991812,4cyclictest3961624-21ssh21:37:192
3801668991812,4cyclictest3907727-21/usr/sbin/munin21:00:192
380166599188,1cyclictest4045209-21cpuspeed_turbos22:35:121
380166599182,5cyclictest521-21gdbus20:25:011
3801665991814,3cyclictest4066191-21ssh22:48:321
3801662991813,4cyclictest3912651-21latency_hist21:05:010
3801662991813,3cyclictest3896757-21latency_hist20:50:010
3801672991714,2cyclictest74-21kswapd023:15:003
3801672991714,2cyclictest74-21kswapd023:15:003
3801672991714,2cyclictest74-21kswapd022:40:003
3801672991713,3cyclictest253-21systemd-journal21:14:553
3801672991712,3cyclictest36779-21latency_hist00:40:013
3801668991715,2cyclictest3855373-21kworker/u8:3+events_unbound20:10:272
3801668991715,1cyclictest0-21swapper/223:15:002
3801668991715,1cyclictest0-21swapper/223:15:002
3801668991714,2cyclictest3902569-21kworker/u8:1+events_unbound22:05:282
380166599175,4cyclictest3954345-21ssh21:32:321
3801665991715,1cyclictest516-21rs:main1
3801665991713,3cyclictest3976889-21ssh21:48:091
380166599170,7cyclictest3896826-21cat20:50:011
3801662991714,2cyclictest61150irq/132-enp0s29f1:rx-020:35:140
3801662991714,2cyclictest4170974-21turbostat.cron00:00:000
3801662991714,2cyclictest171ktimers/023:31:350
3801662991714,2cyclictest0-21swapper/020:20:000
3801662991712,3cyclictest3834560-21diskstats19:45:100
380167299162,2cyclictest191rcu_preempt00:15:253
3801672991614,2cyclictest74-21kswapd023:00:013
3801672991613,2cyclictest21898-21sh00:30:003
3801672991612,3cyclictest74-21kswapd000:05:013
380167299160,14cyclictest516-21rs:main3
3801668991613,2cyclictest4211-21kworker/u8:1+events_unbound00:35:252
3801668991613,2cyclictest4103968-21kworker/u8:2+flush-259:023:45:112
3801668991613,2cyclictest4103968-21kworker/u8:2+flush-259:023:30:372
3801668991613,2cyclictest3918500-21kworker/u8:2+events_unbound21:15:242
3801668991613,2cyclictest2904-21cron22:30:002
3801668991612,3cyclictest3839946-21kworker/u8:1+flush-259:020:10:002
3801668991611,3cyclictest4059960-21ssh22:45:162
3801665991612,3cyclictest4185213-21ssh00:09:001
3801662991614,1cyclictest61150irq/132-enp0s29f1:rx-023:50:080
3801662991614,1cyclictest61150irq/132-enp0s29f1:rx-023:44:030
3801662991614,1cyclictest61150irq/132-enp0s29f1:rx-023:37:170
3801662991614,1cyclictest61150irq/132-enp0s29f1:rx-023:15:360
3801662991614,1cyclictest61150irq/132-enp0s29f1:rx-023:08:400
3801662991614,1cyclictest61150irq/132-enp0s29f1:rx-022:50:460
3801662991614,1cyclictest61150irq/132-enp0s29f1:rx-021:49:580
3801662991614,1cyclictest61150irq/132-enp0s29f1:rx-000:01:560
3801662991613,2cyclictest61950irq/133-enp0s29f1:tx-021:55:230
3801662991613,2cyclictest61150irq/132-enp0s29f1:rx-023:27:480
3801662991613,2cyclictest61150irq/132-enp0s29f1:rx-023:10:300
3801662991613,2cyclictest61150irq/132-enp0s29f1:rx-023:10:300
3801662991613,2cyclictest61150irq/132-enp0s29f1:rx-023:00:170
3801662991613,2cyclictest61150irq/132-enp0s29f1:rx-022:57:190
3801662991613,2cyclictest61150irq/132-enp0s29f1:rx-022:37:370
3801662991613,2cyclictest61150irq/132-enp0s29f1:rx-022:20:140
3801662991613,2cyclictest61150irq/132-enp0s29f1:rx-022:02:010
3801662991613,2cyclictest61150irq/132-enp0s29f1:rx-021:15:110
3801662991613,2cyclictest4062334-21kthreadcore22:45:240
3801662991613,2cyclictest15338-21irqcore00:25:190
3801662991612,3cyclictest514-21in:imuxsock00:33:170
3801662991612,3cyclictest3941189-21apt-get21:25:100
3801672991513,2cyclictest491ktimers/323:36:363
3801672991513,1cyclictest0-21swapper/320:05:003
3801672991512,2cyclictest3891167-21unixbench_multi20:40:283
3801672991512,2cyclictest3865984-21apt-get20:20:123
3801672991512,2cyclictest3817370-21cpuspeed_turbos19:25:133
3801672991512,2cyclictest3806786-21kworker/u8:2+flush-259:019:30:183
3801672991512,2cyclictest3801945-21cpuspeed_turbos19:10:083
3801668991513,1cyclictest3902569-21kworker/u8:1+events_unbound21:25:132
3801668991512,2cyclictest4156209-21kworker/u8:1+events_unbound23:55:022
3801668991512,2cyclictest4103968-21kworker/u8:2+flush-259:023:40:112
3801668991511,3cyclictest4043193-21kworker/u8:1+flush-259:022:35:282
3801668991511,3cyclictest3923776-21ssh21:11:312
3801668991510,2cyclictest4015061-21kworker/u8:4+flush-259:022:30:202
380166599158,1cyclictest4082037-21kworker/u8:3+xprtiod00:35:511
380166599156,2cyclictest40203752sleep122:16:031
380166599156,1cyclictest3902987-21ethtool20:55:301
380166599156,1cyclictest311ktimers/120:40:121
380166599155,4cyclictest3834659-21fschecks_time19:45:101
3801665991512,2cyclictest4163569-21cron23:55:001
3801665991512,2cyclictest0-21swapper/123:30:001
3801665991511,2cyclictest3833502-21apt19:40:011
380166299159,4cyclictest253-21systemd-journal20:55:000
3801662991513,1cyclictest61150irq/132-enp0s29f1:rx-022:33:320
3801662991513,1cyclictest61150irq/132-enp0s29f1:rx-022:17:080
3801662991513,1cyclictest61150irq/132-enp0s29f1:rx-022:05:510
3801662991513,1cyclictest61150irq/132-enp0s29f1:rx-000:35:270
3801662991513,1cyclictest61150irq/132-enp0s29f1:rx-000:17:300
3801662991513,1cyclictest171ktimers/021:37:530
3801662991513,1cyclictest0-21swapper/021:10:010
3801662991512,2cyclictest61150irq/132-enp0s29f1:rx-021:12:200
3801662991512,2cyclictest4171004-21kworker/u8:2+flush-259:000:20:180
3801662991511,3cyclictest3907497-21latency_hist21:00:000
3801662991510,4cyclictest3833499-21latency_hist19:40:020
3801672991411,2cyclictest3896777-21kworker/u8:0+flush-259:021:00:253
3801672991410,3cyclictest4022227-21ssh22:19:333
380166899149,4cyclictest74-21kswapd000:25:002
3801668991412,1cyclictest4149073-21kworker/u8:0+flush-259:000:12:162
3801668991412,1cyclictest0-21swapper/220:35:002
3801668991411,2cyclictest4082037-21kworker/u8:3+flush-259:000:30:022
3801668991411,2cyclictest3902569-21kworker/u8:1+events_unbound22:10:122
3801668991411,2cyclictest3896777-21kworker/u8:0+events_unbound21:05:152
3801668991410,2cyclictest4083-21unixbench-2d00:15:252
380166599148,1cyclictest311ktimers/123:14:061
380166599148,1cyclictest311ktimers/123:14:061
380166599148,1cyclictest311ktimers/122:00:211
380166599148,1cyclictest311ktimers/100:17:221
380166599148,1cyclictest311ktimers/100:12:351
380166599147,1cyclictest311ktimers/121:51:011
3801665991411,2cyclictest4171625-21fschecks_time00:00:161
3801665991410,3cyclictest4146394-21ssh23:40:321
3801662991413,1cyclictest61150irq/132-enp0s29f1:rx-022:41:060
3801662991412,1cyclictest61150irq/132-enp0s29f1:rx-019:35:000
3801662991411,2cyclictest3840156-21kworker/u8:4+events_unbound20:00:220
3801662991411,2cyclictest3839946-21kworker/u8:1+flush-259:019:55:140
380167299139,3cyclictest4161921-21ssh23:51:523
380167299139,3cyclictest3961355-21ssh21:36:473
3801672991311,1cyclictest3839911-21idleruntime-cro19:54:593
3801672991311,1cyclictest0-21swapper/300:10:133
3801672991310,2cyclictest4067394-21cpuspeed_turbos22:50:123
3801672991310,2cyclictest3860742-21kworker/u8:4+flush-259:020:50:333
380166899139,3cyclictest3876117-21kworker/u8:3+events_unbound20:41:302
3801668991310,2cyclictest3970732-21kworker/u8:2+events_unbound21:55:102
380166599136,5cyclictest4149338-21ssh23:45:131
3801665991311,1cyclictest311ktimers/122:24:231
3801665991310,2cyclictest4057074-21unixbench_multi22:40:301
3801665991310,2cyclictest0-21swapper/120:55:001
3801665991310,2cyclictest0-21swapper/120:25:151
3801665991310,1cyclictest0-21swapper/122:55:011
380166299137,4cyclictest253-21systemd-journal19:45:000
380167299129,2cyclictest3914351-21kthreadcore21:05:193
380167299129,2cyclictest3881811-21fschecks_time20:35:183
380167299129,1cyclictest0-21swapper/320:50:003
380167299128,3cyclictest3967684-21ssh21:40:253
380167299127,3cyclictest4128529-21cat23:30:213
380167299121,1cyclictest3918226-21diskmemload21:22:073
380167299121,1cyclictest191rcu_preempt23:57:473
380167299121,1cyclictest191rcu_preempt22:08:023
3801672991210,1cyclictest491ktimers/321:54:563
380167299120,1cyclictest191rcu_preempt22:10:063
380166899127,3cyclictest4119678-21ssh23:25:132
380166599128,3cyclictest3941121-21kworker/u8:0+flush-259:021:27:041
380166599126,4cyclictest14832-21/usr/sbin/munin00:25:161
380166599121,3cyclictest3811883-21awk19:20:001
3801665991210,1cyclictest3970474-21idleruntime-cro21:45:001
380166599120,11cyclictest0-21swapper/120:19:541
380166299129,2cyclictest3876581-21fschecks_time20:30:120
380166299127,3cyclictest253-21systemd-journal19:13:000
380167299119,1cyclictest0-21swapper/322:40:303
380167299119,1cyclictest0-21swapper/320:30:263
380167299119,1cyclictest0-21swapper/319:45:003
380167299118,2cyclictest491ktimers/300:30:153
380167299118,2cyclictest3840252-21cpuspeed_turbos19:55:093
380167299117,3cyclictest4028380-21ssh22:22:093
380167299111,2cyclictest191rcu_preempt22:49:583
380167299110,0cyclictest0-21swapper/322:30:243
380166899118,2cyclictest3791265-21kworker/u8:1+events_unbound19:20:162
380166899110,9cyclictest3839281-21munin-run19:49:592
380166599118,2cyclictest0-21swapper/121:15:111
380166299117,3cyclictest3834364-21kworker/u8:3+flush-259:019:55:000
380167299108,1cyclictest4123814-21sensors23:25:273
380167299108,1cyclictest3855988-21mii-tool20:10:213
380166899107,2cyclictest3866050-21cpuspeed_turbos20:20:152
380166899104,5cyclictest0-21swapper/220:00:222
380166599108,1cyclictest842-21snmpd20:30:111
380166599108,1cyclictest842-21snmpd00:31:021
380166599107,2cyclictest4066999-21kworker/u8:2+xprtiod23:05:061
380166599106,3cyclictest4132118-21ssh23:31:301
380166299105,4cyclictest3866647-21sed20:20:190
38016729997,1cyclictest491ktimers/320:55:193
38016729997,1cyclictest0-21swapper/319:45:083
38016729995,3cyclictest3828851-21cut19:35:143
38016689997,1cyclictest401ktimers/219:40:032
38016689997,1cyclictest0-21swapper/219:55:152
38016659996,2cyclictest3817030-21cron19:25:011
38016659995,2cyclictest3860505-21latency_hist20:15:011
38016689985,2cyclictest3800872-21kworker/u8:0+events_unbound19:51:022
38016659980,7cyclictest3801713-21cat19:10:001
38016659974,2cyclictest3839933-21/usr/sbin/munin19:55:001
38016659971,4cyclictest3834126-21apt19:45:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional