You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-01 - 01:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: reduced 
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8s.osadl.org (updated Thu Nov 30, 2023 12:44:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
392581299732729,2cyclictest4121185-21turbostat10:49:561
392581299730727,2cyclictest3974479-21turbostat08:24:551
392582099727724,2cyclictest5921-21turbostat12:04:552
392582899720717,2cyclictest4188741-21turbostat11:54:553
392582899719712,6cyclictest3925483-21turbostat07:14:563
392581299718715,2cyclictest3951946-21turbostat07:59:551
392582099716713,2cyclictest4143727-21turbostat11:14:542
392582899713710,2cyclictest22916-21turbostat12:24:563
392582899711708,2cyclictest4041826-21turbostat09:34:553
392582899710708,1cyclictest4115572-21turbostat10:44:563
392582899710707,2cyclictest4126751-21turbostat10:54:563
392582899709706,2cyclictest4019286-21turbostat09:09:553
392582099709706,2cyclictest3939578-21turbostat07:34:562
392582899708706,1cyclictest4154912-21turbostat11:24:553
392582899708705,2cyclictest3934015-21turbostat07:29:563
392581299708705,2cyclictest3968875-21turbostat08:19:541
392582899705702,2cyclictest4070655-21turbostat10:04:553
392582899703700,2cyclictest4194280-21turbostat11:59:553
392581299701693,7cyclictest4053360-21turbostat09:44:551
392581299699697,1cyclictest3957769-21turbostat08:09:551
392582899697694,2cyclictest3980060-21turbostat08:29:553
392581299696693,2cyclictest3926139-21turbostat07:19:551
392581299695692,2cyclictest4104389-21turbostat10:34:561
392581299692683,6cyclictest4160505-21turbostat11:29:551
392582899691688,2cyclictest3980317-21turbostat08:34:553
392582899689686,2cyclictest4024869-21turbostat09:14:563
392582899689680,5cyclictest4030421-21turbostat09:19:563
392582099686677,5cyclictest4109980-21turbostat10:39:562
392580399686679,6cyclictest4132320-21turbostat10:59:560
392582899684681,2cyclictest4081795-21turbostat10:14:553
392581299684681,2cyclictest4076233-21turbostat10:09:551
392581299684681,2cyclictest4076233-21turbostat10:09:551
392581299684681,2cyclictest4036266-21turbostat09:29:561
392581299683676,6cyclictest4183151-21turbostat11:49:551
392582899682680,1cyclictest3985889-21turbostat08:39:543
392581299682680,1cyclictest4008167-21turbostat08:59:561
392582899681678,2cyclictest4035992-21turbostat09:24:553
392582099681678,2cyclictest4137891-21turbostat11:04:562
392582899679677,1cyclictest4171629-21turbostat11:39:553
392582099679677,1cyclictest4166058-21turbostat11:34:552
392582899677674,2cyclictest4065092-21turbostat09:59:553
392582899677674,2cyclictest4064832-21turbostat09:54:543
392582899677674,2cyclictest3940186-21turbostat07:39:553
392581299677675,1cyclictest11493-21turbostat12:09:551
392581299676673,2cyclictest3991475-21turbostat08:44:561
392581299675672,2cyclictest4177237-21turbostat11:44:551
392581299675672,2cyclictest4177237-21turbostat11:44:551
392582899673671,1cyclictest4098504-21turbostat10:29:563
392580399673671,1cyclictest4149339-21turbostat11:19:550
392582899672670,1cyclictest3957487-21turbostat08:04:553
392582899667664,2cyclictest3946118-21turbostat07:49:563
392581299666663,2cyclictest3933730-21turbostat07:24:561
392582899662660,1cyclictest3997066-21turbostat08:49:563
392582899662659,2cyclictest3945839-21turbostat07:44:563
392582899661658,1cyclictest17344-21turbostat12:19:563
392582899661655,5cyclictest3951683-21turbostat07:54:553
392581299661659,1cyclictest34094-21turbostat12:34:551
392582899658655,2cyclictest4138165-21turbostat11:09:553
392582899657655,1cyclictest3963307-21turbostat08:14:553
392581299657654,2cyclictest4013697-21turbostat09:04:551
392581299655652,2cyclictest4087322-21turbostat10:19:551
392582899654652,1cyclictest4092882-21turbostat10:24:553
392582899652646,3cyclictest171rcu_preempt12:39:553
392582899649646,2cyclictest4058897-21turbostat09:49:553
392582899647640,6cyclictest28524-21turbostat12:29:563
392582899644642,1cyclictest4047424-21turbostat09:39:543
392582899634630,3cyclictest3951667-21kworker/3:1+pm08:15:103
392581299633631,1cyclictest4002606-21turbostat08:54:561
392582899622619,2cyclictest11746-21turbostat12:14:553
392582899617610,6cyclictest4036231-21kworker/3:2+pm12:00:113
392581299598596,1cyclictest4155268-21kworker/1:0+pm11:50:101
392582099596593,2cyclictest4143752-21kworker/2:3+pm12:05:102
392582099584582,1cyclictest4092997-21kworker/2:2+pm11:05:112
392582099584580,3cyclictest4143752-21kworker/2:3+pm11:15:112
392582099582580,1cyclictest4092997-21kworker/2:2+pm10:40:102
392581299573571,1cyclictest4013671-21kworker/1:2+pm09:05:121
392581299563561,1cyclictest3946099-21kworker/1:2+pm08:10:111
392582899338336,1cyclictest3951667-21kworker/3:1+pm08:05:093
392582899336333,2cyclictest4036231-21kworker/3:2+pm09:40:103
392581299322320,1cyclictest4028545-21kworker/1:1+pm10:20:111
392581299322319,2cyclictest4155268-21kworker/1:0+pm11:30:121
392581299321318,2cyclictest4028545-21kworker/1:1+pm10:10:091
392581299313310,2cyclictest3996933-21kworker/1:1+pm08:45:101
392581299310307,2cyclictest3946099-21kworker/1:2+pm08:25:101
392580399307304,2cyclictest4126828-21kworker/0:1+pm11:00:120
392581299298295,2cyclictest4028545-21kworker/1:1+pm09:45:101
392582899293290,2cyclictest4036231-21kworker/3:2+pm11:10:113
392581299290288,1cyclictest4028545-21kworker/1:1+pm10:50:111
392582899283281,1cyclictest4036231-21kworker/3:2+pm09:25:113
392581299281279,1cyclictest4155268-21kworker/1:0+pm12:10:131
392582099280277,2cyclictest4143752-21kworker/2:3+pm11:35:122
392581299280277,2cyclictest4028545-21kworker/1:1+pm10:35:121
392582899265262,2cyclictest4036231-21kworker/3:2+pm10:15:103
392580399263260,2cyclictest4138125-21kworker/0:0+pm11:20:090
392582899259254,4cyclictest4036231-21kworker/3:2+pm11:40:113
392582899259254,4cyclictest4036231-21kworker/3:2+pm11:40:113
392582899257255,1cyclictest4036231-21kworker/3:2+pm10:55:123
392582899256253,2cyclictest3951667-21kworker/3:1+pm08:50:113
392581299253250,2cyclictest3890585-21kworker/1:1+pm07:25:131
392582899251249,1cyclictest4036231-21kworker/3:2+pm10:05:103
392582899251249,1cyclictest4036231-21kworker/3:2+pm10:05:093
392581299242240,1cyclictest33835-21kworker/1:1+pm12:35:131
392582899240238,1cyclictest3933710-21kworker/3:1+pm07:30:123
392582899237234,2cyclictest17315-21kworker/3:0+pm12:30:113
392582899231229,1cyclictest3310775-21kworker/3:2+pm07:15:133
392582899228225,2cyclictest3951667-21kworker/3:1+pm07:55:113
392581299222219,2cyclictest4028545-21kworker/1:1+pm09:30:111
392582899215212,2cyclictest4036231-21kworker/3:2+pm11:25:103
392582099205202,2cyclictest1756385-21kworker/2:2+pm07:35:122
392581299202200,1cyclictest3946099-21kworker/1:2+pm08:00:101
392582899193191,1cyclictest3951667-21kworker/3:1+pm08:40:123
392582899187185,1cyclictest4036231-21kworker/3:2+pm10:45:103
392582899187185,1cyclictest4036231-21kworker/3:2+pm10:30:103
3925820996764,2cyclictest4115612-21kworker/u8:0+events_unbound10:45:282
3925820996764,2cyclictest3963288-21kworker/u8:1+events_unbound08:15:012
3925820995855,2cyclictest4087341-21kworker/u8:2+flush-8:010:20:012
3925803995852,4cyclictest3926111-21kworker/u8:1+flush-8:007:25:260
3925803995750,4cyclictest3945830-21latency_hist07:40:010
3925820995650,4cyclictest4019270-21latency_hist09:05:002
3925820995347,4cyclictest3945884-21kworker/u8:1+flush-8:007:45:012
3925820995249,2cyclictest3980291-21kworker/u8:3+events_unbound08:45:262
3925820995249,2cyclictest22974-21kworker/u8:3+flush-8:012:25:372
3925820995248,2cyclictest4042082-21kworker/u8:0+flush-8:009:35:202
3925820995146,3cyclictest3940150-21kworker/u8:4+events_unbound08:05:342
3925820995047,2cyclictest4036285-21kworker/u8:1+events_unbound10:30:332
3925820995046,3cyclictest4183115-21kworker/u8:0+flush-8:012:11:152
3925820995046,3cyclictest3991516-21kworker/u8:1+flush-8:008:55:362
3925820994946,2cyclictest3896625-21kworker/u8:3+flush-8:007:15:322
3925803994944,3cyclictest4008416-21apt-get08:55:180
3925803994942,4cyclictest3974348-21systemd-journal09:18:280
3925820994844,3cyclictest4183115-21kworker/u8:0+flush-8:011:50:232
3925820994844,3cyclictest3980291-21kworker/u8:3+flush-8:008:40:342
3925803994842,4cyclictest3974348-21systemd-journal09:40:000
3925820994744,2cyclictest4189900-21kworker/u8:2+flush-8:011:55:272
3925820994744,2cyclictest3963288-21kworker/u8:1+events_unbound08:30:172
3925828994643,2cyclictest3926111-21kworker/u8:1+events_unbound07:25:003
3925803994642,3cyclictest3945884-21kworker/u8:1+events_unbound07:55:290
3925803994642,3cyclictest3896625-21kworker/u8:3+flush-8:007:15:250
3925803994640,4cyclictest3974348-21systemd-journal07:32:260
3925828994542,2cyclictest4143749-21kworker/u8:2+events_unbound11:30:303
3925820994542,2cyclictest4183115-21kworker/u8:0+flush-8:012:30:262
3925820994541,2cyclictest4087341-21kworker/u8:2+flush-8:010:20:322
3925812994541,3cyclictest4087341-21kworker/u8:2+events_unbound10:25:011
3925803994542,2cyclictest4115612-21kworker/u8:0+flush-8:011:05:260
3925828994442,2cyclictest4171649-21kworker/u8:4+flush-8:011:45:373
3925820994441,2cyclictest4137924-21kworker/u8:3+events_unbound11:20:272
3925820994440,3cyclictest3951704-21kworker/u8:2+events_unbound08:20:332
3925820994440,3cyclictest22974-21kworker/u8:3+events_unbound12:35:232
3925820994440,2cyclictest4087341-21kworker/u8:2+events_unbound10:25:262
3925812994440,3cyclictest3896625-21kworker/u8:3+events_unbound07:35:261
3925820994338,3cyclictest4013667-21kworker/u8:2+events_unbound09:25:222
3925812994339,3cyclictest4189900-21kworker/u8:2+flush-8:011:55:311
3925812994337,4cyclictest4025124-21apt-get09:10:191
3925820994238,3cyclictest4036285-21kworker/u8:1+flush-8:009:55:232
3925803994237,3cyclictest3957739-21kworker/u8:0+flush-8:008:15:360
3925820994138,2cyclictest4013667-21kworker/u8:2+events_unbound09:10:332
3925820994135,4cyclictest4177822-21appstreamcli11:40:012
3925820994135,4cyclictest4177822-21appstreamcli11:40:002
3925812994136,3cyclictest3974348-21systemd-journal07:10:011
3925803994135,4cyclictest4047412-21latency_hist09:35:000
3925803994134,4cyclictest39652-21latency_hist12:35:000
3925820994037,2cyclictest4042082-21kworker/u8:0+flush-8:010:05:342
3925820994037,2cyclictest4042082-21kworker/u8:0+flush-8:010:05:342
3925820994037,2cyclictest3896625-21kworker/u8:3+events_unbound07:29:272
3925820994036,3cyclictest4019506-21kworker/u8:1+flush-8:009:05:212
3925812994037,2cyclictest4013667-21kworker/u8:2+flush-8:009:15:231
3925803994035,4cyclictest4087341-21kworker/u8:2+events_unbound10:25:330
3925828993936,2cyclictest3980291-21kworker/u8:3+flush-8:008:55:293
3925828993933,4cyclictest4110235-21apt-get10:35:203
3925820993936,2cyclictest4036285-21kworker/u8:1+events_unbound10:00:262
3925820993935,3cyclictest3980291-21kworker/u8:3+events_unbound09:45:332
3925820993935,3cyclictest11967-21kworker/u8:1+events_unbound12:23:302
3925812993934,4cyclictest4137924-21kworker/u8:3+flush-8:011:05:191
3925803993933,4cyclictest3940150-21kworker/u8:4+flush-8:008:05:300
3925820993834,3cyclictest4013667-21kworker/u8:2+flush-8:009:15:292
3925812993834,3cyclictest3980291-21kworker/u8:3+flush-8:009:55:281
3925820993734,2cyclictest3980291-21kworker/u8:3+events_unbound09:42:302
3925820993733,3cyclictest3991516-21kworker/u8:1+flush-8:008:50:302
3925812993734,2cyclictest4013667-21kworker/u8:2+flush-8:009:25:011
3925812993730,4cyclictest3974348-21systemd-journal08:33:241
3925803993733,3cyclictest3980291-21kworker/u8:3+flush-8:008:30:230
3925803993729,5cyclictest45290-21latency_hist12:40:000
3925812993631,3cyclictest3940139-21appstreamcli07:30:011
3925803993632,3cyclictest4115612-21kworker/u8:0+flush-8:010:45:320
3925803993632,3cyclictest3980291-21kworker/u8:3+flush-8:009:45:270
3925803993630,4cyclictest3963559-21apt-get08:10:190
3925820993532,2cyclictest4171649-21kworker/u8:4+events_unbound11:45:312
3925820993531,3cyclictest4042082-21kworker/u8:0+flush-8:010:10:322
3925803993531,3cyclictest4019506-21kworker/u8:1+events_unbound09:05:250
3925828993431,2cyclictest4137924-21kworker/u8:3+events_unbound11:15:243
3925820993431,2cyclictest4143749-21kworker/u8:2+flush-8:011:25:202
3925820993430,3cyclictest11967-21kworker/u8:1+events_unbound12:15:292
3925803993426,5cyclictest3974348-21systemd-journal11:45:290
3925828993329,3cyclictest4115612-21kworker/u8:0+events_unbound11:01:303
3925820993330,2cyclictest4115612-21kworker/u8:0+events_unbound10:55:202
3925820993330,2cyclictest3957739-21kworker/u8:0+flush-8:008:15:332
3925812993329,3cyclictest4093130-21kworker/u8:3+events_unbound10:40:211
3925803993326,4cyclictest4132304-21latency_hist10:55:000
3925812993228,3cyclictest4042082-21kworker/u8:0+events_unbound09:35:331
3925803993227,4cyclictest4013667-21kworker/u8:2+flush-8:009:10:230
3925803993225,4cyclictest4194272-21latency_hist11:55:010
3925820993128,2cyclictest4036285-21kworker/u8:1+events_unbound09:30:382
3925820993128,2cyclictest3980291-21kworker/u8:3+events_unbound08:35:302
3925820993127,3cyclictest3945884-21kworker/u8:1+flush-8:007:45:262
3925803993127,3cyclictest4143749-21kworker/u8:2+flush-8:011:25:300
3925812993026,3cyclictest4137924-21kworker/u8:3+events_unbound11:20:391
3925820992924,4cyclictest3957752-21latency_hist08:05:002
3925812992925,3cyclictest4183115-21kworker/u8:0+events_unbound12:25:341
3925803992923,4cyclictest4098612-21gpgv10:25:000
3925803992923,4cyclictest3974348-21systemd-journal09:56:270
3925803992922,4cyclictest4058888-21latency_hist09:45:000
3925803992824,3cyclictest4036285-21kworker/u8:1+flush-8:010:00:280
3925812992724,2cyclictest4036285-21kworker/u8:1+events_unbound10:00:241
3925812992723,3cyclictest4137924-21kworker/u8:3+flush-8:011:15:391
3925812992723,3cyclictest11967-21kworker/u8:1+events_unbound12:20:311
3925803992723,3cyclictest4137924-21kworker/u8:3+events_unbound11:35:290
3925803992722,3cyclictest3974743-21apt-get08:20:210
3925812992622,3cyclictest4183115-21kworker/u8:0+events_unbound12:00:401
3925803992618,5cyclictest4115558-21latency_hist10:40:000
3925812992518,5cyclictest482-21rs:main1
3925803992523,1cyclictest3980291-21kworker/u8:3+events_unbound08:40:280
3925803992521,3cyclictest4166080-21kworker/u8:1+flush-8:011:40:230
3925803992521,3cyclictest4166080-21kworker/u8:1+flush-8:011:40:230
3925803992521,3cyclictest4137924-21kworker/u8:3+events_unbound11:30:340
3925803992419,3cyclictest3980291-21kworker/u8:3+events_unbound08:35:240
3925803992418,4cyclictest3974348-21systemd-journal12:10:010
3925803992418,4cyclictest11967-21kworker/u8:1+flush-8:012:15:010
3925820992320,2cyclictest3926111-21kworker/u8:1+events_unbound07:20:242
3925820992319,3cyclictest3940150-21kworker/u8:4+events_unbound07:51:272
392582099231,2cyclictest451-21dbus-daemon07:15:012
392582099231,20cyclictest3980298-21cron08:30:002
3925803992319,3cyclictest4036285-21kworker/u8:1+flush-8:010:05:220
3925803992319,3cyclictest4036285-21kworker/u8:1+flush-8:010:05:210
3925820992218,3cyclictest4064853-21kworker/u8:2+flush-8:009:55:002
3925812992218,3cyclictest4115612-21kworker/u8:0+events_unbound10:55:251
392581299221,19cyclictest3946375-21apt-get07:45:211
3925803992219,2cyclictest4013667-21kworker/u8:2+events_unbound09:00:300
3925803992218,3cyclictest4036285-21kworker/u8:1+events_unbound10:15:020
3925803992217,4cyclictest3980291-21kworker/u8:3+flush-8:008:50:310
3925803992217,3cyclictest4036285-21kworker/u8:1+flush-8:010:15:000
3925803992216,4cyclictest17598-21apt-get12:15:190
392580399220,21cyclictest3951942-21grep07:55:000
3925812992118,2cyclictest4149338-21sh11:15:001
3925803992118,2cyclictest0-21swapper/012:00:220
392580399211,19cyclictest0-21swapper/007:45:220
3925820992017,2cyclictest4121405-21kworker/u8:1+events_unbound10:50:222
3925820992016,3cyclictest3957474-21munin-plugin-st08:00:002
3925812992016,3cyclictest4137924-21kworker/u8:3+flush-8:011:35:351
3925812992016,2cyclictest0-21swapper/112:15:211
3925803992017,2cyclictest4002597-21cron08:50:010
3925803992016,3cyclictest4189900-21kworker/u8:2+events_unbound11:55:290
3925803992016,2cyclictest4036518-21apt-get09:25:140
3925828991913,4cyclictest3980044-21latency_hist08:25:013
392580399190,18cyclictest0-21swapper/012:20:200
3925828991815,2cyclictest4019274-21munin-plugin-st09:05:013
3925820991815,2cyclictest4013667-21kworker/u8:2+events_unbound09:24:302
392581299180,0cyclictest0-21swapper/111:00:011
392580399176,6cyclictest4109980-21turbostat10:35:010
3925803991716,1cyclictest4036226-21kworker/0:2+kacpid09:52:580
3925803991715,1cyclictest4036226-21kworker/0:2+kacpid09:24:070
3925803991713,3cyclictest4093130-21kworker/u8:3+flush-8:010:40:310
392581299160,16cyclictest0-21swapper/109:50:381
392581299160,16cyclictest0-21swapper/107:50:361
392581299160,16cyclictest0-21swapper/107:40:481
3925803991615,1cyclictest3940149-21kworker/0:2+kacpid07:44:070
3925803991612,3cyclictest3951704-21kworker/u8:2+flush-8:008:05:000
3925803991510,3cyclictest4183115-21kworker/u8:0+flush-8:012:25:310
3925803991411,2cyclictest0-21swapper/011:10:220
392582899129,2cyclictest4183115-21kworker/u8:0+events_unbound12:09:303
3925803991110,1cyclictest3970490-21kworker/0:2+kacpid08:28:500
3925803991010,0cyclictest3914660-21kworker/0:0+kacpid07:10:400
39258289996,1cyclictest481ktimers/307:10:003
39258039999,0cyclictest3914660-21kworker/0:0+kacpid07:20:010
39258129970,3cyclictest3925878-21sed07:10:011
39258039972,3cyclictest3925795-21latency_hist07:10:010
28269870,3rtkit-daemon0-21swapper/207:05:262
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional