You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-04-21 - 06:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Mon Apr 21, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
135017994840,5cyclictest426114-21cp00:30:013
135001994639,5cyclictest398188-21latency_hist23:55:011
135011994337,4cyclictest260799-21latency_hist21:10:002
135017994237,3cyclictest134702-21kworker/u8:1+flush-259:019:25:003
135017993734,2cyclictest365264-21kworker/u8:0+flush-259:000:00:203
135017993734,2cyclictest272182-21kworker/u8:0+events_unbound21:40:243
135017993633,2cyclictest293754-21kworker/u8:2+flush-259:022:30:113
135017993632,3cyclictest359713-21kworker/u8:2+flush-259:023:15:213
135017993531,3cyclictest365264-21kworker/u8:0+events_unbound23:55:203
135017993531,2cyclictest134702-21kworker/u8:1+events_unbound20:15:213
135017993530,4cyclictest272182-21kworker/u8:0+events_unbound21:30:143
135017993528,4cyclictest3000860-21systemd-journal21:25:003
135011993529,4cyclictest240031-21latency_hist20:50:012
135011993528,5cyclictest332302-21latency_hist22:35:012
135001993530,3cyclictest326468-21latency_hist22:25:011
134991993529,4cyclictest177062-21gpgv19:50:010
134991993427,4cyclictest276848-21latency_hist21:25:010
135017993326,2cyclictest134702-21kworker/u8:1+events_unbound19:30:193
135011993328,3cyclictest304942-21base6422:00:022
135001993327,4cyclictest309361-21ntpq22:00:201
134991993329,2cyclictest229811-21kworker/u8:0+events_unbound20:45:200
135017993229,2cyclictest181706-21kworker/u8:4+events_unbound20:45:263
135017993226,4cyclictest145553-21latency_hist19:20:003
135001993229,2cyclictest134702-21kworker/u8:1+flush-259:020:02:561
135001993226,4cyclictest176898-21kworker/u8:0+flush-259:020:10:241
134991993229,2cyclictest260837-21kworker/u8:1+flush-259:021:15:210
135017993127,3cyclictest208696-21kworker/u8:3+flush-259:020:50:223
135017993127,3cyclictest181706-21kworker/u8:4+events_unbound21:00:003
134991993125,5cyclictest186834-21ntpq19:55:220
135017993027,2cyclictest359713-21kworker/u8:2+events_unbound23:20:253
135017993027,2cyclictest310910-21kworker/u8:1+flush-259:022:55:223
135017993026,3cyclictest426666-21kworker/u8:1+events_unbound00:30:143
135017993026,3cyclictest305281-21kworker/u8:3+flush-259:022:00:223
135017993026,3cyclictest181706-21kworker/u8:4+events_unbound20:30:003
135017993026,3cyclictest145405-21kworker/u8:2+flush-259:019:35:293
135011993024,4cyclictest3000860-21systemd-journal22:38:132
134991993025,3cyclictest134702-21kworker/u8:1+events_unbound19:15:180
134991993024,4cyclictest3000860-21systemd-journal00:30:130
134991993023,5cyclictest419259-21ntpq00:15:220
135017992926,2cyclictest181706-21kworker/u8:4+flush-259:020:05:003
135017992925,3cyclictest359713-21kworker/u8:2+events_unbound23:35:183
135017992925,3cyclictest150723-21kworker/u8:0+events_unbound19:25:243
135017992923,2cyclictest86386-21kworker/u8:3+flush-259:019:10:213
135017992824,3cyclictest332524-21kworker/u8:0+events_unbound23:00:163
135017992824,3cyclictest181706-21kworker/u8:4+events_unbound21:55:013
135011992822,5cyclictest370428-21latency_hist23:20:012
135017992724,2cyclictest219146-21kworker/u8:5+events_unbound20:35:153
135017992723,3cyclictest181706-21kworker/u8:4+events_unbound20:40:183
135011992720,4cyclictest310868-21latency_hist22:10:012
135001992721,4cyclictest398705-21base6423:55:001
135017992623,2cyclictest176898-21kworker/u8:0+flush-259:019:55:223
135017992623,2cyclictest156014-21kworker/u8:3+flush-259:019:40:263
135017992619,5cyclictest294160-21latency_hist21:50:003
135011992620,5cyclictest192785-21apt-get20:02:552
135011992620,4cyclictest140613-21apt-get19:15:132
135001992620,5cyclictest3000860-21systemd-journal19:45:001
135001992619,5cyclictest3000860-21systemd-journal21:51:131
13500199261,22cyclictest1-21systemd19:55:011
134991992621,3cyclictest3000860-21systemd-journal22:49:550
135017992523,1cyclictest359713-21kworker/u8:2+events_unbound23:35:003
135017992522,2cyclictest409727-21kworker/u8:5+flush-259:000:10:113
135017992522,2cyclictest382107-21kworker/u8:4+flush-259:023:50:213
135017992522,2cyclictest260837-21kworker/u8:1+flush-259:021:15:273
135011992522,2cyclictest365264-21kworker/u8:0+flush-259:000:00:002
135011992518,5cyclictest397280-21ntpq23:50:232
135001992519,4cyclictest235008-21/usr/sbin/munin20:45:161
135017992421,2cyclictest250672-21kworker/u8:2+flush-259:021:05:213
135017992420,3cyclictest272182-21kworker/u8:0+events_unbound21:35:243
135001992420,3cyclictest134702-21kworker/u8:1+flush-259:020:15:171
134991992419,4cyclictest70116-21kworker/u8:0+flush-259:019:10:170
134991992419,3cyclictest310910-21kworker/u8:1+flush-259:023:15:000
134991992417,6cyclictest166412-21latency_hist19:40:010
134991992416,5cyclictest240464-21diskstats20:50:180
135017992320,2cyclictest156014-21kworker/u8:3+events_unbound19:50:203
135017992320,2cyclictest134702-21kworker/u8:1+events_unbound19:45:123
135017992319,3cyclictest310910-21kworker/u8:1+events_unbound23:27:153
135017992319,3cyclictest310910-21kworker/u8:1+events_unbound23:27:153
135017992319,3cyclictest299393-21kworker/u8:0+flush-259:022:15:253
135017992319,3cyclictest250672-21kworker/u8:2+events_unbound21:10:143
135017992319,3cyclictest250672-21kworker/u8:2+events_unbound21:10:133
135017992317,4cyclictest219370-21/usr/sbin/munin20:30:173
135011992317,4cyclictest176856-21apt-get19:50:022
135001992318,4cyclictest208696-21kworker/u8:3+flush-259:020:50:221
135001992317,4cyclictest332994-21apt-get22:35:151
135017992219,2cyclictest229811-21kworker/u8:0+flush-259:021:00:163
135017992218,3cyclictest293754-21kworker/u8:2+events_unbound22:08:153
135011992218,3cyclictest145405-21kworker/u8:2+flush-259:019:20:262
135001992218,3cyclictest409727-21kworker/u8:5+flush-259:000:20:291
135001992217,4cyclictest359713-21kworker/u8:2+flush-259:023:40:181
135001992214,5cyclictest3000860-21systemd-journal19:40:011
134991992218,3cyclictest134702-21kworker/u8:1+flush-259:019:35:010
134991992217,4cyclictest365264-21kworker/u8:0+flush-259:000:00:180
134991992215,5cyclictest293098-21latency_hist21:45:000
134991992215,5cyclictest198700-21diskstats20:10:230
134991992214,5cyclictest135237-21base6419:10:010
135017992117,3cyclictest310910-21kworker/u8:1+events_unbound22:26:143
135017992117,3cyclictest181706-21kworker/u8:4+events_unbound20:10:203
135011992118,2cyclictest359713-21kworker/u8:2+flush-259:023:10:182
135011992114,5cyclictest266681-21/usr/sbin/munin21:15:172
135001992118,2cyclictest359713-21kworker/u8:2+events_unbound23:30:141
135001992117,3cyclictest332524-21kworker/u8:0+flush-259:023:00:181
135001992117,3cyclictest245485-21apt-get20:55:131
135001992117,3cyclictest134702-21kworker/u8:1+flush-259:019:30:001
135001992115,3cyclictest260837-21kworker/u8:1+events_unbound21:20:141
134991992116,3cyclictest310910-21kworker/u8:1+flush-259:022:20:180
134991992115,4cyclictest3000860-21systemd-journal23:10:000
135017992017,2cyclictest359713-21kworker/u8:2+events_unbound23:10:183
135017992017,2cyclictest293754-21kworker/u8:2+events_unbound21:55:193
135017992017,2cyclictest176898-21kworker/u8:0+events_unbound20:05:233
135017992016,3cyclictest409726-21kworker/u8:1+events_unbound00:22:153
135017992014,4cyclictest213054-21ntpq20:20:243
13501199205,6cyclictest250477-21cat21:00:002
13501199204,5cyclictest275988-21ntpq21:20:232
13500199201,17cyclictest1-21systemd22:52:121
135017991916,2cyclictest431880-21kworker/u8:2+events_unbound00:35:133
135017991916,2cyclictest299393-21kworker/u8:0+flush-259:022:10:163
13501199194,4cyclictest431918-21https00:35:122
13501199194,4cyclictest391539-21ntpq23:40:222
135001991915,3cyclictest86386-21kworker/u8:3+events_unbound19:15:221
135001991915,3cyclictest250672-21kworker/u8:2+flush-259:021:05:241
135001991915,2cyclictest1-21systemd20:39:591
135001991913,4cyclictest250647-21/usr/sbin/munin21:00:131
134991991915,2cyclictest193265-21apt-get20:05:120
135017991815,2cyclictest293754-21kworker/u8:2+events_unbound22:20:033
135017991815,2cyclictest261458-21kworker/u8:3+flush-259:021:30:003
135017991814,3cyclictest409727-21kworker/u8:5+events_unbound00:17:383
135017991814,3cyclictest359713-21kworker/u8:2+events_unbound00:05:153
13501199183,4cyclictest358487-21ld23:00:232
135011991812,3cyclictest3000860-21systemd-journal23:20:012
135001991814,3cyclictest134702-21kworker/u8:1+flush-259:020:05:151
135001991813,4cyclictest299393-21kworker/u8:0+events_unbound22:10:091
135001991813,3cyclictest182752-21apt-get19:55:131
13500199181,15cyclictest161246-21unin-run19:35:001
134991991813,3cyclictest261458-21kworker/u8:3+flush-259:021:35:200
134991991812,4cyclictest3000860-21systemd-journal20:30:000
13499199181,15cyclictest1-21systemd20:00:490
13499199180,16cyclictest3000860-21systemd-journal21:30:140
13501199173,6cyclictest343152-21munin-run22:44:592
135011991712,4cyclictest3000860-21systemd-journal20:20:012
135001991714,2cyclictest250672-21kworker/u8:2+flush-259:021:10:221
135001991714,2cyclictest250672-21kworker/u8:2+flush-259:021:10:211
135001991713,3cyclictest310910-21kworker/u8:1+events_unbound22:55:251
135001991713,2cyclictest409726-21kworker/u8:1+flush-259:000:15:001
135001991710,5cyclictest134717-21latency_hist19:10:001
134991991714,2cyclictest60650irq/133-enp0s29f1:rx-023:35:200
134991991714,2cyclictest60650irq/133-enp0s29f1:rx-023:01:590
134991991714,2cyclictest60650irq/133-enp0s29f1:rx-021:12:550
134991991714,2cyclictest60650irq/133-enp0s29f1:rx-021:12:540
134991991714,2cyclictest60650irq/133-enp0s29f1:rx-000:26:010
135017991612,3cyclictest343376-21kworker/u8:3+events_unbound23:05:123
135017991612,3cyclictest343376-21kworker/u8:3+events_unbound22:49:453
135017991612,3cyclictest310910-21kworker/u8:1+events_unbound22:51:033
13501199165,9cyclictest224367-21latency_hist20:35:012
13501199165,4cyclictest213909-21latency_hist20:25:012
13501199164,8cyclictest835-21runrttasks20:44:472
13501199162,4cyclictest376523-21sshd23:28:132
13501199162,4cyclictest376523-21sshd23:28:132
135011991611,3cyclictest155181-21ntpq19:25:242
135001991612,3cyclictest427716-21tr00:30:131
135001991612,3cyclictest310910-21kworker/u8:1+events_unbound23:09:591
135001991612,3cyclictest181706-21kworker/u8:4+flush-259:020:30:261
135001991611,4cyclictest293754-21kworker/u8:2+flush-259:022:30:211
134991991614,1cyclictest60650irq/133-enp0s29f1:rx-022:52:300
134991991614,1cyclictest60650irq/133-enp0s29f1:rx-021:55:350
134991991614,1cyclictest60650irq/133-enp0s29f1:rx-021:04:540
134991991613,2cyclictest60650irq/133-enp0s29f1:rx-023:52:480
134991991613,2cyclictest60650irq/133-enp0s29f1:rx-022:26:110
134991991613,2cyclictest60650irq/133-enp0s29f1:rx-021:08:370
134991991613,2cyclictest60650irq/133-enp0s29f1:rx-019:50:170
134991991613,2cyclictest60650irq/133-enp0s29f1:rx-000:10:430
134991991610,4cyclictest305114-21cat22:00:020
135017991513,2cyclictest332524-21kworker/u8:0+events_unbound22:40:143
135017991512,2cyclictest332524-21kworker/u8:0+events_unbound22:39:203
135017991511,3cyclictest359713-21kworker/u8:2+events_unbound23:48:173
13501199158,5cyclictest414917-21cat00:14:552
13501199158,5cyclictest343776-21cron22:50:002
13501199157,6cyclictest415465-21tune2fs00:15:162
13501199157,6cyclictest303934-21sort21:55:282
13501199157,5cyclictest426950-21fschecks_time00:30:102
13501199157,5cyclictest277833-21grep21:30:082
13501199156,6cyclictest426111-21cmp00:30:012
13501199155,6cyclictest294160-21latency_hist21:50:002
135001991512,2cyclictest316464-21cpuspeed_turbos22:15:131
135001991511,3cyclictest365264-21kworker/u8:0+events_unbound00:00:201
135001991510,4cyclictest272182-21kworker/u8:0+flush-259:021:30:201
135001991510,4cyclictest208696-21kworker/u8:3+flush-259:020:20:251
13500199150,13cyclictest1-21systemd00:20:001
13499199158,5cyclictest371066-21cp23:20:010
13499199155,1cyclictest0-21swapper/021:55:010
134991991513,1cyclictest60650irq/133-enp0s29f1:rx-022:18:100
134991991513,1cyclictest60650irq/133-enp0s29f1:rx-020:55:250
134991991513,1cyclictest60650irq/133-enp0s29f1:rx-019:40:250
134991991513,1cyclictest60650irq/133-enp0s29f1:rx-000:20:200
134991991512,2cyclictest61450irq/134-enp0s29f1:tx-023:55:140
134991991512,2cyclictest60650irq/133-enp0s29f1:rx-023:15:270
134991991512,2cyclictest60650irq/133-enp0s29f1:rx-022:55:230
134991991512,2cyclictest60650irq/133-enp0s29f1:rx-020:40:210
134991991511,3cyclictest60650irq/133-enp0s29f1:rx-022:30:090
135017991410,3cyclictest365264-21kworker/u8:0+events_unbound23:42:323
13501199149,3cyclictest325593-21ntpq22:20:222
13501199149,3cyclictest200538-21grep20:10:262
13501199148,5cyclictest240698-21/usr/sbin/munin20:50:212
13501199148,4cyclictest261752-21sort21:10:152
13501199148,4cyclictest261752-21sort21:10:152
13501199147,6cyclictest349497-21grep22:55:182
13501199147,5cyclictest393008-21turbostat23:50:002
13501199147,4cyclictest135111-21latency_hist19:10:002
13501199146,5cyclictest156423-21df_inode19:30:192
13501199146,5cyclictest140481-21sed19:15:012
13501199145,6cyclictest404235-21expr00:00:142
13501199145,6cyclictest161492-21/usr/sbin/munin19:35:172
13501199143,2cyclictest420592-21diskstats00:20:172
13501199143,2cyclictest387126-21munin-run23:40:002
13501199143,2cyclictest311434-21hddtemp_smartct22:10:112
13501199143,2cyclictest292712-21users21:40:262
13501199142,3cyclictest0-21swapper/200:07:132
13500199149,4cyclictest431914-21apt-get00:35:111
13500199149,4cyclictest382107-21kworker/u8:4+events_unbound23:49:051
13500199149,4cyclictest234753-21journal-offline20:45:001
13500199148,4cyclictest3000063-21rs:main1
135001991411,2cyclictest3000063-21rs:main1
135001991411,2cyclictest181706-21kworker/u8:4+flush-259:021:40:251
135001991410,3cyclictest214172-21apt-get20:25:101
13499199148,4cyclictest3000860-21systemd-journal00:05:010
134991991413,1cyclictest60650irq/133-enp0s29f1:rx-020:35:170
134991991413,1cyclictest60650irq/133-enp0s29f1:rx-020:20:170
134991991412,2cyclictest60650irq/133-enp0s29f1:rx-023:44:410
134991991412,2cyclictest60650irq/133-enp0s29f1:rx-023:30:190
134991991412,2cyclictest60650irq/133-enp0s29f1:rx-019:21:030
134991991412,1cyclictest60650irq/133-enp0s29f1:rx-022:36:540
134991991412,1cyclictest60650irq/133-enp0s29f1:rx-000:35:230
134991991411,2cyclictest60650irq/133-enp0s29f1:rx-023:29:310
134991991411,2cyclictest60650irq/133-enp0s29f1:rx-023:29:310
134991991411,2cyclictest60650irq/133-enp0s29f1:rx-022:41:260
13501199137,4cyclictest294535-21tr21:50:092
13501199137,4cyclictest251100-21sed21:00:152
13501199137,4cyclictest193688-21tail20:05:172
13501199137,4cyclictest172014-21cpuspeed_turbos19:45:122
13501199136,5cyclictest835-21runrttasks21:35:202
13501199135,6cyclictest377027-21expr23:30:082
13501199135,6cyclictest216488-21kthreadcore20:25:172
13501199135,6cyclictest185693-21grep19:55:202
13501199135,6cyclictest167311-21sed19:40:222
13501199135,5cyclictest316439-21expr22:15:132
13501199133,2cyclictest344200-21df22:50:092
13501199132,2cyclictest225611-21irqstats20:35:202
13500199137,4cyclictest3000860-21systemd-journal19:24:141
13500199136,5cyclictest365233-21latency_hist23:15:011
13499199139,3cyclictest60650irq/133-enp0s29f1:rx-022:10:090
134991991311,2cyclictest60650irq/133-enp0s29f1:rx-021:28:340
134991991311,1cyclictest60650irq/133-enp0s29f1:rx-021:47:140
134991991310,2cyclictest60650irq/133-enp0s29f1:rx-022:07:440
13501199127,3cyclictest360023-21cron23:10:002
13501199127,3cyclictest327092-21cron22:30:002
13501199127,3cyclictest277484-21turbostat.cron21:30:002
13500199128,3cyclictest338381-21fschecks_count22:40:151
13500199126,4cyclictest304590-21latency_hist22:00:001
13499199126,2cyclictest60650irq/133-enp0s29f1:rx-020:15:240
13500199117,3cyclictest310910-21kworker/u8:1+flush-259:023:30:001
13500199117,3cyclictest310910-21kworker/u8:1+flush-259:023:30:001
13500199117,3cyclictest261458-21kworker/u8:3+events_unbound21:35:191
13500199117,2cyclictest0-21swapper/100:10:001
13500199116,3cyclictest370610-21cat23:20:001
13500199108,1cyclictest3000063-21rs:main1
13500199106,3cyclictest145405-21kworker/u8:2+events_unbound19:50:021
13499199108,1cyclictest60650irq/133-enp0s29f1:rx-023:45:550
1350019991,7cyclictest343803-21awk22:50:001
1350019991,6cyclictest376044-21cron23:24:591
1350019981,5cyclictest426013-21turbostat.cron00:30:001
1350019981,5cyclictest310853-21cron22:10:011
1350019981,5cyclictest294162-21sh21:50:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional