You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 21:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sun Nov 16, 2025 12:43:59)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1809640211454,55sleep30-21swapper/307:06:433
180492828755,10sleep10-21swapper/107:05:091
180975528252,9sleep20-21swapper/207:08:102
180973328259,6sleep00-21swapper/007:07:520
1810025992616,6cyclictest2045858-21taskset10:16:293
213338722512,9sleep02133398-21sh11:15:080
181002599251,16cyclictest2055022-21taskset10:24:043
181002599251,16cyclictest2055022-21taskset10:24:033
1810025992419,3cyclictest0-21swapper/312:05:203
21871342230,4sleep01939314-21diskmemload11:48:270
181002599239,9cyclictest2024829-21taskset10:05:063
1810025992320,1cyclictest0-21swapper/310:00:233
1810025992320,1cyclictest0-21swapper/308:40:003
1810025992319,2cyclictest0-21swapper/312:25:213
21738902220,3sleep11810019-21cyclictest11:40:181
19547472222,9sleep11954788-21perf09:20:001
181002599227,12cyclictest0-21swapper/311:22:563
181002599222,13cyclictest2271612-21taskset12:37:003
1810025992219,1cyclictest0-21swapper/309:30:133
1810025992218,2cyclictest0-21swapper/311:10:213
181002599221,16cyclictest1886561-21chrt08:20:133
181002599220,18cyclictest2176698-21chrt11:40:233
181002599219,9cyclictest2000370-21chrt09:49:033
181002599212,12cyclictest2117390-21taskset11:05:003
1810025992118,1cyclictest0-21swapper/312:20:173
1810025992118,1cyclictest0-21swapper/307:55:233
181002599211,17cyclictest0-21swapper/311:15:563
181002599211,15cyclictest2124221-21taskset11:08:243
181002599211,15cyclictest1970248-21sleep09:30:003
181002599211,15cyclictest1970248-21sleep09:30:003
22045052202,13sleep00-21swapper/011:59:460
181002599209,8cyclictest2200647-21taskset11:55:203
181002599202,13cyclictest2263079-21taskset12:32:073
181002599202,13cyclictest2228543-21taskset12:12:113
181002599202,12cyclictest2195520-21taskset11:53:543
1810025992016,2cyclictest0-21swapper/310:25:223
1810025992011,7cyclictest15950irq/132-ahci[0000:00:17.0]07:45:153
181002599201,15cyclictest2161971-21chrt11:30:453
181002599201,14cyclictest2093572-21chrt10:48:203
20697722191,2sleep00-21swapper/010:32:110
19527532191,12sleep21939314-21diskmemload09:16:242
181002599198,7cyclictest1870430-21taskset08:04:383
1810025991916,1cyclictest0-21swapper/312:15:293
1810025991916,1cyclictest0-21swapper/312:00:193
1810025991916,1cyclictest0-21swapper/308:05:213
1810025991915,2cyclictest0-21swapper/309:44:093
181002599191,14cyclictest66550irq/131-enp1s0-TxRx-310:57:353
181002599191,14cyclictest2164662-21chrt11:35:023
181002599191,14cyclictest2153666-21chrt11:25:283
181002599191,14cyclictest0-21swapper/310:50:453
181002599191,14cyclictest0-21swapper/309:50:323
181002599191,14cyclictest0-21swapper/309:10:223
181002599191,13cyclictest2084851-21chrt10:41:473
181002599191,13cyclictest2037359-21chrt10:10:453
181002599190,15cyclictest0-21swapper/307:32:533
22558302181,10sleep21939314-21diskmemload12:28:582
21542922181,3sleep10-21swapper/111:26:031
181002599182,12cyclictest2185909-21taskset11:46:333
181002599182,12cyclictest2014646-21taskset09:56:333
181002599182,12cyclictest1952574-21taskset09:16:043
1810025991815,1cyclictest0-21swapper/309:00:173
1810025991815,1cyclictest0-21swapper/308:25:143
1810025991815,1cyclictest0-21swapper/308:10:183
1810025991815,1cyclictest0-21swapper/307:25:003
1810025991814,2cyclictest0-21swapper/308:50:173
181002599181,15cyclictest0-21swapper/310:35:203
181002599181,14cyclictest0-21swapper/309:35:403
181002599181,13cyclictest1955106-21chrt09:20:093
181002599181,13cyclictest0-21swapper/307:25:123
181002599181,13cyclictest0-21swapper/307:25:123
181002599181,12cyclictest2070507-21taskset10:33:253
21927782172,8sleep22192835-21ntpq11:50:222
21319862171,3sleep00-21swapper/011:13:080
20919172171,11sleep02091927-21ssh10:45:330
20288712170,1sleep00-21swapper/010:05:220
19990912171,4sleep21939314-21diskmemload09:46:502
19767442171,4sleep21939314-21diskmemload09:32:592
19759902171,3sleep11939314-21diskmemload09:31:381
19619912171,3sleep00-21swapper/009:24:000
19449222171,3sleep01939314-21diskmemload09:11:100
19080382171,10sleep11908254-21diskstats08:40:161
18892572171,6sleep01890391-21/usr/sbin/munin08:20:190
1810025991714,1cyclictest0-21swapper/307:15:193
181002599171,13cyclictest0-21swapper/307:35:243
181002599171,12cyclictest1939337-21sleep09:09:593
22649182160,3sleep00-21swapper/012:34:390
22438312161,4sleep22243879-21ntpq12:20:202
22109842161,4sleep11939314-21diskmemload12:01:511
21707332161,11sleep02170742-21ssh11:37:530
20449922163,8sleep22045277-21sed10:15:292
20366442161,10sleep22036774-21ssh10:10:242
19173162161,10sleep21917372-21ntp_states08:45:222
18143222161,10sleep11814409-21wc07:10:221
181002599161,8cyclictest1844320-21taskset07:40:013
181002599161,11cyclictest1881286-21chrt08:15:133
181002599160,13cyclictest0-21swapper/307:10:193
181002599160,10cyclictest1928671-21taskset08:59:003
22479722151,9sleep12248183-21cat12:25:001
22383822151,2sleep20-21swapper/212:18:482
21939812151,1sleep10-21swapper/111:51:251
21781782151,9sleep22178189-21sh11:42:002
20945162151,2sleep10-21swapper/110:49:551
20530762151,2sleep20-21swapper/210:20:382
20530762151,2sleep20-21swapper/210:20:382
19546612151,9sleep01954667-21sshd09:19:510
19085382151,9sleep01909878-21cat08:40:200
18392232151,3sleep00-21swapper/007:35:180
181002599151,9cyclictest1918168-21taskset08:48:543
181002599151,9cyclictest1912761-21taskset08:41:533
181002599151,9cyclictest1897031-21taskset08:30:073
181002599150,13cyclictest0-21swapper/307:50:263
22279042141,8sleep02227917-21rm12:11:200
22206392141,7sleep01939314-21diskmemload12:08:120
21867572141,7sleep21939314-21diskmemload11:47:532
21024282141,9sleep22102503-21/usr/sbin/munin10:55:142
20771452141,8sleep02077153-21rm10:36:190
20610402141,6sleep00-21swapper/010:26:050
19985992141,9sleep01998611-21ssh09:46:050
19901012141,9sleep21990115-21ssh09:40:252
19850852141,8sleep21939314-21diskmemload09:39:412
19612392141,7sleep11939314-21diskmemload09:22:401
18559872141,8sleep01857690-21cut07:50:170
22632002131,6sleep11939314-21diskmemload12:32:191
22039432131,3sleep22203953-21ssh11:59:002
21652392131,7sleep12165821-21kthreadcore11:35:171
21375082131,8sleep12137609-21ntpq11:15:211
21307732131,7sleep21939314-21diskmemload11:10:572
21238672134,6sleep21810019-21cyclictest11:07:482
21221762131,8sleep12122266-21cat11:05:271
20923682131,4sleep22092379-21sh10:46:202
20542062131,8sleep02054215-21ssh10:22:370
20542062131,8sleep02054215-21ssh10:22:360
20313742131,6sleep21939314-21diskmemload10:08:392
20060132131,3sleep12006090-21find09:50:261
19463512131,2sleep20-21swapper/209:13:432
19182582131,8sleep01918369-21sed08:50:010
22490232121,7sleep12249786-21cat12:25:201
22212012120,8sleep12221205-21sh12:08:571
21848412121,6sleep11939314-21diskmemload11:45:281
21609562121,3sleep12161073-21ntp_192.168.11411:30:211
21311922121,4sleep11939314-21diskmemload11:11:401
21173332121,4sleep22117387-21munin-run11:05:002
21156522121,3sleep12115662-21ssh11:02:061
21152532121,5sleep01939314-21diskmemload11:01:260
21082232121,3sleep10-21swapper/110:57:211
21004982121,3sleep10-21swapper/110:52:041
20851672120,8sleep02085177-21ssh10:42:220
20800322127,1sleep10-21swapper/110:40:171
20787432121,6sleep21939314-21diskmemload10:39:142
20711262121,6sleep11939314-21diskmemload10:34:361
20160922120,8sleep22016095-21rm09:59:052
20088422121,6sleep22008871-21munin-run09:55:002
19671162121,4sleep11939314-21diskmemload09:25:341
19671162121,4sleep11939314-21diskmemload09:25:331
19028952121,7sleep21903020-21gltestperf08:35:162
22714292111,6sleep11939314-21diskmemload12:36:471
22708482111,3sleep20-21swapper/212:36:002
22655592111,3sleep01939314-21diskmemload12:35:100
22570972111,3sleep20-21swapper/212:30:152
22466652111,4sleep01939314-21diskmemload12:23:160
22354272110,7sleep02235523-21ssh12:15:250
22352012111,5sleep12235260-21ntpq12:15:221
22276872111,4sleep21939314-21diskmemload12:11:022
22191042111,2sleep20-21swapper/212:06:092
22113472111,4sleep21939314-21diskmemload12:02:202
21931722111,5sleep02193440-21grep11:50:290
21610832111,4sleep22161124-21ntpq11:30:212
21569542111,4sleep01939314-21diskmemload11:30:120
21559012111,3sleep20-21swapper/211:28:452
21484622111,6sleep12148471-21ssh11:24:231
21464282111,3sleep00-21swapper/011:20:500
21456672111,4sleep21939314-21diskmemload11:20:232
21249122111,3sleep01939314-21diskmemload11:09:550
21081272111,6sleep02108142-21ssh10:57:120
21006322111,3sleep00-21swapper/010:52:200
20836422111,5sleep22083696-21ntpq10:40:222
20705742111,4sleep21939314-21diskmemload10:33:332
20623962111,3sleep20-21swapper/210:28:242
20614442111,2sleep12061461-21ssh10:26:461
20535222111,4sleep11939314-21diskmemload10:21:251
20535222111,4sleep11939314-21diskmemload10:21:251
20471582111,3sleep11939314-21diskmemload10:18:531
20467142111,3sleep00-21swapper/010:18:030
20387662111,4sleep11939314-21diskmemload10:13:111
20365222111,5sleep02036581-21ntpq10:10:220
20254662113,3sleep10-21swapper/110:05:181
20236502111,6sleep02023659-21ssh10:03:280
20170472111,5sleep20-21swapper/210:00:132
20158072111,4sleep01939314-21diskmemload09:58:360
20150642111,4sleep11939314-21diskmemload09:57:161
19997632111,3sleep10-21swapper/109:48:021
19919252111,6sleep01991932-21ssh09:43:000
19763692111,3sleep00-21swapper/009:32:170
19699462111,3sleep00-21swapper/009:29:300
19699462111,3sleep00-21swapper/009:29:290
19455522111,4sleep11939314-21diskmemload09:12:151
19010662111,5sleep21901113-21ntpq08:30:232
18815622111,5sleep11881774-21ls08:15:151
18812642111,4sleep21881283-21apt-get08:15:122
18695442110,7sleep21865456-21/usr/sbin/munin08:00:252
18651672111,6sleep21865264-21cut08:00:022
18592282111,3sleep11859413-21timerwakeupswit07:50:261
18440502111,5sleep21843996-31find07:40:012
18440452110,5sleep21843996-31find07:39:582
18353592111,6sleep21837152-21kthreadcore07:30:192
18180392111,6sleep11819315-21cat07:15:191
18142642111,5sleep01814320-21ntpq07:10:210
22544262101,3sleep00-21swapper/012:27:040
22223462101,2sleep10-21swapper/112:10:091
22110712101,3sleep00-21swapper/012:01:580
22026752101,3sleep11939314-21diskmemload11:57:181
21723072100,1sleep00-21swapper/011:40:060
21718762101,3sleep20-21swapper/211:39:472
21549032101,4sleep01939314-21diskmemload11:27:030
21340512100,6sleep21939314-21diskmemload11:15:182
21005092101,5sleep22100521-21ssh10:52:052
20767232101,2sleep10-21swapper/110:35:361
20222102101,2sleep10-21swapper/110:01:011
20014862100,4sleep00-21swapper/009:50:090
19903572101,3sleep11939314-21diskmemload09:40:281
19840252100,4sleep11939314-21diskmemload09:37:241
19838532100,1sleep00-21swapper/009:37:060
19632182101,4sleep21963433-21hddtemp_smartct09:25:182
19632182101,4sleep21963433-21hddtemp_smartct09:25:182
19605302101,3sleep21939314-21diskmemload09:21:292
19348622103,3sleep01934905-21kthreadcore09:05:200
19340372100,6sleep11-21systemd09:05:001
19026332101,4sleep11902894-21df_inode08:35:151
18970392101,4sleep11897058-21apt-get08:30:141
18960842103,3sleep00-21swapper/008:25:280
18918112101,4sleep21891830-21apt-get08:25:142
18704622101,4sleep01870510-21munin-run08:05:000
18642122101,5sleep11864272-21ntp_offset07:55:231
18641632101,4sleep01864210-21ntpq07:55:220
18341972100,3sleep11835358-21cat07:30:181
18288292101,4sleep01828900-21irqstats07:25:160
18288292101,4sleep01828900-21irqstats07:25:160
18284142101,4sleep21828617-21diskstats07:25:152
18284142101,4sleep21828617-21diskstats07:25:142
18280692101,5sleep01828096-21munin-run07:25:000
18101972101,3sleep21810216-21apt-get07:10:112
1935649291,4sleep11937406-21cut09:05:211
1932908291,3sleep21933021-21ntpq09:00:212
1928526290,1sleep00-21swapper/008:57:100
1886539291,3sleep20-21swapper/208:20:132
1874808291,4sleep11874860-21open_files08:05:251
1859050291,4sleep21854933-21/usr/sbin/munin07:50:232
1853991291,4sleep01854232-21grep07:45:280
1845024291,4sleep11846660-21grep07:40:201
1843925291,3sleep11843996-31find07:39:481
1827084291,3sleep21827131-21ntpq07:20:222
1823060291,3sleep11823079-21apt-get07:20:121
1817539291,3sleep21817755-21diskstats07:15:162
1919087281,3sleep00-21swapper/008:50:180
1903178280,4sleep01904164-21cut08:35:180
1883653281,3sleep01885090-21load08:15:180
1866506281,3sleep11868218-21cut08:00:201
1928654270,1sleep20-21swapper/208:58:482
1920161271,2sleep10-21swapper/108:50:181
1912357271,3sleep21912594-21unixbench_singl08:40:292
1902057270,1sleep00-21swapper/008:34:510
1886533270,1sleep10-21swapper/108:20:081
1880828271,3sleep0802-21snmpd08:12:490
1871508272,2sleep00-21swapper/008:05:200
1849665270,2sleep10-21swapper/107:45:131
1848379270,1sleep00-21swapper/007:40:230
1833003270,4sleep10-21swapper/107:25:291
1833003270,4sleep10-21swapper/107:25:291
1934289260,1sleep20-21swapper/209:05:052
1933899260,1sleep00-21swapper/009:03:200
1923755260,1sleep10-21swapper/108:55:031
1913841261,2sleep10-21swapper/108:45:171
1877961261,2sleep20-21swapper/208:10:192
1876566261,2sleep10-21swapper/108:10:181
1849687261,2sleep20-21swapper/207:45:152
1838530260,1sleep00-21swapper/007:33:250
1896741250,1sleep10-21swapper/108:29:421
1874574250,1sleep20-21swapper/208:05:222
1822367250,1sleep00-21swapper/007:18:220
1918805240,1sleep20-21swapper/208:50:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional