You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 14:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sat Nov 29, 2025 12:44:00)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
212880621340,3sleep014-21ksoftirqd/011:20:140
1799098212852,69sleep30-21swapper/307:05:253
1799749210588,10sleep10-21swapper/107:09:111
179947528468,10sleep20-21swapper/207:05:402
179890127953,14sleep00-21swapper/007:05:230
20854232350,4sleep20-21swapper/210:50:172
1800148992722,3cyclictest0-21swapper/310:35:143
1800148992624,1cyclictest0-21swapper/309:20:013
1800148992624,1cyclictest0-21swapper/309:20:003
1800148992622,2cyclictest0-21swapper/310:55:203
1800148992612,12cyclictest0-21swapper/311:05:003
180014899260,19cyclictest2080491-21taskset10:45:583
1800148992521,2cyclictest0-21swapper/311:50:193
21479852240,2sleep10-21swapper/111:30:261
1800148992421,1cyclictest0-21swapper/309:10:163
1800148992420,2cyclictest0-21swapper/311:10:203
20731932231,3sleep20-21swapper/210:41:372
194342722312,7sleep11930003-21diskmemload09:16:221
194342722312,7sleep11930003-21diskmemload09:16:211
1800148992320,1cyclictest0-21swapper/312:04:593
1800148992320,1cyclictest0-21swapper/311:30:153
1800148992319,2cyclictest0-21swapper/311:40:193
1800148992319,2cyclictest0-21swapper/310:45:013
180014899231,19cyclictest2236562-21taskset12:28:523
180014899231,16cyclictest2205002-21sleep12:10:003
20566632222,6sleep02056729-21ntpq10:30:200
180014899222,14cyclictest2221622-21taskset12:20:153
180014899222,13cyclictest1983069-21taskset09:44:253
1800148992219,1cyclictest0-21swapper/308:30:133
180014899221,17cyclictest2135927-21chrt11:25:063
180014899221,16cyclictest2166457-21chrt11:45:163
180014899220,19cyclictest0-21swapper/310:15:163
180014899220,14cyclictest2087519-21taskset10:50:273
21662652211,9sleep22166426-21cstates11:45:152
21124602211,2sleep20-21swapper/211:09:432
180014899212,13cyclictest2218207-21taskset12:15:303
1800148992118,1cyclictest0-21swapper/309:05:243
1800148992117,2cyclictest0-21swapper/312:15:003
1800148992117,2cyclictest0-21swapper/310:00:193
21978332201,13sleep12199056-21cut12:05:181
180014899202,13cyclictest2057948-21taskset10:31:313
1800148992014,3cyclictest0-21swapper/309:20:173
1800148992014,3cyclictest0-21swapper/309:20:163
180014899201,15cyclictest2021094-21chrt10:09:093
180014899201,14cyclictest2181647-21chrt11:55:143
180014899201,12cyclictest1844952-21taskset07:50:123
1800148992010,7cyclictest0-21swapper/310:26:253
1800148992010,6cyclictest19886792sleep309:45:493
180014899200,18cyclictest66550irq/131-enp1s0-TxRx-312:36:123
21965812191,11sleep21930003-21diskmemload12:04:362
21205112191,2sleep00-21swapper/011:14:580
19446862191,9sleep21944696-21ssh09:18:512
19446862191,9sleep21944696-21ssh09:18:502
18817172191,9sleep21881718-21cron08:24:592
180014899192,12cyclictest2111722-21taskset11:08:223
180014899192,11cyclictest2131095-21taskset11:20:183
1800148991916,2cyclictest0-21swapper/309:40:003
1800148991916,1cyclictest0-21swapper/309:25:183
1800148991916,1cyclictest0-21swapper/308:40:163
180014899191,14cyclictest2021905-21chrt10:10:113
180014899191,14cyclictest1966743-21chrt09:32:183
180014899190,11cyclictest1800375-21taskset07:10:023
22522052189,4sleep0151ktimers/012:36:520
22283302181,2sleep20-21swapper/212:24:042
21812222181,4sleep01930003-21diskmemload11:55:000
21275952181,2sleep20-21swapper/211:19:092
20593402181,10sleep11930003-21diskmemload10:34:021
19740552180,2sleep11974065-21ssh09:36:461
180014899182,11cyclictest2044467-21taskset10:24:583
1800148991815,1cyclictest0-21swapper/309:50:183
1800148991815,1cyclictest0-21swapper/308:05:183
1800148991814,2cyclictest0-21swapper/309:00:173
180014899181,14cyclictest0-21swapper/311:37:393
180014899181,13cyclictest2237792-21chrt12:30:143
180014899181,13cyclictest0-21swapper/311:15:013
180014899181,12cyclictest1998704-21chrt09:55:063
22527462171,4sleep11930003-21diskmemload12:37:371
21272872171,3sleep11930003-21diskmemload11:18:321
20881132171,3sleep00-21swapper/010:51:010
20286562171,4sleep11930003-21diskmemload10:13:571
20122422171,4sleep21930003-21diskmemload10:01:572
19826752171,3sleep21930003-21diskmemload09:43:402
180014899172,12cyclictest1834037-21turbostat07:40:013
1800148991713,2cyclictest0-21swapper/307:55:183
180014899171,13cyclictest0-21swapper/308:25:143
180014899171,13cyclictest0-21swapper/308:00:133
180014899171,10cyclictest1839665-21taskset07:45:123
180014899170,14cyclictest0-21swapper/308:45:153
22435492161,2sleep00-21swapper/012:31:440
22045872160,3sleep00-21swapper/012:09:220
21038922161,2sleep20-21swapper/211:02:382
21020472161,7sleep12102196-21ntpq11:00:201
20959382160,3sleep00-21swapper/010:56:470
20297672161,4sleep22029905-21diskstats10:15:162
20295522160,9sleep01930003-21diskmemload10:15:100
1800148991613,1cyclictest0-21swapper/308:35:213
180014899161,11cyclictest1871121-21turbostat08:14:593
180014899161,10cyclictest1876100-21taskset08:16:273
180014899160,13cyclictest0-21swapper/308:50:223
180014899160,10cyclictest1919077-21chrt08:59:433
21661722151,2sleep00-21swapper/011:45:050
21631332150,1sleep20-21swapper/211:40:262
21352632150,8sleep21930003-21diskmemload11:24:232
21187392150,1sleep10-21swapper/111:11:341
20810482151,3sleep21930003-21diskmemload10:46:592
20731382153,1sleep1171rcu_preempt10:41:321
20059282150,3sleep11930003-21diskmemload09:59:351
197566121510,2sleep21944775-21kworker/u16:2+xprtiod09:39:522
19584752151,8sleep21930003-21diskmemload09:25:532
19530452151,8sleep21930003-21diskmemload09:24:522
19530452151,8sleep21930003-21diskmemload09:24:512
18965882151,9sleep11896634-21ntpq08:35:211
180014899151,9cyclictest1812542-21taskset07:17:073
1800148991511,2cyclictest0-21swapper/307:25:173
180014899150,13cyclictest0-21swapper/307:40:173
180014899150,13cyclictest0-21swapper/307:20:183
180014899150,12cyclictest0-21swapper/308:20:193
22365242141,2sleep00-21swapper/012:28:500
22126332141,6sleep11930003-21diskmemload12:14:251
22049032141,9sleep21930003-21diskmemload12:09:522
20128562141,7sleep11930003-21diskmemload10:03:151
20080642141,8sleep02009647-21cut10:00:170
19432702141,9sleep01930003-21diskmemload09:16:050
19432702141,9sleep01930003-21diskmemload09:16:040
18435712141,8sleep11843660-21ntpq07:45:191
180014899140,12cyclictest0-21swapper/307:30:133
22376692131,3sleep22237680-21ssh12:30:062
22361752131,8sleep22236189-21ssh12:28:202
22260202130,1sleep00-21swapper/012:20:280
22213282131,5sleep11930003-21diskmemload12:20:141
21896622131,5sleep12189688-21apt-get12:00:141
21571712131,3sleep22157181-21ssh11:37:412
21424982131,8sleep22142505-21ssh11:28:472
21111012131,7sleep02111106-21ssh11:06:490
20951302131,7sleep22095242-21sshd10:55:292
20738432131,8sleep02073855-21ssh10:42:470
20647262131,3sleep12064738-21sensors_temp10:35:271
20514752130,7sleep11930003-21diskmemload10:28:541
20270462131,2sleep20-21swapper/210:11:072
19677142131,6sleep00-21swapper/009:34:080
19303832131,8sleep21930596-21cat09:10:132
18981822131,8sleep11898437-21/usr/sbin/munin08:40:141
18337102130,7sleep11833609-31sort07:36:091
18003912131,7sleep11800409-21apt-get07:10:131
21658962121,7sleep02165910-21latency_hist11:45:010
21561042120,8sleep12156114-21ssh11:35:431
21427972122,4sleep11930003-21diskmemload11:29:221
21418542121,8sleep00-21swapper/011:27:340
21258702121,5sleep01930003-21diskmemload11:15:430
20818412121,5sleep10-21swapper/110:48:261
20659562121,7sleep22065959-21ssh10:37:252
20652302121,4sleep01930003-21diskmemload10:35:460
20451412121,7sleep02045179-21fschecks_count10:25:160
20448522121,4sleep21930003-21diskmemload10:25:142
20422972121,7sleep01930003-21diskmemload10:20:460
20352022120,6sleep11930003-21diskmemload10:17:031
20194642121,6sleep11930003-21diskmemload10:06:171
20049932121,4sleep21930003-21diskmemload09:57:562
19910572122,6sleep21991127-21grep09:50:132
19897612121,4sleep11930003-21diskmemload09:48:131
19810972121,4sleep11930003-21diskmemload09:40:421
19537342121,7sleep01953871-21grep09:25:150
18874112121,6sleep21887575-21diskstats08:30:122
18598572121,7sleep21860021-21timerwakeupswit08:00:262
18502602120,7sleep11850470-21cstates07:55:121
18454582121,7sleep21845758-21kthreadcore07:50:162
22515592111,3sleep20-21swapper/212:35:582
22416812110,2sleep12241866-21strings12:30:211
22294092111,5sleep12229672-21ssh12:25:141
22183022111,3sleep00-21swapper/012:15:390
22181782111,4sleep21930003-21diskmemload12:15:292
22109762111,4sleep21930003-21diskmemload12:11:532
21873442111,3sleep01930003-21diskmemload11:56:550
21798652111,3sleep20-21swapper/211:52:532
21720442111,4sleep11930003-21diskmemload11:47:421
21590512111,4sleep11930003-21diskmemload11:40:171
21574552111,4sleep01930003-21diskmemload11:38:150
21498492111,3sleep00-21swapper/011:33:190
21485072111,4sleep21930003-21diskmemload11:30:482
21182362111,4sleep21930003-21diskmemload11:10:442
21119642111,6sleep11930003-21diskmemload11:08:511
20871372111,5sleep11930003-21diskmemload10:50:211
20796242111,5sleep02079686-21ntp_states10:45:240
20585692111,4sleep21930003-21diskmemload10:32:392
20372612111,3sleep10-21swapper/110:20:071
20219172111,4sleep01930003-21diskmemload10:10:120
20215582111,4sleep02021613-21munin-run10:10:000
19964562111,4sleep11930003-21diskmemload09:51:361
19896722111,4sleep21930003-21diskmemload09:48:022
19814952111,4sleep01930003-21diskmemload09:41:290
19662282111,3sleep21930003-21diskmemload09:31:212
19594002111,3sleep11930003-21diskmemload09:27:321
19504842111,3sleep00-21swapper/009:20:260
19504842111,3sleep00-21swapper/009:20:250
19356292111,4sleep11930003-21diskmemload09:11:161
19349332111,4sleep01930003-21diskmemload09:10:270
18981602111,4sleep01898179-21apt-get08:40:130
18336962111,5sleep21833648-31find07:35:592
22191462101,3sleep11930003-21diskmemload12:16:571
22094012101,4sleep02209447-21ntpq12:10:230
21873232101,3sleep10-21swapper/111:56:521
21856592101,4sleep22185713-21ntpq11:55:212
21350442100,3sleep10-21swapper/111:24:021
21035982101,3sleep01930003-21diskmemload11:02:070
20966302100,3sleep10-21swapper/110:58:041
20415762101,3sleep20-21swapper/210:20:242
20191662100,2sleep22019180-21ssh10:05:452
20036892101,3sleep00-21swapper/009:55:410
19983372100,4sleep01930003-21diskmemload09:54:520
19756102101,3sleep00-21swapper/009:39:460
19615182100,6sleep11962084-21kthreadcore09:30:161
19510292101,3sleep11930003-21diskmemload09:21:111
19510292101,3sleep11930003-21diskmemload09:21:101
19255232102,3sleep11926167-21kthreadcore09:05:181
19196642101,4sleep01919683-21apt-get09:00:130
19145952102,4sleep01914844-21tr08:55:150
19087702101,4sleep21908789-21apt-get08:50:132
19075642101,4sleep01907618-21ntpq08:45:210
18913182101,4sleep11891372-21ntpq08:30:191
18820642101,4sleep11882267-21diskstats08:25:151
18714502104,3sleep2431ktimers/208:15:142
18396442101,4sleep21839662-21apt-get07:45:122
18361812101,4sleep11838006-21grep07:40:191
18288932101,4sleep01828912-21apt-get07:35:160
18130602101,4sleep11813079-21apt-get07:20:151
18055182101,5sleep11805898-21grep07:15:011
2196098290,2sleep01930003-21diskmemload12:03:500
2178712291,2sleep10-21swapper/111:51:001
1988657290,2sleep00-21swapper/009:45:470
1925010291,4sleep21925248-21df09:05:152
1923701290,5sleep10-21swapper/109:00:221
1867613291,4sleep01869396-21tr08:10:190
1839925291,4sleep01840051-21perl07:45:130
1810512291,4sleep21811615-21memory07:15:182
1807715291,3sleep00-21swapper/007:15:120
1876011280,1sleep00-21swapper/008:15:270
1871379281,3sleep11871449-21cpuspeed_turbos08:15:131
1866093280,1sleep20-21swapper/208:10:122
1859801280,1sleep00-21swapper/008:00:250
1855569281,3sleep10-21swapper/108:00:141
1850746281,3sleep01850846-21kthreadcore07:55:150
1849109281,3sleep11849236-21systemctl07:50:251
1838007281,3sleep20-21swapper/207:40:192
1823601281,2sleep01823620-21apt-get07:30:110
1801931281,3sleep21803694-21kthreadcore07:10:182
1918768270,1sleep20-21swapper/208:55:452
1908229270,1sleep20-21swapper/208:46:292
1902952270,1sleep20-21swapper/208:41:552
1897579270,1sleep00-21swapper/008:39:480
1893692270,3sleep21895395-21cut08:35:182
1891969270,1sleep00-21swapper/008:30:390
1886686270,1sleep00-21swapper/008:25:410
1881371270,1sleep10-21swapper/108:20:371
1878026271,2sleep00-21swapper/008:20:170
1870444271,3sleep10-21swapper/108:10:271
1864442270,2sleep21864785-21strings08:05:202
1861383270,3sleep00-21swapper/008:05:170
1855093270,1sleep20-21swapper/207:57:552
1823623270,1sleep10-21swapper/107:30:131
1823302270,1sleep00-21swapper/007:29:340
1823117270,1sleep10-21swapper/107:27:151
1800803271,3sleep01800959-21awk07:10:160
1928840260,3sleep00-21swapper/009:05:210
1913811260,4sleep10-21swapper/108:54:571
1907019260,2sleep11907461-21strings08:45:181
1882268261,2sleep20-21swapper/208:25:162
1862631261,2sleep10-21swapper/108:05:181
1849287261,2sleep00-21swapper/007:50:280
1838364261,2sleep00-21swapper/007:40:220
1828308260,1sleep20-21swapper/207:31:012
1817970260,1sleep00-21swapper/007:23:300
1814306261,2sleep20-21swapper/207:20:202
1919829250,1sleep20-21swapper/209:00:152
1914101250,2sleep10-21swapper/108:55:121
1818351250,1sleep20-21swapper/207:25:122
1908963240,1sleep00-21swapper/008:50:150
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional