You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 13:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot3s.osadl.org (updated Sun Nov 23, 2025 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
224967622430,9sleep10-21swapper/100:35:221
1778114213661,68sleep30-21swapper/319:05:143
178269128856,10sleep20-21swapper/219:08:542
178248828352,10sleep10-21swapper/119:06:201
178263627555,14sleep00-21swapper/019:08:120
20405472261,3sleep00-21swapper/022:25:320
178290299252,17cyclictest2125133-21taskset23:20:113
178290299252,17cyclictest1937317-21taskset21:25:143
1782902992511,11cyclictest0-21swapper/321:19:053
19456912241,11sleep01945721-21forks21:30:120
1782902992420,2cyclictest0-21swapper/300:30:373
22212392231,2sleep00-21swapper/000:19:510
1782902992320,1cyclictest0-21swapper/323:17:453
1782902992320,1cyclictest0-21swapper/320:40:013
1782902992320,1cyclictest0-21swapper/300:20:193
1782902992317,2cyclictest0-21swapper/322:41:033
1782902992311,9cyclictest1846350-21chrt20:05:173
178290299231,16cyclictest1975765-21chrt21:46:513
178290299222,13cyclictest2197231-21taskset00:04:093
178290299222,13cyclictest2197231-21taskset00:04:083
1782902992219,1cyclictest0-21swapper/300:35:173
1782902992218,2cyclictest0-21swapper/323:30:003
1782902992217,3cyclictest0-21swapper/321:31:173
1782902992217,2cyclictest0-21swapper/323:05:113
21315302211,3sleep10-21swapper/123:22:551
20102282216,9sleep01912456-21diskmemload22:09:330
19693982211,3sleep00-21swapper/021:44:390
178290299219,9cyclictest1843074-21chrt20:00:573
178290299212,13cyclictest2156598-21taskset23:38:483
178290299212,13cyclictest2041113-21taskset22:26:243
178290299212,12cyclictest1991279-21taskset21:55:333
178290299211,9cyclictest2057941-21taskset22:37:223
1782902992117,2cyclictest0-21swapper/322:55:153
1782902992117,2cyclictest0-21swapper/300:15:013
178290299211,16cyclictest2204831-21chrt00:08:413
178290299211,15cyclictest2093376-21chrt23:00:143
20914412204,4sleep2431ktimers/222:58:342
178290299202,14cyclictest2165485-21sleep23:44:593
1782902992018,1cyclictest0-21swapper/321:00:213
1782902992018,1cyclictest0-21swapper/300:25:123
1782902992017,1cyclictest0-21swapper/322:46:183
1782902992016,2cyclictest0-21swapper/319:19:423
1782902992015,2cyclictest0-21swapper/322:50:153
1782902992014,3cyclictest0-21swapper/323:56:243
1782902992014,3cyclictest0-21swapper/322:21:103
178290299201,15cyclictest1969384-21chrt21:44:373
178290299200,13cyclictest2017948-21taskset22:13:483
178290299192,13cyclictest2049906-21taskset22:32:283
178290299192,13cyclictest2024729-21taskset22:16:333
178290299192,11cyclictest1978751-21taskset21:50:153
1782902991916,2cyclictest0-21swapper/322:04:593
1782902991916,1cyclictest0-21swapper/321:20:233
1782902991916,1cyclictest0-21swapper/320:30:173
1782902991916,1cyclictest0-21swapper/319:25:193
1782902991914,3cyclictest0-21swapper/300:15:173
178290299191,15cyclictest0-21swapper/323:10:033
178290299191,14cyclictest2170644-21chrt23:45:293
178290299191,14cyclictest0-21swapper/323:50:163
19431792181,3sleep10-21swapper/121:27:051
1782902991813,2cyclictest0-21swapper/320:20:153
178290299181,13cyclictest1955625-21taskset21:35:203
178290299181,13cyclictest0-21swapper/323:30:193
178290299181,13cyclictest0-21swapper/320:50:123
178290299181,12cyclictest1916886-21chrt21:10:183
178290299180,11cyclictest1831326-21taskset19:50:193
22365772171,4sleep21912456-21diskmemload00:29:142
21653602171,11sleep12165363-21ssh23:44:521
21496182171,2sleep2171rcu_preempt23:35:102
20927252171,4sleep21912456-21diskmemload23:00:072
20688432171,11sleep12069033-21sed22:45:141
20669122171,10sleep21912456-21diskmemload22:43:312
20495292171,4sleep01912456-21diskmemload22:31:550
20335402171,4sleep11912456-21diskmemload22:22:221
20245662171,4sleep11912456-21diskmemload22:16:231
20239752171,3sleep00-21swapper/022:15:300
20185122171,4sleep21912456-21diskmemload22:14:402
20097552170,2sleep20-21swapper/222:08:512
20020132171,4sleep01912456-21diskmemload22:04:330
19607242171,9sleep00-21swapper/021:38:580
19267382171,3sleep10-21swapper/121:16:581
18049622171,10sleep21804986-21ntpq19:25:212
178290299175,10cyclictest0-21swapper/321:05:233
1782902991714,1cyclictest0-21swapper/320:15:163
178290299171,13cyclictest2008136-21chrt22:06:233
178290299171,13cyclictest0-21swapper/319:30:193
178290299171,12cyclictest1795762-21chrt19:20:133
178290299171,12cyclictest1795762-21chrt19:20:123
178290299170,13cyclictest0-21swapper/319:10:233
178290299170,11cyclictest1816247-21chrt19:35:473
22295232162,8sleep12229537-21apt-get00:25:131
21982272161,3sleep10-21swapper/100:05:131
21878852161,3sleep20-21swapper/223:57:072
21726472161,2sleep20-21swapper/223:48:372
21557122161,4sleep11912456-21diskmemload23:37:251
21555752161,8sleep01912456-21diskmemload23:37:120
21482092161,8sleep11912456-21diskmemload23:33:261
21164452160,3sleep20-21swapper/223:14:462
20753662161,4sleep21912456-21diskmemload22:48:322
20660442161,2sleep00-21swapper/022:42:130
20421102161,4sleep11912456-21diskmemload22:27:541
19372852163,6sleep21937313-21apt-get21:25:142
18741292161,10sleep01874177-21ntpq20:30:210
178290299162,9cyclictest1869549-21taskset20:25:503
178290299162,9cyclictest1869549-21taskset20:25:493
178290299161,9cyclictest1886339-21taskset20:45:023
1782902991611,3cyclictest0-21swapper/320:55:133
178290299160,12cyclictest0-21swapper/319:55:143
22370542151,7sleep02237079-21munin-run00:30:010
22035302151,8sleep21912456-21diskmemload00:06:212
21967222151,9sleep02196732-21ssh00:03:150
21967222151,9sleep02196732-21ssh00:03:140
21956662151,9sleep22195678-21ssh00:01:292
21956662151,9sleep22195678-21ssh00:01:282
21788012151,9sleep02178811-21ssh23:50:360
21080212150,2sleep10-21swapper/123:09:151
20978612150,2sleep10-21swapper/123:00:501
20978482151,8sleep01912456-21diskmemload23:00:490
20645642151,9sleep11912456-21diskmemload22:40:261
20286032151,9sleep22030226-21kthreadcore22:20:172
19016102150,8sleep272-21kcompactd020:56:002
18959882151,10sleep21896057-21/usr/sbin/munin20:50:262
178290299151,9cyclictest1821634-21taskset19:41:483
1782902991512,2cyclictest0-21swapper/319:45:143
1782902991511,2cyclictest0-21swapper/320:10:283
178290299150,13cyclictest0-21swapper/320:40:173
22283122141,9sleep02228321-21sh00:23:310
22204682140,1sleep20-21swapper/200:18:292
21969462140,2sleep10-21swapper/100:03:401
21969462140,2sleep10-21swapper/100:03:391
21484852141,9sleep22148497-21ssh23:33:512
21250992142,6sleep22125125-21apt-get23:20:122
20268442141,3sleep22026854-21ssh22:19:452
19761612140,3sleep21912456-21diskmemload21:47:272
19258522140,2sleep21912456-21diskmemload21:15:362
19192232141,9sleep11919234-21sh21:12:581
18761752141,9sleep11877827-21grep20:35:181
21455572131,6sleep02145605-21ntpq23:30:200
21385182131,2sleep10-21swapper/123:26:031
21007052131,6sleep01912456-21diskmemload23:05:010
20829032131,5sleep21912456-21diskmemload22:52:412
20569312131,8sleep22056942-21ssh22:35:492
20489962131,5sleep20-21swapper/222:31:052
20161722131,8sleep12016180-21ssh22:11:071
20011832131,3sleep12001193-21ssh22:03:121
19912932131,2sleep00-21swapper/021:55:340
19672552131,5sleep10-21swapper/121:41:241
19515322131,3sleep21951541-21ssh21:32:272
19072222131,8sleep11907312-21latency_hist21:05:001
17963002131,3sleep21797116-21cut19:20:162
17963002131,3sleep21797116-21cut19:20:162
22506532120,8sleep02250663-21ssh00:36:060
22206142121,3sleep10-21swapper/100:18:441
22127532121,2sleep02212758-21ssh00:13:490
22114092121,7sleep22211412-21ssh00:11:302
21888522121,4sleep10-21swapper/123:58:381
21152322121,4sleep11912456-21diskmemload23:12:531
21144862121,7sleep02114489-21ssh23:11:440
21082372120,1sleep20-21swapper/223:09:362
20908472121,5sleep10-21swapper/122:57:371
20904202121,5sleep00-21swapper/022:56:550
20833872121,6sleep00-21swapper/022:53:280
20596182121,5sleep11912456-21diskmemload22:39:531
19827612121,3sleep20-21swapper/221:50:272
19791062121,6sleep01980669-21cut21:50:170
19505262120,1sleep10-21swapper/121:30:531
19293332122,5sleep01929459-21diskstats21:20:150
18863542121,5sleep11886373-21apt-get20:45:131
18052922121,7sleep01805367-21/usr/sbin/munin19:25:270
22453782111,4sleep21912456-21diskmemload00:35:072
22441872111,4sleep21912456-21diskmemload00:33:592
22279412111,3sleep20-21swapper/200:22:532
22275012110,5sleep10-21swapper/100:22:071
22117432111,4sleep11912456-21diskmemload00:12:051
22038862111,3sleep00-21swapper/000:07:000
21801812111,4sleep21912456-21diskmemload23:52:472
21739822111,5sleep12174194-21diskstats23:50:131
21724542111,4sleep11912456-21diskmemload23:48:201
21619622111,5sleep087-21kswapd023:40:220
21389442111,3sleep00-21swapper/023:26:420
21375752111,4sleep21912456-21diskmemload23:25:242
20823782111,4sleep11912456-21diskmemload22:51:481
20762802111,4sleep01912456-21diskmemload22:49:580
20406922111,3sleep20-21swapper/222:25:462
20343632111,4sleep01912456-21diskmemload22:23:370
20069122111,7sleep12006968-21/usr/sbin/munin22:05:211
20002472111,4sleep21912456-21diskmemload22:01:452
19936122111,4sleep11912456-21diskmemload21:59:081
19903842111,5sleep21990508-21ssh21:55:212
19763912111,4sleep01912456-21diskmemload21:47:480
19596492111,4sleep21912456-21diskmemload21:37:212
19588522110,1sleep10-21swapper/121:36:071
19438122111,4sleep01912456-21diskmemload21:28:040
19180042111,4sleep01912456-21diskmemload21:11:010
18956802111,5sleep11895729-21ntpq20:50:211
18871522111,6sleep21888426-21kthreadcore20:45:182
18654432111,6sleep01866351-21kthreadcore20:25:200
18654432111,6sleep01866351-21kthreadcore20:25:190
18548122111,6sleep11855240-21cut20:15:181
18530582110,2sleep20-21swapper/220:10:232
18260942111,5sleep01826207-21ntpq19:45:190
22429362101,3sleep01912456-21diskmemload00:31:400
22426532101,3sleep11912456-21diskmemload00:31:091
21870252101,3sleep00-21swapper/023:55:450
21716942101,3sleep01912456-21diskmemload23:47:070
21642482101,4sleep21912456-21diskmemload23:43:092
21292802101,4sleep02129331-21ntpq23:20:220
21229732101,3sleep11912456-21diskmemload23:17:211
21209862101,4sleep02121078-21ntpq23:15:200
21168672101,3sleep20-21swapper/223:15:012
20583142101,4sleep01912456-21diskmemload22:37:550
20437522101,4sleep11912456-21diskmemload22:30:071
20107482101,3sleep00-21swapper/022:10:020
19831822101,3sleep11912456-21diskmemload21:50:421
19749642101,3sleep10-21swapper/121:45:371
19356982101,5sleep11935706-21ssh21:23:071
19347892101,3sleep21912456-21diskmemload21:21:442
19267802100,4sleep01912456-21diskmemload21:17:010
19165742101,4sleep21916880-21strings21:10:182
19124462101,4sleep01912500-21munin-run21:10:000
19022122101,4sleep21902231-21apt-get21:00:122
18801472103,1sleep2171rcu_preempt20:35:562
18707242101,5sleep11872241-21tr20:30:171
18689152101,5sleep11868967-21ntpq20:25:241
18689152101,5sleep11868967-21ntpq20:25:241
18646132103,4sleep1618-21gdbus20:25:011
18487052101,5sleep01848800-21cat20:10:000
18274832101,4sleep01827502-21apt-get19:50:120
18118252101,4sleep21812035-21hddtemp_smartct19:35:152
18065362101,4sleep01806745-21hddtemp_smartct19:30:170
17954552101,4sleep21795526-21munin-run19:20:002
17830652100,3sleep00-21swapper/019:10:040
1967229291,2sleep20-21swapper/221:41:212
1908051291,4sleep21909057-21cut21:05:182
1892146290,1sleep00-21swapper/020:50:170
1858326291,3sleep21858373-21ntpq20:15:232
1844204291,4sleep11844627-21tr20:05:161
1836969291,3sleep01837050-21ntpq19:55:210
1833606291,4sleep11833026-21/usr/sbin/munin19:55:181
1821198291,4sleep11821310-21ls19:40:281
1812136291,4sleep01813658-21cut19:35:170
1798871291,4sleep01799583-21memory19:20:190
1798871291,4sleep01799583-21memory19:20:180
1881031281,3sleep21881050-21apt-get20:40:122
1869800280,1sleep20-21swapper/220:29:022
1869800280,1sleep20-21swapper/220:29:012
1860140281,3sleep00-21swapper/020:20:180
1826208281,3sleep11826260-21ld19:45:201
1823685281,3sleep21825458-21kthreadcore19:45:172
1821312281,3sleep21821518-21users19:40:292
1912187270,2sleep10-21swapper/121:06:441
1897512271,2sleep01897546-21kthreadcore20:55:170
1885400270,3sleep01885471-21find20:40:270
1881017270,4sleep10-21swapper/120:40:031
1875440270,2sleep00-21swapper/020:35:130
1858808270,4sleep00-21swapper/020:15:300
1853723270,1sleep10-21swapper/120:11:211
1838372271,3sleep00-21swapper/020:00:140
1833190270,2sleep21833313-21diskstats19:55:152
1817440271,3sleep00-21swapper/019:40:190
1806818271,3sleep21808213-21kthreadcore19:30:192
1788019270,1sleep10-21swapper/119:13:381
1783645271,2sleep20-21swapper/219:10:192
1898707261,2sleep10-21swapper/120:55:181
1891088260,1sleep00-21swapper/020:46:150
1875009260,1sleep20-21swapper/220:32:542
1848543260,1sleep20-21swapper/220:07:572
1843396261,2sleep10-21swapper/120:05:011
1827758261,2sleep20-21swapper/219:50:152
1810226261,2sleep10-21swapper/119:30:221
1801304261,2sleep10-21swapper/119:25:161
1794544260,2sleep10-21swapper/119:15:231
1791665261,1sleep00-21swapper/019:15:180
1849439250,1sleep00-21swapper/020:10:160
1843195250,1sleep20-21swapper/220:02:292
1828394250,1sleep10-21swapper/119:50:181
1811710250,2sleep10-21swapper/119:35:141
1795739250,2sleep10-21swapper/119:20:121
1795739250,2sleep10-21swapper/119:20:111
1902121240,1sleep00-21swapper/021:00:030
1859869240,1sleep20-21swapper/220:20:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional