You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-11 - 05:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Mar 11, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2012639212157,57sleep30-21swapper/319:08:423
200777629059,9sleep20-21swapper/219:05:082
201247328453,12sleep10-21swapper/119:06:351
201205228152,6sleep00-21swapper/019:05:250
22126892401,12sleep02212700-21sshd21:49:190
22126892401,12sleep02212700-21sshd21:49:180
201283999245,15cyclictest2500486-21taskset00:31:593
2012839992412,8cyclictest2231160-21taskset22:00:103
2012839992219,1cyclictest0-21swapper/322:05:163
24722822212,14sleep12472428-21/usr/sbin/munin00:15:201
2012839992118,1cyclictest0-21swapper/321:58:003
2012839992117,2cyclictest0-21swapper/322:55:203
201283999211,16cyclictest2308156-21taskset22:42:403
201283999210,2cyclictest0-21swapper/323:10:583
24860472201,15sleep12486210-21head00:25:141
22977632201,2sleep00-21swapper/022:35:370
22474462201,2sleep20-21swapper/222:08:572
201283999205,10cyclictest2132816-21taskset21:00:103
2012839992017,1cyclictest0-21swapper/321:20:283
2012839992016,2cyclictest0-21swapper/323:58:393
2012839992015,2cyclictest0-21swapper/300:19:373
201283999201,15cyclictest0-21swapper/323:26:503
24826272190,2sleep00-21swapper/000:21:230
21165732190,7sleep0151ktimers/020:44:590
201283999198,8cyclictest2397418-21turbostat23:39:553
2012839991917,1cyclictest0-21swapper/321:05:203
2012839991916,1cyclictest0-21swapper/323:46:143
2012839991916,1cyclictest0-21swapper/323:02:003
2012839991916,1cyclictest0-21swapper/300:10:413
2012839991915,2cyclictest0-21swapper/323:16:433
2012839991914,2cyclictest0-21swapper/321:50:143
2012839991913,4cyclictest0-21swapper/322:49:533
2012839991912,4cyclictest0-21swapper/319:35:333
201283999191,15cyclictest0-21swapper/322:39:373
201283999191,15cyclictest0-21swapper/322:28:563
201283999191,14cyclictest0-21swapper/323:40:113
201283999191,14cyclictest0-21swapper/323:07:203
2012839991910,2cyclictest0-21swapper/322:19:583
24071632182,11sleep12407954-21cat23:40:151
23687962180,3sleep10-21swapper/123:16:551
23154072181,3sleep00-21swapper/022:45:430
21771552180,1sleep20-21swapper/221:28:362
2012839991816,1cyclictest0-21swapper/319:15:133
2012839991815,1cyclictest0-21swapper/321:18:163
2012839991814,2cyclictest0-21swapper/322:50:123
2012839991814,2cyclictest0-21swapper/321:42:313
2012839991814,2cyclictest0-21swapper/300:00:013
2012839991812,3cyclictest0-21swapper/323:31:273
2012839991812,3cyclictest0-21swapper/321:36:383
201283999181,14cyclictest2430994-21taskset23:52:403
201283999181,14cyclictest0-21swapper/321:45:423
201283999181,14cyclictest0-21swapper/321:45:423
201283999181,14cyclictest0-21swapper/321:30:113
2012839991811,4cyclictest0-21swapper/321:25:513
201283999181,14cyclictest0-21swapper/321:10:183
201283999181,14cyclictest0-21swapper/300:35:123
201283999181,13cyclictest0-21swapper/320:40:113
201283999180,14cyclictest0-21swapper/322:10:163
23931972171,3sleep20-21swapper/223:30:182
23771122171,6sleep0171rcu_preempt23:21:180
23670462171,3sleep02367109-21ntpq23:15:220
23508782171,3sleep00-21swapper/023:06:190
22277312171,3sleep00-21swapper/021:56:040
2012839991714,1cyclictest0-21swapper/300:25:183
2012839991713,2cyclictest0-21swapper/323:20:063
2012839991713,2cyclictest0-21swapper/300:08:333
201283999171,14cyclictest0-21swapper/322:20:463
2012839991711,3cyclictest0-21swapper/300:20:203
201283999171,12cyclictest0-21swapper/322:30:073
201283999171,12cyclictest0-21swapper/320:55:243
201283999170,13cyclictest0-21swapper/320:35:223
201283999170,13cyclictest0-21swapper/319:40:123
24314642161,5sleep02431481-21ssh23:53:180
22654902161,4sleep12143206-21diskmemload22:19:421
22537622161,4sleep22143206-21diskmemload22:10:462
22476232161,2sleep00-21swapper/022:09:100
21930092161,4sleep12143206-21diskmemload21:36:291
2012839991614,1cyclictest0-21swapper/320:15:173
2012839991614,1cyclictest0-21swapper/320:10:163
2012839991612,2cyclictest0-21swapper/320:25:013
201283999160,13cyclictest0-21swapper/320:00:203
201283999160,13cyclictest0-21swapper/319:50:253
201283999160,12cyclictest0-21swapper/319:25:103
24400642151,3sleep12143206-21diskmemload23:58:031
24308262151,11sleep22430841-21ssh23:52:282
23185642151,7sleep12318577-21sh22:49:531
21952032150,3sleep22143206-21diskmemload21:39:232
21917022151,3sleep02143206-21diskmemload21:35:230
21827492151,10sleep12182810-21ldconfig21:30:191
21218602151,10sleep22121880-21cut20:50:002
201283999157,4cyclictest0-21swapper/319:12:583
2012839991511,2cyclictest0-21swapper/319:30:123
2012839991510,2cyclictest0-21swapper/320:30:133
201283999150,13cyclictest0-21swapper/319:55:163
201283999150,12cyclictest0-21swapper/320:25:123
201283999150,12cyclictest0-21swapper/319:45:113
25105802141,6sleep00-21swapper/000:38:470
24926672141,3sleep22143206-21diskmemload00:28:122
24222342141,9sleep02422248-21ssh23:47:400
23862272141,4sleep02386239-21ssh23:26:480
23495222141,6sleep22349581-21ntpq23:05:202
23443192141,8sleep22143206-21diskmemload23:04:132
23076642141,9sleep02307675-21ssh22:42:010
22891732142,1sleep00-21swapper/022:31:150
22790802141,9sleep12279139-21ntpq22:25:201
22311272141,4sleep20-21swapper/222:00:102
21500652142,7sleep22143206-21diskmemload21:12:332
20565222141,9sleep02056609-21/usr/sbin/munin19:45:220
2012839991412,1cyclictest0-21swapper/319:20:163
2012839991411,2cyclictest0-21swapper/320:05:173
201283999140,12cyclictest0-21swapper/320:50:163
24851732131,8sleep12485184-21ssh00:24:451
24819472131,6sleep22143206-21diskmemload00:20:292
24423232131,8sleep02442542-21cat00:00:150
24396792131,7sleep22143206-21diskmemload23:57:322
24231082131,8sleep12143206-21diskmemload23:48:501
23357522130,6sleep12335767-21ssh22:59:261
23231732131,6sleep12323263-21ntpq22:50:211
23171562131,4sleep22317171-21ssh22:48:022
22362362131,3sleep02236247-21ssh22:00:410
22049932131,7sleep22205166-21diskstats21:45:122
22049932131,7sleep22205166-21diskstats21:45:112
22035102131,5sleep20-21swapper/221:43:432
22026392130,3sleep10-21swapper/121:42:341
21759522131,4sleep02175966-21ssh21:27:020
21673772131,8sleep22143206-21diskmemload21:22:152
21565062131,6sleep22156562-21ntpq21:15:202
201283999130,11cyclictest0-21swapper/320:45:153
25016792121,7sleep02501689-21ssh00:33:350
24558772121,3sleep10-21swapper/100:05:451
24461502121,6sleep12446208-21ntpq00:00:201
24409072121,5sleep00-21swapper/023:59:100
23810432121,7sleep12382639-21cut23:25:171
23001412121,7sleep12143206-21diskmemload22:38:471
22879502121,7sleep22288008-21/usr/sbin/munin22:30:222
22830952120,2sleep0171rcu_preempt22:29:480
21689172121,7sleep02168931-21ssh21:24:160
21614252121,7sleep12143206-21diskmemload21:20:121
21581962121,5sleep00-21swapper/021:16:440
21209442121,7sleep02116792-21/usr/sbin/munin20:45:200
20886252121,6sleep22088674-21ntpq20:15:212
20616792121,6sleep12061727-21ntpq19:50:191
20571592120,1sleep10-21swapper/119:46:511
20457782121,5sleep22045825-21ntpq19:35:212
20416892121,5sleep12041708-21apt-get19:35:131
20174502121,7sleep02017694-21unixbench_singl19:10:270
24985712111,5sleep22498618-21ntpq00:30:192
24765252111,4sleep22143206-21diskmemload00:19:542
24685532111,6sleep02468666-21ssh00:15:150
24664752110,7sleep12143206-21diskmemload00:13:111
24656742111,3sleep20-21swapper/200:12:082
24583862111,4sleep00-21swapper/000:09:050
24566612111,4sleep22143206-21diskmemload00:06:462
24121632111,3sleep20-21swapper/223:41:202
24044372111,2sleep20-21swapper/223:37:452
24036122111,5sleep12143206-21diskmemload23:36:391
23962052111,1sleep00-21swapper/023:33:250
23955132110,2sleep10-21swapper/123:32:291
23858372111,3sleep20-21swapper/223:26:182
23606042111,2sleep20-21swapper/223:12:392
23503402111,4sleep12143206-21diskmemload23:05:361
23438982111,5sleep12143206-21diskmemload23:03:401
23278412111,3sleep00-21swapper/022:55:120
23232642111,5sleep22323324-21ntpq22:50:212
23079402111,3sleep20-21swapper/222:42:232
23071192111,4sleep12143206-21diskmemload22:41:221
22991022111,3sleep20-21swapper/222:37:232
22803772111,4sleep22143206-21diskmemload22:26:122
22650532111,3sleep20-21swapper/222:19:062
22473262111,5sleep12143206-21diskmemload22:08:471
22394422111,4sleep12143206-21diskmemload22:04:551
22291332111,4sleep22143206-21diskmemload21:57:552
22211972111,6sleep02221211-21ssh21:53:590
21747312110,7sleep12143206-21diskmemload21:25:271
21603872111,3sleep10-21swapper/121:19:371
21485712111,3sleep00-21swapper/021:10:340
21434602111,3sleep10-21swapper/121:10:031
21062322111,5sleep22106484-21df20:35:132
20635522111,6sleep02064781-21kthreadcore19:55:150
20574082111,5sleep22057456-21munin-run19:50:012
20514892111,5sleep22051733-21users19:40:262
20313302111,6sleep02031515-21cut19:25:130
20258012111,5sleep12026004-21diskstats19:20:141
25083572101,3sleep12143206-21diskmemload00:35:511
25004192101,3sleep12143206-21diskmemload00:31:551
24657022101,2sleep00-21swapper/000:12:090
24477012101,3sleep20-21swapper/200:01:302
24222602101,2sleep20-21swapper/223:47:432
24056212101,3sleep00-21swapper/023:39:190
23798252100,1sleep20-21swapper/223:24:542
23780852101,6sleep12378098-21ssh23:22:351
23627692102,4sleep22362798-21apt-get23:15:112
23605802101,3sleep00-21swapper/023:12:370
23594282101,2sleep10-21swapper/123:11:061
23272432101,3sleep00-21swapper/022:54:450
22897912101,3sleep10-21swapper/122:32:041
22722292101,3sleep12143206-21diskmemload22:22:031
22710912101,2sleep20-21swapper/222:20:322
22703602101,4sleep02270415-21ntpq22:20:220
22573752101,3sleep00-21swapper/022:15:120
22549182101,4sleep12143206-21diskmemload22:12:171
22544032101,3sleep00-21swapper/022:11:360
22223992101,4sleep10-21swapper/121:55:131
22186642101,4sleep22143206-21diskmemload21:50:362
22136072102,3sleep12213629-21apt-get21:50:111
21854002101,3sleep00-21swapper/021:32:580
21853722101,3sleep20-21swapper/221:32:562
21382202101,4sleep02138409-21diskstats21:05:120
21102592101,4sleep12110289-21ntpq20:35:201
21049042101,5sleep22104961-21ntpq20:30:202
20957392101,4sleep02095805-21irqstats20:25:140
20683332101,4sleep02068352-21apt-get20:00:130
20244782101,4sleep12024535-21ntpq19:15:201
2503164290,3sleep20-21swapper/200:35:132
2492541291,5sleep02492555-21ssh00:28:020
2411584291,2sleep00-21swapper/023:40:350
2341664290,1sleep00-21swapper/023:00:420
2333525290,2sleep20-21swapper/222:56:322
2212294290,2sleep10-21swapper/121:48:481
2212294290,2sleep10-21swapper/121:48:471
2204145291,3sleep00-21swapper/021:44:340
2132384290,2sleep10-21swapper/120:58:201
2122120290,4sleep22122139-21apt-get20:50:112
2113033291,4sleep12114804-21cut20:40:161
2100039290,1sleep10-21swapper/120:26:391
2073635291,3sleep22073654-21apt-get20:05:112
2073091292,1sleep10-21swapper/120:01:221
2064784291,4sleep12066606-21kthreadcore19:55:171
2020655291,4sleep22020865-21hddtemp_smartct19:15:132
2428720280,1sleep10-21swapper/123:50:221
2133327281,3sleep22133671-21kthreadcore21:00:142
2110401281,3sleep02110518-21/usr/sbin/munin20:35:230
2091568281,3sleep12093347-21kthreadcore20:20:181
2090481280,1sleep00-21swapper/020:20:160
2084543280,3sleep12084562-21apt-get20:15:121
2082993281,3sleep22083042-21ntpq20:10:212
2040378281,3sleep02040429-21ntpq19:30:180
2143033270,1sleep20-21swapper/221:08:032
2132494271,4sleep2431ktimers/220:59:462
2132156270,1sleep00-21swapper/020:55:300
2126829270,1sleep00-21swapper/020:50:300
2103088271,3sleep0151ktimers/020:30:180
2083985270,1sleep10-21swapper/120:14:441
2078605270,1sleep00-21swapper/020:08:450
2073084270,3sleep20-21swapper/220:01:182
2067751270,1sleep20-21swapper/219:56:142
2062563270,1sleep00-21swapper/019:52:580
2058240271,2sleep20-21swapper/219:50:162
2041349270,1sleep20-21swapper/219:34:022
2041258270,1sleep10-21swapper/119:32:521
2031610271,3sleep22031645-21kthreadcore19:25:152
2025096270,4sleep00-21swapper/019:15:470
2138410260,2sleep10-21swapper/121:05:131
2137601260,1sleep10-21swapper/121:01:421
2121655260,1sleep10-21swapper/120:47:231
2095227260,4sleep20-21swapper/220:25:022
2094965260,4sleep20-21swapper/220:24:512
2083694260,1sleep00-21swapper/020:11:030
2078474260,1sleep10-21swapper/120:07:061
2051951260,1sleep00-21swapper/019:43:040
2051854260,1sleep10-21swapper/119:41:491
2046720260,1sleep00-21swapper/019:39:330
2035795260,4sleep10-21swapper/119:26:101
2030426261,1sleep20-21swapper/219:20:312
2026288261,2sleep02027533-21kthreadcore19:20:170
2017007260,1sleep10-21swapper/119:10:201
2013598261,2sleep20-21swapper/219:10:162
2133070250,2sleep00-21swapper/021:00:120
2104758250,1sleep10-21swapper/120:30:191
2089273250,1sleep00-21swapper/020:15:350
2122418240,1sleep10-21swapper/120:50:151
2111841240,1sleep20-21swapper/220:40:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional