You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-17 - 06:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Wed Dec 17, 2025 00:43:55)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2398381212059,57sleep30-21swapper/319:06:493
239844128256,19sleep00-21swapper/019:07:350
239851628063,9sleep20-21swapper/219:08:322
239844227761,10sleep10-21swapper/119:07:361
239876599261,18cyclictest2438273-21taskset19:45:003
2398765992320,1cyclictest0-21swapper/322:45:013
2398765992320,1cyclictest0-21swapper/300:35:233
2398765992320,1cyclictest0-21swapper/300:30:123
239876599228,12cyclictest0-21swapper/322:35:123
2398765992218,2cyclictest0-21swapper/323:41:253
239876599212,13cyclictest2464801-21taskset20:10:123
2398765992119,1cyclictest0-21swapper/321:35:133
2398765992118,1cyclictest0-21swapper/321:20:223
2398765992118,1cyclictest0-21swapper/320:30:013
2398765992117,2cyclictest0-21swapper/300:05:003
2398765992116,1cyclictest0-21swapper/323:35:003
239876599198,2cyclictest0-21swapper/321:01:003
239876599197,1cyclictest0-21swapper/321:55:163
2398765991917,1cyclictest0-21swapper/319:30:013
2398765991916,1cyclictest0-21swapper/323:55:003
239876599191,15cyclictest0-21swapper/320:35:103
239876599191,14cyclictest2548906-21taskset21:25:283
239876599188,6cyclictest2654179-21taskset23:07:413
2398765991815,1cyclictest0-21swapper/322:55:163
2398765991815,1cyclictest0-21swapper/322:10:183
2398765991815,1cyclictest0-21swapper/320:59:593
2398765991815,1cyclictest0-21swapper/319:15:203
239876599181,4cyclictest0-21swapper/319:50:113
2398765991814,2cyclictest0-21swapper/319:55:253
2398765991812,3cyclictest2732602-21taskset00:20:263
2398765991812,2cyclictest228-1kworker/3:1H+kblockd19:45:173
239876599181,10cyclictest2602375-21taskset22:20:153
239876599178,6cyclictest2648869-21turbostat23:05:003
239876599177,6cyclictest2585939-21taskset22:03:593
239876599174,6cyclictest2480306-21taskset20:21:283
2398765991714,1cyclictest0-21swapper/321:40:163
2398765991713,2cyclictest0-21swapper/321:15:143
2398765991713,2cyclictest0-21swapper/321:05:143
2398765991713,1cyclictest0-21swapper/320:50:173
239876599171,14cyclictest0-21swapper/319:35:223
239876599171,13cyclictest0-21swapper/322:50:123
239876599171,13cyclictest0-21swapper/321:45:203
239876599171,13cyclictest0-21swapper/300:15:183
239876599171,12cyclictest2607241-21chrt22:25:123
239876599171,11cyclictest2416422-21taskset19:24:563
239876599171,10cyclictest2533095-21taskset21:10:543
239876599170,14cyclictest0-21swapper/320:40:303
239876599170,11cyclictest2490760-21taskset20:30:543
27434402167,5sleep12743775-21sed00:35:011
25797412161,9sleep02579775-21ntpq21:55:190
25244402161,10sleep12526150-21kthreadcore21:05:151
239876599169,5cyclictest2665115-21chrt23:20:113
239876599169,4cyclictest0-21swapper/320:45:103
239876599166,6cyclictest2601374-21taskset22:15:543
239876599161,4cyclictest0-21swapper/319:10:113
239876599161,3cyclictest0-21swapper/320:15:133
2398765991612,1cyclictest0-21swapper/323:35:173
239876599161,12cyclictest0-21swapper/323:45:203
239876599161,11cyclictest2733255-21chrt00:25:123
239876599161,10cyclictest2664677-21taskset23:18:193
239876599160,13cyclictest0-21swapper/322:45:193
239876599160,13cyclictest0-21swapper/322:05:223
239876599160,13cyclictest0-21swapper/319:30:203
239876599160,13cyclictest0-21swapper/300:10:253
239876599160,13cyclictest0-21swapper/300:05:173
239876599160,12cyclictest0-21swapper/323:25:183
25380392151,2sleep10-21swapper/121:15:251
239876599151,9cyclictest2659423-21taskset23:13:063
239876599151,9cyclictest2659423-21taskset23:13:063
239876599151,9cyclictest2617165-21taskset22:31:543
239876599151,9cyclictest2575411-21taskset21:53:253
239876599151,2cyclictest0-21swapper/323:55:203
239876599150,13cyclictest0-21swapper/320:05:163
27048282140,9sleep22705619-21sed23:55:182
26282092142,8sleep22628284-21tr22:45:122
25758192141,8sleep12576015-21expr21:55:131
239876599140,2cyclictest0-21swapper/320:00:103
239876599140,12cyclictest0-21swapper/321:30:153
26594722132,8sleep2800-21snmpd23:13:462
26594722132,8sleep2800-21snmpd23:13:462
25271302131,7sleep02527180-21ntpq21:05:180
24971912131,8sleep22498395-21cut20:40:192
24540022131,8sleep22454263-21df_inode20:00:102
24487302131,2sleep20-21swapper/219:55:102
27084622121,7sleep12710212-21kthreadcore00:00:171
26374542121,6sleep12637500-21ntpq22:50:201
25914932121,5sleep02591512-21apt-get22:10:110
25377432122,5sleep02537798-21ntpq21:15:200
25289832121,7sleep12530220-21kthreadcore21:10:161
26793802111,5sleep02679521-21ntpq23:30:190
26336332111,5sleep22633756-21diskstats22:50:132
26330992111,7sleep0800-21snmpd22:48:390
26176742112,3sleep02617693-21apt-get22:35:120
26060682111,6sleep22606156-21ps22:20:232
25967492110,1sleep00-21swapper/022:15:110
25071622111,6sleep22507403-21cstates20:50:132
27439802101,3sleep02743998-21apt-get00:35:120
27320332101,4sleep12732089-21ntp_states00:20:221
27057232101,3sleep02705771-21ntpq23:55:200
26900802101,4sleep00-21swapper/023:40:210
26851782101,4sleep22685422-21users23:35:252
26627262101,4sleep22663648-21memory23:15:172
26598342101,4sleep02659853-21apt-get23:15:120
26585522101,4sleep02658593-21ntpq23:10:190
26585522101,4sleep02658593-21ntpq23:10:180
26533222101,4sleep12653373-21ntpq23:05:191
26392742103,3sleep02640704-21kthreadcore22:55:160
25568362104,3sleep22558554-21latency21:35:182
25553612101,4sleep12556834-21tr21:35:171
25284962101,4sleep22528699-21diskstats21:10:132
25109462100,6sleep02507141-21/usr/sbin/munin20:50:190
24956462101,5sleep02495888-21grep20:35:240
24739222101,4sleep02474012-21ntpq20:15:190
24647802101,3sleep20-21swapper/220:10:132
24586292100,1sleep10-21swapper/120:00:271
24108442100,1sleep00-21swapper/019:15:530
2712907291,4sleep22713851-21cut00:05:162
2712340291,4sleep12712342-21acpi00:05:111
2712078291,4sleep22712102-21munin-plugin-st00:05:002
2695255291,3sleep22695303-21ntpq23:45:202
2654645293,3sleep1331ktimers/123:10:121
2654645293,3sleep1331ktimers/123:10:111
2649374290,3sleep02648871-21apt-get23:05:020
2648857290,5sleep20-21swapper/223:05:002
2610993291,4sleep12611132-21ntp_192.168.11422:25:181
2601990291,4sleep02602008-21apt-get22:20:130
2595488291,4sleep12595543-21ntpq22:10:201
2574538291,3sleep22574586-21ntpq21:50:212
2565299291,2sleep10-21swapper/121:45:111
2553540291,4sleep22553587-21ntpq21:30:202
2539124291,3sleep02539271-21diskstats21:20:120
2522001291,3sleep12522092-21ps21:00:221
2521797291,3sleep22521886-21ntpq21:00:192
2511085291,4sleep12511125-21ntpq20:50:201
2505861291,4sleep02505912-21ld20:45:180
2495202291,4sleep12495319-21ld20:35:161
2481448291,4sleep22482999-21grep20:25:152
2475908290,1sleep20-21swapper/220:20:152
2474013291,3sleep12474060-21ntpq20:15:201
2463205291,3sleep12463251-21ntpq20:05:191
2459858291,4sleep22461475-21cut20:05:162
2444084291,4sleep12444850-21grep19:50:161
2443259291,4sleep12443399-21latency_hist19:50:001
2433099291,4sleep02433317-21df_inode19:40:130
2426083291,4sleep22426221-21netstat19:30:182
2418770291,4sleep12420492-21cat19:25:201
2417282291,3sleep02418764-21ps19:25:190
2403288291,4sleep22403533-21users19:10:262
2402963291,4sleep12403020-21ntp_states19:10:211
2748755280,1sleep20-21swapper/200:37:182
2739029281,3sleep22740474-21cut00:30:162
2726711281,3sleep22726759-21ntpq00:15:182
2717034280,1sleep00-21swapper/000:07:020
2696592280,1sleep10-21swapper/123:50:121
2675564281,2sleep12675583-21apt-get23:30:121
2670893281,3sleep22671751-21kthreadcore23:25:152
2653375281,2sleep22653422-21ntpq23:05:202
2644575281,3sleep00-21swapper/023:00:160
2642425281,3sleep22642555-21needreboot22:55:182
2638437281,3sleep02638486-21munin-run22:54:590
2612562281,3sleep22612704-21diskstats22:30:132
2612437281,3sleep12612561-21cut22:30:121
2607219281,2sleep20-21swapper/222:25:112
2603352281,3sleep12605099-21kthreadcore22:20:181
2595774280,1sleep20-21swapper/222:10:252
2586633281,3sleep22586832-21cat22:05:162
2586306281,3sleep02586562-21df22:05:130
2564062281,3sleep02564115-21ld21:40:210
2549825281,3sleep02550038-21smartctl21:30:150
2544272280,1sleep20-21swapper/221:25:082
2543673280,1sleep10-21swapper/121:20:411
2542970281,2sleep22543018-21ntpq21:20:192
2513061281,3sleep12514415-21kthreadcore20:55:171
2506824280,1sleep10-21swapper/120:49:191
2502124281,4sleep22502235-21grep20:45:112
2496910281,3sleep02497104-21gltestperf20:40:170
2491342280,2sleep20-21swapper/220:35:092
2490888280,1sleep00-21swapper/020:32:320
2490392281,3sleep22490396-21mailstats20:30:252
2486669281,3sleep12486701-21kthreadcore20:30:161
2474327281,3sleep22474363-21sed20:15:262
2469777280,3sleep02469798-21munin-run20:15:000
2468653281,3sleep12468770-21ld20:10:201
2448897281,2sleep02449020-21diskstats19:55:120
2428526281,3sleep22430198-21kthreadcore19:35:192
2427782281,2sleep10-21swapper/119:35:141
2416691281,2sleep22416710-21apt-get19:25:132
2409255281,3sleep12410039-21/usr/sbin/munin19:15:181
2403904281,3sleep02403933-21munin-run19:15:000
2743975270,1sleep10-21swapper/100:35:091
2743246270,1sleep00-21swapper/000:32:310
2732807270,1sleep20-21swapper/200:22:552
2729085271,3sleep00-21swapper/000:20:170
2707601271,2sleep00-21swapper/000:00:160
2694564270,3sleep02695162-21missed_timers23:45:180
2670326270,3sleep12670345-21apt-get23:25:111
2665093270,2sleep20-21swapper/223:20:112
2660991271,2sleep10-21swapper/123:15:161
2623225271,3sleep10-21swapper/122:40:141
2621681271,2sleep12621735-21ntpq22:35:211
2586563271,3sleep12586609-21fschecks_count22:05:141
2581048270,2sleep20-21swapper/222:00:112
2575257270,1sleep00-21swapper/021:51:280
2570016270,1sleep00-21swapper/021:46:150
2564778270,1sleep10-21swapper/121:41:421
2554779270,2sleep00-21swapper/021:35:120
2554485270,1sleep10-21swapper/121:34:361
2549231270,1sleep10-21swapper/121:29:361
2512881271,2sleep00-21swapper/020:55:160
2480533270,1sleep10-21swapper/120:24:231
2480241270,1sleep00-21swapper/020:20:420
2448724270,1sleep10-21swapper/119:55:051
2448311270,1sleep00-21swapper/019:53:000
2444851271,3sleep20-21swapper/219:50:172
2437979270,1sleep10-21swapper/119:44:061
2427167270,4sleep10-21swapper/119:34:001
2411742271,3sleep00-21swapper/019:20:130
2738012260,4sleep00-21swapper/000:27:510
2733233260,2sleep20-21swapper/200:25:122
2727379260,4sleep00-21swapper/000:15:480
2722185260,4sleep20-21swapper/200:11:002
2718103261,2sleep10-21swapper/100:10:151
2717530261,1sleep02717549-21apt-get00:10:110
2703093261,2sleep10-21swapper/123:55:171
2701296260,4sleep20-21swapper/223:52:302
2697091260,2sleep00-21swapper/023:50:140
2695970260,1sleep10-21swapper/123:46:191
2690142261,2sleep10-21swapper/123:40:221
2682863261,2sleep10-21swapper/123:35:171
2680820260,2sleep00-21swapper/023:35:120
2675555260,4sleep20-21swapper/223:30:052
2669381260,3sleep0221rcuc/023:20:250
2642959261,2sleep10-21swapper/122:55:251
2622931260,1sleep20-21swapper/222:40:112
2621737261,2sleep20-21swapper/222:35:212
2612410260,1sleep00-21swapper/022:30:070
2596724260,1sleep20-21swapper/222:15:102
2585959260,4sleep00-21swapper/022:04:140
2566328261,2sleep20-21swapper/221:45:162
2564910260,1sleep20-21swapper/221:43:232
2538476260,4sleep20-21swapper/221:17:112
2533118260,1sleep00-21swapper/021:11:070
2523434261,2sleep20-21swapper/221:05:122
2519378261,2sleep00-21swapper/021:00:170
2517235260,1sleep20-21swapper/220:58:302
2500540260,1sleep10-21swapper/120:40:221
2485668260,4sleep10-21swapper/120:27:551
2463075261,2sleep00-21swapper/020:05:170
2442968260,1sleep20-21swapper/219:46:172
2442571261,2sleep00-21swapper/019:45:270
2427116260,4sleep00-21swapper/019:33:190
2412024261,2sleep20-21swapper/219:20:162
2411455261,2sleep10-21swapper/119:20:101
2404338261,2sleep20-21swapper/219:15:032
2690703250,4sleep20-21swapper/223:40:552
2668909250,1sleep10-21swapper/123:20:181
2628187250,2sleep12628206-21apt-get22:45:111
2623110250,1sleep00-21swapper/022:40:130
2609835250,1sleep00-21swapper/022:25:180
2597008250,2sleep10-21swapper/122:15:121
2580522250,1sleep20-21swapper/221:56:592
2570572250,1sleep10-21swapper/121:50:121
2481138250,1sleep00-21swapper/020:25:130
2454389250,2sleep00-21swapper/020:00:130
2433031250,1sleep20-21swapper/219:40:112
2431588250,2sleep00-21swapper/019:35:210
2733532240,1sleep10-21swapper/100:25:151
2722501240,1sleep10-21swapper/100:15:011
2670616240,1sleep00-21swapper/023:25:140
2644151240,1sleep10-21swapper/123:00:141
2581347240,1sleep10-21swapper/122:00:141
2544562240,1sleep00-21swapper/021:25:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional