You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 08:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  TQ-Systems
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot3s.osadl.org (updated Tue Jul 01, 2025 00:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
261825822260,2sleep10-21swapper/122:08:321
24027842146100,16sleep30-21swapper/319:05:543
2403006214297,15sleep00-21swapper/019:08:430
24029422141100,34sleep10-21swapper/119:07:531
24029332131109,15sleep20-21swapper/219:07:462
247759221300,2sleep00-21swapper/020:20:200
2403232993833,3cyclictest0-21swapper/322:45:163
2403232993421,10cyclictest0-21swapper/321:10:203
2403232992923,4cyclictest0-21swapper/322:25:303
2403232992722,3cyclictest0-21swapper/300:30:173
2403232992520,3cyclictest1-21systemd19:39:593
2403232992512,9cyclictest262-21systemd-journal22:01:473
2403232992420,2cyclictest0-21swapper/323:27:303
2403232992420,2cyclictest0-21swapper/300:20:003
2403232992419,3cyclictest1-21systemd00:00:013
2403232992419,3cyclictest0-21swapper/323:20:263
240323299241,19cyclictest2834392-21sshd00:27:463
2403232992319,2cyclictest0-21swapper/320:05:173
2403232992314,6cyclictest2683756-21sh22:50:163
2403232992313,7cyclictest2809243-21grep00:10:223
2403232992312,8cyclictest2778743-21tr23:50:173
2403232992218,2cyclictest0-21swapper/321:35:033
2403232992217,3cyclictest2722809-21grep23:15:143
2403232992217,3cyclictest2588733-21head21:50:163
2403232992216,4cyclictest1-21systemd22:16:473
2403232992215,5cyclictest2439602-21grep19:40:313
2403232992214,6cyclictest2773979-21grep23:45:283
2403232992214,5cyclictest2557248-21fschecks_count21:30:163
2403232992211,7cyclictest262-21systemd-journal00:00:473
2403232992211,7cyclictest2419329-21/usr/sbin/munin19:20:283
2403232992210,8cyclictest2585151-21wc21:45:323
240323299214,4cyclictest2490995-21/usr/sbin/munin20:35:163
2403232992118,2cyclictest2534018-21kthreadcore21:15:163
240323299211,7cyclictest2407237-21ntpq19:10:213
2403232992117,2cyclictest2720496-21sh23:12:343
2403232992117,2cyclictest2667361-21packagekitd22:40:033
2403232992117,2cyclictest0-21swapper/323:40:173
2403232992117,2cyclictest0-21swapper/300:20:233
2403232992116,3cyclictest1056-21cron19:55:013
2403232992114,5cyclictest2504107-21cut20:45:183
2403232992112,6cyclictest943-21runrttasks23:30:463
240323299211,18cyclictest2596623-21runrttasks21:55:193
240323299211,18cyclictest2579984-21taskset21:44:523
240323299211,17cyclictest2707027-21cut23:05:163
2403232992110,8cyclictest2613682-21cat22:05:203
268279222010,4sleep2421ktimers/222:49:462
240323299208,8cyclictest2638060-21tr22:20:203
240323299202,14cyclictest2525370-21cron21:10:013
240323299201,6cyclictest2848769-21runrttasks00:36:303
2403232992016,2cyclictest0-21swapper/320:50:253
240323299201,5cyclictest2625545-21sh22:12:473
2403232992015,3cyclictest0-21swapper/322:30:013
2403232992014,4cyclictest511ktimers/322:36:393
2403232992013,5cyclictest2477723-21cut20:20:213
2403232992011,6cyclictest2699141-21/usr/sbin/munin23:00:143
240323299201,16cyclictest2415141-21/usr/sbin/munin19:20:003
240323299200,8cyclictest2547183-21seq21:22:253
26966002190,2sleep20-21swapper/222:56:422
2403232991915,3cyclictest2429033-21ntp_kernel_err19:30:223
2403232991911,6cyclictest2758848-21sh23:35:583
2403232991911,6cyclictest2440663-21grep19:45:143
2403232991911,6cyclictest2440663-21grep19:45:133
2403232991910,7cyclictest0-21swapper/321:27:143
2403232991910,6cyclictest2802538-21ssh00:07:083
2403232991910,6cyclictest2515784-21/usr/sbin/munin21:00:143
2403232991910,6cyclictest2470066-21dpkg20:15:013
2403232991814,2cyclictest0-21swapper/322:55:143
2403232991811,5cyclictest2513216-21cut20:55:183
240323299180,5cyclictest2484736-21/usr/sbin/munin20:25:263
28354152171,4sleep22525376-21diskmemload00:29:392
27915312171,7sleep02791545-21ssh23:57:580
26104012170,3sleep10-21swapper/122:03:151
240323299177,7cyclictest2489326-21ntpq20:30:213
240323299171,3cyclictest2450550-21fschecks_count19:55:143
28501002161,9sleep20-21swapper/200:38:422
26509512161,10sleep12651405-21chmod22:30:011
26410982160,2sleep10-21swapper/122:22:161
25210962161,4sleep12522119-21grep21:05:181
240323299161,3cyclictest2499372-21ntpq20:40:203
2403232991612,2cyclictest0-21swapper/320:00:133
2403232991612,2cyclictest0-21swapper/319:25:143
240323299160,14cyclictest2474873-21grep20:15:293
28429062151,10sleep22842916-21ssh00:34:292
27841982151,3sleep02525376-21diskmemload23:53:550
27745202151,8sleep12525376-21diskmemload23:46:091
27612522150,6sleep10-21swapper/123:39:491
27060292151,3sleep22706039-21ssh23:04:302
26505552150,1sleep10-21swapper/122:29:201
25951342151,2sleep10-21swapper/121:54:051
25636012150,7sleep22539210-21kworker/u16:0+rpciod21:34:022
27616582141,7sleep02525376-21diskmemload23:40:040
27130992140,1sleep00-21swapper/023:08:130
27029752141,5sleep02703034-21ntp_states23:00:220
26030272141,9sleep02603041-21ssh21:58:580
25447352141,9sleep02541402-21/usr/sbin/munin21:20:210
24640622140,2sleep20-21swapper/220:05:202
28331552132,5sleep1945-21snmpd00:25:381
27827632130,1sleep20-21swapper/223:51:442
27779932131,7sleep12779560-21kthreadcore23:50:191
27704822131,8sleep22772101-21tr23:45:192
27130622131,5sleep10-21swapper/123:08:101
25056232131,8sleep22505779-21diskstats20:50:132
24903342131,8sleep02490533-21latency_hist20:35:000
24470422130,2sleep20-21swapper/219:50:172
28093422120,8sleep22809356-21ssh00:10:322
27857292121,7sleep22785855-21kthreadcore23:55:162
27666652120,1sleep20-21swapper/223:41:052
27387182121,7sleep20-21swapper/223:25:182
26737432121,7sleep02525376-21diskmemload22:43:030
26634882121,7sleep12663640-21ssh22:35:261
26294472126,3sleep0171ktimers/022:15:200
26237152121,2sleep10-21swapper/122:10:281
26120572120,1sleep20-21swapper/222:05:172
25943322122,7sleep02525376-21diskmemload21:52:430
25718482121,4sleep10-21swapper/121:39:231
25647072120,8sleep22564269-21apt-get21:35:022
25640182121,4sleep00-21swapper/021:34:400
24507752121,7sleep02451226-21kthreadcore19:55:160
28447462111,6sleep12846494-21kthreadcore00:35:161
28427652111,6sleep02842771-21sh00:34:100
28173642111,5sleep02525376-21diskmemload00:15:470
28118732111,3sleep10-21swapper/100:14:281
27913142111,3sleep10-21swapper/123:57:391
27432972111,3sleep00-21swapper/023:26:210
27432582111,3sleep10-21swapper/123:26:181
27351952111,7sleep10-21swapper/123:20:511
27277462111,3sleep00-21swapper/023:16:390
27215552111,3sleep10-21swapper/123:14:141
27137422111,5sleep22525376-21diskmemload23:09:102
26652542110,6sleep00-21swapper/022:37:330
26565472111,3sleep20-21swapper/222:31:032
26408152111,3sleep00-21swapper/022:21:510
26400662111,3sleep20-21swapper/222:20:352
26254062111,3sleep00-21swapper/022:12:370
26086832111,3sleep20-21swapper/222:00:392
25995912111,4sleep12599945-21strings21:55:221
25941342111,4sleep22525376-21diskmemload21:52:252
25798752111,6sleep02579885-21ssh21:44:420
25698462111,3sleep00-21swapper/021:36:050
25463532110,1sleep10-21swapper/121:21:071
24037182110,5sleep016-21ksoftirqd/019:10:150
28416642100,6sleep12841675-21ssh00:32:151
28334622101,3sleep02525376-21diskmemload00:26:100
28271002101,3sleep20-21swapper/200:23:232
28256852101,2sleep10-21swapper/100:21:131
28095892101,2sleep00-21swapper/000:10:540
28012472101,2sleep20-21swapper/200:05:102
27999272101,3sleep02525376-21diskmemload00:03:330
27997222101,4sleep22525376-21diskmemload00:03:152
27985812101,3sleep10-21swapper/100:01:291
27754772101,3sleep00-21swapper/023:47:360
27617762101,4sleep12761795-21apt-get23:40:131
27606922101,2sleep00-21swapper/023:38:550
27586262102,4sleep22753851-21kworker/u16:0+xprtiod23:35:412
27516992101,3sleep02525376-21diskmemload23:32:100
27515942100,2sleep10-21swapper/123:31:581
27363082101,3sleep20-21swapper/223:22:372
27342502101,4sleep02734299-21ntpq23:20:220
27273802101,3sleep12525376-21diskmemload23:16:031
27270532100,3sleep20-21swapper/223:15:322
27199142101,3sleep00-21swapper/023:11:400
27145482100,2sleep20-21swapper/223:10:032
27043902101,4sleep12525376-21diskmemload23:01:481
26984192101,2sleep10-21swapper/122:59:391
26894482101,3sleep02525376-21diskmemload22:52:490
26880582100,1sleep20-21swapper/222:50:392
26838292101,4sleep12683868-21kthreadcore22:50:181
26801832100,3sleep10-21swapper/122:45:441
26753952101,2sleep00-21swapper/022:45:070
26728512101,3sleep20-21swapper/222:41:412
26717502100,1sleep10-21swapper/122:40:281
26659762101,2sleep20-21swapper/222:38:432
26567142101,3sleep02525376-21diskmemload22:31:210
26332372101,3sleep10-21swapper/122:17:471
26234412100,6sleep20-21swapper/222:10:252
26180112101,3sleep00-21swapper/022:08:080
26099132101,2sleep00-21swapper/022:02:300
26011312101,2sleep20-21swapper/221:56:112
25870852101,3sleep20-21swapper/221:48:252
25867642101,3sleep10-21swapper/121:47:521
25867602101,3sleep00-21swapper/021:47:510
25793912101,2sleep20-21swapper/221:43:572
25785252100,2sleep10-21swapper/121:42:301
25624072101,2sleep10-21swapper/121:32:051
25542132101,2sleep00-21swapper/021:26:150
25413992101,4sleep22541421-21apt-get21:20:152
25408032101,3sleep20-21swapper/221:19:422
25401062100,5sleep02525376-21diskmemload21:18:320
25385792101,3sleep10-21swapper/121:16:171
25324532101,3sleep00-21swapper/021:14:000
25303042101,3sleep22525376-21diskmemload21:10:292
25259812100,2sleep10-21swapper/121:10:151
24954392100,2sleep00-21swapper/020:40:010
24189972101,4sleep22419071-21ntpq19:20:242
2843532291,3sleep02843558-21apt-get00:35:110
2825999290,2sleep00-21swapper/000:21:400
2818393291,3sleep12525376-21diskmemload00:17:171
2812531291,3sleep22525376-21diskmemload00:15:042
2801636290,2sleep00-21swapper/000:05:440
2801458291,2sleep10-21swapper/100:05:291
2691112290,3sleep02525376-21diskmemload22:55:060
2650769291,2sleep20-21swapper/222:29:422
2650006291,3sleep02525376-21diskmemload22:28:270
2632307291,3sleep22525376-21diskmemload22:16:162
2555854290,3sleep12525376-21diskmemload21:28:471
2515247290,1sleep10-21swapper/120:58:571
2510763291,4sleep22510777-21diskstats20:55:142
2510597291,3sleep02510616-21apt-get20:55:110
2494350291,3sleep12494397-21ntpq20:35:221
2469922296,1sleep00-21swapper/020:13:390
2464011291,4sleep12464060-21ntpq20:05:191
2459037291,4sleep22459088-21ntpq20:00:202
2420073291,4sleep12420088-21munin-run19:25:001
2410846290,5sleep22412470-21cut19:15:142
2403428291,4sleep22403581-21df_inode19:10:142
2749362281,4sleep22749837-21grep23:30:182
2554416281,3sleep22554423-21ssh21:26:322
2519957281,4sleep02519970-21head21:00:290
2458958281,4sleep12459036-21ld20:00:191
2449703280,1sleep00-21swapper/019:50:280
2424871280,1sleep00-21swapper/019:27:470
2414047281,2sleep10-21swapper/119:15:171
2524325270,2sleep00-21swapper/021:05:210
2515619270,3sleep12515637-21apt-get21:00:141
2500776271,3sleep12501006-21idleruntime20:45:161
2500584270,2sleep20-21swapper/220:45:132
2471037271,3sleep02472467-21ps20:15:210
2454765270,1sleep20-21swapper/219:56:102
2445822271,3sleep12447039-21kthreadcore19:50:171
2440341270,2sleep10-21swapper/119:45:121
2440341270,2sleep10-21swapper/119:45:111
2435980272,2sleep0221rcuc/019:40:210
2425295270,1sleep20-21swapper/219:30:022
2525079260,1sleep20-21swapper/221:06:242
2520280260,4sleep20-21swapper/221:03:522
2505602260,2sleep10-21swapper/120:50:121
2500018260,1sleep20-21swapper/220:41:112
2499365260,2sleep00-21swapper/020:40:220
2489981260,1sleep20-21swapper/220:30:372
2484212260,3sleep22484333-21netstat20:25:212
2481120261,2sleep00-21swapper/020:25:190
2479339260,2sleep20-21swapper/220:20:242
2475198260,1sleep20-21swapper/220:18:102
2474653261,2sleep12474803-21timerwakeupswit20:15:301
2465701261,2sleep20-21swapper/220:10:182
2464921260,1sleep00-21swapper/020:08:510
2455525261,2sleep00-21swapper/020:00:140
2451227261,2sleep10-21swapper/119:55:171
2444405261,2sleep20-21swapper/219:45:252
2444405261,2sleep20-21swapper/219:45:242
2435327260,1sleep10-21swapper/119:40:151
2434001261,3sleep10-21swapper/119:35:251
2425477261,2sleep02425523-21grep19:30:160
2420335260,2sleep20-21swapper/219:25:122
2416633261,2sleep00-21swapper/019:20:220
2403406260,2sleep10-21swapper/119:10:131
2509289250,2sleep00-21swapper/020:50:200
2488033250,1sleep10-21swapper/120:30:201
2434686250,1sleep20-21swapper/219:35:342
2420527250,2sleep12420757-21idleruntime19:25:151
2500747240,1sleep00-21swapper/020:45:140
2495782240,1sleep10-21swapper/120:40:141
2490667240,1sleep20-21swapper/220:35:152
2482696240,1sleep10-21swapper/120:25:201
2475816240,1sleep10-21swapper/120:20:181
2468890240,1sleep10-21swapper/120:10:211
2440618240,1sleep00-21swapper/019:45:150
2440618240,1sleep00-21swapper/019:45:140
2435519240,1sleep20-21swapper/219:40:202
2430022240,1sleep00-21swapper/019:35:010
2425046240,1sleep10-21swapper/119:30:011
2413927240,1sleep00-21swapper/019:15:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional