You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-17 - 19:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Nov 17, 2025 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30188211627101,11473sleep229712-21apt-get12:04:072
30188211627101,11473sleep229712-21apt-get12:04:072
1946428426,798sleep219508-21ntpq11:29:222
17389281529,748sleep217425-21ntpq09:39:222
26629260167,467sleep226716-21latency_hist11:53:572
26629260167,467sleep226716-21latency_hist11:53:572
1085623380,9sleep012-21rcu_sched07:38:560
1080422959,269sleep210925-21latency_hist11:03:582
1172922559,225sleep311771-21ntpq07:39:373
1172922559,225sleep311771-21ntpq07:39:373
11378221835,166sleep211461-21timerandwakeup09:19:272
11378221835,166sleep211461-21timerandwakeup09:19:272
32246221365,130sleep232260-21kernelversion07:04:222
23551220931,137sleep223695-21latency_hist08:18:572
23775220720,67sleep20-21swapper/208:19:092
15732220513,176sleep315879-21latency_hist11:18:573
28376220121,41sleep128393-21cron08:33:541
659921998,151sleep26750-21sed09:04:332
15735219353,74sleep215764-21latency_hist07:53:582
15735219353,74sleep215764-21latency_hist07:53:582
21416217441,70sleep20-21swapper/208:09:322
21416217441,70sleep20-21swapper/208:09:322
3634217113,143sleep13679-21ntpq10:39:211
870216622,120sleep10-21swapper/107:08:581
2619521628,14sleep20-21swapper/208:24:432
2619521628,14sleep20-21swapper/208:24:432
8985216022,67sleep29125-21latency_hist09:13:582
4092215578,25sleep20-21swapper/212:24:182
16056215516,125sleep216146-21fschecks_time11:19:162
5773215413,122sleep05823-21/usr/sbin/munin12:29:210
23833214710,124sleep223922-21cut11:44:162
18148214712,119sleep318223-21timerandwakeup07:59:283
18148214712,119sleep318223-21timerandwakeup07:59:283
12944214612,121sleep213026-21fschecks_time07:44:162
29063214520,71sleep231751-21cyclictest08:34:202
27207214529,55sleep20-21swapper/208:29:192
18827214530,103sleep218931-21latency_hist11:28:572
32146214417,112sleep032173-21awk12:09:220
3250821428,14sleep20-21swapper/208:44:302
3250821428,14sleep20-21swapper/208:44:302
12301213832,53sleep212324-21modprobe11:08:542
12301213832,53sleep212324-21modprobe11:08:542
24536213758,63sleep224555-21ls10:03:572
2404213612,106sleep2362-21snmpd08:53:542
2404213612,106sleep2362-21snmpd08:53:542
618021359,85sleep26297-21cat12:29:312
27767213528,69sleep227778-21vmstat11:54:302
24786213514,109sleep324790-21grep10:04:143
3533213118,93sleep33599-21grep07:14:253
3519213015,75sleep20-21swapper/207:14:242
3148921309,104sleep230998-21/usr/sbin/munin10:24:242
2642221309,108sleep226476-21grep10:09:182
7561212848,62sleep27558-21munin-run07:28:582
29729212815,96sleep329808-21apt-key12:03:553
19280212886,22sleep019418-21grep09:44:290
2049121277,74sleep220534-21awk09:49:272
1033212759,32sleep21067-21munin-node07:08:592
671212344,62sleep20-21swapper/212:13:562
4610212334,60sleep20-21swapper/207:19:132
16219212211,53sleep20-21swapper/207:54:182
24237211817,84sleep124238-21ntp_states08:19:251
1131121189,95sleep011361-21cpuspeed07:39:140
1131121189,95sleep011361-21cpuspeed07:39:140
15717211715,86sleep00-21swapper/009:34:190
16512211612,90sleep016579-21swap11:19:260
7677211514,81sleep37785-21latency_hist07:28:593
3721211516,85sleep33751-21basename12:23:543
460421149,55sleep24688-21cat10:43:592
30652211425,31sleep20-21swapper/206:59:182
2694021148,92sleep127014-21find10:09:291
6026211214,78sleep36046-21/usr/sbin/munin12:29:283
4160211243,56sleep24206-21date08:58:592
4160211243,56sleep24206-21date08:58:592
19596211215,81sleep10-21swapper/108:04:261
541021097,53sleep25486-21awk10:44:272
23529210913,83sleep123550-21sort08:18:561
2227021099,43sleep20-21swapper/211:39:152
918210812,82sleep3968-21cpuspeed12:14:133
27420210821,71sleep327490-21grep11:54:273
662210716,16sleep20-21swapper/210:29:252
429921070,52sleep34303-21kernelversion12:24:223
2628821075,12sleep00-21swapper/011:50:320
2628821075,12sleep00-21swapper/011:50:320
5275210613,78sleep05327-21processes10:44:250
1450210616,75sleep11519-21/usr/sbin/munin12:14:271
14407210622,43sleep00-21swapper/007:49:110
14407210622,43sleep00-21swapper/007:49:110
17740210517,64sleep117881-21unixbench_singl09:39:291
28705210433,59sleep228796-21load11:59:222
28451210413,75sleep328522-21latency_hist08:33:563
12353210424,57sleep112476-21latency_hist11:08:571
12353210424,57sleep112476-21latency_hist11:08:571
10860210414,70sleep310949-21fschecks_time09:19:163
10860210414,70sleep310949-21fschecks_time09:19:163
10777210444,47sleep210815-21http07:38:552
2223210332,44sleep20-21swapper/212:18:572
16218210317,12sleep20-21swapper/209:34:292
15208210228,62sleep215294-21grep09:33:562
7942210141,45sleep20-21swapper/207:29:102
14035210124,57sleep013884-21latency_hist11:13:570
23008210010,74sleep023151-21cut09:58:580
838829945,37sleep20-21swapper/210:54:252
1447029918,40sleep014471-21apt-get11:14:500
2974629816,21sleep20-21swapper/210:19:202
2945029818,36sleep30-21swapper/310:19:083
1476529730,54sleep214826-21latency11:15:162
489329627,47sleep34955-21fschecks_time10:44:163
3273529519,52sleep10-21swapper/108:44:371
3273529519,52sleep10-21swapper/108:44:371
3108329513,63sleep331141-21/usr/sbin/munin12:04:443
3108329513,63sleep331141-21/usr/sbin/munin12:04:443
3170629419,48sleep131851-21tail10:24:271
635629311,27sleep20-21swapper/210:49:122
635629311,27sleep20-21swapper/210:49:122
1169929328,18sleep20-21swapper/207:39:362
1169929328,18sleep20-21swapper/207:39:362
2040329213,63sleep020540-21cat08:08:580
3164029141,28sleep30-21swapper/307:03:553
1879829129,49sleep218838-21modprobe08:03:552
1879829129,49sleep218838-21modprobe08:03:552
1280029125,23sleep231751-21cyclictest11:09:192
2190329027,47sleep20-21swapper/211:38:552
172712906,64sleep00-21swapper/009:39:200
839128920,45sleep08527-21unixbench_singl09:09:520
71428919,55sleep3858-21latency_hist12:13:573
238772897,13sleep30-21swapper/309:59:303
1075328919,54sleep0313-21rs:main0
502628811,62sleep15075-21irqstats10:44:181
2010328733,20sleep20-21swapper/208:05:452
1875628718,43sleep118523-21/usr/sbin/munin09:44:171
1607328613,56sleep016165-21perl07:54:160
314942855,69sleep00-21swapper/007:02:020
1909028512,60sleep319091-21cpu08:04:123
1363128515,54sleep313765-21cut09:28:573
1363128515,54sleep313765-21cut09:28:573
45072849,60sleep20-21swapper/208:59:202
282928414,54sleep12852-21/usr/sbin/munin12:19:201
1879728434,15sleep20-21swapper/209:44:182
92302838,17sleep20-21swapper/209:14:142
2884428332,22sleep20-21swapper/210:14:352
455728214,53sleep34625-21latency08:59:213
309652826,61sleep20-21swapper/208:39:452
309652826,61sleep20-21swapper/208:39:452
2265428231,19sleep20-21swapper/209:55:132
1336328210,56sleep113491-21uptime11:09:301
969228113,53sleep09698-21open_files09:14:280
251428132,10sleep20-21swapper/212:19:132
247852808,57sleep20-21swapper/210:04:132
229628016,48sleep0225-21systemd-journal08:52:310
229628016,48sleep0225-21systemd-journal08:52:310
1278228031,34sleep20-21swapper/209:24:242
1278228031,34sleep20-21swapper/209:24:242
833827917,45sleep38386-21grep09:09:513
1438727915,45sleep113877-21apt-get11:14:061
2292827817,37sleep023058-21grep11:39:290
213527718,24sleep30-21swapper/307:09:323
592527610,51sleep05933-21df09:04:140
2142727624,28sleep30-21swapper/309:53:563
1633627620,28sleep30-21swapper/311:19:213
3171527520,23sleep10-21swapper/107:03:571
304522757,53sleep10-21swapper/108:39:031
304522757,53sleep10-21swapper/108:39:031
2538527520,27sleep10-21swapper/111:49:161
2538527520,27sleep10-21swapper/111:49:161
197127529,11sleep20-21swapper/210:34:182
711727416,24sleep00-21swapper/010:49:300
711727416,24sleep00-21swapper/010:49:300
431427428,29sleep10-21swapper/107:18:551
122712740,8sleep10-21swapper/109:23:571
2517127310,45sleep125208-21ntpq10:04:241
399727119,33sleep04133-21grep10:39:290
2760327111,47sleep327675-21/usr/sbin/munin10:13:553
24832717,12sleep10-21swapper/107:10:441
217152716,53sleep00-21swapper/011:36:310
68732706,11sleep10-21swapper/109:05:531
2882327016,39sleep128837-21sshd10:14:301
2499927026,26sleep00-21swapper/011:48:540
1434827027,22sleep00-21swapper/009:29:260
611126920,26sleep10-21swapper/112:29:301
318626914,38sleep33192-21ntp_states08:54:283
318626914,38sleep33192-21ntp_states08:54:283
2648926912,37sleep126498-21/usr/sbin/munin08:24:461
2648926912,37sleep126498-21/usr/sbin/munin08:24:461
995426816,33sleep19987-21/usr/sbin/munin10:59:241
387126823,21sleep30-21swapper/310:39:283
2121426817,32sleep121289-21grep11:34:261
1972326822,27sleep00-21swapper/011:29:280
1815126819,35sleep118267-21timerwakeupswit11:24:281
811826719,26sleep00-21swapper/010:54:190
211932678,48sleep00-21swapper/009:51:470
135226715,26sleep10-21swapper/108:49:291
135226715,26sleep10-21swapper/108:49:291
2960326616,23sleep30-21swapper/308:34:373
1333926622,27sleep313401-21/usr/sbin/munin07:44:263
1294026612,40sleep112943-21entropy07:44:151
3227626512,33sleep332345-21/usr/sbin/munin07:04:233
305812658,41sleep00-21swapper/008:39:230
305812658,41sleep00-21swapper/008:39:230
2881126521,26sleep00-21swapper/011:59:240
2294326522,26sleep122962-21munin-run09:58:551
143426517,30sleep31530-21memory08:49:313
143426517,30sleep31530-21memory08:49:313
128002657,37sleep00-21swapper/009:24:260
128002657,37sleep00-21swapper/009:24:260
107026514,27sleep00-21swapper/007:09:010
803126418,21sleep00-21swapper/007:29:160
683926422,19sleep10-21swapper/107:24:291
676126412,36sleep06838-21/usr/sbin/munin07:24:280
320926418,20sleep10-21swapper/108:54:301
320926418,20sleep10-21swapper/108:54:301
2141126415,25sleep30-21swapper/311:34:283
846226320,21sleep10-21swapper/110:54:281
534826219,19sleep10-21swapper/112:28:571
3226526212,23sleep10-21swapper/112:09:271
2927626210,39sleep029342-21cat08:34:340
2749726215,26sleep00-21swapper/011:54:280
2515326218,24sleep00-21swapper/010:04:230
2113326211,38sleep321213-21mailstats08:09:293
2113326211,38sleep321213-21mailstats08:09:293
1113426213,24sleep30-21swapper/311:04:153
1113426213,24sleep30-21swapper/311:04:153
1053526216,20sleep10-21swapper/109:18:571
586226111,27sleep10-21swapper/107:23:561
3226526112,32sleep032322-21irqstats08:44:230
3226526112,32sleep032322-21irqstats08:44:230
2922326117,25sleep329232-21munin-run10:18:563
2231726114,25sleep30-21swapper/308:14:183
1316726114,26sleep30-21swapper/311:09:283
1309026118,26sleep00-21swapper/007:44:180
725526013,20sleep10-21swapper/110:49:311
725526013,20sleep10-21swapper/110:49:311
644126018,22sleep30-21swapper/310:49:173
644126018,22sleep30-21swapper/310:49:173
2923426017,29sleep029360-21latency_hist10:18:570
2564526018,26sleep325651-21diskstats08:24:153
2564526018,26sleep325651-21diskstats08:24:153
2299426016,24sleep30-21swapper/309:58:573
2227826013,19sleep30-21swapper/311:39:163
2056826018,23sleep10-21swapper/109:49:301
1458826012,19sleep30-21swapper/309:29:293
1144926011,33sleep111452-21/usr/sbin/munin07:39:311
1144926011,33sleep111452-21/usr/sbin/munin07:39:311
1013126016,20sleep10-21swapper/107:34:301
763625912,23sleep10-21swapper/109:09:121
3219325913,20sleep30-21swapper/308:44:223
3219325913,20sleep30-21swapper/308:44:223
3196525913,21sleep00-21swapper/007:04:080
2767825913,25sleep00-21swapper/010:13:560
2605925911,35sleep026111-21processes08:24:410
2605925911,35sleep026111-21processes08:24:410
247342599,36sleep324736-21sshd11:45:303
2345025917,27sleep123469-21munin-run11:43:541
2190825919,25sleep021913-21kworker/u8:308:13:540
2190825919,25sleep021913-21kworker/u8:308:13:540
2128725921,20sleep10-21swapper/108:09:311
2128725921,20sleep10-21swapper/108:09:311
1730425918,16sleep10-21swapper/107:58:571
2995025810,34sleep00-21swapper/010:19:260
249002585,41sleep00-21swapper/008:22:100
1565825813,26sleep30-21swapper/309:34:183
1539925816,19sleep30-21swapper/307:50:313
1539925816,19sleep30-21swapper/307:50:313
73425712,31sleep3831-21sed10:29:263
435325711,24sleep10-21swapper/108:59:151
3018725717,24sleep10-21swapper/112:04:061
3018725717,24sleep10-21swapper/112:04:061
1963725720,20sleep30-21swapper/311:29:273
1498425720,17sleep10-21swapper/107:49:281
1498425720,17sleep10-21swapper/107:49:281
1259125714,30sleep112620-21cut09:24:181
1259125714,30sleep112620-21cut09:24:181
1212425715,17sleep00-21swapper/009:23:570
633925612,29sleep36408-21mailstats09:04:303
3047625614,24sleep030509-21irqstats12:04:330
3047625614,24sleep030509-21irqstats12:04:330
1882025614,19sleep30-21swapper/309:44:193
1103825615,24sleep011034-21apt-get11:04:080
1103825615,24sleep011034-21apt-get11:04:080
973725516,23sleep39793-21irqstats07:34:203
3258025515,24sleep032573-21apt-get10:29:110
2947325512,25sleep129468-21apt-get10:19:101
1794625516,23sleep017970-21ntp_states11:24:230
1771225518,19sleep10-21swapper/107:59:181
1771225518,19sleep10-21swapper/107:59:181
1754325522,18sleep00-21swapper/007:59:120
1754325522,18sleep00-21swapper/007:59:120
120152557,11sleep10-21swapper/109:23:421
804425418,21sleep10-21swapper/107:29:171
591525417,21sleep30-21swapper/307:23:583
469725416,19sleep00-21swapper/012:24:320
460625415,23sleep04601-21apt-get07:19:110
2711025412,17sleep00-21swapper/008:29:090
128625411,19sleep00-21swapper/012:14:220
404525315,19sleep00-21swapper/008:58:570
404525315,19sleep00-21swapper/008:58:570
2434025312,21sleep10-21swapper/111:44:271
149102536,37sleep10-21swapper/109:31:051
291625214,22sleep02922-21ntp_states12:19:230
260125218,18sleep10-21swapper/110:34:291
2330525213,24sleep123312-21df_inode09:59:161
1952625210,22sleep10-21swapper/111:29:251
1726725212,23sleep117276-21munin-run11:23:561
794325111,23sleep37938-21apt-get07:29:113
321522517,35sleep30-21swapper/310:27:083
1542225114,18sleep10-21swapper/109:34:081
1077125113,22sleep010767-21apt-get09:19:090
438325012,18sleep00-21swapper/008:59:180
2613925013,22sleep326192-21users11:49:303
2613925013,22sleep326192-21users11:49:303
2348825015,19sleep00-21swapper/008:18:550
1699725012,23sleep316993-21apt-get09:39:103
164362507,17sleep30-21swapper/307:54:233
3097324910,18sleep30-21swapper/308:39:463
3097324910,18sleep30-21swapper/308:39:463
2869624917,20sleep128705-21cpu08:34:111
2820824914,18sleep10-21swapper/111:58:571
2385424912,21sleep323860-21diskstats08:19:143
170824911,15sleep00-21swapper/010:34:120
347424812,21sleep03511-21ntp_states07:14:220
3162624712,22sleep031701-21threads10:24:260
2323724711,21sleep023304-21df09:59:150
179124611,15sleep30-21swapper/310:34:153
1048324611,17sleep30-21swapper/309:18:553
893824510,20sleep38939-21sshd10:55:313
145824415,15sleep10-21swapper/110:33:561
1006624413,14sleep30-21swapper/310:59:263
1751524311,15sleep30-21swapper/311:24:103
294102409,18sleep129412-21sshd12:00:221
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional