You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-06-19 - 02:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Wed Jun 18, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28923294212,913sleep328924-21memory12:00:523
16946261720,581sleep216962-21/usr/sbin/munin08:00:482
16946261720,581sleep216962-21/usr/sbin/munin08:00:482
1476235411,329sleep21479-21ntpq12:15:502
261382346112,149sleep07257-1kworker/0:0H08:30:270
261382346112,149sleep07257-1kworker/0:0H08:30:270
11706232158,217sleep011769-21sendmail-msp07:45:230
32377231014,125sleep032471-21fschecks_time10:30:550
31406229277,198sleep031458-21apt-config12:10:220
3231228845,199sleep03303-21switchtime07:15:550
30092226212,230sleep030193-21fschecks_time07:00:280
43132257126,110sleep00-21swapper/007:20:430
58502240100,120sleep00-21swapper/009:05:570
58502240100,120sleep00-21swapper/009:05:570
206042223105,54sleep00-21swapper/011:35:250
19549222245,163sleep019581-21tune2fs09:50:450
18349222130,108sleep00-21swapper/008:05:430
22909221630,169sleep023017-21latency_hist08:20:250
5900221510,155sleep05990-21fschecks_time07:25:440
1666822150,14sleep028294-21cyclictest08:00:370
1666822150,14sleep028294-21cyclictest08:00:370
8679220110,175sleep08830-21latency_hist07:35:250
19897219914,140sleep019935-21fschecks_count08:10:440
2508621917,146sleep025158-21sed10:06:360
2508621917,146sleep025158-21sed10:06:360
825821905,143sleep00-21swapper/010:56:110
483421796,162sleep00-21swapper/009:04:240
15052217353,106sleep015132-21latency09:35:460
29996216528,124sleep029341-21/usr/sbin/munin08:41:010
29996216528,124sleep029341-21/usr/sbin/munin08:41:010
5234216434,115sleep05310-21sendmail_mailtr10:45:540
11392216288,53sleep00-21swapper/011:05:440
14620215989,22sleep00-21swapper/011:15:420
14620215989,22sleep00-21swapper/011:15:420
21248215112,123sleep121446-21latency_hist08:15:251
13819215012,125sleep313853-21irqstats07:50:473
27591214912,117sleep00-21swapper/010:15:400
27591214912,117sleep00-21swapper/010:15:400
18267214733,93sleep00-21swapper/009:45:480
1666214612,77sleep01109-21/usr/sbin/munin07:10:540
10956214611,112sleep00-21swapper/011:05:240
892214126,98sleep00-21swapper/007:10:250
32502214134,92sleep032624-21gpgv08:50:260
1018214122,102sleep00-21swapper/012:15:360
3054213515,99sleep33099-21/usr/sbin/munin07:15:493
22229213517,102sleep122340-21timerandwakeup08:16:061
1897221358,113sleep018943-21latency_hist11:30:240
2230213015,100sleep00-21swapper/008:55:470
7470212945,73sleep0370-21snmpd10:54:250
7750212619,33sleep37959-21modprobe10:55:253
3210921247,104sleep032164-21latency_hist10:30:270
3210921247,104sleep032164-21latency_hist10:30:270
28245212424,30sleep20-21swapper/206:55:252
28245212424,30sleep20-21swapper/206:55:252
23146212413,80sleep00-21swapper/008:20:370
23880212116,63sleep00-21swapper/010:05:230
12941211918,79sleep112942-21cron09:30:221
12941211918,79sleep112942-21cron09:30:221
22070211715,86sleep322044-21latency_hist11:40:253
21288211610,69sleep021375-21cat09:55:500
29130211513,26sleep029150-21latency_hist08:40:240
29130211513,26sleep029150-21latency_hist08:40:240
27903211516,84sleep127905-21sshd06:51:571
27903211516,84sleep127905-21sshd06:51:571
2675721158,92sleep00-21swapper/008:31:030
137121159,70sleep01436-21ls10:35:400
137121159,70sleep01436-21ls10:35:400
15262211214,80sleep215315-21sendmail_mailqu09:35:522
13715211210,86sleep013779-21mailstats09:30:530
31334211117,79sleep131344-21needreboot08:45:591
9820211048,45sleep00-21swapper/009:20:230
9861210920,70sleep19877-21latency_hist09:20:241
25201210947,45sleep00-21swapper/011:50:230
23568210910,65sleep023610-21sendmail11:45:220
28542210810,22sleep028545-21apt-get12:00:370
23291210810,79sleep223418-21/usr/sbin/munin10:00:562
12030210716,75sleep312074-21/usr/sbin/munin07:45:423
12030210716,75sleep312074-21/usr/sbin/munin07:45:423
27266210513,69sleep227335-21sort11:55:462
24408210319,64sleep323912-21apt-get10:05:343
24408210319,64sleep323912-21apt-get10:05:343
3253210222,59sleep13385-21tail12:20:581
2654210233,54sleep028294-21cyclictest12:20:450
3114210116,69sleep33169-21mailstats12:20:563
16351210011,15sleep00-21swapper/011:20:470
544029918,61sleep24943-21apt-get09:05:352
544029918,61sleep24943-21apt-get09:05:352
2691729830,52sleep326971-21cpuspeed06:50:423
2691729830,52sleep326971-21cpuspeed06:50:423
1779129821,51sleep217851-21grep11:25:462
12842986,82sleep00-21swapper/008:51:520
125329829,48sleep31322-21latency_hist10:35:353
1237129823,56sleep312504-21sed09:26:003
1237129823,56sleep312504-21sed09:26:003
2613329715,11sleep00-21swapper/010:10:440
363529620,54sleep20-21swapper/210:41:482
363529620,54sleep20-21swapper/210:41:482
2942429616,62sleep00-21swapper/010:20:530
203402960,6sleep20-21swapper/208:10:532
1617829623,55sleep116207-21fschecks_count11:20:441
1273229616,59sleep212733-21/usr/sbin/munin11:10:352
1273229616,59sleep212733-21/usr/sbin/munin11:10:352
3192229526,16sleep00-21swapper/012:10:390
281312946,41sleep00-21swapper/011:58:510
1632929422,16sleep00-21swapper/009:40:390
1210429410,17sleep10-21swapper/107:45:441
1210429410,17sleep10-21swapper/107:45:441
113429415,55sleep20-21swapper/207:10:392
3084829217,61sleep330866-21fschecks_count10:25:493
3084829217,61sleep330866-21fschecks_count10:25:493
2571829139,27sleep00-21swapper/011:50:480
1385329110,66sleep213975-21timerwakeupswit09:30:552
74332908,43sleep00-21swapper/009:10:500
2996128914,48sleep20-21swapper/210:21:042
3145928819,49sleep131590-21apt-key12:10:231
1814228811,58sleep218176-21/usr/sbin/munin09:45:462
1287828849,28sleep00-21swapper/009:29:360
1287828849,28sleep00-21swapper/009:29:360
105632889,20sleep00-21swapper/009:20:520
105632889,20sleep00-21swapper/009:20:520
837928713,61sleep28399-21kernelversion10:56:132
1478328711,17sleep10-21swapper/109:35:391
371628612,60sleep13725-21df_inode09:00:411
350328613,58sleep33549-21sed10:41:453
350328613,58sleep33549-21sed10:41:453
459828512,51sleep24206-21/usr/sbin/munin07:20:492
1603028510,61sleep116041-21vmstat07:55:581
99552845,12sleep00-21swapper/007:37:480
231428420,48sleep32323-21unin-run12:20:233
1330828413,57sleep113314-21df09:30:421
130552845,11sleep00-21swapper/007:47:210
130552845,11sleep00-21swapper/007:47:210
1637628311,57sleep316402-21df09:40:423
152532839,24sleep015339-21fschecks_time07:55:450
330628227,31sleep10-21swapper/107:15:561
2440528222,43sleep1312-21rs:main1
295852816,42sleep00-21swapper/006:58:090
2601328113,53sleep326014-21cpu10:10:393
2524528125,33sleep10-21swapper/111:50:241
1620928124,29sleep20-21swapper/211:20:452
1019628113,51sleep110290-21cut09:20:451
1019628113,51sleep110290-21cut09:20:451
894727927,36sleep28990-21ntp_kernel_pll_09:15:532
1057327929,28sleep20-21swapper/209:20:542
1057327929,28sleep20-21swapper/209:20:542
212192788,17sleep10-21swapper/111:35:501
1865427815,42sleep218692-21/usr/sbin/munin08:05:492
183327828,27sleep31852-21apt-get08:55:343
1246327817,45sleep112498-21munin-plugin-st11:10:221
303122770,8sleep10-21swapper/107:00:321
1492727717,42sleep115016-21latency_hist07:55:261
2674727610,47sleep30-21swapper/308:31:023
757027518,39sleep17616-21apt-config10:55:231
637327513,46sleep26385-21/usr/sbin/munin10:50:442
3104827513,40sleep331082-21fschecks_count08:45:493
2379927519,33sleep223925-21wc08:20:562
385327316,16sleep00-21swapper/010:41:540
385327316,16sleep00-21swapper/010:41:540
330727321,31sleep10-21swapper/110:41:301
330727321,31sleep10-21swapper/110:41:301
1983227326,26sleep10-21swapper/109:50:501
1735627321,25sleep30-21swapper/308:00:573
1735627321,25sleep30-21swapper/308:00:573
3159227218,28sleep231798-21modprobe12:10:242
1385527230,29sleep00-21swapper/007:50:480
898727117,30sleep29079-21tune2fs07:35:442
548427125,24sleep30-21swapper/307:25:233
3260127125,25sleep10-21swapper/112:11:081
3113227016,23sleep10-21swapper/110:25:551
3113227016,23sleep10-21swapper/110:25:551
2936627013,16sleep20-21swapper/208:40:392
2936627013,16sleep20-21swapper/208:40:392
2580727015,16sleep10-21swapper/110:10:241
2580727015,16sleep10-21swapper/110:10:241
1121027019,27sleep30-21swapper/311:05:393
2563026923,25sleep225665-21basename11:50:462
2073626930,29sleep00-21swapper/011:35:330
1958026917,26sleep30-21swapper/311:30:483
1794126921,23sleep10-21swapper/111:25:491
865226816,33sleep38676-21sed07:35:243
3002226823,26sleep30-21swapper/307:00:263
2980826823,30sleep129899-21nin-node07:00:241
2897726824,26sleep30-21swapper/306:55:583
2553526819,25sleep10-21swapper/111:50:451
2277326827,22sleep10-21swapper/111:40:511
2266726818,20sleep30-21swapper/310:00:413
1359126824,23sleep30-21swapper/311:10:563
1359126824,23sleep30-21swapper/311:10:563
99912677,12sleep20-21swapper/211:00:502
728526719,26sleep10-21swapper/109:10:471
361726717,24sleep30-21swapper/309:00:353
3189026712,27sleep30-21swapper/312:10:343
202002675,10sleep20-21swapper/211:31:522
109222679,31sleep30-21swapper/309:20:593
109222679,31sleep30-21swapper/309:20:593
2886926610,24sleep20-21swapper/206:55:492
265926620,22sleep20-21swapper/207:15:362
1585026619,20sleep20-21swapper/211:20:252
1585026619,20sleep20-21swapper/211:20:252
1568426613,40sleep215748-21switchtime07:55:552
2838526510,39sleep328453-21/usr/sbin/munin08:35:593
2838526510,39sleep328453-21/usr/sbin/munin08:35:593
2509226518,24sleep30-21swapper/308:25:493
2509226518,24sleep30-21swapper/308:25:493
220952655,17sleep20-21swapper/209:57:252
176692658,41sleep30-21swapper/311:25:393
1521826518,23sleep10-21swapper/111:16:161
1521826518,23sleep10-21swapper/111:16:161
149026522,22sleep10-21swapper/110:35:451
149026522,22sleep10-21swapper/110:35:451
139572655,12sleep00-21swapper/011:14:410
139572655,12sleep00-21swapper/011:14:410
105552656,11sleep10-21swapper/111:01:131
1956926418,27sleep10-21swapper/108:10:241
1129326420,24sleep20-21swapper/211:05:422
2598226314,32sleep126030-21unin-run08:30:241
2598226314,32sleep126030-21unin-run08:30:241
2927426220,27sleep129316-21fschecks_time10:20:501
2700526219,28sleep226971-21systemd10:11:392
262026215,33sleep22627-21modprobe10:40:222
262026215,33sleep22627-21modprobe10:40:222
244326216,20sleep30-21swapper/307:15:253
225182625,12sleep20-21swapper/211:40:442
167426219,18sleep30-21swapper/308:55:263
1669026212,35sleep216699-21/usr/sbin/munin09:40:492
1334326226,20sleep213394-21ls07:50:262
1334326226,20sleep213394-21ls07:50:262
832226118,22sleep20-21swapper/209:15:242
3222226119,26sleep232016-21/usr/sbin/munin07:05:472
1847926121,21sleep30-21swapper/309:45:553
1648826110,38sleep316523-21ntp_kernel_pll_11:20:503
1457926114,24sleep30-21swapper/309:35:253
1195626123,21sleep20-21swapper/209:25:502
1195626123,21sleep20-21swapper/209:25:502
983126021,23sleep39871-21if_eth011:00:473
224822609,33sleep30-21swapper/308:16:083
1320826014,17sleep10-21swapper/111:10:501
1320826014,17sleep10-21swapper/111:10:501
1072026017,26sleep10-21swapper/107:40:481
7632597,17sleep30-21swapper/307:09:233
637725922,19sleep20-21swapper/207:25:542
53282594,11sleep10-21swapper/107:23:231
382125915,26sleep23894-21systemd-run12:25:232
3272925913,27sleep232233-21apt-get08:50:362
2772625914,31sleep327850-21unixbench_singl11:55:563
1951825918,26sleep319548-21munin-run08:10:223
173825921,22sleep10-21swapper/107:10:551
146202595,20sleep20-21swapper/207:52:362
738625817,20sleep30-21swapper/309:10:493
625825816,25sleep36264-21ntp_states07:25:513
611025819,20sleep30-21swapper/309:06:043
611025819,20sleep30-21swapper/309:06:043
44925819,19sleep20-21swapper/210:31:092
3204125820,21sleep10-21swapper/107:05:421
3184625814,30sleep231874-21users08:46:082
3072925818,26sleep20-21swapper/210:25:442
3072925818,26sleep20-21swapper/210:25:442
2989325817,23sleep20-21swapper/212:05:242
2922125816,20sleep20-21swapper/208:40:262
2922125816,20sleep20-21swapper/208:40:262
2448425816,22sleep30-21swapper/308:25:253
2160025813,28sleep221591-21apt-get08:15:382
73952576,17sleep30-21swapper/310:53:283
716825715,24sleep17312-21modprobe07:30:261
315262576,38sleep20-21swapper/207:02:562
3138825719,20sleep30-21swapper/312:10:213
334225616,22sleep20-21swapper/209:00:262
2910325618,18sleep20-21swapper/210:20:392
2910325618,18sleep20-21swapper/210:20:392
236325616,24sleep10-21swapper/108:55:521
176725614,30sleep31810-21ntpq10:35:513
176725614,30sleep31810-21ntpq10:35:513
1633025613,26sleep116324-21apt-get09:40:401
1500125614,24sleep315025-21ntp_states11:16:093
1500125614,24sleep315025-21ntp_states11:16:093
57712554,10sleep20-21swapper/210:47:292
2880425514,18sleep10-21swapper/112:00:501
2816725515,22sleep10-21swapper/110:15:581
2816725515,22sleep10-21swapper/110:15:581
2608625518,23sleep126092-21df_inode10:10:411
2598125516,22sleep326051-21sensors11:50:553
2469025514,18sleep20-21swapper/210:06:262
2469025514,18sleep20-21swapper/210:06:262
2396025517,22sleep223967-21df_inode11:45:412
2175525519,15sleep10-21swapper/109:55:591
2157125513,18sleep20-21swapper/211:35:562
161125513,25sleep11620-21systemd-run08:55:241
1051925517,22sleep30-21swapper/307:40:423
100292554,19sleep10-21swapper/107:38:461
604625418,18sleep30-21swapper/310:50:253
2980125410,17sleep30-21swapper/308:40:563
2980125410,17sleep30-21swapper/308:40:563
2481125410,30sleep224848-21fschecks_count08:25:442
2481125410,30sleep224848-21fschecks_count08:25:442
2449425414,17sleep30-21swapper/311:45:543
2076925417,17sleep30-21swapper/309:55:243
1485425416,22sleep314861-21df_inode09:35:413
1178625418,20sleep20-21swapper/207:45:252
808425312,24sleep18072-21apt-get10:55:441
1922425315,23sleep119196-21latency_hist09:50:251
491225211,23sleep24943-21apt-get09:05:232
2950625215,21sleep129546-21grep08:40:501
2950625215,21sleep129546-21grep08:40:501
2783025213,22sleep00-21swapper/006:50:590
2783025213,22sleep00-21swapper/006:50:590
22582528,36sleep10-21swapper/112:20:091
2230825213,23sleep322304-21apt-get11:40:373
2101525211,24sleep321011-21apt-get09:55:403
1964425215,23sleep119668-21ntp_states11:30:501
1946125213,23sleep219458-21apt-get09:50:382
735325112,22sleep27350-21apt-get07:30:392
58122515,16sleep10-21swapper/110:48:001
303872516,34sleep30-21swapper/310:25:133
3013625111,22sleep230203-21df12:05:402
2762325113,23sleep20-21swapper/208:35:262
2306925110,19sleep10-21swapper/110:00:511
2870025012,22sleep128747-21users08:36:051
2870025012,22sleep128747-21users08:36:051
547424914,16sleep10-21swapper/109:05:421
547424914,16sleep10-21swapper/109:05:421
3271724912,22sleep132741-21ntp_states10:31:041
2649724916,16sleep10-21swapper/108:30:561
1020924918,14sleep30-21swapper/307:40:253
2771224817,15sleep30-21swapper/310:15:483
2771224817,15sleep30-21swapper/310:15:483
2107024617,16sleep30-21swapper/308:15:233
96272455,31sleep30-21swapper/309:17:563
2741224411,15sleep10-21swapper/111:55:501
166792449,18sleep116676-21apt-get08:00:391
166792449,18sleep116676-21apt-get08:00:391
163092445,31sleep30-21swapper/307:59:243
59942408,16sleep15782-21/usr/sbin/munin07:25:451
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional