You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-06 - 03:32

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Jul 06, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
6712275817,721sleep36823-21/usr/sbin/munin19:44:543
26430272313,693sleep326452-21/usr/sbin/munin00:14:483
7146266916,637sleep17210-21/usr/sbin/munin23:14:411
7146266916,637sleep17210-21/usr/sbin/munin23:14:411
2570026118,591sleep025705-21ntp_states22:29:500
2570026118,591sleep025705-21ntp_states22:29:500
21386244713,417sleep321361-21latency_hist20:34:223
13566236524,164sleep013569-21cron20:09:190
2950422909,265sleep029596-21tune2fs20:59:390
23441225998,146sleep023467-21latency_hist22:24:210
1706822479,47sleep021865-21cyclictest23:45:000
24923224629,202sleep025014-21cut00:09:490
1895224639,136sleep02028-21unixbench_singl19:29:530
25897221011,178sleep025846-21apt-get19:05:050
5616219926,161sleep05667-21fschecks_time23:09:560
2632219916,58sleep00-21swapper/022:59:500
10904219912,173sleep010907-21cron21:44:180
2477421986,28sleep00-21swapper/022:27:070
18622219812,166sleep018715-21fschecks_time20:24:420
1207121989,171sleep012185-21latency_hist20:04:220
5099219415,160sleep24974-21latency_hist23:09:212
11480217917,51sleep00-21swapper/023:29:220
11480217917,51sleep00-21swapper/023:29:220
13836217816,60sleep00-21swapper/020:09:350
5866217616,51sleep00-21swapper/019:44:230
791721756,103sleep00-21swapper/019:49:440
833221749,93sleep08401-21latency_hist23:19:200
833221749,93sleep08401-21latency_hist23:19:200
16068217110,144sleep016094-40ogrotate23:44:200
13357217116,58sleep00-21swapper/021:49:520
16082217015,56sleep00-21swapper/020:14:520
21334216111,58sleep021347-21modprobe20:34:190
18718216016,129sleep018720-21cron22:09:180
18718216016,129sleep018720-21cron22:09:180
817721508,16sleep00-21swapper/021:34:370
817721508,16sleep00-21swapper/021:34:370
12940214920,38sleep212998-21/usr/sbin/munin21:49:412
303214214,109sleep0370-21snmpd22:54:210
303214214,109sleep0370-21snmpd22:54:210
15163214121,104sleep115226-21sed20:14:211
17223213931,93sleep017319-21latency_hist22:04:220
9831213816,104sleep19835-21cron23:24:181
26311213820,40sleep00-21swapper/020:49:340
26311213820,40sleep00-21swapper/020:49:340
7173213727,99sleep00-21swapper/019:47:090
28307213715,87sleep00-21swapper/022:39:310
28307213715,87sleep00-21swapper/022:39:310
12619213515,12sleep00-21swapper/023:30:020
12619213515,12sleep00-21swapper/023:30:020
368921345,29sleep00-21swapper/023:04:350
31371213431,88sleep031410-21ntpq21:04:440
21168213316,93sleep10-21swapper/122:14:531
8570213022,92sleep00-21swapper/023:19:330
6637213016,101sleep213145-1kworker/2:2H21:29:382
6637213016,101sleep213145-1kworker/2:2H21:29:382
26995212919,90sleep227133-21grep20:49:522
26995212919,90sleep227133-21grep20:49:522
24516212929,83sleep024588-21date20:44:210
24516212929,83sleep024588-21date20:44:210
16275212722,83sleep316242-21gpgv23:44:233
1143721259,85sleep00-21swapper/021:44:430
30772212431,73sleep00-21swapper/019:19:430
27847212228,84sleep0370-21snmpd22:36:480
28298212112,84sleep00-21swapper/019:09:530
26454212128,80sleep0286-21dbus-daemon00:14:490
25025211711,91sleep025089-21latency20:44:410
23336211436,64sleep023346-21modprobe18:59:190
23237211427,75sleep022843-21/usr/sbin/munin00:04:440
15585211322,74sleep215600-21apt21:59:192
15636211230,44sleep015715-21latency_hist21:59:210
30999211116,27sleep30-21swapper/321:04:353
87221098,17sleep00-21swapper/021:09:530
2680210815,75sleep22440-21apt-get19:34:232
21076210713,76sleep021170-21find18:49:510
32077210618,35sleep00-21swapper/019:24:350
17925210620,68sleep20-21swapper/222:04:482
17925210620,68sleep20-21swapper/222:04:482
21097210527,28sleep00-21swapper/022:14:520
19009210530,62sleep0370-21snmpd22:09:350
19009210530,62sleep0370-21snmpd22:09:350
16949210521,14sleep00-21swapper/020:19:330
2124821049,77sleep00-21swapper/023:59:310
6635210325,26sleep20-21swapper/219:44:532
2824721039,17sleep00-21swapper/020:54:470
2824721039,17sleep00-21swapper/020:54:470
21173210320,69sleep121267-21wc18:49:521
8658210222,61sleep38700-21fschecks_count23:19:403
1973121028,15sleep00-21swapper/023:54:380
24436210122,62sleep224438-21cron20:44:182
24436210122,62sleep224438-21cron20:44:182
17889210119,62sleep217908-21munin-run23:49:192
509321008,18sleep00-21swapper/021:24:390
31607210024,51sleep231688-21cat21:04:502
185121008,44sleep01367-21apt-get21:14:300
185121008,44sleep01367-21apt-get21:14:300
1384721008,72sleep00-21swapper/023:34:510
567029919,55sleep10-21swapper/123:09:571
299092998,16sleep00-21swapper/022:44:360
521029815,56sleep30-21swapper/323:09:223
1039229723,48sleep310542-21grep21:39:533
94532967,15sleep00-21swapper/019:54:450
3055129617,62sleep230477-21/usr/sbin/munin19:19:372
2371829610,71sleep323828-21cpu18:59:263
229202966,16sleep00-21swapper/020:39:200
229202966,16sleep00-21swapper/020:39:200
2085929628,33sleep20-21swapper/218:49:442
66222956,45sleep00-21swapper/021:29:360
66222956,45sleep00-21swapper/021:29:360
1441729515,61sleep114306-21/usr/sbin/munin21:54:411
3192429412,68sleep331964-21ps22:49:483
3192429412,68sleep331964-21ps22:49:483
2226029415,66sleep122303-21cat18:54:401
386929319,18sleep10-21swapper/123:04:401
3049229316,61sleep130586-21threads22:44:531
2678929315,65sleep226791-21grep22:34:362
2583629316,22sleep30-21swapper/322:29:533
2583629316,22sleep30-21swapper/322:29:533
2027929317,52sleep220351-21cat23:54:522
894129212,64sleep19051-21latency_hist19:54:211
2547829224,38sleep225673-21gpgv19:04:222
988729120,52sleep19921-21/usr/sbin/munin21:39:431
44992906,72sleep00-21swapper/021:21:530
221529026,43sleep12279-21cat21:15:321
221529026,43sleep12279-21cat21:15:321
2917828916,21sleep20-21swapper/220:59:212
2917828916,21sleep20-21swapper/220:59:212
2791728910,63sleep00-21swapper/000:19:460
2791728910,63sleep00-21swapper/000:19:460
2691828918,54sleep126994-21grep20:49:511
2691828918,54sleep126994-21grep20:49:511
1199028919,55sleep212064-21/usr/sbin/munin20:04:202
2105128833,31sleep321198-21latency_hist23:59:213
1487928817,18sleep10-21swapper/123:39:401
1169528825,38sleep111690-21apt-get23:29:351
1169528825,38sleep111690-21apt-get23:29:351
923228714,55sleep29257-21/usr/sbin/munin19:54:392
62728726,17sleep00-21swapper/022:54:370
62728726,17sleep00-21swapper/022:54:370
34812870,9sleep20-21swapper/221:19:362
250192875,71sleep00-21swapper/019:00:490
2459828711,62sleep324642-21idleruntime00:09:393
1671228625,32sleep10-21swapper/120:19:211
41722856,28sleep00-21swapper/019:38:490
819228417,50sleep28282-21/usr/sbin/munin21:34:392
819228417,50sleep28282-21/usr/sbin/munin21:34:392
2337128424,42sleep123448-21awk00:04:491
192222846,30sleep00-21swapper/023:51:350
114428421,47sleep1310-21in:imuxsock21:11:401
1200828316,51sleep312127-21users21:44:543
1132728310,58sleep111404-21xargs19:59:511
288482817,62sleep30-21swapper/320:55:423
288482817,62sleep30-21swapper/320:55:423
1797528116,50sleep118027-21latency_hist23:49:221
1131328115,44sleep211343-21grep21:44:412
299432805,27sleep00-21swapper/019:15:510
2080628023,34sleep220950-21grep20:29:532
2021528011,28sleep10-21swapper/123:54:511
237642796,16sleep20-21swapper/222:24:392
2169827923,27sleep20-21swapper/220:34:372
2169827923,27sleep20-21swapper/220:34:372
1752427927,31sleep20-21swapper/220:19:512
2933027814,50sleep229374-21ntp_kernel_err19:14:482
242527822,40sleep32454-21munin-run19:34:203
102027724,33sleep30-21swapper/322:54:483
102027724,33sleep30-21swapper/322:54:483
218727616,31sleep10-21swapper/122:59:391
1277627517,42sleep312771-21apt-get21:49:343
59132747,16sleep20-21swapper/223:10:022
324527413,41sleep2370-21snmpd19:35:002
255827419,39sleep12677-21latency_hist19:34:221
1525427415,45sleep215255-21ntp_states23:39:492
1118927422,30sleep30-21swapper/319:59:493
268327324,25sleep30-21swapper/321:15:423
268327324,25sleep30-21swapper/321:15:423
935727213,41sleep19388-21munin-run21:39:181
935727213,41sleep19388-21munin-run21:39:181
64227220,35sleep2693-21fschecks_count22:54:392
64227220,35sleep2693-21fschecks_count22:54:392
400427216,25sleep30-21swapper/321:19:503
1864127211,13sleep30-21swapper/323:49:483
35927123,29sleep20-21swapper/221:09:432
2893427120,37sleep329012-21awk22:39:503
2893427120,37sleep329012-21awk22:39:503
2866427117,38sleep228695-21munin-plugin-st19:14:202
2164527114,42sleep221691-21/usr/sbin/munin23:59:452
1073127120,24sleep10-21swapper/123:24:551
1073127120,24sleep10-21swapper/123:24:551
2811327011,12sleep10-21swapper/120:54:441
2811327011,12sleep10-21swapper/120:54:441
1081427038,15sleep20-21swapper/223:24:562
1081427038,15sleep20-21swapper/223:24:562
884226917,27sleep20-21swapper/223:19:432
3212826911,43sleep332054-21/usr/sbin/munin19:24:383
1811926927,21sleep10-21swapper/122:04:511
1811926927,21sleep10-21swapper/122:04:511
1332926917,36sleep113382-21fschecks_count23:34:401
1061726917,33sleep310653-21proc_pri23:24:533
1061726917,33sleep310653-21proc_pri23:24:533
2967026823,20sleep10-21swapper/119:14:551
2295226820,23sleep30-21swapper/300:04:383
2201226815,39sleep122057-21ntp_kernel_pll_20:34:441
2201226815,39sleep122057-21ntp_kernel_pll_20:34:441
92826714,38sleep1968-21cut19:29:211
3263426713,36sleep332698-21grep21:09:403
2749026724,20sleep20-21swapper/200:19:322
2749026724,20sleep20-21swapper/200:19:322
262422679,14sleep20-21swapper/200:14:432
770526618,27sleep20-21swapper/219:49:382
3146226612,24sleep20-21swapper/222:49:342
3146226612,24sleep20-21swapper/222:49:342
3059026625,25sleep130648-21fschecks_time19:19:401
2778526628,24sleep10-21swapper/100:19:431
2778526628,24sleep10-21swapper/100:19:431
1306226520,24sleep10-21swapper/120:04:521
515626414,34sleep15219-21idleruntime21:24:401
2968926421,21sleep30-21swapper/320:59:423
2424126413,23sleep10-21swapper/118:59:411
231372649,19sleep30-21swapper/322:21:213
210242649,19sleep10-21swapper/120:30:201
888826320,18sleep30-21swapper/319:54:193
468526313,11sleep30-21swapper/319:39:413
345026324,21sleep20-21swapper/223:04:202
3076126323,18sleep10-21swapper/121:04:211
3037326322,21sleep30-21swapper/322:44:513
1899626312,32sleep119030-21proc_pri20:24:511
1795026324,15sleep30-21swapper/322:04:493
1795026324,15sleep30-21swapper/322:04:493
1320526318,29sleep213218-21/usr/sbin/munin20:04:532
964426215,31sleep19659-21/usr/sbin/munin19:54:501
729726217,19sleep30-21swapper/321:29:523
729726217,19sleep30-21swapper/321:29:523
40452626,11sleep10-21swapper/119:37:131
352626216,22sleep10-21swapper/121:19:401
2887526224,20sleep30-21swapper/300:24:213
2887526224,20sleep30-21swapper/300:24:213
2697226214,25sleep30-21swapper/322:34:413
2529026211,37sleep325361-21sensors20:44:483
2509526213,22sleep20-21swapper/200:09:512
2339126214,24sleep323157-21/usr/sbin/munin20:39:433
2339126214,24sleep323157-21/usr/sbin/munin20:39:433
217062627,42sleep30-21swapper/322:17:533
171372626,44sleep10-21swapper/122:04:141
1216326222,22sleep30-21swapper/323:29:493
1216326222,22sleep30-21swapper/323:29:493
3173326115,23sleep10-21swapper/122:49:421
3173326115,23sleep10-21swapper/122:49:421
2346826121,19sleep10-21swapper/122:24:221
2269026120,19sleep30-21swapper/318:54:503
2182326116,28sleep321772-21latency_hist18:54:213
1645326117,19sleep20-21swapper/221:59:512
82726019,20sleep10-21swapper/122:54:421
82726019,20sleep10-21swapper/122:54:421
2288826020,25sleep30-21swapper/320:39:193
2288826020,25sleep30-21swapper/320:39:193
154902605,11sleep30-21swapper/320:14:393
710725921,21sleep10-21swapper/121:29:491
710725921,21sleep10-21swapper/121:29:491
575725917,26sleep15825-21modprobe19:44:201
2503625920,22sleep10-21swapper/122:29:221
2501525919,26sleep125094-21/usr/sbin/munin00:09:501
186525913,19sleep20-21swapper/221:14:322
186525913,19sleep20-21swapper/221:14:322
1826125920,24sleep218265-21diskstats23:49:392
751325815,20sleep20-21swapper/223:14:502
751325815,20sleep20-21swapper/223:14:502
1742525813,27sleep317545-21/usr/sbin/munin23:45:083
1703725817,28sleep317057-21fschecks_count20:19:403
626325718,22sleep30-21swapper/321:29:193
366425718,23sleep33817-21awk19:35:093
2031625717,18sleep10-21swapper/122:14:201
2031625717,18sleep10-21swapper/122:14:201
108182576,10sleep20-21swapper/221:43:112
96925620,19sleep20-21swapper/219:29:222
3117025617,25sleep231190-21modprobe22:49:182
2545125611,30sleep225508-21irqstats22:29:442
2545125611,30sleep225508-21irqstats22:29:442
2086825613,25sleep220930-21memory22:14:452
1958525622,16sleep30-21swapper/322:09:503
1958525622,16sleep30-21swapper/322:09:503
194922565,39sleep30-21swapper/320:25:333
1633225617,24sleep316338-21ntp_states21:59:483
2640925515,27sleep126488-21systemctl19:05:261
2134925513,18sleep30-21swapper/323:59:383
2127525514,16sleep10-21swapper/123:59:361
1410425516,23sleep314133-21irqstats20:09:463
918025415,22sleep39176-21apt-get19:54:363
2870225412,24sleep10-21swapper/122:39:441
2870225412,24sleep10-21swapper/122:39:441
2668225416,23sleep326727-21awk20:49:453
2668225416,23sleep326727-21awk20:49:453
2666325416,22sleep10-21swapper/122:34:231
2666325416,22sleep10-21swapper/122:34:231
2484025419,16sleep10-21swapper/120:44:381
1475725422,17sleep20-21swapper/220:09:582
1324125414,24sleep313238-21apt-get23:34:333
1178125417,22sleep20-21swapper/223:29:402
1178125417,22sleep20-21swapper/223:29:402
727425311,18sleep30-21swapper/323:14:433
727425311,18sleep30-21swapper/323:14:433
500425313,24sleep25000-21apt-get21:24:332
322102537,33sleep10-21swapper/121:08:411
2833025312,23sleep228326-21apt-get22:39:332
2833025312,23sleep228326-21apt-get22:39:332
2618925315,21sleep326273-21memory19:05:203
2596725313,24sleep125962-21apt-get00:14:341
2279725315,22sleep222931-21unixbench_multi22:19:542
673225213,17sleep30-21swapper/323:14:193
56272525,15sleep20-21swapper/219:42:392
2475825215,22sleep224826-21df20:44:352
1481725219,17sleep30-21swapper/321:54:513
1344125219,15sleep10-21swapper/121:49:531
738425111,24sleep17393-21munin-run19:49:201
2740825119,17sleep30-21swapper/319:09:363
2740625114,21sleep127401-21apt-get19:09:341
2249425112,23sleep222531-21ntp_states18:54:452
1839225116,16sleep10-21swapper/123:49:421
940925012,17sleep30-21swapper/321:39:203
940925012,17sleep30-21swapper/321:39:203
2919425013,21sleep329229-21irqstats19:14:453
2616825020,17sleep20-21swapper/219:05:192
2036024915,16sleep20-21swapper/222:14:212
1716224914,15sleep10-21swapper/120:19:421
229452488,16sleep20-21swapper/200:04:372
1230624810,22sleep312302-21apt-get20:04:333
304582474,34sleep20-21swapper/221:01:272
2676224712,22sleep126781-21apt-get22:34:311
2326924711,17sleep10-21swapper/120:39:411
2326924711,17sleep10-21swapper/120:39:411
1921724714,18sleep20-21swapper/222:09:412
2426724614,13sleep30-21swapper/322:24:523
1324524513,16sleep20-21swapper/223:34:372
305772446,15sleep30-21swapper/319:19:383
1877024416,14sleep20-21swapper/220:24:442
154832449,16sleep20-21swapper/220:14:382
1252524412,16sleep10-21swapper/121:49:201
29522395,16sleep30-21swapper/322:59:533
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional