You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 20:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Dec 01, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26922223149,2287sleep226913-21apt-get11:48:212
26922223149,2287sleep226913-21apt-get11:48:212
17653273614,707sleep322178-1kworker/3:2H09:37:053
17653273614,707sleep322178-1kworker/3:2H09:37:053
5953262610,598sleep36032-21/usr/sbin/munin08:58:543
2827125618,537sleep112172-1kworker/1:0H08:28:241
12198248222,446sleep312208-21unin-run07:38:083
7956243144,377sleep05666-21snmpd09:04:120
790723760,92sleep00-21swapper/010:48:270
23782235910,331sleep023826-21ntpq08:13:330
23782235910,331sleep023826-21ntpq08:13:330
28993230427,254sleep329027-21idleruntime-cro10:13:063
2929523020,7sleep30-21swapper/310:13:183
11771222817,191sleep111853-21cut09:18:261
11771222817,191sleep111853-21cut09:18:261
25682222012,184sleep00-21swapper/011:43:510
25682222012,184sleep00-21swapper/011:43:510
1377022160,13sleep0900-21cyclictest07:43:080
9826221515,183sleep19916-21grep10:53:381
2490222037,131sleep00-21swapper/008:18:180
2490222037,131sleep00-21swapper/008:18:180
12210220316,158sleep00-21swapper/007:38:090
18388219223,149sleep118380-21apt-get09:38:251
2290921846,163sleep00-21swapper/009:53:090
29714218210,158sleep129759-21ntpq10:13:331
2823321718,150sleep05666-21snmpd08:28:150
31981216554,98sleep031983-21entropy12:04:060
31040216341,94sleep00-21swapper/010:18:290
21501216114,134sleep321549-21unin-run08:08:073
10962215833,27sleep00-21swapper/010:58:250
5822215683,34sleep05869-21idleruntime12:23:280
2140215674,65sleep02154-21fschecks_count08:48:260
13322215227,111sleep013401-21fschecks_time09:23:260
3811215059,78sleep01-21systemd12:18:070
18426214915,107sleep00-21swapper/011:23:080
16513214878,21sleep00-21swapper/009:33:270
16513214878,21sleep00-21swapper/009:33:270
2455214754,73sleep02544-21open_inodes07:08:090
540214321,23sleep00-21swapper/010:23:410
22035214317,109sleep322099-21cpuspeed11:33:243
9028214078,19sleep00-21swapper/009:08:360
9028214078,19sleep00-21swapper/009:08:360
4669213954,49sleep04697-21grep07:13:220
30151213914,111sleep230187-21grep11:58:272
30150213911,109sleep00-21swapper/010:13:400
26631213959,23sleep00-21swapper/010:03:350
15357213912,104sleep00-21swapper/007:48:100
1038921389,118sleep35666-21snmpd07:30:413
5433213730,20sleep05425-21apt-get08:58:230
3268213724,93sleep33411-21sed12:13:383
1939721348,114sleep018879-21/usr/sbin/munin11:23:570
4000213310,109sleep04014-21netstat08:53:300
14515213249,28sleep00-21swapper/011:08:390
49221289,17sleep00-21swapper/008:43:210
2694221288,106sleep00-21swapper/008:23:350
22749212426,79sleep122783-21idleruntime-cro09:53:061
5087212319,88sleep35227-21date08:58:103
7475212217,87sleep37496-21latency_hist07:23:063
27811212213,95sleep227831-21grep10:08:252
28766211716,37sleep028832-21wc11:53:280
19631211544,11sleep00-21swapper/007:59:000
2179121149,59sleep00-21swapper/008:08:210
1805211423,73sleep21815-21unin-run08:48:072
28258211315,27sleep05666-21snmpd11:53:080
28258211315,27sleep05666-21snmpd11:53:080
26265211319,68sleep126329-21fschecks_time10:03:281
18787211239,31sleep218840-21fschecks_count07:58:342
1069821120,7sleep10-21swapper/107:33:081
473121088,82sleep00-21swapper/010:38:270
25115210766,24sleep00-21swapper/009:58:380
12430210715,23sleep10-21swapper/111:03:231
31673210629,63sleep00-21swapper/008:38:330
31673210629,63sleep00-21swapper/008:38:330
25139210616,75sleep225191-21/usr/sbin/munin08:18:282
25139210616,75sleep225191-21/usr/sbin/munin08:18:282
10218210612,79sleep110221-21entropy09:13:261
10218210612,79sleep110221-21entropy09:13:261
6195210518,60sleep30-21swapper/310:43:223
14088210526,58sleep114102-21grep07:43:271
24841210411,79sleep024871-21awk11:43:060
2030621045,86sleep00-21swapper/009:43:320
7502210311,66sleep00-21swapper/010:48:080
16906210312,72sleep10-21swapper/109:33:361
16906210312,72sleep10-21swapper/109:33:361
31456210211,35sleep031451-21munin-run12:03:090
29074210218,64sleep129166-21latency_hist10:13:091
24711210216,71sleep224715-21diskstats09:58:282
22829210222,63sleep322906-21sed09:53:083
31149210117,63sleep00-21swapper/008:38:080
11613210128,51sleep111770-21sed10:58:391
3151210014,69sleep23267-21timerandwakeup12:13:372
1132210025,59sleep31159-21cpu07:03:243
3188929914,71sleep331955-21cpuspeed12:03:573
1321929915,67sleep112427-21/usr/sbin/munin07:38:421
619529810,73sleep16214-21apt-get07:18:201
216702986,53sleep00-21swapper/011:32:330
2862429717,18sleep00-21swapper/010:08:390
682729617,17sleep00-21swapper/007:18:390
2210329621,14sleep00-21swapper/011:33:250
81932955,10sleep00-21swapper/012:29:500
151292959,65sleep00-21swapper/009:28:300
77722949,16sleep00-21swapper/007:23:230
2257029415,63sleep122009-21/usr/sbin/munin11:33:391
124292949,16sleep00-21swapper/011:03:220
1177129414,59sleep211780-21vmstat10:58:402
318229311,27sleep00-21swapper/007:08:300
3050829214,62sleep10-21swapper/111:58:341
26042926,53sleep00-21swapper/010:30:520
3230429015,60sleep332313-21vmstat08:38:443
3230429015,60sleep332313-21vmstat08:38:443
155552898,69sleep015550-21apt-get07:48:220
83882880,7sleep20-21swapper/209:08:092
19552885,72sleep00-21swapper/012:09:470
854928716,58sleep38598-21awk09:08:233
854928716,58sleep38598-21awk09:08:233
631228717,51sleep36432-21timerwakeupswit12:23:393
503728724,32sleep10-21swapper/107:13:311
265728711,63sleep15666-21snmpd10:31:341
183612877,66sleep018380-21apt-get09:38:200
1253928619,48sleep312581-21fschecks_count07:38:283
978728426,44sleep00-21swapper/010:53:370
3237328432,28sleep10-21swapper/106:58:341
2537828422,29sleep30-21swapper/311:43:263
2537828422,29sleep30-21swapper/311:43:263
746028313,46sleep36971-21/usr/sbin/munin09:03:353
420328342,29sleep00-21swapper/010:36:090
121382837,57sleep00-21swapper/009:18:330
121382837,57sleep00-21swapper/009:18:330
675128117,49sleep16782-21latency_hist09:03:081
621628113,56sleep26311-21sed12:23:382
595528122,31sleep10-21swapper/110:43:081
161428113,52sleep11733-21cat12:08:591
31828016,27sleep20-21swapper/210:23:352
2506128027,32sleep30-21swapper/311:43:093
1641628012,52sleep20-21swapper/209:33:212
1641628012,52sleep20-21swapper/209:33:212
159692807,33sleep016016-21ntpq11:13:310
277627913,48sleep12513-21/usr/sbin/munin12:13:281
203192796,13sleep00-21swapper/008:03:240
1269327916,50sleep312709-21kernelversion11:03:293
1183727910,54sleep311839-21sshd10:59:173
1565327826,28sleep20-21swapper/211:13:252
106832787,59sleep00-21swapper/009:13:370
106832787,59sleep00-21swapper/009:13:370
2634427717,46sleep326209-21latency_hist08:23:093
2634427717,46sleep326209-21latency_hist08:23:093
2000027720,32sleep10-21swapper/108:03:091
986427647,17sleep09926-21sshd07:28:360
718827616,43sleep27190-21/usr/sbin/munin12:28:222
3172027610,44sleep10-21swapper/108:38:341
3172027610,44sleep10-21swapper/108:38:341
1711727610,51sleep20-21swapper/211:18:222
2939327511,47sleep229399-21diskstats10:13:262
272227523,30sleep20-21swapper/207:08:112
226327514,46sleep22315-21/usr/sbin/munin08:48:282
902727418,37sleep39053-21munin-run07:28:073
2175027416,44sleep00-21swapper/009:48:290
1350427413,43sleep313572-21latency09:23:293
753527320,38sleep27688-21latency_hist07:23:092
466327321,35sleep24739-21wc12:18:372
401527317,39sleep14061-21ntpq08:53:311
198727316,21sleep30-21swapper/310:28:373
181627322,30sleep30-21swapper/308:48:083
418827242,14sleep00-21swapper/012:18:270
1222127225,24sleep30-21swapper/309:18:363
1222127225,24sleep30-21swapper/309:18:363
284827125,25sleep10-21swapper/107:08:141
188727117,21sleep20-21swapper/210:28:322
1872127115,43sleep218771-21latency09:39:142
180332716,53sleep00-21swapper/007:53:390
1721227118,33sleep317321-21grep11:18:273
745127021,32sleep27475-21munin-plugin-st10:48:062
539827012,43sleep15494-21latency_hist12:23:081
1740027018,22sleep10-21swapper/107:53:281
1063827018,24sleep30-21swapper/310:58:083
335826912,39sleep33057-21/usr/sbin/munin07:08:333
3271026927,26sleep3415-21sed06:58:443
845426818,31sleep38598-21unixbench_singl10:48:393
678326815,36sleep26914-21sed09:03:092
3268726814,35sleep1317-21memory10:23:341
2545126817,21sleep10-21swapper/111:43:281
2545126817,21sleep10-21swapper/111:43:281
173826816,33sleep21828-21timerandwakeup07:03:392
932026718,31sleep29365-21/usr/sbin/munin07:28:222
2952626724,27sleep329537-21modprobe08:33:063
2032626736,17sleep10-21swapper/108:03:251
1856926714,19sleep10-21swapper/111:23:091
98226614,35sleep21045-21/usr/sbin/munin12:08:442
32122660,12sleep112-21rcu_sched10:33:301
2353026619,24sleep10-21swapper/109:53:351
2060826623,18sleep20-21swapper/208:03:302
899926521,25sleep29057-21awk10:53:062
2518026523,19sleep10-21swapper/109:58:391
2365026520,28sleep10-21swapper/108:13:301
2365026520,28sleep10-21swapper/108:13:301
1853826517,26sleep30-21swapper/309:39:113
1711626513,35sleep117111-21apt-get11:18:211
1371326517,31sleep113772-21/usr/sbin/munin09:23:351
2984526414,34sleep229820-21/usr/sbin/munin08:33:222
2739426417,30sleep326455-21/usr/sbin/munin08:23:423
2189026424,26sleep321993-21cut08:08:283
45026323,22sleep30-21swapper/310:23:403
2487326322,20sleep10-21swapper/111:43:071
1505726314,32sleep315125-21ls09:28:293
1061826319,22sleep30-21swapper/309:13:363
1061826319,22sleep30-21swapper/309:13:363
87226217,29sleep01007-21latency_hist07:03:090
360326220,23sleep20-21swapper/210:33:412
317826212,33sleep33184-21diskstats10:33:293
2873626212,33sleep228743-21ntp_states08:28:502
1205326216,33sleep212087-21/usr/sbin/munin09:18:312
1205326216,33sleep212087-21/usr/sbin/munin09:18:312
94782618,19sleep10-21swapper/109:08:541
94782618,19sleep10-21swapper/109:08:541
2669326110,32sleep226750-21irqstats08:23:292
2127526117,19sleep10-21swapper/111:28:381
814026013,24sleep10-21swapper/107:23:321
784526016,25sleep20-21swapper/207:23:252
497326019,18sleep20-21swapper/208:58:092
2732226011,26sleep10-21swapper/108:23:401
2622526017,27sleep126238-21munin-run08:23:071
2622526017,27sleep126238-21munin-run08:23:071
2384326013,30sleep323867-21/usr/sbin/munin08:13:363
2384326013,30sleep323867-21/usr/sbin/munin08:13:363
1603326019,25sleep316071-21/usr/sbin/munin11:13:343
1434926022,20sleep20-21swapper/211:08:332
1338826013,33sleep213390-21sshd11:04:172
915825912,28sleep29260-21/usr/sbin/munin09:08:382
915825912,28sleep29260-21/usr/sbin/munin09:08:382
780025922,19sleep10-21swapper/112:28:371
120622598,14sleep20-21swapper/207:36:542
1155625920,17sleep10-21swapper/107:33:381
113025912,27sleep11125-21apt-get07:03:221
409025813,27sleep34085-21apt-get12:18:223
3265625820,20sleep20-21swapper/208:43:072
3265625820,20sleep20-21swapper/208:43:072
3225425823,19sleep10-21swapper/112:04:111
2751425813,30sleep227660-21latency_hist10:08:092
263125817,17sleep30-21swapper/308:48:373
255782587,11sleep30-21swapper/309:59:103
2211825811,24sleep20-21swapper/211:33:282
1480425815,17sleep20-21swapper/207:43:402
80025716,24sleep2833-21munin-run07:03:072
493725722,21sleep30-21swapper/307:13:293
2410125718,19sleep20-21swapper/209:55:052
2364225714,19sleep30-21swapper/309:53:373
1687725711,28sleep116891-21unin-run07:53:081
1272225716,28sleep212730-21uname07:38:312
1087625721,17sleep20-21swapper/209:13:392
1087625721,17sleep20-21swapper/209:13:392
662425615,26sleep16647-21ntp_states10:43:331
486325614,19sleep20-21swapper/210:38:292
464425613,25sleep34639-21apt-get10:38:223
319802564,9sleep30-21swapper/310:21:183
2624925611,30sleep326258-21df_inode10:03:263
1850125618,19sleep20-21swapper/207:58:092
816025514,24sleep18196-21ntp_states10:48:331
613925510,27sleep16215-21/usr/sbin/munin12:23:371
2383625515,17sleep20-21swapper/208:13:352
2383625515,17sleep20-21swapper/208:13:352
215825513,24sleep12261-21perl08:48:271
1592925510,21sleep20-21swapper/207:48:372
3045725417,19sleep30-21swapper/308:33:393
2405925413,19sleep20-21swapper/211:38:362
2179325411,25sleep221785-21apt-get08:08:232
1993725412,25sleep119931-21apt-get09:43:211
1617825416,18sleep20-21swapper/209:33:092
1407125416,23sleep114123-21tune2fs11:08:281
9192535,36sleep30-21swapper/312:08:133
831025315,15sleep10-21swapper/109:08:091
2865525315,18sleep20-21swapper/211:53:262
2055425314,18sleep30-21swapper/309:43:403
2045325312,24sleep320446-21apt-get11:28:203
200552537,36sleep20-21swapper/211:26:412
1536425315,21sleep30-21swapper/311:13:103
1531225315,23sleep315354-21latency_hist07:48:093
3136225212,26sleep31-21systemd12:03:083
2890625214,23sleep328948-21ntp_states11:53:313
2700225213,23sleep127006-21diskstats11:48:241
2700225213,23sleep127006-21diskstats11:48:241
215212527,36sleep20-21swapper/211:30:372
1993825216,18sleep20-21swapper/209:43:222
2692325112,23sleep326993-21df11:48:223
2692325112,23sleep326993-21df11:48:223
1712625112,22sleep317123-21apt-get07:53:183
1687725117,16sleep10-21swapper/111:18:081
1596425114,17sleep30-21swapper/307:48:383
1323325112,23sleep213229-21apt-get09:23:202
2984425015,19sleep10-21swapper/108:33:211
2849825011,16sleep10-21swapper/111:53:211
1887825012,23sleep318898-21apt-get11:23:193
1479125011,24sleep114861-21df09:28:231
318372499,20sleep20-21swapper/212:03:192
3146524911,23sleep231542-21find10:18:392
216022498,15sleep20-21swapper/209:48:272
1994724911,21sleep319980-21munin-run08:03:073
2537724811,20sleep225368-21apt-get11:43:252
2537724811,20sleep225368-21apt-get11:43:252
2222424811,14sleep10-21swapper/108:08:341
474224711,22sleep24751-21diskstats07:13:242
3067424710,15sleep10-21swapper/110:18:091
766324613,17sleep30-21swapper/312:28:353
176524617,16sleep10-21swapper/108:48:061
95912379,12sleep20-21swapper/210:53:312
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional