You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-05 - 18:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Jul 05, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26403276166,7598sleep323052-21kworker/3:009:32:423
28144273116,698sleep228188-21ntpq11:19:492
8931267218,638sleep08937-21df_inode10:19:410
26606260414,572sleep226649-21ntpq11:14:482
5934242313,396sleep15978-21unin-run11:54:211
15922353126,39sleep31742-21unixbench_singl09:55:123
25933234212,315sleep325969-21awk11:14:203
1130823337,314sleep311355-21ntpq12:09:463
203822324133,174sleep320536-21latency_hist09:14:243
22763229959,46sleep322873-21timerandwakeup09:19:543
3835229228,190sleep33978-21unixbench_singl11:44:543
12919222550,20sleep30-21swapper/308:49:443
16571222227,175sleep316772-21cat10:44:233
8618220014,164sleep38749-21latency_hist10:19:233
1246219814,87sleep31250-21modprobe11:39:203
18819219414,163sleep318959-21latency_hist09:09:233
6875218823,152sleep26957-21memory08:29:482
18158218511,161sleep01-21systemd10:49:210
26626217516,28sleep126675-21latency07:49:451
30656217421,24sleep30-21swapper/311:29:233
22690217315,144sleep322699-21modprobe07:39:203
14161217348,72sleep314280-21cat10:34:543
4233217210,11sleep30-21swapper/308:19:563
2810921729,151sleep328161-21if_eth007:54:433
23603217015,70sleep30-21swapper/307:39:533
15599216313,13sleep30-21swapper/307:14:493
2883721628,15sleep30-21swapper/309:39:533
26781216110,60sleep30-21swapper/311:14:533
185472161114,36sleep318641-21fschecks_time10:49:453
185472161114,36sleep318641-21fschecks_time10:49:453
863421589,89sleep38670-21cat06:54:453
11262215812,110sleep30-21swapper/308:44:353
26311215720,109sleep026346-21fschecks_count11:14:420
19042157120,21sleep3370-21snmpd11:39:453
12655215711,113sleep30-21swapper/310:29:533
995721548,68sleep30-21swapper/308:39:433
1038321517,133sleep3370-21snmpd12:06:183
31567214718,105sleep30-21swapper/311:29:553
14295214515,65sleep014401-21timerandwakeup07:10:010
1421121459,61sleep30-21swapper/312:19:313
922821448,124sleep39277-21sed10:19:483
705421448,124sleep37076-21latency_hist10:14:223
31610214310,65sleep30-21swapper/308:04:533
31610214310,65sleep30-21swapper/308:04:533
2503521438,119sleep30-21swapper/309:29:233
14356214217,92sleep30-21swapper/308:54:333
25274214123,100sleep125268-21apt-get09:29:371
1460021405,11sleep30-21swapper/307:10:083
3102821395,12sleep30-21swapper/309:47:003
8328213818,56sleep30-21swapper/311:59:523
518221345,12sleep30-21swapper/310:05:423
11691213410,52sleep30-21swapper/310:29:213
22890213210,104sleep322980-21sort11:04:233
17606213020,85sleep217700-21cut09:04:422
18648212914,97sleep018695-21cat10:49:460
18648212914,97sleep018695-21cat10:49:460
6293212767,45sleep30-21swapper/308:29:213
144321259,91sleep30-21swapper/308:14:223
144321259,91sleep30-21swapper/308:14:223
2370421239,56sleep30-21swapper/311:04:513
1303121235,83sleep30-21swapper/307:07:583
1303121235,83sleep30-21swapper/307:07:583
9738212017,87sleep09782-21sed12:04:460
27207212030,71sleep30-21swapper/307:49:553
11883211828,46sleep20-21swapper/207:04:382
11883211828,46sleep20-21swapper/207:04:382
9540211721,79sleep29686-21unixbench_multi10:19:552
8408211511,90sleep08530-21cat11:59:530
27573211518,81sleep127721-21cat11:19:231
1915211338,58sleep30-21swapper/308:14:393
13878211011,61sleep314083-21latency_hist12:19:233
1611121079,54sleep00-21swapper/008:59:430
20462210614,77sleep120477-21open_inodes10:54:521
7061210531,56sleep07080-21kernelversion06:49:580
12459210520,68sleep112453-21apt-get12:14:401
26354210422,63sleep226348-21apt-get07:49:372
2430210450,34sleep30-21swapper/309:59:243
1592721046,82sleep30-21swapper/308:59:383
32204210315,74sleep232292-21latency_hist11:34:232
17787210231,60sleep30-21swapper/307:21:193
29670210111,46sleep30-21swapper/307:59:443
29670210111,46sleep30-21swapper/307:59:443
8135210035,33sleep30-21swapper/308:34:383
5787210026,48sleep15933-21sed08:24:581
368210012,69sleep30-21swapper/309:54:233
368210012,69sleep30-21swapper/309:54:233
25706210012,22sleep00-21swapper/009:29:480
2563029931,51sleep30-21swapper/307:45:023
2026329915,71sleep320299-21ntp_kernel_err07:29:473
2026329915,71sleep320299-21ntp_kernel_err07:29:473
1769929829,14sleep30-21swapper/310:45:043
803129739,36sleep38053-21idleruntime-cro06:54:213
471729742,37sleep04758-21tr08:24:210
2058029731,38sleep30-21swapper/309:14:323
1128929729,11sleep30-21swapper/307:00:553
729929621,44sleep27296-21apt-get10:14:342
720529630,30sleep30-21swapper/311:55:483
720529630,30sleep30-21swapper/311:55:483
2910929620,54sleep029263-21latency_hist11:24:230
968129524,42sleep19694-21munin-node06:59:241
2972529514,65sleep129869-21latency_hist09:44:231
1963629529,50sleep30-21swapper/307:29:233
1107229510,67sleep00-21swapper/006:59:560
3271729417,63sleep1335-21latency11:34:461
563829329,11sleep30-21swapper/311:50:353
3103529317,60sleep131099-21cpuspeed08:04:391
3103529317,60sleep131099-21cpuspeed08:04:391
3051929218,61sleep230595-21wc09:44:532
200762925,43sleep30-21swapper/310:54:413
865029118,56sleep28729-21mailstats08:34:532
738629117,59sleep07404-21fschecks_count10:14:400
475629122,46sleep24815-21fschecks_count11:49:412
1353129115,35sleep10-21swapper/110:34:361
2989529015,60sleep129958-21/usr/sbin/munin11:24:521
2777228914,36sleep30-21swapper/311:19:373
136272890,9sleep113167-21apt-get07:09:241
1997728520,49sleep20-21swapper/207:29:422
1997728520,49sleep20-21swapper/207:29:422
78622847,17sleep30-21swapper/308:34:223
280272847,63sleep10-21swapper/109:38:421
2203428418,43sleep222113-21mailstats10:59:532
1802028420,41sleep218044-21munin-run07:24:212
31132838,18sleep10-21swapper/111:44:351
2845228313,56sleep128582-21unixbench_multi11:19:561
2270328317,49sleep022760-21unin-run07:39:210
3228728227,34sleep20-21swapper/208:09:202
3228728227,34sleep20-21swapper/208:09:202
3130828215,45sleep031009-21/usr/sbin/munin08:04:460
3130828215,45sleep031009-21/usr/sbin/munin08:04:460
3017728223,39sleep130326-21grep07:59:541
3017728223,39sleep130326-21grep07:59:541
243502828,18sleep024351-21sshd07:42:410
1200328218,41sleep112134-21sed08:44:541
418328123,37sleep04202-21apt-get10:04:340
1050528111,55sleep110544-21if_err_eth006:59:441
904528022,36sleep09184-21grep06:54:550
2467828027,29sleep10-21swapper/111:09:381
2457128015,49sleep024632-21users09:24:560
2730427915,31sleep20-21swapper/209:34:512
58812788,16sleep00-21swapper/011:53:430
2736827810,13sleep30-21swapper/309:34:523
2385027816,41sleep223976-21/usr/sbin/munin07:40:002
2348827817,42sleep023630-21latency_hist09:24:230
2060827821,35sleep020602-21apt-get09:14:370
1342927717,43sleep013625-21cat07:09:240
12702775,16sleep10-21swapper/108:12:511
12702775,16sleep10-21swapper/108:12:511
1414327619,42sleep114235-21latency_hist08:54:231
2496627524,29sleep10-21swapper/107:44:441
1943127515,28sleep00-21swapper/009:09:490
2312627419,33sleep223123-21apt-get11:04:342
3253427324,22sleep20-21swapper/211:34:432
2133927322,21sleep00-21swapper/010:59:230
2016427311,48sleep120210-21sed07:29:451
2016427311,48sleep120210-21sed07:29:451
2010627314,43sleep00-21swapper/007:29:440
2010627314,43sleep00-21swapper/007:29:440
1290827313,42sleep212918-21/usr/sbin/munin08:49:432
1111927318,29sleep00-21swapper/010:24:540
59027213,46sleep0619-21proc_pri08:09:510
59027213,46sleep0619-21proc_pri08:09:510
1964727211,16sleep00-21swapper/007:29:240
103002726,14sleep20-21swapper/206:59:372
75942716,53sleep00-21swapper/008:31:070
3004127118,38sleep030141-21cut09:44:430
174327115,26sleep00-21swapper/011:39:420
1547427120,35sleep215512-21ntp_kernel_pll_10:39:462
86972707,16sleep20-21swapper/212:00:032
2982027014,16sleep20-21swapper/207:59:472
2982027014,16sleep20-21swapper/207:59:472
2920427011,16sleep20-21swapper/207:59:232
1428527015,33sleep014429-21/usr/sbin/munin10:34:550
78242688,12sleep10-21swapper/106:51:411
2601726822,21sleep20-21swapper/209:29:542
2322526816,30sleep123335-21perl11:04:411
2287826819,21sleep20-21swapper/207:39:232
225662685,11sleep10-21swapper/111:01:181
1908226828,19sleep20-21swapper/210:49:562
1908226828,19sleep20-21swapper/210:49:562
185652685,12sleep10-21swapper/109:06:431
303412677,15sleep10-21swapper/109:44:481
1568826719,26sleep00-21swapper/008:59:230
963326614,23sleep20-21swapper/212:04:442
853426615,24sleep10-21swapper/111:59:541
259482666,49sleep00-21swapper/007:47:500
186642665,51sleep00-21swapper/009:07:590
479726520,30sleep24845-21grep08:24:232
32526515,32sleep2367-21apt-config09:54:222
32526515,32sleep2367-21apt-config09:54:222
3215126518,29sleep032180-21munin-run11:34:210
3158326519,28sleep10-21swapper/109:49:401
3158326519,28sleep10-21swapper/109:49:401
2495026515,24sleep00-21swapper/011:09:450
2219126516,25sleep00-21swapper/010:59:550
1646226521,23sleep10-21swapper/108:59:521
1216326516,31sleep112183-21modprobe12:14:201
1017126523,23sleep10-21swapper/108:39:491
857626415,31sleep28633-21irqstats06:54:442
776426413,33sleep17777-21/usr/sbin/munin10:14:491
2816326414,30sleep027875-21/usr/sbin/munin07:54:440
199426418,21sleep20-21swapper/208:14:422
144632649,12sleep20-21swapper/212:19:502
13722644,12sleep20-21swapper/211:39:232
622226314,32sleep26293-21df11:54:382
622226314,32sleep26293-21df11:54:382
51226315,32sleep0721-21latency_hist09:54:240
1652226323,21sleep10-21swapper/107:19:231
443626223,18sleep00-21swapper/011:49:230
1575426220,20sleep00-21swapper/010:39:520
1377426216,22sleep20-21swapper/212:19:222
11292627,12sleep20-21swapper/209:55:032
860126120,20sleep00-21swapper/008:34:510
28512618,43sleep00-21swapper/008:15:120
211226117,22sleep10-21swapper/111:39:511
2106926125,20sleep10-21swapper/109:14:511
1224326115,33sleep212255-21kernelversion10:29:442
1117626122,20sleep00-21swapper/012:09:430
71832606,11sleep10-21swapper/111:55:341
71832606,11sleep10-21swapper/111:55:341
213926016,23sleep10-21swapper/109:59:221
1530026015,31sleep015307-21/usr/sbin/munin07:14:420
1512726023,19sleep10-21swapper/108:54:561
1303326014,22sleep10-21swapper/110:31:411
12812605,45sleep20-21swapper/208:13:002
12812605,45sleep20-21swapper/208:13:002
125526012,35sleep01297-21munin-run11:39:210
460625918,26sleep24643-21sed10:04:482
336425915,18sleep20-21swapper/209:59:562
2378225912,22sleep10-21swapper/107:39:591
232612595,10sleep20-21swapper/209:22:272
1486925919,25sleep214929-21/usr/sbin/munin08:54:522
1126425917,24sleep011331-21df08:44:360
966125815,24sleep09680-21apt-get08:39:320
679825815,20sleep00-21swapper/011:54:520
679825815,20sleep00-21swapper/011:54:520
2235925814,30sleep10-21swapper/109:19:431
1841225814,27sleep218419-21sshd07:24:412
1744625814,20sleep00-21swapper/007:19:550
1471425818,24sleep00-21swapper/012:19:570
117592589,30sleep10-21swapper/110:29:231
1137325814,29sleep211437-21ps12:09:492
2934925716,26sleep229343-21apt-get11:24:372
2006925712,21sleep20-21swapper/210:54:402
1689425712,25sleep116888-21apt-get10:44:381
146192575,10sleep20-21swapper/207:10:232
80325613,21sleep00-21swapper/009:54:320
46602566,37sleep10-21swapper/108:23:371
270725618,23sleep12734-21fschecks_count09:59:421
2549425613,26sleep225629-21awk07:45:012
1975725611,20sleep10-21swapper/110:54:231
1975725611,20sleep10-21swapper/110:54:231
1508225615,21sleep10-21swapper/112:20:021
1250725615,22sleep00-21swapper/007:05:000
1097025614,18sleep20-21swapper/210:24:522
707825516,17sleep00-21swapper/010:14:230
673525513,25sleep26260-21apt-get06:49:282
3210025513,28sleep132101-21sshd11:33:411
3207925513,28sleep032162-21cat09:49:520
2743825511,20sleep00-21swapper/009:34:530
2143925514,26sleep121433-21apt-get07:34:381
1757325513,24sleep117671-21users07:19:561
168972558,35sleep20-21swapper/209:00:382
107612557,39sleep20-21swapper/208:41:562
493625418,16sleep10-21swapper/110:04:551
2380425417,17sleep00-21swapper/011:04:520
1905925415,24sleep219053-21apt-get09:09:392
1328825414,23sleep00-21swapper/008:49:540
701225316,20sleep10-21swapper/110:14:201
3006325318,18sleep00-21swapper/007:59:530
3006325318,18sleep00-21swapper/007:59:530
2863925315,16sleep00-21swapper/009:39:460
2798525317,17sleep10-21swapper/107:54:411
239425314,22sleep12434-21ps08:15:011
2067725315,22sleep220683-21df_inode09:14:402
1329325316,16sleep00-21swapper/010:34:230
1313725311,25sleep10-21swapper/107:09:211
1313725311,25sleep10-21swapper/107:09:211
124222535,37sleep00-21swapper/012:14:270
106062535,38sleep10-21swapper/112:09:111
1396825213,24sleep213987-21ntp_states10:34:492
685625114,21sleep10-21swapper/108:29:471
2487325113,18sleep20-21swapper/211:09:432
2418025115,20sleep20-21swapper/209:24:492
1736525113,17sleep20-21swapper/210:44:592
99562506,33sleep10-21swapper/110:22:161
249582504,37sleep10-21swapper/109:29:001
1843725012,23sleep118529-21perl07:24:441
509024913,21sleep25119-21df08:24:432
43502495,35sleep20-21swapper/211:48:572
1929224911,22sleep119329-21irqstats09:09:461
163022495,34sleep20-21swapper/207:17:102
1640024810,16sleep10-21swapper/110:44:211
1152324814,18sleep20-21swapper/208:44:432
2897724713,16sleep20-21swapper/209:39:562
583124610,21sleep25838-21diskstats10:09:412
531824611,15sleep00-21swapper/008:24:490
3021224611,20sleep030253-21users11:24:560
255724610,21sleep02106-21apt-get09:59:250
861024513,15sleep10-21swapper/108:34:521
2656824211,16sleep00-21swapper/007:49:440
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional