You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-19 - 05:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Fri Sep 19, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
416295866,9567sleep030954-21kworker/0:221:30:580
416295866,9567sleep030954-21kworker/0:221:30:580
29679246749,4652sleep216936-21kworker/2:123:00:512
1462229597,940sleep014663-21ntpq22:12:490
1105729578,914sleep011065-21ntpq23:43:040
1143264411,617sleep31146-21/usr/sbin/munin21:32:433
1143264411,617sleep31146-21/usr/sbin/munin21:32:433
1775525316,470sleep017801-21ntpq22:22:500
19216248626,444sleep219299-21memory22:27:492
10417238220,341sleep310473-21fschecks_count20:17:463
262602368147,198sleep026298-21fschecks_count00:32:410
1279823526,314sleep0362-21snmpd20:23:040
313522302139,22sleep00-21swapper/023:07:230
137682290100,173sleep013793-21fschecks_count23:52:450
11650224535,187sleep011790-21unixbench_singl22:03:000
27467224218,207sleep127513-21unin-run00:37:231
2233822408,219sleep022352-21grep19:12:430
215652240102,79sleep021615-21modprobe20:57:230
31331222760,141sleep0225-21systemd-journal19:42:220
18179221810,124sleep018334-21latency_hist00:07:260
271922099,180sleep00-21swapper/019:52:520
4821219825,152sleep24964-21unixbench_multi23:23:112
762721929,170sleep07665-21ntpq23:32:580
762721929,170sleep07665-21ntpq23:32:580
695521929,168sleep07059-21latency_hist20:07:260
695521929,168sleep07059-21latency_hist20:07:260
4637218926,14sleep00-21swapper/021:42:330
24386218446,122sleep00-21swapper/000:27:240
1943021808,158sleep019574-21date20:47:260
3119121777,159sleep0284-21dbus-daemon21:27:230
27309217515,18sleep00-21swapper/019:27:490
27309217515,18sleep00-21swapper/019:27:490
8982217410,128sleep09033-21idleruntime23:37:590
23685217313,149sleep023750-21ls21:02:400
19900217214,23sleep00-21swapper/000:12:350
3073217016,134sleep00-21swapper/021:37:530
243072167127,21sleep024349-21proc_pri19:17:560
1265321666,14sleep00-21swapper/022:07:280
1265321666,14sleep00-21swapper/022:07:280
10009216612,136sleep010048-21sendmail20:17:230
932021649,55sleep00-21swapper/021:57:350
18376216410,22sleep00-21swapper/000:07:380
5159216312,21sleep00-21swapper/020:02:250
161142155107,30sleep316171-21users23:57:563
21967215417,119sleep221978-21cat19:12:292
428214712,111sleep0284-21dbus-daemon19:47:230
709214624,97sleep2908-21apt-config21:32:262
709214624,97sleep2908-21apt-config21:32:262
2895521448,124sleep028989-21ps19:32:540
1434221435,12sleep00-21swapper/020:28:000
22111214117,105sleep121596-21apt-get20:57:321
2005221415,11sleep00-21swapper/022:30:370
16575214111,97sleep00-21swapper/022:17:580
1250721418,78sleep012548-21ntpq23:47:480
1250721418,78sleep012548-21ntpq23:47:480
22809213812,105sleep022879-21unixbench_multi22:37:560
1062213810,115sleep01065-21grep21:32:400
1062213810,115sleep01065-21grep21:32:400
1228213724,51sleep01291-21mailstats23:12:520
1228213724,51sleep01291-21mailstats23:12:520
25942213627,69sleep026093-21sed00:32:250
20977213610,90sleep021130-21awk19:07:550
2850821359,68sleep028526-21apt-get22:57:380
2550121318,42sleep025598-21perl19:22:440
108821307,90sleep01124-21ntpq19:47:480
233221299,57sleep00-21swapper/023:17:400
7530212817,39sleep37593-21sed21:52:253
7530212817,39sleep37593-21sed21:52:253
1526921279,57sleep00-21swapper/023:57:370
2566221269,99sleep00-21swapper/022:47:530
2524321238,74sleep00-21swapper/021:07:370
777021228,56sleep00-21swapper/021:52:380
2361621228,55sleep00-21swapper/022:42:360
2935221195,86sleep00-21swapper/021:18:430
2278121185,85sleep00-21swapper/000:22:200
2343221167,52sleep00-21swapper/021:02:240
2671621158,16sleep00-21swapper/022:52:370
19386211522,68sleep319544-21grep19:02:543
16877211517,31sleep30-21swapper/300:02:413
605321135,27sleep00-21swapper/021:47:250
605321135,27sleep00-21swapper/021:47:250
22660211218,78sleep322705-21ntp_states19:12:503
16742211019,74sleep316800-21if_err_eth020:37:483
27391210816,74sleep227393-21ntp_states19:27:512
27391210816,74sleep227393-21ntp_states19:27:512
1655921077,14sleep00-21swapper/020:37:410
7520210614,73sleep21-21systemd21:52:242
7520210614,73sleep21-21systemd21:52:242
1965321068,59sleep00-21swapper/020:47:380
14341210616,77sleep314441-21fschecks_time22:12:443
10863210616,73sleep310879-21/usr/sbin/munin23:42:583
914121050,6sleep30-21swapper/321:57:253
719121048,14sleep00-21swapper/020:07:380
29745210418,68sleep329781-21wc21:22:253
986210315,69sleep11013-21irqstats23:12:451
986210315,69sleep11013-21irqstats23:12:451
21663210320,30sleep00-21swapper/022:36:140
411421027,13sleep00-21swapper/023:22:370
18795210226,60sleep018871-21fschecks_time19:02:430
45421017,62sleep0495-21systemd-run23:12:230
30647210118,59sleep30-21swapper/323:02:543
4318210011,72sleep14371-21/usr/sbin/munin23:22:581
4139210022,63sleep24146-21modprobe21:42:232
2788421005,83sleep00-21swapper/021:15:080
161842984,13sleep10-21swapper/123:57:581
1983129712,70sleep319867-21sort20:47:453
1109829617,60sleep210558-21/usr/sbin/munin23:43:062
74129515,67sleep3792-21cpuspeed23:12:393
74129515,67sleep3792-21cpuspeed23:12:393
233029510,16sleep30-21swapper/323:17:393
1213629512,69sleep212149-21uname20:22:462
103282958,31sleep00-21swapper/020:17:390
3127429415,61sleep231479-21latency_hist21:27:252
1408729317,60sleep314209-21cat20:27:553
1317029324,21sleep30-21swapper/322:08:033
1317029324,21sleep30-21swapper/322:08:033
469229221,43sleep34658-21apt-get21:42:563
1332329214,62sleep213444-21cat22:08:062
1332329214,62sleep213444-21cat22:08:062
81229115,56sleep3889-21fschecks_time19:47:423
2852729121,20sleep20-21swapper/221:17:442
2717229110,65sleep127179-21ntpq22:52:501
161852917,30sleep00-21swapper/020:36:440
66952905,30sleep00-21swapper/023:29:270
66952905,30sleep00-21swapper/023:29:270
2211328727,38sleep322106-21apt-get20:57:343
2337828619,17sleep10-21swapper/122:42:251
2167228636,22sleep20-21swapper/220:57:252
1894028622,43sleep218684-21/usr/sbin/munin19:02:452
2596428416,53sleep226057-21sed19:22:552
2509828426,35sleep325123-21idleruntime-cro19:22:233
1717228420,46sleep117279-21ls20:37:591
3178228217,46sleep231855-21/usr/sbin/munin23:07:452
2522028213,51sleep325239-21apt-get21:07:363
240128221,34sleep12570-21gpgv21:37:251
240128221,34sleep12570-21gpgv21:37:251
2820028121,36sleep10-21swapper/119:32:221
2820028121,36sleep10-21swapper/119:32:221
852628016,48sleep38673-21latency_hist20:12:253
719328017,43sleep27229-21cpu20:07:402
1714128026,29sleep30-21swapper/322:22:253
1339028018,38sleep2362-21snmpd23:52:222
1339028018,38sleep2362-21snmpd23:52:222
1475527918,37sleep214903-21latency_hist20:32:262
98662775,10sleep10-21swapper/123:40:021
578927713,52sleep15871-21fschecks_time20:03:031
578927713,52sleep15871-21fschecks_time20:03:031
1997027725,32sleep120006-21munin-run19:07:231
1502327724,35sleep315009-40mandb23:57:243
99972767,10sleep30-21swapper/320:17:133
256012768,56sleep10-21swapper/100:28:471
2057727614,47sleep120578-21diskstats22:32:441
1096627617,27sleep10-21swapper/123:43:011
812527522,35sleep18139-21needreboot21:52:471
1968727523,25sleep20-21swapper/200:12:252
382427415,22sleep30-21swapper/319:57:353
2179627417,42sleep121855-21/usr/sbin/munin00:17:491
1229727421,32sleep112440-21gpgv22:07:251
3002027324,27sleep30-21swapper/321:22:463
1992427319,32sleep319920-21apt-get00:12:383
159912734,11sleep30-21swapper/320:34:173
1494027316,34sleep114970-21modprobe23:57:221
982927215,24sleep10-21swapper/121:57:521
701227223,27sleep30-21swapper/323:32:253
701227223,27sleep30-21swapper/323:32:253
2707127211,15sleep20-21swapper/200:32:562
2569927210,16sleep20-21swapper/221:07:512
3168727114,42sleep131206-21apt-get21:27:321
3168727114,42sleep131206-21apt-get21:27:321
1392427120,23sleep10-21swapper/120:27:531
517927020,32sleep15369-21date20:02:261
295032705,11sleep30-21swapper/319:34:143
2688027015,32sleep226772-21/usr/sbin/munin21:12:432
2384427012,27sleep20-21swapper/219:17:422
1612427017,22sleep10-21swapper/122:17:461
2806326920,31sleep128090-21munin-run21:17:231
2494026915,29sleep224945-21modprobe21:07:222
2161226918,23sleep20-21swapper/200:17:462
1061126919,27sleep310641-21munin-run22:02:223
927426817,37sleep39293-21open_inodes23:38:063
2906726815,30sleep20-21swapper/219:32:562
273026714,37sleep12734-21sh19:52:531
201026727,21sleep20-21swapper/223:17:222
201026727,21sleep20-21swapper/223:17:222
3006826617,28sleep130087-21apt-get23:02:351
2189526622,29sleep321929-21ntp_kernel_err00:17:513
2076326622,25sleep10-21swapper/119:07:521
1783626613,38sleep217842-21ntp_states22:22:522
132626621,28sleep21428-21/usr/sbin/munin19:47:542
89372655,48sleep20-21swapper/221:56:112
1905726515,34sleep119185-21unixbench_multi00:07:561
120002655,48sleep30-21swapper/322:05:213
83122645,10sleep30-21swapper/323:35:193
83122645,10sleep30-21swapper/323:35:193
2899226410,25sleep10-21swapper/119:32:551
1965426419,28sleep119707-21ls20:47:391
1254526422,21sleep30-21swapper/320:22:553
659426316,32sleep16606-21vmstat23:28:191
659426316,32sleep16606-21vmstat23:28:191
3123526323,26sleep131273-21sed21:27:241
2630926314,34sleep126311-21sshd22:50:531
1920026313,23sleep10-21swapper/122:27:481
190692635,11sleep10-21swapper/120:43:401
82222628,44sleep10-21swapper/123:34:081
82222628,44sleep10-21swapper/123:34:081
615526222,18sleep20-21swapper/220:03:122
615526222,18sleep20-21swapper/220:03:122
2711426217,26sleep327160-21grep22:52:483
130892625,45sleep10-21swapper/123:48:341
130892625,45sleep10-21swapper/123:48:341
1266226211,24sleep10-21swapper/122:07:331
1266226211,24sleep10-21swapper/122:07:331
561626118,26sleep30-21swapper/320:02:363
561626118,26sleep30-21swapper/320:02:363
3164626117,27sleep131640-21apt-get23:07:401
2480626119,27sleep324864-21if_err_eth000:27:443
2057926114,32sleep220674-21fschecks_time22:32:452
2986626013,34sleep330012-21date23:02:253
2903726016,27sleep10-21swapper/121:17:551
279326019,20sleep10-21swapper/123:17:531
2307926019,21sleep10-21swapper/100:22:411
2117526021,18sleep30-21swapper/322:32:583
2056326014,33sleep320589-21kernelversion19:07:463
1974326017,24sleep20-21swapper/220:47:442
1041326014,31sleep210416-21/usr/sbin/munin20:17:452
3239525920,21sleep30-21swapper/321:28:073
3239525920,21sleep30-21swapper/321:28:073
2390325911,20sleep20-21swapper/222:42:462
1552825916,18sleep10-21swapper/120:32:581
1418625916,29sleep213659-21/usr/sbin/munin23:52:552
610725816,24sleep26144-21ntp_states23:28:082
610725816,24sleep26144-21ntp_states23:28:082
48902586,44sleep20-21swapper/219:59:322
463925811,19sleep20-21swapper/221:42:352
3064525819,22sleep230733-21sed19:37:572
2702625822,17sleep30-21swapper/319:27:443
2702625822,17sleep30-21swapper/319:27:443
249862585,41sleep10-21swapper/119:20:561
2368125815,26sleep123678-21apt-get21:02:371
1869125813,27sleep30-21swapper/322:27:253
2741025719,14sleep30-21swapper/321:12:573
2526425715,24sleep325373-21perl22:47:443
2406425715,20sleep10-21swapper/122:42:491
2159725719,22sleep221618-21munin-node19:12:252
1596225718,17sleep20-21swapper/223:57:552
1421925715,22sleep20-21swapper/222:12:352
132702575,43sleep30-21swapper/323:50:543
132702575,43sleep30-21swapper/323:50:543
460625615,25sleep34664-21processes23:23:083
3073425620,19sleep30-21swapper/319:37:583
284925612,23sleep32354-21apt-get21:37:353
277725616,19sleep30-21swapper/319:52:553
2538825614,25sleep125385-21apt-get19:22:371
2516725612,25sleep225161-21apt-get22:47:392
2361225613,24sleep323680-21sensors_temp00:22:553
1837725615,20sleep30-21swapper/320:42:493
1595125615,20sleep20-21swapper/222:17:432
143125619,22sleep20-21swapper/221:32:502
143125619,22sleep20-21swapper/221:32:502
1018225612,28sleep210334-21latency_hist23:42:252
873725514,25sleep18734-21apt-get20:12:351
493525514,17sleep10-21swapper/121:43:061
3162325512,24sleep231616-21apt-get19:42:412
2605825520,19sleep30-21swapper/319:22:563
2579825515,24sleep125857-21/usr/sbin/munin21:07:541
236202558,19sleep30-21swapper/322:42:393
2264425512,16sleep20-21swapper/220:58:302
2218525515,24sleep122202-21ntp_states19:12:321
2128525516,23sleep121368-21latency_hist00:17:241
1940425517,23sleep219406-21systemd-run20:47:242
1756825519,18sleep20-21swapper/200:02:562
1728125520,19sleep20-21swapper/220:38:002
913925415,25sleep20-21swapper/220:12:482
3140425419,19sleep231381-21latency_hist23:07:252
2867425415,20sleep10-21swapper/122:58:241
2799625420,21sleep228060-21unin-run22:57:242
232325416,16sleep20-21swapper/219:52:422
203225412,20sleep10-21swapper/119:52:251
622125312,25sleep16215-21apt-get21:47:391
622125312,25sleep16215-21apt-get21:47:391
1623625315,25sleep216261-21munin-run20:37:232
1424425313,23sleep114239-21apt-get22:12:381
2462525215,19sleep20-21swapper/200:27:392
232925214,22sleep22325-21apt-get23:17:382
2014125219,15sleep20-21swapper/200:12:452
1098325210,24sleep210992-21diskstats22:02:452
1972225115,18sleep30-21swapper/300:12:263
1162825116,17sleep10-21swapper/120:22:241
225212509,16sleep30-21swapper/322:37:513
2206725013,22sleep222063-21apt-get22:37:382
768424911,22sleep27690-21ntp_states23:33:002
768424911,22sleep27690-21ntp_states23:33:002
2433424914,15sleep20-21swapper/200:27:222
47424815,19sleep10-21swapper/119:47:241
33162489,16sleep10-21swapper/121:37:581
318562489,14sleep30-21swapper/323:07:463
3174224811,19sleep331828-21perl19:42:463
2376724816,20sleep323793-21fschecks_count21:02:433
267372479,15sleep30-21swapper/300:32:533
1930024712,16sleep30-21swapper/322:27:503
745824614,16sleep10-21swapper/120:07:471
1187424611,18sleep10-21swapper/120:22:361
934324513,15sleep20-21swapper/221:57:372
3014124310,19sleep130148-21df_inode19:37:431
1738224310,16sleep10-21swapper/122:22:391
1808524211,15sleep10-21swapper/100:07:231
2162423811,13sleep10-21swapper/120:57:241
214362317,13sleep30-21swapper/320:55:453
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional