You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-18 - 07:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Thu Jun 18, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
51372680919,6772sleep114897-21kworker/1:022:37:381
2029269710,669sleep12157-21/usr/sbin/munin20:43:151
2555325867,566sleep225566-21ntp_states21:57:482
31579247067,384sleep231608-21munin-plugin-st20:37:222
854424267,409sleep2363-21snmpd21:04:482
13058241823,379sleep113061-21/usr/sbin/munin19:37:421
21128240712,381sleep321177-21ntpq20:02:473
21128240712,381sleep321177-21ntpq20:02:473
22516236811,339sleep222527-21apt-get20:07:342
573123547,337sleep2363-21snmpd00:20:252
1703235312,325sleep21719-21modprobe22:27:222
406233313,201sleep2466-21sendmail_mailqu00:02:532
295482293101,176sleep229706-21latency_hist22:12:252
9495229016,256sleep19544-21unin-run22:52:231
23241228118,189sleep223282-21grep23:32:492
23241228118,189sleep223282-21grep23:32:492
29254223261,43sleep229318-21mailstats20:27:492
29254223261,43sleep229318-21mailstats20:27:492
1992222760,40sleep20-21swapper/222:27:342
12049222550,27sleep20-21swapper/222:58:422
12049222550,27sleep20-21swapper/222:58:422
48282223121,29sleep20-21swapper/219:12:292
5226222140,148sleep20-21swapper/222:37:432
4040222010,193sleep24121-21mailstats22:32:512
11200221548,119sleep20-21swapper/222:57:252
15909220415,95sleep316035-21sed21:27:543
15909220415,95sleep316035-21sed21:27:543
32072218735,128sleep232226-21unixbench_singl22:17:542
26501218512,158sleep126585-21find23:42:531
4511218418,138sleep20-21swapper/219:12:252
18645217916,148sleep218713-21ls21:37:382
16718217917,48sleep216880-21cat21:32:252
16718217917,48sleep216880-21cat21:32:252
28193217613,151sleep028211-21apt22:07:340
23012217112,142sleep223077-21apt-config21:52:232
2028721718,14sleep00-21swapper/023:22:540
10053217012,144sleep210109-21interrupts19:27:422
10435216913,64sleep20-21swapper/200:33:572
297932165113,24sleep20-21swapper/223:52:522
3284215313,123sleep03340-21tune2fs19:07:420
21194215215,117sleep20-21swapper/223:27:322
21194215215,117sleep20-21swapper/223:27:322
223621517,109sleep229847-21kworker/2:220:44:042
9545214959,78sleep29616-21latency_hist22:52:242
24479214316,20sleep20-21swapper/223:37:422
24479214316,20sleep20-21swapper/223:37:422
20260214217,18sleep20-21swapper/221:42:382
20260214217,18sleep20-21swapper/221:42:382
32145214116,14sleep20-21swapper/200:02:242
27168214111,115sleep227228-21systemd-run23:47:222
26935213711,62sleep20-21swapper/222:02:442
31033213621,92sleep331056-21munin-plugin-st22:17:223
2532821365,51sleep20-21swapper/220:16:592
2821321346,55sleep20-21swapper/220:23:542
2821321346,55sleep20-21swapper/220:23:542
11393213110,67sleep20-21swapper/219:32:392
18956212912,70sleep20-21swapper/219:57:252
18956212912,70sleep20-21swapper/219:57:252
16253212714,71sleep20-21swapper/219:47:442
21019212610,17sleep10-21swapper/121:42:541
21019212610,17sleep10-21swapper/121:42:541
838921259,95sleep20-21swapper/219:22:422
1679121247,15sleep20-21swapper/223:12:512
1679121247,15sleep20-21swapper/223:12:512
8987212318,32sleep19027-21http00:32:221
30828212323,77sleep00-21swapper/023:57:380
17592212331,38sleep00-21swapper/019:52:350
17592212331,38sleep00-21swapper/019:52:350
2780212145,61sleep20-21swapper/220:47:362
2769321219,77sleep227762-21df23:47:392
2769321219,77sleep227762-21df23:47:392
2198821215,53sleep20-21swapper/220:07:062
2198821215,53sleep20-21swapper/220:07:062
1470921205,101sleep00-21swapper/021:23:470
339621179,86sleep20-21swapper/200:12:462
15778211715,85sleep115812-21awk19:47:231
7911211614,83sleep07969-21/usr/sbin/munin00:27:410
1712821166,22sleep20-21swapper/221:32:542
20512211522,68sleep120555-21latency_hist20:02:251
23206211321,64sleep023410-21latency_hist21:52:250
19655211310,14sleep00-21swapper/021:39:090
2927211143,48sleep22960-21cyclictest19:07:242
2927211143,48sleep22960-21cyclictest19:07:242
2924421106,20sleep10-21swapper/123:52:381
7224210913,77sleep17258-21cut19:17:531
26156210975,17sleep20-21swapper/220:17:522
26156210975,17sleep20-21swapper/220:17:522
1970821097,58sleep20-21swapper/223:22:442
1372521097,59sleep20-21swapper/221:22:362
6297210813,80sleep16300-21/usr/sbin/munin20:57:481
2475121088,67sleep2363-21snmpd21:56:512
10283210818,72sleep110434-21sed00:33:561
8973210417,70sleep19100-21/usr/sbin/munin19:22:531
1805521047,57sleep20-21swapper/223:17:432
1805521047,57sleep20-21swapper/223:17:432
22141210315,74sleep122186-21ntpq21:47:461
27310210222,58sleep327343-21fschecks_count20:22:413
27310210222,58sleep327343-21fschecks_count20:22:413
21245210226,45sleep121214-21apt-get23:27:481
21245210226,45sleep121214-21apt-get23:27:481
772721018,18sleep20-21swapper/200:27:352
4426210160,22sleep20-21swapper/220:52:402
3314210116,21sleep00-21swapper/022:32:240
32269210119,68sleep332308-21ntpq20:37:473
14783210119,63sleep214801-21kernelversion19:42:442
12104210020,56sleep112265-21sed19:32:541
2415129916,62sleep124306-21sed23:37:241
1919329859,19sleep20-21swapper/219:57:372
955129710,69sleep39505-21apt-get00:33:013
715729760,18sleep20-21swapper/219:17:492
1662629627,46sleep116677-21munin-run21:32:241
1662629627,46sleep116677-21munin-run21:32:241
1519329617,65sleep315258-21awk23:07:513
1519329617,65sleep315258-21awk23:07:513
2250729426,42sleep122024-21apt-get20:07:331
2784229316,55sleep127662-21/usr/sbin/munin23:47:461
2784229316,55sleep127662-21/usr/sbin/munin23:47:461
904529259,21sleep29038-21apt-get21:07:382
904529259,21sleep29038-21apt-get21:07:382
67222929,66sleep20-21swapper/200:22:512
2951929213,66sleep129547-21latency_hist22:12:241
217129239,36sleep31200-21/usr/sbin/munin20:43:173
832329115,58sleep38211-21/usr/sbin/munin22:47:413
2349729110,16sleep30-21swapper/321:52:323
144162917,14sleep20-21swapper/223:07:222
144162917,14sleep20-21swapper/223:07:222
142632919,19sleep214278-21munin-run19:42:232
119529118,49sleep1415-21/usr/sbin/munin22:22:531
1130229119,51sleep311515-21latency_hist22:57:263
2853129013,51sleep128581-21latency_hist20:27:241
2853129013,51sleep128581-21latency_hist20:27:241
161552907,12sleep00-21swapper/019:47:420
633428916,54sleep16397-21/usr/sbin/munin00:22:421
3238528915,59sleep00-21swapper/000:02:390
1078128927,39sleep010806-21munin-run00:37:220
793228743,12sleep20-21swapper/222:47:122
743228720,45sleep07579-21grep19:17:560
3077628743,12sleep20-21swapper/222:14:102
242928743,12sleep20-21swapper/200:08:262
242928743,12sleep20-21swapper/200:08:262
1784528717,52sleep117914-21df23:17:381
1784528717,52sleep117914-21df23:17:381
1186128630,35sleep111887-21apt21:17:221
69012857,18sleep26904-21sshd20:58:382
126112857,14sleep20-21swapper/221:17:482
1011028526,44sleep310126-21cut19:27:433
2796428419,16sleep30-21swapper/323:47:483
2796428419,16sleep30-21swapper/323:47:483
2722528415,54sleep027293-21/usr/sbin/munin20:22:380
2722528415,54sleep027293-21/usr/sbin/munin20:22:380
3106628312,50sleep331215-21uptime20:32:553
2879328215,40sleep30-21swapper/322:07:543
2737428216,38sleep327529-21grep22:02:533
2096128220,38sleep320714-21apt-get23:27:253
1570828244,28sleep20-21swapper/223:09:042
1570828244,28sleep20-21swapper/223:09:042
1099428211,57sleep311040-21ntp_offset21:12:483
157112818,17sleep00-21swapper/023:09:060
157112818,17sleep00-21swapper/023:09:060
1154628141,11sleep20-21swapper/221:13:282
644828012,16sleep00-21swapper/022:42:250
1166228024,33sleep00-21swapper/019:32:450
334327919,43sleep13169-21/usr/sbin/munin19:07:431
1957127915,51sleep319621-21ntp_kernel_pll_19:57:463
123027915,46sleep0363-21snmpd20:42:370
69627840,11sleep20-21swapper/220:42:192
561027813,15sleep00-21swapper/020:57:230
517427718,28sleep00-21swapper/000:17:520
3196227720,37sleep132070-21grep22:17:531
2027827720,29sleep00-21swapper/021:42:400
2027827720,29sleep00-21swapper/021:42:400
1857927717,41sleep018728-21unixbench_singl23:17:540
1857927717,41sleep018728-21unixbench_singl23:17:540
1840727717,34sleep10-21swapper/121:37:251
668727614,45sleep06755-21sshd22:42:380
2307827622,34sleep323202-21apt-config21:52:243
330627518,39sleep13395-21/usr/sbin/munin00:12:451
2363527520,36sleep123652-21uniq21:52:541
1974027518,26sleep00-21swapper/019:57:520
1760327512,47sleep317746-21latency_hist23:17:243
1760327512,47sleep317746-21latency_hist23:17:243
2732027420,26sleep00-21swapper/023:47:250
2637027418,39sleep326410-21ps23:42:513
941727310,32sleep00-21swapper/021:07:480
941727310,32sleep00-21swapper/021:07:480
742727318,31sleep37575-21unixbench_singl22:42:533
496827315,26sleep30-21swapper/320:52:543
401627314,40sleep14039-21/usr/sbin/munin22:32:501
287327311,45sleep12972-21perl20:47:431
3150127216,33sleep10-21swapper/123:57:591
215127218,24sleep10-21swapper/100:07:521
215127218,24sleep10-21swapper/100:07:521
206612727,11sleep10-21swapper/123:26:551
1446827211,45sleep014601-21sed23:07:240
1446827211,45sleep014601-21sed23:07:240
106427223,25sleep00-21swapper/022:22:520
911627115,27sleep30-21swapper/300:32:243
751927110,41sleep30-21swapper/321:02:393
723027115,24sleep10-21swapper/122:42:511
40072718,51sleep00-21swapper/020:51:450
327127112,44sleep03305-21irqstats00:12:440
2792727121,22sleep00-21swapper/022:07:220
205627116,30sleep00-21swapper/000:07:510
205627116,30sleep00-21swapper/000:07:510
701827015,42sleep37076-21sed19:17:463
2411727024,26sleep00-21swapper/023:37:230
2411727024,26sleep00-21swapper/023:37:230
1741727019,27sleep30-21swapper/319:52:253
2799826916,23sleep00-21swapper/023:47:490
2799826916,23sleep00-21swapper/023:47:490
254326926,18sleep30-21swapper/322:27:523
1226126915,34sleep011570-21/usr/sbin/munin22:58:480
1226126915,34sleep011570-21/usr/sbin/munin22:58:480
2273526815,19sleep00-21swapper/020:08:370
2214826818,26sleep10-21swapper/120:07:251
2214826818,26sleep10-21swapper/120:07:251
2149326820,22sleep30-21swapper/321:47:233
2149326820,22sleep30-21swapper/321:47:233
775626714,32sleep17457-21/usr/sbin/munin21:02:451
33512670,7sleep30-21swapper/320:47:523
3265326724,20sleep00-21swapper/022:22:240
3247326720,32sleep10-21swapper/100:02:441
1662526717,24sleep10-21swapper/123:12:461
1662526717,24sleep10-21swapper/123:12:461
1031226716,35sleep110361-21processes19:27:491
978226617,31sleep19777-21apt-get22:52:351
2487126616,27sleep00-21swapper/020:12:530
904626520,28sleep39038-21apt-get21:07:393
904626520,28sleep39038-21apt-get21:07:393
83826518,25sleep00-21swapper/020:42:250
296726517,18sleep10-21swapper/122:28:381
2913226516,23sleep10-21swapper/120:27:441
2913226516,23sleep10-21swapper/120:27:441
2550926514,20sleep10-21swapper/121:57:471
189302657,13sleep10-21swapper/121:37:451
1536626515,33sleep015460-21cut21:27:430
1536626515,33sleep015460-21cut21:27:430
1467926515,23sleep10-21swapper/123:07:331
1467926515,23sleep10-21swapper/123:07:331
2932926411,36sleep029390-21fschecks_time23:52:420
2561326410,45sleep00-21swapper/023:42:200
2561326410,45sleep00-21swapper/023:42:200
2463126419,26sleep10-21swapper/123:37:451
2463126419,26sleep10-21swapper/123:37:451
2328426417,21sleep30-21swapper/323:32:503
2328426417,21sleep30-21swapper/323:32:503
197426419,23sleep30-21swapper/300:07:503
197426419,23sleep30-21swapper/300:07:503
1430526425,22sleep00-21swapper/019:42:260
1427926414,37sleep314303-21latency_hist19:42:253
1291226421,25sleep10-21swapper/123:02:241
1291226421,25sleep10-21swapper/123:02:241
1263526411,36sleep312641-21ntp_states21:17:493
511026313,31sleep35172-21/usr/sbin/munin00:17:513
2214926319,19sleep00-21swapper/023:28:110
2214926319,19sleep00-21swapper/023:28:110
1708626318,28sleep317052-21apt-get21:32:503
463626213,32sleep30-21swapper/319:12:263
3058126113,28sleep130293-21/usr/sbin/munin20:32:451
2133226115,29sleep30-21swapper/323:27:543
2133226115,29sleep30-21swapper/323:27:543
1736126118,19sleep00-21swapper/021:33:000
1693826111,25sleep00-21swapper/023:12:530
1693826111,25sleep00-21swapper/023:12:530
993126015,28sleep00-21swapper/019:27:400
887826014,19sleep00-21swapper/019:22:520
515726011,34sleep10-21swapper/119:12:361
3172326021,21sleep30-21swapper/322:17:463
3080326015,19sleep00-21swapper/020:32:520
27162606,16sleep00-21swapper/019:05:280
27162606,16sleep00-21swapper/019:05:280
2519526014,30sleep325207-21/usr/sbin/munin21:57:413
1286726018,25sleep312896-21awk23:02:223
1286726018,25sleep312896-21awk23:02:223
46002595,10sleep30-21swapper/322:34:333
2577625915,20sleep10-21swapper/120:17:431
2577625915,20sleep10-21swapper/120:17:431
2205625915,26sleep022146-21dpkg20:07:240
2205625915,26sleep022146-21dpkg20:07:240
1917925915,28sleep319249-21sensors21:37:513
1421325914,22sleep10-21swapper/121:22:511
89125813,31sleep3363-21snmpd00:03:343
879225815,21sleep30-21swapper/319:22:513
341125818,19sleep30-21swapper/300:12:473
1436725817,17sleep30-21swapper/321:22:533
655925718,19sleep00-21swapper/020:57:550
2951125717,19sleep30-21swapper/323:52:453
2629325711,20sleep00-21swapper/020:17:540
2629325711,20sleep00-21swapper/020:17:540
2517925710,17sleep30-21swapper/320:15:023
121792577,34sleep00-21swapper/021:17:370
648425618,17sleep30-21swapper/320:57:543
515625616,24sleep05152-21apt-get19:12:350
2884525613,24sleep30-21swapper/320:27:383
2884525613,24sleep30-21swapper/320:27:383
2277625614,26sleep122768-21apt-get23:32:361
158025612,20sleep10-21swapper/119:02:351
158025612,20sleep10-21swapper/119:02:351
547225515,19sleep10-21swapper/120:55:381
352625510,27sleep00-21swapper/022:32:330
2307025514,25sleep323076-21ntp_states20:08:443
1807225511,30sleep318078-21ntp_states19:52:503
1807225511,30sleep318078-21ntp_states19:52:503
160325520,16sleep30-21swapper/319:02:373
160325520,16sleep30-21swapper/319:02:373
1451725514,24sleep014585-21df19:42:380
1138825513,25sleep311384-21apt-get19:32:363
2922525413,26sleep029231-21ntp_states20:27:470
2922525413,26sleep029231-21ntp_states20:27:470
187312548,33sleep10-21swapper/119:55:301
187312548,33sleep10-21swapper/119:55:301
925625314,15sleep10-21swapper/121:07:451
925625314,15sleep10-21swapper/121:07:451
524225312,20sleep10-21swapper/100:17:531
2708125313,24sleep127118-21ntp_states22:02:471
2510525313,23sleep025101-21apt-get21:57:340
2349925315,25sleep023017-21apt-get21:52:330
1604325311,24sleep116062-21apt-get19:47:351
663225214,22sleep36638-21ntp_states00:22:483
653425213,21sleep00-21swapper/000:22:450
551225220,17sleep30-21swapper/322:37:483
3037725216,17sleep10-21swapper/122:12:521
2408625213,23sleep124081-21apt-get20:12:341
1811325214,19sleep30-21swapper/323:17:443
1811325214,19sleep30-21swapper/323:17:443
1371425211,14sleep10-21swapper/123:02:531
1371425211,14sleep10-21swapper/123:02:531
1059125212,23sleep010585-21apt-get21:12:370
870025117,21sleep00-21swapper/022:47:500
748025115,18sleep00-21swapper/021:02:350
1582925115,16sleep30-21swapper/319:47:253
133802517,26sleep10-21swapper/121:21:381
3198124911,22sleep131985-21diskstats20:37:411
3018224912,20sleep030218-21ntp_states22:12:470
99124813,21sleep31059-21switchtime22:22:513
983324814,16sleep00-21swapper/022:52:380
327824812,20sleep33282-21diskstats19:07:413
2565524811,23sleep325663-21systemd-run23:42:233
2565524811,23sleep325663-21systemd-run23:42:233
131332438,15sleep30-21swapper/323:02:343
131332438,15sleep30-21swapper/323:02:343
256752408,15sleep30-21swapper/320:17:373
256752408,15sleep30-21swapper/320:17:373
3082623910,16sleep30-21swapper/323:57:373
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional