You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 11:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Wed Mar 25, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
287502297351,2909sleep1590-1kworker/1:0H00:17:431
6097228168,2794sleep14259-21kworker/1:121:22:571
15694251518,483sleep315735-21ntpq21:53:083
2595240732,280sleep12675-21timerandwakeup22:53:031
21308228416,126sleep121382-21latency20:27:571
1817622400,102sleep127716-21cyclictest22:02:341
15247223413,207sleep014768-21apt-get21:52:400
22796222222,103sleep10-21swapper/123:58:011
29318221714,189sleep329393-21sendmail_mailtr20:53:073
3380221119,102sleep13402-21awk19:32:321
25681220845,58sleep1590-1kworker/1:0H20:42:431
19489220310,141sleep119505-21cpu20:22:481
3158722018,177sleep11-21systemd00:27:331
30431220137,53sleep10-21swapper/122:38:211
26014218916,155sleep326123-21timerandwakeup00:08:043
27523218713,161sleep027554-21grep20:47:550
119782186105,20sleep10-21swapper/119:57:571
24427217735,127sleep124563-21cut22:22:341
12985217513,52sleep127716-21cyclictest23:27:521
1128216928,41sleep10-21swapper/122:48:061
984621635,13sleep30-21swapper/323:17:483
9323216214,39sleep10-21swapper/119:48:281
46452158110,34sleep10-21swapper/121:18:011
8300215511,131sleep18340-21cut23:12:501
8300215511,131sleep18340-21cut23:12:501
21996215434,99sleep10-21swapper/122:13:011
21996215434,99sleep10-21swapper/122:13:011
20938215417,117sleep10-21swapper/123:52:561
674221538,125sleep10-21swapper/123:07:491
26748215327,110sleep126894-21latency_hist00:12:341
1044621509,61sleep10-21swapper/123:18:031
188721468,115sleep00-21swapper/019:27:350
18395214633,98sleep10-21swapper/120:18:001
966121438,122sleep00-21swapper/019:50:120
15971214264,22sleep10-21swapper/121:53:141
12606214060,40sleep112730-21sort21:43:041
12606214060,40sleep112730-21sort21:43:041
8125213810,111sleep18129-21munin-run19:47:351
16530213659,17sleep10-21swapper/123:38:021
29726213548,50sleep129739-21awk20:53:171
25647213414,69sleep125799-21sed22:23:231
844213313,102sleep1926-21latency19:22:551
4326213114,20sleep10-21swapper/119:33:051
31213213124,15sleep10-21swapper/119:17:341
31213213124,15sleep10-21swapper/119:17:341
2305213014,72sleep12427-21latency_hist00:37:341
13893213014,60sleep10-21swapper/121:47:571
20717212811,86sleep30-21swapper/323:52:503
321212612,97sleep00-21swapper/019:22:340
10496212625,65sleep110104-21/usr/sbin/munin19:52:561
30390212531,59sleep130382-21apt-get20:57:491
5642212420,44sleep10-21swapper/119:37:581
28525212430,79sleep128733-21date00:17:351
25162212430,57sleep125163-21munin-run00:07:341
26445212341,55sleep126486-21fschecks_count19:02:531
29243212239,34sleep129246-21cron22:37:321
27255212229,54sleep10-21swapper/120:47:481
24689212242,60sleep10-21swapper/120:38:031
1610821219,61sleep11-21systemd20:12:341
1610821219,61sleep11-21systemd20:12:341
26727212018,89sleep226773-21ntpq19:02:582
5094211818,28sleep10-21swapper/123:02:441
12608211837,28sleep10-21swapper/123:27:321
9453211712,90sleep29576-21expr21:33:042
3064211715,62sleep10-21swapper/121:13:001
2805211313,80sleep22863-21/usr/sbin/munin21:12:532
13725211121,50sleep10-21swapper/120:03:011
17713211012,81sleep317776-21/usr/sbin/munin23:42:523
17622211025,67sleep217636-21modprobe20:17:322
30220210821,70sleep330257-21ntp_kernel_pll_22:38:153
23566210413,77sleep223664-21cat00:02:342
813210346,14sleep10-21swapper/121:07:351
813210346,14sleep10-21swapper/121:07:351
32606210324,56sleep332610-21modprobe22:47:323
29771210317,68sleep029267-21apt-get22:37:430
235821037,12sleep30-21swapper/319:27:543
13255210314,74sleep213303-21latency_hist21:47:342
13255210314,74sleep213303-21latency_hist21:47:342
1063021020,9sleep30-21swapper/319:53:013
31245210019,64sleep231335-21latency_hist19:17:352
31245210019,64sleep231335-21latency_hist19:17:352
1850229810,49sleep10-21swapper/122:02:551
1850229810,49sleep10-21swapper/122:02:551
430929725,49sleep24457-21unixbench_singl22:58:052
2803829752,20sleep10-21swapper/119:07:531
2322929715,62sleep022560-21/usr/sbin/munin20:33:050
224382979,13sleep00-21swapper/023:57:510
1750029712,72sleep117502-21sshd20:15:581
47729628,34sleep10-21swapper/100:30:031
316612968,73sleep331815-21latency_hist00:27:353
2548829613,67sleep025557-21fschecks_time00:07:530
2484229618,55sleep324992-21unixbench_multi20:38:053
2435029626,43sleep324364-21apt-get22:22:323
1975729617,57sleep019464-21/usr/sbin/munin20:22:550
195702967,67sleep00-21swapper/023:47:580
170129624,52sleep01842-21sed00:33:050
1253329511,32sleep312611-21users19:58:073
805129313,66sleep38052-21cron19:47:333
322392935,11sleep30-21swapper/300:27:553
2694629223,14sleep10-21swapper/100:12:471
282152917,68sleep00-21swapper/019:07:560
2703329116,59sleep327084-21fschecks_count00:12:533
2703329116,59sleep327084-21fschecks_count00:12:533
46432908,44sleep10-21swapper/123:00:061
2048529020,16sleep10-21swapper/123:52:351
306482888,61sleep00-21swapper/019:13:050
306482888,61sleep00-21swapper/019:13:050
98702879,43sleep10-21swapper/121:34:491
214562879,57sleep00-21swapper/020:28:000
1838928715,51sleep318407-21/usr/sbin/munin22:02:453
1838928715,51sleep318407-21/usr/sbin/munin22:02:453
81672866,42sleep10-21swapper/121:28:061
645128626,26sleep30-21swapper/323:07:353
548528413,56sleep25521-21latency_hist21:22:342
3184728425,37sleep331456-21/usr/sbin/munin19:17:573
74728311,55sleep30-21swapper/321:07:333
74728311,55sleep30-21swapper/321:07:333
56928318,44sleep2544-21/usr/sbin/munin19:22:482
3175228320,31sleep20-21swapper/219:17:562
3148028310,16sleep10-21swapper/119:17:461
308632835,65sleep20-21swapper/222:40:002
2409628315,52sleep20-21swapper/222:19:162
1756828319,15sleep10-21swapper/123:42:461
1453328211,15sleep10-21swapper/123:32:531
1168528220,27sleep20-21swapper/219:57:452
265052819,17sleep10-21swapper/122:27:511
2393728113,17sleep00-21swapper/000:02:520
2122028151,15sleep10-21swapper/122:12:321
2122028151,15sleep10-21swapper/122:12:321
1890828131,33sleep018990-21mailstats22:03:060
1890828131,33sleep018990-21mailstats22:03:060
3163128027,27sleep20-21swapper/200:27:342
3158528031,34sleep10-21swapper/122:42:571
2129928013,52sleep321448-21latency_hist22:12:343
2129928013,52sleep321448-21latency_hist22:12:343
1353027918,30sleep30-21swapper/320:02:553
10512798,14sleep10-21swapper/121:07:471
2709327816,46sleep327206-21grep22:28:053
2462827812,45sleep024688-21/usr/sbin/munin20:38:020
2160327816,43sleep021696-21fschecks_time22:12:520
2160327816,43sleep021696-21fschecks_time22:12:520
395527714,43sleep33994-21awk22:57:583
2768827717,42sleep227721-21basename22:32:322
1969827715,48sleep319755-21irqstats20:22:543
104012778,14sleep10-21swapper/121:37:471
104012778,14sleep10-21swapper/121:37:471
2619127525,27sleep30-21swapper/320:43:013
125927516,30sleep00-21swapper/021:07:540
3040527415,24sleep00-21swapper/000:22:510
2857127423,27sleep30-21swapper/320:52:343
2701727412,46sleep027036-21date20:47:350
175522748,18sleep30-21swapper/321:58:053
1694227415,33sleep20-21swapper/221:57:512
1357927419,24sleep30-21swapper/321:47:513
1239327411,41sleep00-21swapper/021:42:590
1239327411,41sleep00-21swapper/021:42:590
376727313,45sleep23873-21fschecks_time19:32:542
195062738,16sleep20-21swapper/220:22:492
2392427221,26sleep20-21swapper/220:37:352
207527211,16sleep20-21swapper/222:52:522
1802927214,38sleep218122-21cut20:17:522
955727119,33sleep00-21swapper/023:17:330
955727119,33sleep00-21swapper/023:17:330
857427117,37sleep38071-21apt-get19:47:453
71527117,30sleep00-21swapper/022:47:570
698527115,24sleep00-21swapper/019:42:530
3199627129,18sleep20-21swapper/221:02:512
3199627129,18sleep20-21swapper/221:02:512
2619127124,24sleep20-21swapper/222:27:362
1599827128,24sleep30-21swapper/323:37:433
1087927121,32sleep30-21swapper/321:38:023
1087927121,32sleep30-21swapper/321:38:023
2819627014,38sleep028253-21/usr/sbin/munin22:32:550
1841027019,27sleep30-21swapper/320:18:023
182327016,36sleep31951-21/usr/sbin/munin21:08:053
1269227013,36sleep012841-21latency_hist23:27:350
1111327014,40sleep211252-21/usr/sbin/munin21:38:052
1111327014,40sleep211252-21/usr/sbin/munin21:38:052
984026921,28sleep20-21swapper/219:52:322
882526920,35sleep38878-21ls21:32:493
412026919,31sleep00-21swapper/022:58:030
2720926915,36sleep026402-21/usr/sbin/munin22:28:060
2934426821,26sleep30-21swapper/322:37:343
2799226816,29sleep20-21swapper/219:07:502
225862688,17sleep10-21swapper/120:32:501
1149426814,26sleep00-21swapper/019:57:350
444226720,28sleep34156-21/usr/sbin/munin21:17:553
2057826719,22sleep20-21swapper/222:08:052
2057826719,22sleep20-21swapper/222:08:052
190352677,14sleep00-21swapper/020:20:250
1575726722,30sleep215797-21ps21:53:112
138922677,48sleep00-21swapper/023:29:080
600626615,18sleep00-21swapper/019:38:050
394326615,28sleep30-21swapper/321:17:343
1919026613,22sleep30-21swapper/323:47:493
1695826621,27sleep30-21swapper/320:13:013
754426515,36sleep27602-21sed21:27:552
558726514,36sleep25651-21ps23:03:022
287482657,38sleep00-21swapper/000:17:420
274442658,10sleep00-21swapper/019:05:180
2438326513,31sleep024420-21munin-run22:22:330
1488326515,25sleep215025-21gpgv21:52:342
84052645,48sleep00-21swapper/021:30:520
2838226418,26sleep00-21swapper/000:17:340
2838226418,26sleep00-21swapper/000:17:340
1967226415,24sleep20-21swapper/222:07:332
1967226415,24sleep20-21swapper/222:07:332
1816426417,32sleep018156-21grep22:02:340
111132649,15sleep00-21swapper/023:22:340
693726315,29sleep27002-21latency23:07:542
2448026314,33sleep324606-21unixbench_multi00:03:033
2188426313,26sleep30-21swapper/320:28:073
1765426317,26sleep217712-21fschecks_count23:42:512
1091626310,33sleep211031-21/usr/sbin/munin19:53:052
3233026215,24sleep20-21swapper/200:27:582
2772826217,28sleep227773-21cut20:48:012
2148226214,33sleep321483-21munin-node22:12:423
2148226214,33sleep321483-21munin-node22:12:423
921626119,16sleep00-21swapper/021:32:580
392826113,33sleep03942-21uname19:32:560
3172326121,23sleep00-21swapper/022:43:000
2947526123,24sleep30-21swapper/300:18:183
2931026114,23sleep329323-21munin-node19:12:353
105226112,21sleep30-21swapper/319:23:013
589026014,13sleep30-21swapper/319:38:043
570326016,26sleep05722-21apt-get21:22:440
426726018,24sleep00-21swapper/021:17:510
3143326016,20sleep20-21swapper/222:42:542
297392604,12sleep20-21swapper/200:18:412
2888526011,18sleep00-21swapper/020:52:520
268226018,22sleep00-21swapper/021:12:510
2261226017,28sleep30-21swapper/323:57:553
2158526011,19sleep20-21swapper/222:12:502
2158526011,19sleep20-21swapper/222:12:502
206626014,32sleep02072-21diskstats22:52:500
1488726013,30sleep014896-21diskstats20:07:530
1488726013,30sleep014896-21diskstats20:07:530
1266026012,28sleep30-21swapper/323:27:343
1026426010,27sleep20-21swapper/223:18:002
511925913,27sleep05113-21apt-get23:02:470
2822225915,24sleep228284-21http00:17:322
2822225915,24sleep228284-21http00:17:322
2807725915,26sleep228166-21perl22:32:532
2544625912,20sleep30-21swapper/322:23:173
2315525916,29sleep323226-21switchtime20:33:043
1292425912,31sleep212971-21df23:27:492
2893825820,20sleep20-21swapper/220:52:582
1529725822,20sleep30-21swapper/320:08:043
1529725822,20sleep30-21swapper/320:08:043
1481925813,29sleep314852-21/usr/sbin/munin23:32:593
550425711,20sleep20-21swapper/219:37:542
503725716,16sleep30-21swapper/319:37:353
340925714,27sleep23431-21munin-run19:32:332
3047625719,20sleep30-21swapper/320:57:553
2549525719,18sleep20-21swapper/220:42:362
82422569,19sleep20-21swapper/219:47:362
2516425614,25sleep225260-21latency_hist00:07:352
2150125615,20sleep20-21swapper/220:28:022
1777825613,18sleep00-21swapper/023:42:530
1601125617,22sleep016019-21apt-get23:37:440
1444825615,25sleep014443-21apt-get23:32:480
599725516,20sleep30-21swapper/321:22:553
2690625517,16sleep30-21swapper/319:03:033
2044225519,21sleep00-21swapper/022:08:030
2044225519,21sleep00-21swapper/022:08:030
1647125514,23sleep216478-21ntp_states23:37:592
1456925513,17sleep20-21swapper/220:07:352
1347225515,22sleep013490-21apt-get21:47:440
1143225516,22sleep211436-21diskstats23:22:522
78862546,36sleep00-21swapper/023:12:000
3138825410,32sleep213441-21snmpd20:59:082
3048425411,18sleep00-21swapper/020:57:570
147882549,19sleep20-21swapper/220:07:472
147882549,19sleep20-21swapper/220:07:472
128912547,10sleep30-21swapper/321:43:313
128912547,10sleep30-21swapper/321:43:313
113725417,23sleep21156-21fschecks_count21:07:522
105125415,24sleep31063-21fschecks_count00:32:523
1040025415,24sleep010395-21apt-get21:37:460
1040025415,24sleep010395-21apt-get21:37:460
675725317,24sleep36838-21fschecks_time23:07:513
235725319,18sleep30-21swapper/322:52:573
2143825317,16sleep20-21swapper/223:53:052
1911425314,22sleep219111-21apt-get23:47:442
1330225316,18sleep30-21swapper/323:27:583
1326225316,23sleep013269-21wc20:02:480
97825212,16sleep30-21swapper/322:48:043
872125214,24sleep38815-21mailstats23:13:003
872125214,24sleep38815-21mailstats23:13:003
703725214,21sleep20-21swapper/221:27:352
2696525215,21sleep226996-21munin-run20:47:332
2109825211,23sleep021122-21ntp_states23:52:590
820925112,23sleep28207-21apt-get23:12:432
820925112,23sleep28207-21apt-get23:12:432
760325113,22sleep37642-21memory21:27:563
3194525112,23sleep031939-21apt-get21:02:490
3194525112,23sleep031939-21apt-get21:02:490
2989725112,23sleep329892-21apt-get19:12:483
2989725112,23sleep329892-21apt-get19:12:483
212325011,24sleep02117-21apt-get19:27:470
2540024912,22sleep225396-21apt-get00:07:472
2485624910,25sleep024875-21apt-get22:22:450
20812497,31sleep20-21swapper/200:35:412
1635324913,18sleep00-21swapper/020:12:460
1170824911,23sleep011703-21apt-get19:57:470
1513124814,16sleep20-21swapper/223:33:062
3652475,31sleep00-21swapper/000:28:370
81524010,16sleep20-21swapper/222:47:592
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional