You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-13 - 13:23

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Jun 13, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
39562138069,13748sleep329242-21kworker/3:022:58:083
78772166015,1629sleep17635-21apt-get21:29:531
78772166015,1629sleep17635-21apt-get21:29:531
7713295610,933sleep07635-21apt-get21:28:470
7713295610,933sleep07635-21apt-get21:28:470
2885278110,759sleep32903-21ntpq21:13:073
7638266314,634sleep23758-21kworker/u8:221:27:522
7638266314,634sleep23758-21kworker/u8:221:27:522
1469924497,431sleep130855-1kworker/1:2H20:08:021
1016243133,373sleep31154-21grep22:48:113
2930424300,9sleep127301-21cobalt_printf20:53:101
518024000,25sleep327290-21cyclictest23:02:593
1623722917,240sleep316285-21awk20:12:563
23204227178,108sleep323274-21unixbench_singl23:58:233
13142225058,30sleep30-21swapper/321:43:113
2783223819,202sleep12794-21/usr/sbin/munin22:53:141
28717222424,75sleep30-21swapper/322:33:093
31491221471,95sleep331542-21latency_hist00:27:423
32412220957,105sleep31-21systemd19:22:423
9194220416,170sleep09188-21apt-get21:32:550
9194220416,170sleep09188-21apt-get21:32:550
13408219111,163sleep113535-21timerwakeupswit23:28:091
29416218121,140sleep229566-21grep20:53:122
11957217315,144sleep312064-21timerandwakeup19:58:113
11957217315,144sleep312064-21timerandwakeup19:58:113
29230217219,93sleep010586-1kworker/0:0H20:53:100
11320216712,141sleep011339-21apt-get19:57:510
11320216712,141sleep011339-21apt-get19:57:510
2235216669,26sleep32299-21find19:28:113
10310216457,90sleep310389-21sendmail_mailtr19:53:423
5823215717,74sleep30-21swapper/321:22:483
19448215719,21sleep30-21swapper/320:22:573
12076215418,118sleep20-21swapper/221:42:422
12076215418,118sleep20-21swapper/221:42:422
11008215047,88sleep311158-21latency_hist23:22:423
18055214950,79sleep318140-21awk23:43:093
30745214751,77sleep330787-21systemctl19:17:413
30306214310,84sleep329665-21/usr/sbin/munin22:38:103
26846214219,68sleep326861-21modprobe20:47:403
8062213624,78sleep10-21swapper/119:47:591
8062213624,78sleep10-21swapper/119:47:591
31648213515,102sleep017514-21kworker/u8:221:02:430
14108213119,90sleep214268-21latency_hist23:32:432
5847212914,99sleep15342-21apt-get21:22:501
20337212912,65sleep320442-21apt-config23:52:423
13169212713,101sleep313225-21latency20:03:033
129021267,73sleep31354-21ps00:33:073
9773212418,87sleep19757-21apt-get19:53:001
9782212220,84sleep09807-21fschecks_count23:17:590
24037212220,85sleep324079-21fschecks_count00:03:033
24037212220,85sleep324079-21fschecks_count00:03:033
1508212015,75sleep30-21swapper/319:27:423
14695212018,83sleep214678-21apt-get20:07:592
135921187,18sleep20-21swapper/221:08:062
1122721177,70sleep00-21swapper/023:22:520
11032211716,90sleep311048-21cut21:38:023
11032211716,90sleep311048-21cut21:38:023
2507211544,51sleep30-21swapper/322:53:123
12996211519,74sleep213139-21unixbench_multi21:43:102
15620211418,76sleep215143-21apt-get21:52:512
30023211380,19sleep330032-21unin-run20:57:413
19235211315,84sleep019242-21ntp_states22:03:070
21723211235,58sleep30-21swapper/323:53:313
17003211222,65sleep117153-21latency_hist21:57:431
28182211112,74sleep20-21swapper/222:32:562
9739211012,83sleep29250-21apt-get19:52:532
25012211012,59sleep327290-21cyclictest22:22:563
1889121090,6sleep10-21swapper/123:47:421
1644210911,81sleep21721-21cut22:52:422
1551921096,46sleep30-21swapper/323:36:393
10392210913,80sleep010486-21threads19:53:430
19080210512,32sleep30-21swapper/320:22:413
6934210413,74sleep16981-21ntpq23:08:031
31587210312,18sleep30-21swapper/321:02:423
19789210365,21sleep30-21swapper/322:04:273
7875210216,27sleep00-21swapper/023:12:420
3725210238,15sleep30-21swapper/319:33:073
31449210215,70sleep231497-21cat22:43:012
22463210134,16sleep30-21swapper/320:32:553
14264210119,68sleep114305-21awk21:48:061
14264210119,68sleep114305-21awk21:48:061
12025210022,59sleep012032-21modprobe21:42:400
12025210022,59sleep012032-21modprobe21:42:400
919229930,16sleep30-21swapper/321:32:543
919229930,16sleep30-21swapper/321:32:543
3255229844,39sleep332695-21date22:47:433
1630029813,71sleep316432-21timerwakeupswit21:53:123
1416029819,58sleep014174-21apt-get20:07:400
40682966,12sleep20-21swapper/221:17:552
1818629619,34sleep00-21swapper/020:18:020
1913129513,67sleep119201-21latency_hist20:22:431
1637529522,54sleep215891-21/usr/sbin/munin23:38:082
2465029414,67sleep224750-21timerandwakeup20:38:102
2161429416,61sleep121736-21awk20:28:121
1362029420,30sleep10-21swapper/121:47:411
93529313,67sleep3956-21cpu21:07:553
2870229335,41sleep30-21swapper/300:17:573
2568929313,63sleep225792-21perl20:42:592
1947129312,66sleep019479-21awk23:48:070
1766329229,13sleep30-21swapper/321:58:053
1179129211,18sleep30-21swapper/323:23:113
305872918,71sleep130596-21/usr/sbin/munin22:38:121
2370629117,55sleep223781-21latency_hist00:02:422
367529012,64sleep23724-21sed19:33:062
2920929011,64sleep029305-21switchtime00:18:100
2655429018,53sleep30-21swapper/322:27:523
2090029014,56sleep020770-21/usr/sbin/munin23:53:110
761328930,13sleep30-21swapper/321:27:493
761328930,13sleep30-21swapper/321:27:493
622328914,20sleep36382-21latency_hist19:42:443
2703828922,40sleep027034-21apt-get00:12:530
2165828913,61sleep321815-21latency_hist22:12:423
888528811,18sleep30-21swapper/323:13:143
837528811,59sleep38416-21ntpq19:48:053
837528811,59sleep38416-21ntpq19:48:053
1302928814,24sleep00-21swapper/023:28:000
1149128817,55sleep111558-21sort19:58:001
1149128817,55sleep111558-21sort19:58:001
2576228719,20sleep10-21swapper/100:08:021
2576228719,20sleep10-21swapper/100:08:021
258622869,24sleep325898-21ntpq00:08:043
258622869,24sleep325898-21ntpq00:08:043
640228514,29sleep20-21swapper/219:42:512
413428519,33sleep10-21swapper/122:58:101
1466028510,25sleep314678-21apt-get20:07:523
2078628323,38sleep10-21swapper/123:52:521
283592828,59sleep328393-21munin-run00:17:413
1792028215,39sleep00-21swapper/023:43:060
1681528223,15sleep30-21swapper/323:38:133
52482819,25sleep20-21swapper/219:38:032
153128119,35sleep01684-21latency_hist19:27:430
3154528014,15sleep00-21swapper/000:27:430
2813928022,38sleep128271-21grep19:08:101
719827912,51sleep30-21swapper/323:08:093
52862799,13sleep30-21swapper/321:22:083
470727923,34sleep10-21swapper/121:18:101
383027823,26sleep20-21swapper/221:17:432
3154827813,52sleep231616-21readlink19:18:092
2841527821,27sleep10-21swapper/100:17:431
2222727815,48sleep322320-21latency_hist20:32:423
3082527724,27sleep10-21swapper/119:17:431
3005327716,37sleep029452-21/usr/sbin/munin19:13:100
2782427722,36sleep227833-21modprobe22:32:402
711527612,19sleep20-21swapper/223:08:082
3144127622,38sleep20-21swapper/200:27:412
2813627614,47sleep30-21swapper/320:49:143
2026927623,40sleep220336-21apt-config23:52:412
170327617,31sleep00-21swapper/021:08:120
84027523,23sleep10-21swapper/122:48:091
2889427420,32sleep229036-21diskstats19:12:442
2857327419,25sleep00-21swapper/022:33:060
2764427320,22sleep20-21swapper/220:48:082
141462737,15sleep30-21swapper/321:48:043
141462737,15sleep30-21swapper/321:48:043
950927216,25sleep00-21swapper/023:17:430
208427215,41sleep02102-21idleruntime-cro00:37:400
1957427113,39sleep119630-21/usr/sbin/munin20:22:591
819827016,39sleep28234-21fschecks_count23:13:012
69627016,37sleep3825-21latency_hist21:07:423
287812706,13sleep30-21swapper/319:12:403
2786627022,20sleep00-21swapper/022:32:420
2267927038,15sleep30-21swapper/322:13:113
1853927013,16sleep00-21swapper/022:02:420
1442427014,26sleep20-21swapper/223:32:592
275226911,44sleep0363-21snmpd19:29:170
2719926917,31sleep01-21systemd19:07:410
2719926917,31sleep01-21systemd19:07:410
2639426929,22sleep10-21swapper/119:03:061
2639426929,22sleep10-21swapper/119:03:061
1632026921,31sleep216325-21/usr/sbin/munin20:12:592
160192697,48sleep00-21swapper/021:53:050
127526910,14sleep10-21swapper/100:33:051
2522026814,23sleep00-21swapper/022:23:010
370826713,35sleep023758-21sshd21:17:080
3132726722,24sleep00-21swapper/022:42:590
3106426724,24sleep10-21swapper/119:17:551
2947826711,12sleep20-21swapper/219:12:562
2457826717,25sleep10-21swapper/120:38:091
76882669,15sleep37719-21munin-run19:47:413
703226615,36sleep17109-21/usr/sbin/munin19:43:111
485426622,28sleep14862-21/usr/sbin/munin19:37:501
2431926612,35sleep024347-21/usr/sbin/munin00:03:080
2431926612,35sleep024347-21/usr/sbin/munin00:03:080
188926613,38sleep21890-21sshd00:35:122
136882668,13sleep30-21swapper/323:28:113
2789326512,27sleep10-21swapper/122:32:431
355226419,20sleep20-21swapper/222:57:592
3274626416,25sleep10-21swapper/119:23:001
326852645,11sleep30-21swapper/300:28:473
3075526423,20sleep10-21swapper/100:23:121
2712426416,25sleep20-21swapper/200:12:592
2689926416,33sleep126978-21latency_hist20:47:421
250752645,11sleep30-21swapper/320:39:453
245432646,11sleep30-21swapper/322:20:223
1849326412,37sleep218516-21basename22:02:402
1516226421,28sleep015188-21munin-run21:52:410
1516226421,28sleep015188-21munin-run21:52:410
1444026420,24sleep20-21swapper/221:48:112
1444026420,24sleep20-21swapper/221:48:112
1016426421,30sleep110213-21processes23:18:081
737226319,28sleep27583-21gpgv21:27:442
584426316,28sleep05842-21apt-get21:22:490
547726317,24sleep20-21swapper/223:03:082
324726317,23sleep20-21swapper/222:57:432
2977526319,23sleep00-21swapper/022:37:590
2723326317,22sleep10-21swapper/120:47:581
2616926311,23sleep20-21swapper/200:08:112
2616926311,23sleep20-21swapper/200:08:112
1670626314,34sleep00-21swapper/020:13:090
1188126320,18sleep20-21swapper/219:58:102
1188126320,18sleep20-21swapper/219:58:102
2976326217,27sleep2225-21systemd-journal00:21:032
2590126216,29sleep00-21swapper/000:08:050
2590126216,29sleep00-21swapper/000:08:050
2213226214,32sleep222149-21munin-run23:57:412
1722726214,28sleep10-21swapper/121:57:551
956126111,25sleep20-21swapper/221:33:052
956126111,25sleep20-21swapper/221:33:052
496026112,34sleep04965-21diskstats19:37:570
3209726123,18sleep10-21swapper/100:28:041
1981526118,26sleep219817-21ntp_states20:23:052
18432615,12sleep20-21swapper/219:28:022
1721926121,23sleep00-21swapper/023:42:420
4782608,15sleep20-21swapper/200:32:012
309512606,12sleep00-21swapper/020:58:110
233472606,46sleep00-21swapper/023:58:440
204362605,11sleep00-21swapper/020:25:130
131326010,33sleep11357-21ntpq21:08:051
110526017,24sleep20-21swapper/219:23:132
95832590,6sleep20-21swapper/219:52:432
95832590,6sleep20-21swapper/219:52:432
2365625918,23sleep10-21swapper/100:02:411
2330225914,18sleep10-21swapper/120:33:131
2314625918,26sleep00-21swapper/022:17:400
197832597,39sleep20-21swapper/222:04:222
1420325914,27sleep114289-21gpgv20:07:411
1357625916,23sleep00-21swapper/021:47:400
2255225811,27sleep222678-21/usr/sbin/munin22:13:102
2146925819,22sleep20-21swapper/222:11:082
2110025814,24sleep20-21swapper/220:28:012
3077525717,15sleep20-21swapper/220:58:092
2648125722,18sleep30-21swapper/319:03:083
2648125722,18sleep30-21swapper/319:03:083
2525625712,15sleep20-21swapper/200:07:422
2525625712,15sleep20-21swapper/200:07:422
246552575,12sleep20-21swapper/222:21:502
2451325729,12sleep00-21swapper/020:38:080
2392925719,19sleep10-21swapper/122:18:101
819025611,32sleep18197-21diskstats23:13:001
632025615,20sleep00-21swapper/023:07:420
330525618,20sleep10-21swapper/119:32:571
330425614,25sleep03297-21apt-get19:32:560
3180425612,25sleep131823-21apt-get21:02:521
2603325618,23sleep00-21swapper/020:43:050
2152525614,26sleep00-21swapper/020:28:110
2070025617,24sleep120744-21ntpq22:08:041
1763525615,23sleep217787-21modprobe20:17:442
159425612,25sleep01604-21modprobe22:52:400
1494525618,15sleep10-21swapper/123:33:101
1411325619,23sleep214115-21sshd20:07:062
1134125615,18sleep10-21swapper/123:23:011
852325512,24sleep08591-21/usr/sbin/munin19:48:100
852325512,24sleep08591-21/usr/sbin/munin19:48:100
3242325511,19sleep00-21swapper/019:22:430
2214225517,18sleep10-21swapper/122:13:011
1758425523,17sleep10-21swapper/120:17:431
535825414,19sleep00-21swapper/021:22:410
3229225415,22sleep232314-21/usr/sbin/munin21:03:092
322862545,38sleep10-21swapper/122:44:541
282762545,38sleep00-21swapper/020:51:000
2779125416,18sleep00-21swapper/019:08:010
2255125421,17sleep00-21swapper/020:33:000
2079225414,17sleep00-21swapper/022:08:070
1636825411,27sleep10-21swapper/123:38:071
147202549,24sleep00-21swapper/020:08:170
3063625312,20sleep10-21swapper/120:58:041
2930625314,20sleep10-21swapper/100:18:111
2690325318,18sleep20-21swapper/219:03:562
2690325318,18sleep20-21swapper/219:03:562
1941125311,25sleep10-21swapper/122:03:121
1753125314,23sleep217537-21df_inode23:42:562
984925214,21sleep10-21swapper/121:33:121
984925214,21sleep10-21swapper/121:33:121
301542529,18sleep00-21swapper/000:22:500
2416025215,19sleep10-21swapper/100:03:051
2416025215,19sleep10-21swapper/100:03:051
237125215,17sleep10-21swapper/119:28:131
1214325216,15sleep20-21swapper/223:23:142
1203325212,23sleep112074-21systemd-run21:42:411
1203325212,23sleep112074-21systemd-run21:42:411
773025114,22sleep07740-21unin-run19:47:420
1475325113,17sleep00-21swapper/023:33:050
2632224912,22sleep226460-21latency_hist22:27:432
2501024912,21sleep125005-21apt-get22:22:541
2380224912,23sleep223841-21cat20:37:422
314324814,17sleep00-21swapper/022:57:410
3023424812,21sleep230305-21awk22:38:092
267324812,17sleep10-21swapper/121:13:011
1021424815,17sleep20-21swapper/223:18:092
1901324711,21sleep119080-21df23:47:551
528624616,16sleep10-21swapper/123:03:031
43762456,22sleep010586-1kworker/0:0H22:58:130
281142457,14sleep10-21swapper/122:32:551
2392024511,18sleep023928-21ntp_states22:18:080
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional