You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-18 - 04:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Dec 18, 2025 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
913827839,757sleep09147-21/usr/sbin/munin23:00:000
2517026638,643sleep325214-21ntpq23:50:063
2517026638,643sleep325214-21ntpq23:50:063
6317263015,599sleep06340-21/usr/sbin/munin22:50:070
3018925586,540sleep230668-1kworker/2:1H22:25:182
15272493135,223sleep31651-21apt-key19:14:443
17842249199,31sleep30-21swapper/321:45:133
17842249199,31sleep30-21swapper/321:45:133
4191247271,386sleep34255-21if_eth019:20:053
950241523,365sleep3979-21munin-plugin-st20:54:423
15404238512,360sleep315497-21sed19:55:143
19536234118,200sleep319688-21unixbench_singl21:50:143
18153232031,278sleep3369-21snmpd23:27:373
9046229312,266sleep39183-21cut21:19:453
23942227295,22sleep30-21swapper/322:05:123
7347225662,171sleep37551-21latency_hist21:14:453
59612253103,57sleep30-21swapper/300:30:203
28353225392,57sleep328370-21apt22:19:543
28353225392,57sleep328370-21apt22:19:543
7090221917,187sleep17139-21awk19:30:001
7682221125,34sleep00-21swapper/021:14:540
25569220912,141sleep325638-21cat22:10:133
32193220526,47sleep30-21swapper/322:30:113
23367219610,171sleep323378-21ntpq20:20:583
30685219412,168sleep030707-21cut20:45:020
30685219412,168sleep030707-21cut20:45:020
19762219413,161sleep319810-21latency20:10:043
12766219212,166sleep212831-21find23:10:122
2619219133,28sleep30-21swapper/322:39:563
18221219119,97sleep318243-21tr20:05:073
16674219111,167sleep016705-21grep20:00:070
13696218625,137sleep30-21swapper/323:14:453
21606218271,95sleep321674-21modprobe21:59:433
247702180106,18sleep30-21swapper/320:25:103
9397217717,134sleep30-21swapper/319:35:153
9261217510,155sleep39282-21dpkg21:19:533
2147621748,143sleep321614-21latency_hist23:39:453
16932217111,148sleep317006-21mailstats20:00:153
24105216914,139sleep024121-21ls20:24:430
31280216212,136sleep331294-21grep00:10:053
23347215815,121sleep323428-21fschecks_time23:45:013
23347215815,121sleep323428-21fschecks_time23:45:013
29478215715,51sleep30-21swapper/300:05:003
27680215514,126sleep327667-40mandb23:59:433
1884215512,128sleep32004-21timerwakeupswit20:55:143
21803215375,25sleep30-21swapper/323:40:013
21803215375,25sleep30-21swapper/323:40:013
19577215210,21sleep00-21swapper/020:10:010
6594214922,43sleep30-21swapper/322:50:143
795121468,127sleep37985-21ps22:55:113
679021468,17sleep36817-21munin-run19:29:423
3568214420,98sleep30-21swapper/321:00:193
11559214415,105sleep311705-21head21:25:153
3043213917,41sleep30-21swapper/319:15:413
3043213917,41sleep30-21swapper/319:15:413
26563213812,40sleep30-21swapper/320:30:193
16214213811,20sleep00-21swapper/021:40:300
16214213811,20sleep00-21swapper/021:40:300
4118213020,89sleep24189-21fschecks_time19:20:042
28099213012,39sleep30-21swapper/320:35:163
16006212916,94sleep30-21swapper/323:20:143
4017212717,96sleep34052-21latency00:25:193
1137212419,27sleep20-21swapper/222:35:032
1137212419,27sleep20-21swapper/222:35:032
1404221239,38sleep30-21swapper/323:15:183
1404221239,38sleep30-21swapper/323:15:183
1428221227,37sleep30-21swapper/321:35:083
10195212210,36sleep30-21swapper/319:39:563
2656121207,37sleep30-21swapper/322:14:573
2792321199,34sleep30-21swapper/323:59:553
8070211813,91sleep18144-21systemd-run00:39:431
18570211819,85sleep118622-21sort21:49:431
18570211819,85sleep118622-21sort21:49:431
4372211632,68sleep34437-21cpuspeed21:04:593
4372211632,68sleep34437-21cpuspeed21:04:593
29425211418,74sleep30-21swapper/320:40:083
2869621140,7sleep00-21swapper/000:00:160
7595211222,36sleep30-21swapper/319:30:153
5734211212,22sleep25735-21ntp_states00:30:152
1191211262,31sleep31197-21fschecks_count22:35:203
1191211262,31sleep31197-21fschecks_count22:35:203
3605211110,89sleep0369-21snmpd19:18:450
3605211110,89sleep0369-21snmpd19:18:450
30134211111,85sleep330153-21apt-get22:24:543
18606211132,66sleep318640-21cpu23:29:593
22706210917,34sleep30-21swapper/322:00:153
22706210917,34sleep30-21swapper/322:00:153
1912210914,76sleep21416-21apt-get00:19:532
12638210911,84sleep012706-21systemctl23:10:100
13962210810,14sleep00-21swapper/023:15:150
13962210810,14sleep00-21swapper/023:15:150
12122210715,30sleep01-21systemd21:29:440
1269210421,12sleep30-21swapper/300:18:123
861210316,68sleep2990-21/usr/sbin/munin19:10:112
7257210211,13sleep30-21swapper/300:35:133
4184210214,33sleep30-21swapper/322:44:453
399921025,88sleep00-21swapper/021:03:390
1219210216,23sleep20-21swapper/220:54:532
9207210114,15sleep30-21swapper/323:00:033
27900210114,65sleep127901-21/usr/sbin/munin23:59:531
12134210012,71sleep112238-21date21:29:451
463129919,59sleep24376-21/usr/sbin/munin22:45:212
322329911,36sleep30-21swapper/300:24:433
3223129912,68sleep032291-21/usr/sbin/munin20:50:020
2103429929,43sleep121179-21sed21:55:131
2074129922,50sleep120762-21munin-plugin-st20:14:421
1267029919,12sleep30-21swapper/319:45:293
2205629825,60sleep30-21swapper/320:16:423
1678629731,56sleep3369-21snmpd21:42:103
1678629731,56sleep3369-21snmpd21:42:103
1444729720,12sleep30-21swapper/319:53:323
1444729720,12sleep30-21swapper/319:53:323
3175729619,11sleep30-21swapper/320:48:283
3175729619,11sleep30-21swapper/320:48:283
1344529619,10sleep30-21swapper/321:32:213
1155329616,11sleep30-21swapper/323:05:373
1155329616,11sleep30-21swapper/323:05:373
2757329522,52sleep027638-21uniq20:35:040
2692929512,52sleep0369-21snmpd20:31:130
2062129512,69sleep020652-21kernelversion21:55:030
757129411,67sleep17575-21diskstats22:55:011
182529417,57sleep12024-21grep19:14:461
182529417,57sleep12024-21grep19:14:461
3061029313,65sleep130613-21needreboot22:26:051
2680329213,63sleep226818-21/usr/sbin/munin23:55:102
1677429215,61sleep216805-21nfs_client20:00:092
1568929213,63sleep215183-21apt-get21:39:512
1568929213,63sleep215183-21apt-get21:39:512
652529124,35sleep10-21swapper/100:34:421
2507129117,54sleep125134-21latency23:50:041
2507129117,54sleep125134-21latency23:50:041
2014629116,62sleep220171-21cpu23:34:592
788828919,54sleep07931-21awk22:55:080
637728714,58sleep00-21swapper/021:10:090
3120128633,31sleep30-21swapper/319:04:593
3120128633,31sleep30-21swapper/319:04:593
66628515,45sleep0877-21latency_hist22:34:450
464128516,53sleep04657-21kernelversion21:05:090
464128516,53sleep04657-21kernelversion21:05:090
400228521,28sleep20-21swapper/222:44:442
1390428511,58sleep113421-21apt-get23:14:511
1390428511,58sleep113421-21apt-get23:14:511
3257328411,57sleep00-21swapper/000:14:580
123652840,6sleep20-21swapper/221:29:532
874028319,43sleep28798-21idleruntime-cro22:59:422
735928327,36sleep00-21swapper/019:30:070
2773928340,19sleep027761-1kworker/0:1H23:59:440
1560928321,40sleep115625-21fschecks_count23:20:041
1234628315,51sleep112364-21/usr/sbin/munin21:29:531
387228225,43sleep03890-21munin-run22:44:420
2960128219,30sleep00-21swapper/000:05:050
1995828122,37sleep119876-21latency_hist23:34:461
76228011,52sleep132612-21/usr/sbin/munin19:10:101
1863528014,48sleep018763-21unixbench_multi20:05:160
408827916,49sleep14117-21grep19:20:031
2855027918,30sleep00-21swapper/022:20:190
2855027918,30sleep00-21swapper/022:20:190
1239727922,37sleep112518-21cat19:45:151
911427810,51sleep09196-21mailstats19:35:120
282927810,14sleep20-21swapper/219:15:362
282927810,14sleep20-21swapper/219:15:362
2627127818,38sleep026297-21basename22:14:420
3168127714,40sleep131701-21apt-get22:29:531
2338427720,35sleep123420-21grep20:21:001
3181727614,47sleep231899-21grep22:30:022
309127623,24sleep10-21swapper/121:00:091
2914127614,46sleep229157-21fschecks_count20:40:032
270927622,24sleep10-21swapper/122:40:031
1889727516,44sleep218939-21fschecks_count21:50:012
1774527517,41sleep117895-21latency_hist20:04:451
1622127515,23sleep10-21swapper/121:40:311
1622127515,23sleep10-21swapper/121:40:311
2884127419,26sleep10-21swapper/120:39:461
2665027415,39sleep126715-21fschecks_time22:15:031
1338027416,43sleep113428-21fschecks_count19:50:031
1338027416,43sleep113428-21fschecks_count19:50:031
2969627319,35sleep129808-21apt-config22:24:441
2969627319,35sleep129808-21apt-config22:24:441
2082027313,48sleep220867-21processes21:55:092
1766527317,43sleep117749-21timerandwakeup23:25:131
1588527310,48sleep215899-21needreboot23:20:092
1223427313,44sleep112238-21diskstats23:09:591
327027213,28sleep00-21swapper/000:24:440
2213327221,23sleep00-21swapper/022:00:040
2213327221,23sleep00-21swapper/022:00:040
566727122,27sleep00-21swapper/019:25:070
488927113,22sleep10-21swapper/122:45:281
3252227114,45sleep132560-21ntp_kernel_err20:50:081
2352727120,28sleep20-21swapper/223:45:052
2352727120,28sleep20-21swapper/223:45:052
229632717,12sleep00-21swapper/023:44:380
229632717,12sleep00-21swapper/023:44:380
915327018,28sleep20-21swapper/223:00:022
3185227017,36sleep1287-21dbus-daemon20:49:431
3185227017,36sleep1287-21dbus-daemon20:49:431
128227022,22sleep10-21swapper/122:35:221
128227022,22sleep10-21swapper/122:35:221
42092697,49sleep00-21swapper/000:25:240
2848326912,43sleep228563-21cat00:00:152
1727026914,40sleep017315-21cpu21:44:580
1727026914,40sleep017315-21cpu21:44:580
896226810,20sleep20-21swapper/221:19:402
2770526813,40sleep227758-21sshd20:35:062
2550326819,33sleep225568-21awk22:10:112
2455226816,32sleep00-21swapper/023:49:430
2455226816,32sleep00-21swapper/023:49:430
2192626812,38sleep121984-21/usr/sbin/munin23:40:031
2192626812,38sleep121984-21/usr/sbin/munin23:40:031
726326720,26sleep10-21swapper/122:54:441
166526716,32sleep01824-21munin-node19:14:450
1403226713,38sleep214041-21df_inode23:15:172
1403226713,38sleep214041-21df_inode23:15:172
974926619,33sleep09784-21grep21:20:110
721726618,21sleep20-21swapper/219:30:052
3268926611,32sleep232118-21/usr/sbin/munin20:50:132
3132626624,19sleep10-21swapper/119:05:011
3132626624,19sleep10-21swapper/119:05:011
2586326610,44sleep0369-21snmpd23:52:130
2586326610,44sleep0369-21snmpd23:52:130
67882656,48sleep20-21swapper/200:34:512
609426520,19sleep20-21swapper/221:10:032
2981426522,19sleep20-21swapper/222:24:452
2981426522,19sleep20-21swapper/222:24:452
1870426513,37sleep018809-21fschecks_time23:30:050
1842226519,30sleep118458-21grep20:05:131
1516426520,26sleep115223-21munin-run21:39:421
601526420,21sleep10-21swapper/121:10:021
2415626416,26sleep20-21swapper/220:24:452
2011426416,27sleep10-21swapper/121:54:441
156426418,28sleep21768-21latency_hist00:19:452
1549926419,22sleep00-21swapper/019:55:150
147026420,20sleep10-21swapper/100:19:441
1144126411,37sleep211558-21timerandwakeup21:25:132
597526313,34sleep26000-21ntp_states19:25:132
2979726320,19sleep10-21swapper/120:40:181
2782426316,29sleep027852-21munin-run22:19:420
200826316,23sleep00-21swapper/020:55:150
197226312,32sleep11931-21apt-get00:20:201
947226217,29sleep19535-21if_eth021:20:041
321392626,45sleep20-21swapper/200:12:592
273602628,13sleep10-21swapper/123:56:381
92712615,13sleep20-21swapper/219:35:142
2467726117,21sleep20-21swapper/220:25:092
1070226114,27sleep010783-21run-parts19:40:130
63422607,41sleep00-21swapper/000:32:190
1523426021,24sleep215263-21munin-run21:39:432
1119026021,23sleep10-21swapper/123:05:141
1119026021,23sleep10-21swapper/123:05:141
919725917,27sleep19270-21/usr/sbin/munin19:35:131
604125912,29sleep16120-21/usr/sbin/munin19:25:161
454325912,24sleep00-21swapper/022:45:180
3086725914,22sleep00-21swapper/022:26:130
3077625922,21sleep20-21swapper/200:09:442
1615925923,20sleep10-21swapper/119:59:441
1502125917,25sleep215079-21if_eth019:55:052
1367325915,18sleep00-21swapper/021:34:440
1186325916,21sleep00-21swapper/023:09:410
1186325916,21sleep00-21swapper/023:09:410
257352585,44sleep20-21swapper/223:50:342
257352585,44sleep20-21swapper/223:50:342
2356225813,16sleep20-21swapper/222:05:032
220062589,32sleep10-21swapper/122:00:011
220062589,32sleep10-21swapper/122:00:011
2112025815,22sleep10-21swapper/120:15:021
1334025815,23sleep20-21swapper/219:50:002
1334025815,23sleep20-21swapper/219:50:002
120125811,33sleep01281-21fw_packets22:35:210
120125811,33sleep01281-21fw_packets22:35:210
3238225716,19sleep00-21swapper/019:09:440
3238225716,19sleep00-21swapper/019:09:440
2925325714,21sleep00-21swapper/020:40:050
1836425712,23sleep00-21swapper/023:29:440
1180025714,27sleep211819-21df_inode19:44:592
933125616,18sleep10-21swapper/123:00:051
3099425614,26sleep031013-21apt-get00:09:540
2612925611,30sleep226193-21cat20:30:062
2448625614,22sleep10-21swapper/120:25:041
2347525614,25sleep123469-21apt-get22:04:581
2152525612,29sleep021602-21basename20:15:140
2035425611,25sleep020407-21/usr/sbin/munin23:35:050
369925512,25sleep13216-21apt-get00:24:531
36122558,17sleep20-21swapper/200:24:462
257725517,17sleep10-21swapper/120:59:451
2440025512,26sleep024393-21apt-get20:24:590
1970425514,17sleep20-21swapper/220:10:032
1391225513,24sleep013907-21apt-get21:34:560
1191525517,24sleep212005-21sort23:09:432
1191525517,24sleep212005-21sort23:09:432
1059025514,22sleep20-21swapper/221:24:452
1020125515,22sleep210246-21/usr/sbin/munin19:39:592
727825414,24sleep07284-21ntp_states00:35:140
2387825414,20sleep00-21swapper/022:05:090
2237325418,18sleep20-21swapper/220:19:442
1549625415,24sleep00-21swapper/023:19:550
3114325321,18sleep20-21swapper/219:04:582
3114325321,18sleep20-21swapper/219:04:582
1328725313,24sleep013279-21apt-get19:49:580
1328725313,24sleep013279-21apt-get19:49:580
76962525,11sleep10-21swapper/121:14:551
619725214,22sleep26220-21irqstats22:50:052
3099525219,19sleep20-21swapper/220:45:082
3099525219,19sleep20-21swapper/220:45:082
2548125216,16sleep10-21swapper/122:10:101
1706125220,17sleep20-21swapper/223:24:582
397525113,18sleep00-21swapper/019:19:540
2458025113,15sleep10-21swapper/123:49:441
2458025113,15sleep10-21swapper/123:49:441
2179925113,19sleep20-21swapper/223:40:002
2179925113,19sleep20-21swapper/223:40:002
1835625115,18sleep20-21swapper/220:05:102
466025011,22sleep14741-21memory21:05:101
466025011,22sleep14741-21memory21:05:101
2014525013,24sleep120139-21apt-get23:34:581
1726725013,21sleep217263-21apt-get21:44:562
1726725013,21sleep217263-21apt-get21:44:562
1217325014,18sleep00-21swapper/019:45:090
518024913,18sleep20-21swapper/221:05:192
518024913,18sleep20-21swapper/221:05:192
2798324913,19sleep228133-21gpgv22:19:442
2238024912,21sleep222386-21ntp_states22:00:102
2238024912,21sleep222386-21ntp_states22:00:102
1969024913,20sleep00-21swapper/021:50:150
192042499,24sleep219282-21find23:30:152
311402477,14sleep10-21swapper/100:10:031
1063324712,20sleep110656-21ntp_states19:40:101
32802468,23sleep23286-21ntp_states21:00:142
2968824614,16sleep10-21swapper/100:05:061
2727024514,14sleep10-21swapper/120:34:451
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional