You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-06 - 06:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Sat Dec 06, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
377521412813,14093sleep33111-21kworker/u8:120:48:313
23985286656,8647sleep120873-21kworker/1:023:35:121
2785628246,807sleep221461-1kworker/2:1H20:23:382
8350247460,357sleep18371-21idleruntime-cro19:22:511
9992237415,342sleep110079-21apt-config22:52:521
30563233618,303sleep230599-21ntpq22:13:192
9878230752,67sleep10-21swapper/121:08:081
7340230411,190sleep17371-21munin-plugin-st00:27:501
2845227212,192sleep12904-21sendmail_mailqu22:28:211
285762263116,21sleep10-21swapper/122:08:091
285762263116,21sleep10-21swapper/122:08:091
10487225142,16sleep10-21swapper/122:53:291
13418224313,216sleep313464-21ntpq21:18:203
200692231127,55sleep10-21swapper/119:58:131
27346222924,189sleep127414-21latency_hist23:47:521
25164222911,149sleep125294-21unixbench_multi23:38:231
77152226118,22sleep17742-21fschecks_count00:28:111
12773222512,138sleep112867-21latency_hist21:17:531
12773222512,138sleep112867-21latency_hist21:17:531
27366222310,79sleep127493-21apt-config20:22:531
25643221731,129sleep125693-21idleruntime21:58:121
1476221720,181sleep01590-21fschecks_time00:08:110
16390221452,56sleep10-21swapper/123:12:511
16390221452,56sleep10-21swapper/123:12:511
14850220830,158sleep114537-21/usr/sbin/munin21:23:151
4627220416,124sleep14757-21unixbench_multi22:33:261
13346220416,172sleep113488-21latency_hist19:37:541
6187220131,129sleep16282-21/usr/sbin/munin22:38:221
19812219927,123sleep10-21swapper/121:38:211
19812219927,123sleep10-21swapper/121:38:211
1731219212,166sleep01748-21apt19:03:030
24156218414,115sleep10-21swapper/120:12:511
24156218414,115sleep10-21swapper/120:12:511
21066218018,138sleep221091-21munin-plugin-st23:27:502
4965216915,138sleep04964-21systemd-run20:52:520
16384216810,142sleep116454-21cat21:28:151
22750216513,135sleep022637-21latency_hist23:32:530
8622216430,115sleep10-21swapper/119:23:031
29819216312,22sleep10-21swapper/120:28:181
2143021638,16sleep10-21swapper/123:28:091
18973216234,105sleep10-21swapper/119:53:241
22799216144,27sleep10-21swapper/121:48:181
2217215916,118sleep20-21swapper/219:03:212
28478215755,15sleep10-21swapper/120:24:021
1178215785,52sleep10-21swapper/122:23:161
3682215578,54sleep10-21swapper/120:48:091
4613214636,95sleep14671-21fschecks_count00:18:131
846214551,64sleep21011-21grep18:58:212
5571214582,43sleep10-21swapper/119:13:141
20549214525,97sleep020635-21grep21:42:520
20549214525,97sleep020635-21grep21:42:520
21344214039,43sleep10-21swapper/120:03:101
17157213818,26sleep10-21swapper/119:48:201
8023213611,83sleep18066-21awk21:02:511
8023213611,83sleep18066-21awk21:02:511
413921345,17sleep10-21swapper/100:16:401
23890213410,14sleep10-21swapper/121:53:031
4993213214,103sleep25040-21processes00:18:222
20291213211,36sleep120356-21sendmail_mailst23:23:211
21710213011,12sleep20-21swapper/220:03:192
15081213027,47sleep115100-21apt19:43:031
178021267,32sleep10-21swapper/100:08:161
8891212423,82sleep08917-21munin-run00:32:500
1605212412,98sleep31700-21cut20:42:543
3115321218,21sleep10-21swapper/120:33:021
1567821217,98sleep00-21swapper/023:08:300
1567821217,98sleep00-21swapper/023:08:300
13102212153,53sleep10-21swapper/121:18:141
11435212112,19sleep30-21swapper/321:13:033
11435212112,19sleep30-21swapper/321:13:033
32085212011,21sleep10-21swapper/122:18:191
29457211814,85sleep10-21swapper/123:53:181
878621178,31sleep18820-21ps21:03:201
2739321168,32sleep10-21swapper/122:03:171
1925321165,53sleep11529-21cyclictest23:19:291
122421145,56sleep10-21swapper/120:38:481
470821137,52sleep10-21swapper/119:09:261
1866821135,56sleep10-21swapper/121:34:171
1042121128,82sleep10-21swapper/119:28:041
27774210919,24sleep20-21swapper/223:48:142
10425210817,72sleep29920-21apt-get22:53:012
5292210523,57sleep35332-21sort20:53:123
25827210512,77sleep325934-21sed20:17:543
25827210512,77sleep325934-21sed20:17:543
1326421045,50sleep10-21swapper/123:02:371
2735210314,74sleep22870-21cut00:12:532
1671121037,37sleep10-21swapper/123:13:071
12147210237,43sleep212179-21fschecks_time22:58:182
28192210016,60sleep027562-21/usr/sbin/munin23:48:240
914029915,69sleep29216-21awk19:23:212
455029911,71sleep04624-21find22:33:250
305829953,12sleep10-21swapper/119:07:531
2729529920,61sleep327301-21modprobe23:47:503
1058529912,72sleep010586-21/usr/sbin/munin19:29:040
2261729814,67sleep222693-21systemd-run23:32:512
3195029613,65sleep231984-21irqstats22:18:162
3071729614,66sleep330786-21switchtime22:13:243
1611529519,23sleep20-21swapper/221:28:082
1169029521,56sleep311706-21sh19:32:513
301302946,77sleep10-21swapper/122:12:581
2440129317,58sleep224475-21mailstats21:53:202
2749929227,31sleep20-21swapper/220:22:542
191212928,70sleep219135-21vmstat19:53:262
279292919,27sleep10-21swapper/123:48:171
2455429112,63sleep024680-21sed21:53:220
2452429113,60sleep224450-21/usr/sbin/munin20:13:082
2452429113,60sleep224450-21/usr/sbin/munin20:13:082
241729115,60sleep32510-21cat20:43:223
241729115,60sleep32510-21cat20:43:223
156829121,32sleep30-21swapper/319:02:543
1010229119,47sleep210068-21gpgv22:52:532
1641729019,56sleep316557-21sed19:47:533
318092898,29sleep10-21swapper/100:00:371
29822895,30sleep10-21swapper/120:46:191
29822895,30sleep10-21swapper/120:46:191
1957628924,49sleep319725-21cut19:57:553
9802885,28sleep10-21swapper/100:06:301
9802885,28sleep10-21swapper/100:06:301
963628813,53sleep29685-21latency_hist21:07:532
61502885,28sleep10-21swapper/120:53:501
2983328817,32sleep30-21swapper/320:28:203
398928711,55sleep20-21swapper/219:08:142
265328723,47sleep32669-21modprobe00:12:503
2361928723,42sleep023753-21unixbench_singl20:08:260
2361928723,42sleep023753-21unixbench_singl20:08:260
218002876,27sleep10-21swapper/121:44:431
218002876,27sleep10-21swapper/121:44:431
136562877,27sleep10-21swapper/123:03:101
83002865,29sleep10-21swapper/122:47:111
486228613,55sleep20-21swapper/222:33:312
274702857,16sleep30-21swapper/322:03:193
209932855,13sleep30-21swapper/321:43:123
209932855,13sleep30-21swapper/321:43:123
1486828522,32sleep00-21swapper/019:42:530
1194628522,46sleep012030-21grep21:13:200
1194628522,46sleep012030-21grep21:13:200
955528412,57sleep3369-21sshd19:23:303
145482845,27sleep10-21swapper/119:39:271
1122328427,25sleep20-21swapper/221:12:542
245682839,54sleep10-21swapper/120:13:111
245682839,54sleep10-21swapper/120:13:111
1349428211,18sleep30-21swapper/323:02:563
98127910,54sleep2983-21sshd00:06:312
98127910,54sleep2983-21sshd00:06:312
310127919,32sleep03133-21munin-run20:47:520
310127919,32sleep03133-21munin-run20:47:520
97922786,11sleep00-21swapper/019:26:300
423527818,40sleep34269-21modprobe00:17:513
80927717,17sleep00-21swapper/022:23:070
3062627720,36sleep230655-21awk20:32:512
1249127720,29sleep30-21swapper/322:58:283
2078727618,26sleep20-21swapper/221:43:062
2078727618,26sleep20-21swapper/221:43:062
3109427515,15sleep30-21swapper/323:58:163
3048627522,30sleep30-21swapper/323:57:523
202342750,8sleep00-21swapper/019:58:150
2770027419,28sleep00-21swapper/022:03:240
1385627410,47sleep213923-21memory19:38:152
628327314,18sleep20-21swapper/222:38:232
2678527325,26sleep20-21swapper/222:02:532
104692737,48sleep20-21swapper/219:28:382
316227218,24sleep00-21swapper/000:13:110
309227220,32sleep03235-21head22:28:240
1791327221,27sleep00-21swapper/019:52:510
78372717,16sleep30-21swapper/300:28:133
2426527118,27sleep30-21swapper/320:12:533
2426527118,27sleep30-21swapper/320:12:533
1400127113,46sleep00-21swapper/021:19:260
73682700,7sleep20-21swapper/219:18:592
69592700,10sleep029602-1kworker/0:2H22:42:530
302972709,11sleep00-21swapper/023:56:070
2834227021,27sleep30-21swapper/320:24:003
197782707,38sleep20-21swapper/219:58:022
1549327015,22sleep20-21swapper/223:08:242
1549327015,22sleep20-21swapper/223:08:242
1265227016,27sleep00-21swapper/019:33:240
753826915,40sleep07545-21ntp_states22:43:180
2579626915,37sleep225913-21latency_hist23:42:532
1402226917,28sleep30-21swapper/319:38:203
43526715,37sleep0550-21cat00:03:220
43526715,37sleep0550-21cat00:03:220
2204026720,24sleep30-21swapper/323:28:243
1435126716,36sleep214482-21sshd23:03:312
2447826614,36sleep324552-21sed21:53:213
808626519,25sleep30-21swapper/321:02:533
808626519,25sleep30-21swapper/321:02:533
668426519,32sleep36770-21ls00:23:243
2984326515,27sleep30-21swapper/322:12:523
2984326515,27sleep30-21swapper/322:12:523
2884326520,28sleep328911-40mandb23:52:513
1893826514,33sleep319068-21unixbench_multi23:18:263
656126420,29sleep26587-21munin-run19:17:512
643926420,23sleep00-21swapper/000:23:170
578226413,35sleep25849-21switchtime20:53:232
3137126413,19sleep30-21swapper/320:33:173
270602646,48sleep00-21swapper/023:44:530
2367726415,19sleep30-21swapper/321:52:533
1691326417,27sleep20-21swapper/219:48:132
1473726417,29sleep214771-21/usr/sbin/munin21:23:112
64426316,34sleep2659-21netstat20:38:122
601926322,24sleep30-21swapper/322:38:203
3024926315,22sleep00-21swapper/022:13:130
297226318,27sleep22965-21apt-get00:13:052
2111826314,33sleep00-21swapper/023:27:520
1519226318,22sleep20-21swapper/219:43:122
479626217,27sleep04837-21ls00:18:160
3118426220,24sleep20-21swapper/220:33:082
191282625,11sleep30-21swapper/321:37:543
191282625,11sleep30-21swapper/321:37:543
1778826215,26sleep30-21swapper/323:15:323
1628026214,22sleep30-21swapper/321:28:133
987526120,25sleep39871-21apt-get21:08:063
676526114,30sleep26759-21apt-get20:58:082
676526114,30sleep26759-21apt-get20:58:082
66626119,23sleep00-21swapper/018:58:190
284102616,12sleep00-21swapper/020:24:010
2742626119,22sleep20-21swapper/222:03:182
2679826117,18sleep30-21swapper/322:02:543
1900126116,29sleep219126-21grep21:37:532
1831726117,19sleep00-21swapper/019:53:110
1393026114,20sleep20-21swapper/221:18:312
898826018,19sleep20-21swapper/200:32:522
411526016,23sleep00-21swapper/019:08:160
2608026018,22sleep00-21swapper/021:58:230
2474526016,25sleep024839-21memory20:13:150
2474526016,25sleep024839-21memory20:13:150
2298026022,19sleep00-21swapper/021:48:220
1647026015,18sleep30-21swapper/323:12:533
1647026015,18sleep30-21swapper/323:12:533
141426016,21sleep30-21swapper/322:23:223
1326926016,30sleep313313-21systemd-run19:37:523
1221326015,19sleep00-21swapper/022:58:200
7532599,33sleep1843-21find18:58:201
578525923,15sleep30-21swapper/319:13:203
38952595,12sleep30-21swapper/322:32:553
3112425918,16sleep00-21swapper/023:58:170
263125912,32sleep22341-21/usr/sbin/munin22:28:142
2348525913,22sleep30-21swapper/323:33:213
2182725912,30sleep021873-21processes23:28:210
911025813,29sleep39170-21processes22:48:223
74212588,13sleep30-21swapper/319:19:013
406125816,19sleep20-21swapper/220:48:382
3235925818,22sleep30-21swapper/300:03:093
3235925818,22sleep30-21swapper/300:03:093
3077225814,23sleep00-21swapper/020:32:530
2238925815,18sleep20-21swapper/221:48:072
216992585,12sleep00-21swapper/021:43:250
216992585,12sleep00-21swapper/021:43:250
184125815,27sleep21838-21apt-get20:43:052
184125815,27sleep21838-21apt-get20:43:052
1803925819,19sleep20-21swapper/223:17:532
1531025819,17sleep00-21swapper/019:43:140
1219925814,27sleep20-21swapper/219:33:142
843625718,18sleep00-21swapper/019:22:540
62462575,15sleep00-21swapper/020:54:540
504125717,24sleep35127-21mailstats00:18:233
2905525710,27sleep229112-21sendmail_mailqu22:08:232
2905525710,27sleep229112-21sendmail_mailqu22:08:232
1980825711,22sleep30-21swapper/323:23:073
1768425715,26sleep317680-21apt-get21:33:023
1665325716,27sleep316648-21apt-get19:48:053
122525714,28sleep21261-21ntp_states22:23:172
1191425713,20sleep20-21swapper/221:13:182
1191425713,20sleep20-21swapper/221:13:182
958525616,24sleep09608-21munin-run21:07:510
72992565,12sleep30-21swapper/322:43:133
2997325616,22sleep230073-21users23:53:272
2456625614,18sleep00-21swapper/023:38:100
1061625615,17sleep30-21swapper/322:53:393
669225511,17sleep00-21swapper/019:17:530
498425519,17sleep30-21swapper/319:12:513
2603425515,25sleep226026-21apt-get23:43:052
990525420,14sleep00-21swapper/022:52:510
755325412,29sleep27590-21ps22:43:202
3175025420,16sleep00-21swapper/020:33:260
2423125413,20sleep20-21swapper/220:12:522
2423125413,20sleep20-21swapper/220:12:522
1867825414,25sleep018714-21ntp_states23:18:190
1415525415,17sleep00-21swapper/023:03:290
1033025416,22sleep010336-21ntp_states21:08:190
3171525316,16sleep00-21swapper/022:18:100
3171325313,24sleep331708-21apt-get22:18:093
271382539,32sleep00-21swapper/020:20:580
1483025318,18sleep00-21swapper/021:23:140
2498925217,15sleep30-21swapper/323:38:213
234025211,22sleep30-21swapper/322:28:023
2182925216,18sleep30-21swapper/320:03:243
1793525214,24sleep017958-21irqstats21:33:110
2423825116,15sleep20-21swapper/223:37:522
615525014,22sleep26161-21diskstats00:23:112
2951125011,26sleep05666-21snmpd22:08:450
2951125011,26sleep05666-21snmpd22:08:450
212625019,16sleep30-21swapper/319:03:183
2031825012,22sleep320388-21sensors19:58:193
1951224915,17sleep20-21swapper/221:38:142
1951224915,17sleep20-21swapper/221:38:142
104162499,14sleep00-21swapper/022:52:550
3273024812,20sleep032725-21apt-get20:38:020
156552486,33sleep30-21swapper/321:25:433
224172475,12sleep00-21swapper/020:05:310
814324610,15sleep00-21swapper/021:02:540
1751524611,20sleep017613-21users23:13:270
165302466,14sleep00-21swapper/021:28:180
683524510,20sleep06844-21df_inode20:58:100
2602324511,19sleep30-21swapper/320:18:063
1862324213,18sleep218238-21/usr/sbin/munin23:18:172
307152417,15sleep20-21swapper/223:58:032
83242397,18sleep30-21swapper/321:03:053
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional