You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-11 - 12:41

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Sat Jul 11, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
206752506311,5036sleep2225-21systemd-journal23:23:572
32288234359,3412sleep031627-21kworker/u8:222:19:350
323242139888,1297sleep032325-21ntp_states23:59:240
7802211205,1106sleep228199-1kworker/2:2H22:44:242
7802211205,1106sleep228199-1kworker/2:2H22:44:242
19013211004,1083sleep125551-1kworker/1:0H19:54:501
19013211004,1083sleep125551-1kworker/1:0H19:54:501
1109526247,604sleep111134-21ntpq19:29:221
1902926196,527sleep125551-1kworker/1:0H19:54:541
1902926196,527sleep125551-1kworker/1:0H19:54:541
27182239455,122sleep10-21swapper/123:44:101
19700236514,333sleep019789-21/usr/sbin/munin23:19:270
909723549,329sleep19060-21latency_hist22:48:591
909723549,329sleep19060-21latency_hist22:48:591
48132334148,167sleep14964-21head19:09:281
490523289,305sleep14938-21awk20:53:571
264642318163,63sleep126484-21modprobe20:18:561
32331225710,97sleep132393-21processes23:59:251
6129225439,196sleep16217-21fschecks_time19:14:161
30510225213,225sleep330546-21irqstats23:54:183
3819224947,176sleep03860-21latency_hist19:08:580
7256224317,24sleep37298-21gpgv22:43:563
253842226100,44sleep125396-21modprobe21:58:571
3425221943,128sleep13498-21xargs22:29:291
3425221943,128sleep13498-21xargs22:29:291
32265221315,184sleep232312-21ntpq23:59:222
8903221116,174sleep28981-21ls21:04:272
12700220736,121sleep10-21swapper/122:59:131
16915220446,102sleep10-21swapper/121:29:301
31506219838,104sleep10-21swapper/120:34:151
31506219838,104sleep10-21swapper/120:34:151
5657219314,163sleep05718-21latency_hist00:18:580
5657219314,163sleep05718-21latency_hist00:18:580
2781421927,174sleep1364-21snmpd20:21:021
30043218614,87sleep130182-21latency_hist23:53:591
16415218556,20sleep10-21swapper/123:09:271
32289218211,146sleep10-21swapper/122:19:361
2011921778,48sleep10-21swapper/121:39:311
1071421728,151sleep110796-21mailstats21:10:241
6452217124,127sleep06483-21idleruntime-cro20:58:560
26125217012,139sleep10-21swapper/123:39:261
24961216717,55sleep10-21swapper/120:14:001
3027921628,139sleep130254-21/usr/sbin/munin23:54:121
12672216111,26sleep00-21swapper/022:59:080
30490215713,17sleep20-21swapper/220:29:272
30490215713,17sleep20-21swapper/220:29:272
7140215110,124sleep10-21swapper/120:59:221
7140215110,124sleep10-21swapper/120:59:221
115321499,95sleep10-21swapper/120:39:271
13723214796,25sleep13848-21cyclictest21:19:241
30431214529,24sleep10-21swapper/122:14:151
899221419,26sleep10-21swapper/100:29:121
27735213821,96sleep10-21swapper/122:04:261
8965213624,90sleep00-21swapper/000:29:070
1912621336,41sleep10-21swapper/121:38:261
684121315,43sleep10-21swapper/100:19:561
6206213017,91sleep10-21swapper/122:39:181
23889213012,102sleep023861-21latency_hist21:53:590
10388212621,87sleep10-21swapper/119:28:211
2178212519,19sleep20-21swapper/220:44:172
9604212213,18sleep10-21swapper/122:49:201
9604212213,18sleep10-21swapper/122:49:201
2907521219,97sleep129166-21load22:09:181
1235121197,101sleep1360-21ntpd22:58:321
1235121197,101sleep1360-21ntpd22:58:321
1942821189,98sleep119429-21cpu23:19:201
25699211728,57sleep10-21swapper/121:59:091
1508121178,89sleep10-21swapper/121:24:211
2258021159,92sleep122570-21apt-get21:49:121
4896211482,16sleep10-21swapper/100:14:351
4896211482,16sleep10-21swapper/100:14:351
1267211310,48sleep10-21swapper/122:24:081
3053721129,14sleep00-21swapper/022:14:180
26374211224,67sleep026510-21grep23:39:290
30064211122,70sleep330078-21idleruntime-cro22:13:573
817621108,60sleep18310-21grep00:24:281
8120211022,16sleep18184-21mailstats19:19:291
2263921108,21sleep10-21swapper/120:04:271
16131211023,65sleep016271-21unixbench_singl19:44:310
11135211016,37sleep211113-21apt-get22:54:212
11135211016,37sleep211113-21apt-get22:54:212
18617210910,17sleep10-21swapper/123:15:571
2594210748,46sleep12682-21tune2fs00:09:141
25611210725,54sleep325630-21apt-get23:39:083
26385210613,75sleep026546-21awk21:59:280
6622210415,73sleep06691-21switchtime19:14:280
26768210411,79sleep326770-21sshd22:01:213
648421038,23sleep16531-21munin-run20:58:571
27938210340,30sleep228045-21unixbench_singl23:44:282
947321025,11sleep10-21swapper/121:07:291
31078210218,74sleep10-21swapper/120:33:141
31078210218,74sleep10-21swapper/120:33:141
29167210114,73sleep229179-21grep22:09:192
2450921019,13sleep10-21swapper/123:34:221
1384121019,14sleep10-21swapper/119:39:071
31946210020,56sleep331971-21fschecks_count23:59:143
22087210019,52sleep10-21swapper/123:27:111
1856221005,11sleep00-21swapper/023:15:150
386229943,42sleep14020-21sort19:08:591
210322997,15sleep10-21swapper/121:44:111
2190329715,65sleep222052-21latency_hist20:03:592
1766829713,69sleep217762-21latency_hist21:33:592
575629611,72sleep15819-21latency_hist22:38:581
575629611,72sleep15819-21latency_hist22:38:581
2251029636,17sleep10-21swapper/123:29:091
2886529512,66sleep228940-21fw_packets23:49:162
218622958,49sleep10-21swapper/120:03:581
21102956,32sleep10-21swapper/100:07:281
209229433,17sleep10-21swapper/120:44:121
106032949,46sleep10-21swapper/100:34:131
106032949,46sleep10-21swapper/100:34:131
1227029334,17sleep10-21swapper/119:34:111
283262929,67sleep10-21swapper/120:24:091
283262929,67sleep10-21swapper/120:24:091
943029113,58sleep29531-21fschecks_time22:49:172
943029113,58sleep29531-21fschecks_time22:49:172
80902909,50sleep30-21swapper/319:19:273
439328917,45sleep20-21swapper/200:14:152
439328917,45sleep20-21swapper/200:14:152
144872898,65sleep10-21swapper/123:04:181
1887828824,41sleep218831-21apt-get19:54:102
1887828824,41sleep218831-21apt-get19:54:102
1838228825,28sleep00-21swapper/021:34:250
771328713,57sleep07787-21cat00:24:190
2933428718,46sleep029409-21find23:49:260
1739428715,56sleep217499-21find23:14:002
1493528716,44sleep315073-21grep23:04:283
1016328715,52sleep010164-21cpu21:09:340
1452828626,40sleep014684-21grep19:39:270
3260128514,56sleep332641-21ntp_kernel_pll_22:19:423
777928430,16sleep00-21swapper/022:44:100
777928430,16sleep00-21swapper/022:44:100
1654028424,30sleep20-21swapper/223:09:282
129332845,44sleep10-21swapper/121:18:481
3134128310,58sleep031342-21sshd23:56:250
2904128313,52sleep029186-21/usr/sbin/munin20:24:280
2904128313,52sleep029186-21/usr/sbin/munin20:24:280
152542835,41sleep10-21swapper/119:44:001
1079828314,48sleep210884-21/usr/sbin/munin21:10:252
2677928126,29sleep30-21swapper/320:19:113
1997228121,36sleep30-21swapper/323:19:343
3216628020,31sleep20-21swapper/220:34:282
3216628020,31sleep20-21swapper/220:34:282
263322805,64sleep00-21swapper/020:17:140
227427916,42sleep32068-21/usr/sbin/munin20:44:183
1715027913,49sleep016669-21apt-get19:49:070
1715027913,49sleep016669-21apt-get19:49:070
110652790,5sleep00-21swapper/019:29:200
225812789,13sleep20-21swapper/221:49:132
1566927813,47sleep315762-21/usr/sbin/munin19:44:223
983627719,41sleep09962-21sort19:24:290
975527721,25sleep30-21swapper/319:24:283
653527621,40sleep36666-21latency_hist20:58:593
2490827618,29sleep20-21swapper/220:13:572
226572767,14sleep00-21swapper/023:29:160
1930427618,42sleep019387-21sed19:55:050
1930427618,42sleep019387-21sed19:55:050
1294427622,36sleep212960-21modprobe21:18:572
869827517,37sleep08302-21/usr/sbin/munin21:04:210
748227519,28sleep00-21swapper/020:59:290
748227519,28sleep00-21swapper/020:59:290
615527521,29sleep00-21swapper/000:19:180
3188227513,30sleep20-21swapper/222:19:002
2473527526,24sleep30-21swapper/321:54:273
783527426,23sleep30-21swapper/321:01:223
783527426,23sleep30-21swapper/321:01:223
457027418,25sleep20-21swapper/219:09:252
2259727418,30sleep30-21swapper/323:29:153
1710327435,21sleep317256-21sed21:29:323
1401627320,21sleep20-21swapper/223:03:582
1376127315,43sleep313762-21sshd23:01:213
1126127322,25sleep20-21swapper/219:29:272
682927213,34sleep26948-21/usr/sbin/munin19:14:302
309227213,31sleep00-21swapper/000:09:250
2256527210,48sleep322573-21awk20:04:253
2005827213,42sleep020118-21sed21:39:300
1224627222,31sleep312265-21apt-get19:34:093
762127121,37sleep27662-21idleruntime00:24:172
2990627124,32sleep029948-21df20:29:130
2990627124,32sleep029948-21df20:29:130
279427117,30sleep00-21swapper/022:29:080
279427117,30sleep00-21swapper/022:29:080
2477927120,28sleep20-21swapper/223:34:272
144927114,14sleep30-21swapper/322:24:143
1442727118,40sleep014485-21basename23:04:170
111392718,28sleep30-21swapper/322:54:223
111392718,28sleep30-21swapper/322:54:223
2648927012,33sleep20-21swapper/220:18:572
2540227020,26sleep20-21swapper/221:58:582
2453427026,23sleep30-21swapper/323:34:243
3162026916,37sleep3278-21dbus-daemon22:18:573
268726915,38sleep22755-21if_eth019:04:172
1836026918,24sleep20-21swapper/221:34:232
1133526915,25sleep30-21swapper/319:29:283
2234126822,25sleep00-21swapper/021:48:590
50502676,36sleep30-21swapper/319:10:063
3275326721,28sleep00-21swapper/020:38:580
3275326721,28sleep00-21swapper/020:38:580
3237326712,41sleep3364-21snmpd20:34:573
3237326712,41sleep3364-21snmpd20:34:573
3032726713,20sleep20-21swapper/222:14:082
231212676,50sleep20-21swapper/220:05:492
1437526712,37sleep214449-21/usr/sbin/munin19:39:252
818826617,28sleep20-21swapper/219:19:302
3046026624,25sleep20-21swapper/223:54:172
250952665,20sleep00-21swapper/021:55:120
2217626619,24sleep20-21swapper/223:28:212
841326520,24sleep30-21swapper/321:04:163
837226520,30sleep38413-21processes22:45:543
837226520,30sleep38413-21processes22:45:543
545026515,24sleep20-21swapper/220:54:182
318926521,28sleep30-21swapper/322:29:233
318926521,28sleep30-21swapper/322:29:233
2478526518,20sleep00-21swapper/020:12:210
1799226519,26sleep30-21swapper/321:34:163
495926416,32sleep05094-21latency_hist20:54:000
2725926422,23sleep00-21swapper/023:44:140
180326416,23sleep20-21swapper/220:43:562
163626411,39sleep21757-21sed00:04:302
1452326413,37sleep214601-21cat23:04:192
1296226419,27sleep313022-21unin-run21:18:583
470526311,35sleep04811-21ls19:09:270
2248126319,22sleep00-21swapper/020:04:220
2024326317,21sleep20-21swapper/221:39:322
1986626320,27sleep319916-21awk21:39:253
1062226314,24sleep30-21swapper/300:34:153
1062226314,24sleep30-21swapper/300:34:153
377326214,28sleep33811-21http00:13:563
2565126213,20sleep30-21swapper/320:14:223
230926218,18sleep30-21swapper/300:08:593
479026112,34sleep04793-21/usr/sbin/munin22:34:210
479026112,34sleep04793-21/usr/sbin/munin22:34:210
448926116,29sleep34528-21/usr/sbin/munin00:14:243
448926116,29sleep34528-21/usr/sbin/munin00:14:243
1805426117,17sleep30-21swapper/319:50:203
1805426117,17sleep30-21swapper/319:50:203
1227126119,26sleep212265-21apt-get19:34:122
1048626123,19sleep20-21swapper/219:28:592
704026014,26sleep20-21swapper/200:22:312
2987626018,22sleep30-21swapper/320:29:123
2987626018,22sleep30-21swapper/320:29:123
291326018,22sleep20-21swapper/222:29:182
291326018,22sleep20-21swapper/222:29:182
262826014,20sleep00-21swapper/020:44:270
2382026012,22sleep20-21swapper/223:33:582
2297526011,24sleep30-21swapper/321:49:223
1594926026,20sleep20-21swapper/219:44:292
109426010,30sleep00-21swapper/022:23:590
638125916,26sleep36463-21mailstats00:19:253
502925921,18sleep30-21swapper/322:34:283
502925921,18sleep30-21swapper/322:34:283
2894125915,27sleep328998-21irqstats23:49:173
2100725920,22sleep30-21swapper/319:59:243
1807025915,28sleep318074-21ntp_states23:14:423
969025820,22sleep39833-21sshd00:29:313
391225813,26sleep23940-21kernelversion20:49:182
2031125821,17sleep30-21swapper/319:59:003
2031125821,17sleep30-21swapper/319:59:003
1392525817,22sleep313929-21diskstats19:39:133
1325925815,26sleep313255-21apt-get21:19:103
1161525816,27sleep00-21swapper/022:54:440
1161525816,27sleep00-21swapper/022:54:440
730125716,25sleep07328-21munin-run22:43:570
67442577,10sleep20-21swapper/220:59:072
67442577,10sleep20-21swapper/220:59:072
54025712,25sleep3610-21df20:39:133
2880325716,25sleep028798-21apt-get22:09:090
2873725711,28sleep228783-21ntp_states20:24:222
2873725711,28sleep228783-21ntp_states20:24:222
2810625717,17sleep00-21swapper/020:24:000
1761725711,30sleep017645-21munin-run21:33:570
1315825712,26sleep00-21swapper/019:34:300
1016025715,25sleep310118-21apt-get21:09:333
331225615,21sleep20-21swapper/200:09:272
2623725614,25sleep226306-21sensors_temp21:59:262
212982566,12sleep30-21swapper/321:44:173
146325615,21sleep00-21swapper/000:04:280
1302425614,23sleep00-21swapper/021:18:590
899325517,19sleep20-21swapper/200:29:132
61225515,24sleep0618-21df_inode20:39:140
2856925513,25sleep228699-21cat22:08:592
938525417,20sleep30-21swapper/322:49:143
938525417,20sleep30-21swapper/322:49:143
642825412,17sleep20-21swapper/222:39:232
257725416,22sleep32586-21diskstats19:04:133
2447125413,25sleep024508-21ntp_states23:34:210
1904025416,20sleep219107-21df19:54:592
1904025416,20sleep219107-21df19:54:592
1610925418,24sleep216128-21latency_hist21:28:582
1503125411,24sleep015080-21irqstats21:24:200
1365825417,17sleep20-21swapper/221:19:212
568125317,17sleep30-21swapper/319:14:023
2723325314,23sleep027252-21apt-get22:04:090
267402536,36sleep20-21swapper/223:41:572
160252539,19sleep00-21swapper/023:09:170
103192537,34sleep00-21swapper/022:50:240
103192537,34sleep00-21swapper/022:50:240
2160725215,17sleep30-21swapper/323:24:273
1481325213,23sleep314880-21df21:24:153
1063625212,23sleep010744-21perl00:34:160
1063625212,23sleep010744-21perl00:34:160
785825113,22sleep37895-21ntp_states00:24:223
2756125113,19sleep30-21swapper/323:44:213
2717925120,15sleep20-21swapper/220:19:222
2110025114,21sleep221109-21df_inode21:44:122
2107425110,17sleep00-21swapper/023:24:150
2054025114,22sleep020543-21apt-get19:59:090
1053725112,24sleep210533-21apt-get00:34:102
1053725112,24sleep210533-21apt-get00:34:102
36362509,23sleep30-21swapper/320:49:103
2832825015,20sleep328323-21apt-get20:24:113
2832825015,20sleep328323-21apt-get20:24:113
2786625014,17sleep30-21swapper/322:04:283
1706025012,20sleep30-21swapper/323:13:573
1457325014,18sleep00-21swapper/021:24:000
442224910,23sleep24416-21apt-get22:34:102
442224910,23sleep24416-21apt-get22:34:102
2411324811,20sleep324119-21ntp_states20:09:293
126824513,16sleep20-21swapper/222:24:092
78142388,12sleep00-21swapper/019:19:200
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional