You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-12 - 17:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Sat Jul 12, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
240232797115,7937sleep15038-21kworker/u8:110:19:281
240232797115,7937sleep15038-21kworker/u8:110:19:281
2178529259,902sleep110913-1kworker/1:1H11:54:461
2178529259,902sleep110913-1kworker/1:1H11:54:461
539326987,656sleep35429-21ntpq09:19:473
12514236528,298sleep312541-21latency_hist09:43:573
16326235120,316sleep116358-21/usr/sbin/munin09:54:251
4491230956,204sleep328005-1kworker/3:2H09:18:583
4491230956,204sleep328005-1kworker/3:2H09:18:583
27896230217,269sleep30-21swapper/307:08:563
27896230217,269sleep30-21swapper/307:08:563
3740229936,193sleep32794-21/usr/sbin/munin10:54:523
2105228012,182sleep32166-21sendmail_mailqu09:09:293
29439227710,254sleep017776-21kworker/0:212:18:270
29439227710,254sleep017776-21kworker/0:212:18:270
2284322766,233sleep3370-21snmpd06:54:023
16196225127,211sleep316227-21sed08:13:573
16196225127,211sleep316227-21sed08:13:573
15000225044,102sleep30-21swapper/309:49:303
547722458,225sleep0370-21snmpd11:01:550
29890224549,178sleep329954-21awk08:54:313
12381223836,185sleep312468-21sort07:59:363
28809221949,114sleep328892-21xargs12:14:263
28809221949,114sleep328892-21xargs12:14:263
30317221871,54sleep30-21swapper/312:19:313
30317221871,54sleep30-21swapper/312:19:313
1201322180,10sleep3165-21hwrng11:23:583
31211221723,102sleep30-21swapper/308:59:193
341222107,149sleep33468-21apt-key07:33:593
8187220562,59sleep30-21swapper/307:48:583
25258220418,89sleep30-21swapper/312:04:293
6955220322,169sleep26975-21grep07:44:162
4337220011,109sleep30-21swapper/310:59:083
12545218716,95sleep30-21swapper/311:24:233
12262218517,152sleep212354-21tune2fs11:24:182
8156215717,126sleep18165-21df_inode09:29:131
8156215717,126sleep18165-21df_inode09:29:131
22302215724,97sleep3370-21snmpd08:33:033
30111215114,122sleep230220-21unixbench_multi08:54:342
578214936,91sleep30-21swapper/307:24:183
578214936,91sleep30-21swapper/307:24:183
25085214711,119sleep322432-21cyclictest06:59:193
27976214342,80sleep328005-1kworker/3:2H12:13:583
28255214137,57sleep30-21swapper/308:50:033
23436214010,80sleep323623-21apt-config10:18:583
21451213740,44sleep30-21swapper/310:09:333
21451213740,44sleep30-21swapper/310:09:333
8402213410,17sleep30-21swapper/309:29:203
8402213410,17sleep30-21swapper/309:29:203
3547213214,102sleep122322-21kworker/u8:007:34:011
20187213122,94sleep30-21swapper/308:24:313
16125212667,19sleep30-21swapper/309:54:193
25337212521,87sleep30-21swapper/310:23:593
25337212521,87sleep30-21swapper/310:23:593
2007921250,37sleep220083-21fschecks_count11:49:212
22376212457,38sleep30-21swapper/306:53:583
9128212313,91sleep00-21swapper/011:14:190
1934121237,67sleep319484-21date08:23:583
17484212335,76sleep317608-21timerwakeupswit11:39:293
21584212118,58sleep321221-21apt-get11:54:113
11453211931,72sleep311638-21apt-config07:58:593
31676211660,16sleep30-21swapper/310:43:573
16903211644,60sleep316912-21awk08:14:253
9904211413,89sleep39933-21kernelversion09:34:203
21326211425,75sleep121350-21kernelversion06:49:201
28270211211,56sleep31-21systemd10:33:563
28270211211,56sleep31-21systemd10:33:563
26674211252,15sleep30-21swapper/308:44:303
15070211213,83sleep115211-21cat11:33:581
9140211111,79sleep38409-21/usr/sbin/munin07:49:343
10445211111,87sleep210469-21proc_pri07:54:302
22823211022,72sleep122881-21fw_packets08:34:181
1584021095,91sleep00-21swapper/009:54:080
31403210821,62sleep00-21swapper/008:59:240
32289210711,80sleep132405-21cut09:03:581
13369210730,21sleep313464-21cat09:44:303
1296210744,16sleep30-21swapper/310:49:193
29830210614,76sleep129857-21fschecks_count07:14:191
29225210612,15sleep00-21swapper/010:34:380
22758210515,66sleep022819-21fschecks_time08:34:170
19863210527,18sleep30-21swapper/311:49:023
788210337,48sleep30-21swapper/310:48:583
29445210329,58sleep229449-21modprobe07:13:562
29445210329,58sleep229449-21modprobe07:13:562
23013210311,48sleep30-21swapper/311:58:563
23013210311,48sleep30-21swapper/311:58:563
24428210231,22sleep324438-21find08:39:083
2448210019,61sleep22649-21apt-config10:53:582
1379529935,15sleep30-21swapper/311:29:163
2870929828,50sleep30-21swapper/307:09:373
2870929828,50sleep30-21swapper/307:09:373
851429712,59sleep20-21swapper/207:49:212
784229715,67sleep17870-21latency_hist09:28:571
1175029623,51sleep111758-21apt-get07:59:091
3203929512,27sleep332182-21head07:19:343
1847929516,65sleep218571-21fschecks_time11:44:172
1312929514,62sleep013163-21/usr/sbin/munin09:44:230
725829325,48sleep30-21swapper/309:24:353
2340429321,14sleep30-21swapper/311:59:163
1870429216,53sleep00-21swapper/008:19:310
659829113,58sleep06623-21/usr/sbin/munin09:24:200
2522729122,34sleep20-21swapper/210:23:582
2522729122,34sleep20-21swapper/210:23:582
655129014,62sleep31-21systemd07:43:573
2300529019,55sleep223011-21modprobe11:58:552
2300529019,55sleep223011-21modprobe11:58:552
1036429016,51sleep110505-21grep09:34:301
2623728920,50sleep226319-21latency_hist07:03:582
182702898,37sleep30-21swapper/309:59:323
116392890,8sleep20-21swapper/209:39:232
398628824,13sleep30-21swapper/307:34:363
750028718,14sleep30-21swapper/311:09:133
3009828717,14sleep30-21swapper/310:38:573
2554028723,50sleep30-21swapper/310:24:153
2554028723,50sleep30-21swapper/310:24:153
2572028637,30sleep225779-21idleruntime-cro08:43:562
2392728620,49sleep30-21swapper/308:38:563
2337928624,39sleep20-21swapper/211:59:152
2166828619,53sleep021221-21apt-get11:54:120
162192865,37sleep30-21swapper/311:35:013
973828517,53sleep29879-21unixbench_multi11:14:332
404528514,57sleep04100-21cat07:34:370
2030828523,41sleep020444-21grep08:24:320
612428351,14sleep30-21swapper/311:04:213
2101028324,32sleep20-21swapper/210:09:282
2101028324,32sleep20-21swapper/210:09:282
1562728318,43sleep115767-21unixbench_singl08:09:311
1562728318,43sleep115767-21unixbench_singl08:09:311
37502827,37sleep20-21swapper/207:34:152
2514828115,18sleep00-21swapper/010:23:560
2514828115,18sleep00-21swapper/010:23:560
2402528115,52sleep224027-21sshd06:57:282
2110928132,28sleep00-21swapper/006:49:150
1603528119,27sleep00-21swapper/011:34:340
443428027,29sleep10-21swapper/109:18:561
443428027,29sleep10-21swapper/109:18:561
3122828010,15sleep30-21swapper/307:19:103
663727916,49sleep26765-21unixbench_multi11:04:322
53742799,51sleep10-21swapper/107:39:141
3123627932,30sleep231230-21apt-get07:19:132
3047127917,44sleep030528-21cut10:39:180
2649227923,24sleep00-21swapper/012:08:580
12542798,59sleep00-21swapper/009:07:520
54027820,43sleep2560-21open_inodes09:04:272
3273927811,19sleep3334-21awk09:04:203
92012777,14sleep30-21swapper/311:14:223
2340227718,37sleep223433-21cat10:18:572
189722778,15sleep30-21swapper/310:04:083
1775127722,30sleep00-21swapper/008:18:590
1128127723,37sleep211411-21unixbench_multi11:19:302
959227634,34sleep03548-1kworker/0:1H09:34:010
959227634,34sleep03548-1kworker/0:1H09:34:010
2472127619,39sleep024867-21sed12:03:570
2180927614,13sleep20-21swapper/206:49:292
966027515,44sleep19737-21grep11:14:321
813727522,25sleep10-21swapper/107:48:551
293727524,31sleep00-21swapper/009:13:580
1771827513,46sleep117721-21/usr/sbin/munin11:39:321
1527327512,50sleep20-21swapper/211:34:162
797527422,31sleep08035-21find11:09:260
1364027415,45sleep213327-21/usr/sbin/munin08:04:232
1364027415,45sleep213327-21/usr/sbin/munin08:04:232
3072127313,44sleep130767-21/usr/sbin/munin10:39:231
2677227318,28sleep10-21swapper/110:28:571
2677227318,28sleep10-21swapper/110:28:571
2604127323,31sleep00-21swapper/008:44:140
2355527315,44sleep123270-21/usr/sbin/munin11:59:181
2108927321,36sleep121106-21apt08:29:091
1492227311,45sleep214999-21grep09:49:292
1212127315,47sleep00-21swapper/009:39:460
725927212,44sleep00-21swapper/011:08:580
604027213,43sleep16041-21diskstats11:04:191
318527217,24sleep00-21swapper/009:14:180
318527217,24sleep00-21swapper/009:14:180
2741327223,28sleep10-21swapper/112:09:321
2637827215,41sleep226426-21unin-run12:08:562
2236927225,24sleep10-21swapper/110:14:201
1909627218,32sleep10-21swapper/110:04:181
1551427210,46sleep015626-21cat08:09:300
1551427210,46sleep015626-21cat08:09:300
111427220,27sleep00-21swapper/010:49:150
778127110,44sleep27865-21/usr/sbin/munin11:09:202
2490327124,27sleep024969-21fschecks_time06:59:160
1439227120,38sleep114400-21df09:49:161
1381927121,35sleep30-21swapper/308:04:283
1381927121,35sleep30-21swapper/308:04:283
1302927122,29sleep20-21swapper/209:44:212
3038727013,42sleep030493-21timerandwakeup12:19:320
3038727013,42sleep030493-21timerandwakeup12:19:320
2070927014,42sleep120710-21sshd08:27:281
171027016,33sleep21775-21fschecks_time09:09:162
2891326919,25sleep20-21swapper/210:34:322
269782698,12sleep00-21swapper/010:29:000
269782698,12sleep00-21swapper/010:29:000
1965526919,34sleep0227-21systemd-journal11:48:550
68226815,21sleep10-21swapper/107:24:201
68226815,21sleep10-21swapper/107:24:201
3065426825,29sleep030684-21modprobe08:58:560
2499326811,37sleep024995-21ntp_states08:40:230
1576126814,38sleep115846-21cut11:34:311
868826719,28sleep20-21swapper/209:29:272
868826719,28sleep20-21swapper/209:29:272
53852674,15sleep10-21swapper/111:00:511
2728826713,36sleep227381-21memory10:29:232
2728826713,36sleep227381-21memory10:29:232
1478626713,37sleep0227-21systemd-journal11:31:270
1173326712,40sleep011748-21acpi07:59:080
233672669,42sleep023392-21/usr/sbin/munin06:54:390
1966726623,22sleep10-21swapper/111:48:561
1395026614,33sleep10-21swapper/108:04:301
1395026614,33sleep10-21swapper/108:04:301
138682665,41sleep20-21swapper/211:29:192
1323526618,25sleep10-21swapper/109:44:281
1127226621,22sleep10-21swapper/107:58:571
622126515,23sleep10-21swapper/109:23:561
537926516,30sleep25455-21fschecks_time07:39:152
5092655,48sleep10-21swapper/110:45:571
2728626515,27sleep00-21swapper/012:09:310
192062657,14sleep20-21swapper/210:04:192
1815426521,25sleep00-21swapper/011:43:590
787226419,19sleep20-21swapper/209:28:582
49726421,20sleep20-21swapper/207:24:172
49726421,20sleep20-21swapper/207:24:172
2007526417,25sleep10-21swapper/111:49:211
1738726415,33sleep017407-21apt-get09:59:080
3085826313,42sleep014680-1kworker/0:0H07:17:090
2989126319,22sleep10-21swapper/112:19:211
2989126319,22sleep10-21swapper/112:19:211
264292635,12sleep10-21swapper/108:44:231
2486826317,28sleep10-21swapper/112:03:581
1465626315,31sleep00-21swapper/008:08:570
1465626315,31sleep00-21swapper/008:08:570
1172526315,29sleep211247-21apt-get07:59:022
511726213,26sleep20-21swapper/209:19:412
1931826210,25sleep20-21swapper/208:23:572
1478526222,24sleep00-21swapper/009:49:270
1086726217,29sleep110885-21uname11:19:211
1034726212,21sleep00-21swapper/007:54:240
329426118,24sleep20-21swapper/207:33:582
270402619,12sleep20-21swapper/207:04:432
270402619,12sleep20-21swapper/207:04:432
1705026120,21sleep20-21swapper/208:14:292
321852600,9sleep00-21swapper/007:19:340
31412605,45sleep20-21swapper/209:14:002
31412605,45sleep20-21swapper/209:14:002
2606526014,22sleep00-21swapper/010:24:280
2606526014,22sleep00-21swapper/010:24:280
2288926015,25sleep10-21swapper/106:54:161
2087526015,23sleep20-21swapper/208:28:582
1185526019,25sleep011894-21munin-plugin-st11:23:560
2529325915,24sleep10-21swapper/106:59:251
1813825917,20sleep10-21swapper/108:19:201
737225820,20sleep00-21swapper/007:44:260
315662587,11sleep20-21swapper/210:43:242
2953225815,24sleep10-21swapper/107:13:591
2953225815,24sleep10-21swapper/107:13:591
2916825811,32sleep029322-21wc08:53:590
234925819,17sleep10-21swapper/110:53:571
223825815,28sleep12349-21sort09:09:311
2122625816,20sleep10-21swapper/110:09:311
2122625816,20sleep10-21swapper/110:09:311
1089325815,28sleep210931-21modprobe09:38:552
410125714,24sleep14143-21memory07:34:381
307252575,12sleep20-21swapper/212:20:072
307252575,12sleep20-21swapper/212:20:072
292712574,11sleep00-21swapper/007:11:390
292712574,11sleep00-21swapper/007:11:390
2781925713,26sleep20-21swapper/208:49:092
2445325711,20sleep10-21swapper/108:39:111
2248425716,20sleep20-21swapper/208:33:592
2221925718,21sleep00-21swapper/011:55:010
2221925718,21sleep00-21swapper/011:55:010
1927125719,19sleep00-21swapper/010:04:210
266122569,18sleep10-21swapper/107:04:091
266122569,18sleep10-21swapper/107:04:091
2659225615,17sleep20-21swapper/208:44:292
2237525616,24sleep022424-21systemd-run08:33:570
2113325615,17sleep00-21swapper/010:09:300
2113325615,17sleep00-21swapper/010:09:300
1878725613,28sleep218931-21latency_hist10:03:592
138625619,18sleep10-21swapper/109:08:581
131362567,10sleep00-21swapper/011:25:090
2876825510,22sleep10-21swapper/110:34:061
2872025518,16sleep20-21swapper/212:14:252
2872025518,16sleep20-21swapper/212:14:252
2418325518,21sleep224338-21apt-config08:38:592
1725725510,29sleep217268-21ntp_states11:39:242
1348125516,20sleep10-21swapper/111:28:581
42362546,13sleep20-21swapper/210:58:592
40425415,24sleep0399-21apt-get07:24:110
40425415,24sleep0399-21apt-get07:24:110
2666625415,24sleep226665-21apt-get12:09:122
2546925414,22sleep225539-21df10:24:142
2546925414,22sleep225539-21df10:24:142
963925315,22sleep09633-21apt-get09:34:130
345125313,23sleep03554-21/usr/sbin/munin10:54:490
3137125315,18sleep20-21swapper/207:19:172
2868225316,20sleep10-21swapper/112:14:241
2868225316,20sleep10-21swapper/112:14:241
1644325312,26sleep116438-21apt-get08:14:111
77642529,21sleep10-21swapper/111:09:191
3225625214,21sleep032295-21memory10:44:230
3002825214,23sleep10-21swapper/108:54:331
278425213,23sleep22301-21apt-get10:54:062
843225111,22sleep18429-21apt-get07:49:161
582225110,15sleep00-21swapper/009:19:560
529225113,22sleep05286-21apt-get07:39:090
3233725112,21sleep232379-21ntp_states10:44:252
2780125112,23sleep127316-21apt-get08:49:081
2651025111,22sleep026106-21apt-get07:04:000
2651025111,22sleep026106-21apt-get07:04:000
1162725112,22sleep111638-21ntp_states09:39:231
880425015,19sleep00-21swapper/007:49:270
284125014,19sleep10-21swapper/110:54:291
149662498,18sleep20-21swapper/208:09:162
149662498,18sleep20-21swapper/208:09:162
2844024812,13sleep10-21swapper/110:33:581
2844024812,13sleep10-21swapper/110:33:581
193224812,22sleep01951-21apt-get07:29:080
1778924814,15sleep10-21swapper/109:59:221
2251924610,20sleep022556-21ntp_states10:14:230
1635924312,13sleep20-21swapper/209:54:262
317302428,17sleep10-21swapper/107:19:281
311712376,14sleep20-21swapper/208:59:182
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional