You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-14 - 19:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Mon Sep 14, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1241721123224,11164sleep010585-21kworker/0:212:08:070
259572356613,3538sleep021808-21kworker/0:211:08:240
166012133619,1305sleep316603-21kworker/3:008:56:593
12839266717,634sleep112874-21/usr/sbin/munin12:08:211
28875245611,432sleep228321-21kworker/u8:311:18:152
51162276105,52sleep30-21swapper/310:03:183
1015224431,188sleep31036-21modprobe11:32:583
1015224431,188sleep31036-21modprobe11:32:583
980022396,198sleep39875-21find08:33:293
3628223815,208sleep33774-21unixbench_singl11:38:313
18467223011,206sleep318492-21awk07:18:233
7407222868,68sleep37371-21latency_hist08:28:003
5739222024,145sleep35826-21latency_hist11:48:003
16783221616,40sleep30-21swapper/307:13:223
25575220914,179sleep025623-21unin-run07:42:590
17220220948,27sleep30-21swapper/312:23:173
17220220948,27sleep30-21swapper/312:23:173
29873220533,151sleep329986-21timerwakeupswit07:53:363
22481220413,166sleep322561-21fschecks_time10:58:183
21822219214,163sleep321941-21expr07:28:283
28177219111,158sleep327384-21/usr/sbin/munin07:48:313
24883218920,154sleep024917-21fschecks_count09:23:170
24883218920,154sleep024917-21fschecks_count09:23:170
15851218011,115sleep315912-21basename10:37:583
13007218016,148sleep213085-21sed12:08:262
1407217622,76sleep30-21swapper/309:53:003
1407217622,76sleep30-21swapper/309:53:003
28413217316,60sleep30-21swapper/311:18:003
28413217316,60sleep30-21swapper/311:18:003
18316216815,130sleep30-21swapper/309:03:013
18316216815,130sleep30-21swapper/309:03:013
1667521660,10sleep10-21swapper/107:13:191
16262216621,123sleep30-21swapper/312:18:283
16262216621,123sleep30-21swapper/312:18:283
773121640,9sleep20-21swapper/211:53:062
10965216413,86sleep30-21swapper/312:03:143
8958216116,94sleep39022-21latency_hist08:33:013
8270215216,118sleep08345-21sendmail_mailtr11:53:300
2362121498,90sleep323657-21ntpq09:18:243
16090214215,21sleep00-21swapper/007:13:020
14529214134,40sleep00-21swapper/007:08:100
14529214134,40sleep00-21swapper/007:08:100
21370213926,46sleep30-21swapper/309:12:583
24585213730,90sleep30-21swapper/307:38:213
14420213114,101sleep314488-21awk08:48:273
14420213114,101sleep314488-21awk08:48:273
7283213018,34sleep21-21systemd11:52:592
427621300,13sleep04260-21munin-run08:17:590
3022212821,85sleep23179-21ls09:58:002
118721289,104sleep31151-21latency_hist08:08:013
2029212545,42sleep32070-21ntp_kernel_err09:53:243
3221121248,100sleep032197-21apt-get11:28:150
3221121248,100sleep032197-21apt-get11:28:150
28388212013,49sleep328473-21mailstats09:33:343
2477221200,7sleep10-21swapper/109:23:091
2477221200,7sleep10-21swapper/109:23:091
9706211931,76sleep39725-21netstat11:58:213
14267211613,84sleep014326-21/usr/sbin/munin12:13:200
14267211613,84sleep014326-21/usr/sbin/munin12:13:200
6831211523,37sleep30-21swapper/310:08:233
6831211523,37sleep30-21swapper/310:08:233
20028211529,69sleep30-21swapper/307:23:273
822921137,56sleep38266-21ps11:53:293
150721128,90sleep31508-21diskstats08:08:183
22218211110,86sleep322301-21sed09:13:273
28894211014,76sleep328862-21apt-get11:18:293
26102211014,43sleep326192-21ls09:28:013
26102211014,43sleep326192-21ls09:28:013
19050211012,31sleep30-21swapper/312:28:223
16748211013,80sleep30-21swapper/310:38:273
10026211043,14sleep30-21swapper/310:18:163
23154210912,34sleep023196-21ntp_states07:33:260
17719210923,32sleep30-21swapper/310:43:113
4738210812,35sleep30-21swapper/308:18:193
27905210817,68sleep227928-21cpu09:33:212
25765210811,41sleep30-21swapper/311:08:193
3426210716,76sleep30-21swapper/308:13:263
19162210716,42sleep30-21swapper/309:03:303
20475210611,82sleep320539-21df12:33:143
13409210520,33sleep30-21swapper/310:28:213
3723210315,40sleep3367-21snmpd09:59:513
31140210331,36sleep30-21swapper/307:58:243
31087210369,18sleep30-21swapper/309:43:153
1784210316,70sleep00-21swapper/008:08:230
15165210212,41sleep315283-21timerandwakeup07:08:293
15165210212,41sleep315283-21timerandwakeup07:08:293
13585210219,44sleep30-21swapper/312:11:263
23112210110,41sleep323151-21ntpq07:33:253
24035210020,54sleep124051-21sort11:03:161
20789210010,35sleep30-21swapper/310:53:093
15372210010,33sleep30-21swapper/312:18:003
15372210010,33sleep30-21swapper/312:18:003
10118210011,73sleep110177-21idleruntime10:18:181
863529814,40sleep30-21swapper/310:13:193
635629821,63sleep16463-21apt-key10:08:001
241482989,13sleep00-21swapper/009:18:330
845029719,60sleep08518-21uniq10:13:160
2879529720,59sleep028920-21apt-key07:53:000
264192978,34sleep30-21swapper/309:28:173
264192978,34sleep30-21swapper/309:28:173
239482978,35sleep30-21swapper/311:03:103
1457329717,36sleep10-21swapper/110:33:121
1219729717,67sleep012207-21unin-run12:07:590
661229614,14sleep30-21swapper/311:48:283
2288329613,70sleep022948-21mailstats10:58:270
507329511,67sleep05199-21sed11:43:290
2632729413,46sleep30-21swapper/307:43:263
2632729413,46sleep30-21swapper/307:43:263
1746529415,56sleep116980-21/usr/sbin/munin08:58:271
1746529415,56sleep116980-21/usr/sbin/munin08:58:271
646429321,50sleep26612-21apt-key10:08:012
529729124,41sleep25446-21grep08:18:302
410429111,65sleep14172-21readlink10:00:131
3203429125,45sleep1231-21systemd-journal08:02:581
2631829116,58sleep026319-21/usr/sbin/munin09:28:100
2631829116,58sleep026319-21/usr/sbin/munin09:28:100
1273229128,18sleep3367-21snmpd10:27:453
190182906,70sleep00-21swapper/009:03:260
988828918,46sleep10-21swapper/111:58:271
1275028913,59sleep112798-21systemd-run10:27:591
646428812,61sleep16536-21mailstats11:48:261
966628713,24sleep10-21swapper/108:33:271
673028716,54sleep06874-21/usr/sbin/munin11:48:290
2986228719,16sleep30-21swapper/309:38:223
2831428724,44sleep128339-21munin-run11:17:581
2831428724,44sleep128339-21munin-run11:17:581
1954028715,58sleep219580-21sort10:48:192
2060328616,18sleep30-21swapper/310:53:013
3217728510,20sleep20-21swapper/211:28:092
3217728510,20sleep20-21swapper/211:28:092
1936328524,42sleep219511-21grep12:28:292
629928417,46sleep01-21systemd10:07:590
2165428415,50sleep021169-21/usr/sbin/munin07:28:250
1705428416,17sleep20-21swapper/207:13:292
1045328417,50sleep010502-21basename08:37:580
3151128318,47sleep231663-21sed07:58:312
2174528317,51sleep221819-21switchtime07:28:272
1527028319,46sleep1231-21systemd-journal12:17:581
1527028319,46sleep1231-21systemd-journal12:17:581
2917228215,47sleep029253-21latency11:18:420
680628123,38sleep26803-21apt-get10:08:092
680628123,38sleep26803-21apt-get10:08:092
2966128114,30sleep00-21swapper/007:53:330
2091328125,25sleep10-21swapper/110:53:191
57228016,50sleep2584-21open_files09:48:272
57228016,50sleep2584-21open_files09:48:272
131192808,58sleep10-21swapper/110:28:151
1202828018,46sleep0231-21systemd-journal08:42:590
1202828018,46sleep0231-21systemd-journal08:42:590
3077127924,30sleep10-21swapper/109:43:011
1370527924,26sleep30-21swapper/307:03:293
1370527924,26sleep30-21swapper/307:03:293
108022796,33sleep30-21swapper/308:38:143
108022796,33sleep30-21swapper/308:38:143
1421827816,46sleep114237-21/usr/sbin/munin08:48:211
1421827816,46sleep114237-21/usr/sbin/munin08:48:211
1386027827,27sleep00-21swapper/007:03:300
1386027827,27sleep00-21swapper/007:03:300
3191727728,34sleep332108-21latency_hist11:28:023
3191727728,34sleep332108-21latency_hist11:28:023
3110127716,31sleep20-21swapper/211:23:282
3110127716,31sleep20-21swapper/211:23:282
3010127712,52sleep130104-21vmstat07:53:381
1586527719,33sleep215987-21munin-node07:13:002
1586527719,33sleep215987-21munin-node07:13:002
1210227717,39sleep212249-21latency_hist08:43:012
1210227717,39sleep212249-21latency_hist08:43:012
1220927615,18sleep20-21swapper/212:08:012
682027525,31sleep00-21swapper/010:08:160
682027525,31sleep00-21swapper/010:08:160
2834827517,34sleep21-21systemd11:17:592
2834827517,34sleep21-21systemd11:17:592
2400127512,47sleep124059-21sendmail-msp07:37:591
2009127517,41sleep120291-21latency_hist12:33:001
902427416,18sleep00-21swapper/008:33:020
243027418,11sleep30-21swapper/311:36:133
2283027417,35sleep222909-21fschecks_time07:33:202
2095527420,28sleep00-21swapper/012:33:270
144027313,42sleep01588-21latency_hist09:53:010
144027313,42sleep01588-21latency_hist09:53:010
1967727212,44sleep119724-21/usr/sbin/munin10:48:221
3172727117,39sleep131818-21apt-config11:28:001
3172727117,39sleep131818-21apt-config11:28:001
2886427116,39sleep128341-21apt-get11:18:101
2141727116,41sleep021454-21sed09:12:590
2066427118,29sleep20-21swapper/209:08:292
1849427115,23sleep00-21swapper/007:18:240
1689627120,20sleep20-21swapper/212:22:592
1689627120,20sleep20-21swapper/212:22:592
743427013,42sleep17511-21find10:08:411
743427013,42sleep17511-21find10:08:411
71727015,25sleep10-21swapper/109:48:301
71727015,25sleep10-21swapper/109:48:301
2928127012,39sleep229422-21latency_hist09:38:012
2855727017,26sleep00-21swapper/011:18:010
2026827024,24sleep10-21swapper/109:08:201
3149426919,27sleep20-21swapper/209:43:262
2654126914,38sleep226691-21sed07:43:292
2654126914,38sleep226691-21sed07:43:292
2047126928,21sleep20-21swapper/212:33:132
192102690,8sleep00-21swapper/012:28:260
1648926918,28sleep10-21swapper/112:18:301
1648926918,28sleep10-21swapper/112:18:301
1338226918,40sleep30-21swapper/308:45:293
1258726918,20sleep00-21swapper/008:43:200
1157526917,32sleep10-21swapper/112:03:281
1003326917,34sleep010114-21fschecks_time10:18:170
3062526813,40sleep030678-21interrupts11:23:180
3062526813,40sleep030678-21interrupts11:23:180
29582689,44sleep02962-21df_inode11:38:160
2163626813,39sleep121655-21apt-get09:13:091
1718626822,29sleep20-21swapper/208:58:202
1718626822,29sleep20-21swapper/208:58:202
1534226819,22sleep20-21swapper/212:17:592
1534226819,22sleep20-21swapper/212:17:592
1428726826,27sleep20-21swapper/207:08:002
1428726826,27sleep20-21swapper/207:08:002
2471926711,24sleep20-21swapper/207:38:242
2157026717,31sleep10-21swapper/107:28:221
1591726716,22sleep20-21swapper/208:53:272
1497926721,25sleep20-21swapper/210:33:252
1413726718,21sleep00-21swapper/008:48:200
1413726718,21sleep00-21swapper/008:48:200
25652669,15sleep20-21swapper/211:37:582
1305326618,23sleep20-21swapper/208:43:302
982526520,18sleep00-21swapper/011:58:260
3202526510,41sleep0367-21snmpd09:44:490
2095826520,24sleep00-21swapper/007:28:010
1819226518,24sleep10-21swapper/110:43:261
1812326512,23sleep20-21swapper/210:43:232
1279926516,23sleep20-21swapper/210:28:002
2892226421,26sleep129093-21sed07:53:011
2651226419,21sleep20-21swapper/209:28:202
2651226419,21sleep20-21swapper/209:28:202
2597026416,24sleep10-21swapper/111:08:251
478826316,24sleep20-21swapper/210:03:012
3003926311,27sleep030118-21sort09:38:270
280252637,44sleep00-21swapper/011:14:220
1826326323,23sleep00-21swapper/009:02:580
1826326323,23sleep00-21swapper/009:02:580
1207426320,18sleep10-21swapper/108:43:001
1207426320,18sleep10-21swapper/108:43:001
942526213,34sleep29428-21entropy11:58:162
890326215,30sleep18944-21sshd08:32:591
3229026217,22sleep10-21swapper/111:28:241
3229026217,22sleep10-21swapper/111:28:241
23552626,11sleep10-21swapper/111:35:141
2215726220,23sleep20-21swapper/210:58:002
203132625,13sleep00-21swapper/010:50:110
1973626211,21sleep10-21swapper/107:23:211
189372626,42sleep00-21swapper/010:47:240
1556326217,25sleep015582-21apt-get12:18:090
1556326217,25sleep015582-21apt-get12:18:090
1344426220,26sleep013494-21/usr/sbin/munin10:28:220
1149226217,26sleep00-21swapper/010:23:140
326922616,11sleep10-21swapper/108:03:231
315626116,28sleep23222-21latency11:38:222
2922326110,38sleep129280-21latency_hist09:38:001
2193626118,22sleep00-21swapper/009:13:200
431826019,17sleep20-21swapper/208:18:022
3229926021,22sleep00-21swapper/008:03:100
2919426013,30sleep229238-21/usr/sbin/munin07:53:142
2478426018,19sleep10-21swapper/107:38:271
94092598,35sleep29465-21interrupts08:33:202
233222595,17sleep10-21swapper/110:58:401
2109025914,26sleep021186-21sshd10:53:220
1617325917,17sleep10-21swapper/110:38:131
962825822,22sleep19664-21munin-run10:17:581
3272125813,25sleep20-21swapper/208:03:242
26252584,15sleep00-21swapper/009:54:140
2103025819,23sleep121107-21sed12:33:281
1876425817,19sleep20-21swapper/207:18:302
352025715,25sleep03458-21apt-get09:58:470
2586025714,29sleep10-21swapper/107:43:121
2586025714,29sleep10-21swapper/107:43:121
2585925715,26sleep025855-21apt-get07:43:110
2585925715,26sleep025855-21apt-get07:43:110
2450425715,19sleep20-21swapper/209:22:592
157802575,16sleep00-21swapper/010:37:030
651925613,26sleep06556-21ntp_states08:23:250
272325612,20sleep10-21swapper/108:13:001
244582565,10sleep10-21swapper/109:22:211
165925613,26sleep11650-21apt-get09:53:141
1551825614,22sleep10-21swapper/108:53:181
1473725617,18sleep10-21swapper/107:08:191
1473725617,18sleep10-21swapper/107:08:191
312625511,22sleep10-21swapper/111:38:211
586425411,30sleep05953-21latency_hist08:23:000
440125416,23sleep2231-21systemd-journal11:43:062
3175425415,18sleep10-21swapper/109:43:291
2634825414,23sleep126338-21apt-get09:28:151
2634825414,23sleep126338-21apt-get09:28:151
1759625414,22sleep00-21swapper/012:23:260
1759625414,22sleep00-21swapper/012:23:260
1721425415,24sleep217219-21diskstats12:23:162
1721425415,24sleep217219-21diskstats12:23:162
1683825414,19sleep00-21swapper/010:38:280
2675525314,24sleep226790-21idleruntime-cro11:12:582
2498925312,16sleep20-21swapper/209:23:192
2498925312,16sleep20-21swapper/209:23:192
2292825317,21sleep222957-21munin-run09:17:582
1965325317,22sleep219664-21grep07:23:182
1753325317,20sleep10-21swapper/112:23:231
1753325317,20sleep10-21swapper/112:23:231
1435125315,21sleep20-21swapper/208:48:262
1435125315,21sleep20-21swapper/208:48:262
1312425313,23sleep213212-21fschecks_time10:28:162
1262625311,17sleep10-21swapper/108:43:211
1303525214,22sleep113042-21df_inode07:03:151
1303525214,22sleep113042-21df_inode07:03:151
1036525214,22sleep210401-21ntp_states10:18:232
82102518,16sleep10-21swapper/111:53:271
843324911,22sleep28440-21df_inode10:13:142
804124914,20sleep212872-1kworker/2:2H08:28:242
607824916,19sleep10-21swapper/108:23:101
2453024914,20sleep00-21swapper/011:03:270
2441524917,18sleep20-21swapper/211:03:252
453024817,17sleep10-21swapper/108:18:091
3118724811,20sleep031223-21ntp_states07:58:250
2866224812,20sleep20-21swapper/207:52:582
283322489,15sleep00-21swapper/009:33:310
1817824811,20sleep118184-21diskstats07:18:161
1587724810,23sleep015883-21ntp_states08:53:250
274924713,18sleep20-21swapper/208:13:012
1863124712,21sleep218637-21df_inode09:03:162
520024611,20sleep15332-21awk11:43:301
336924616,16sleep10-21swapper/108:13:241
1618624610,20sleep216244-21df10:38:142
677924512,13sleep20-21swapper/208:23:312
264424514,17sleep00-21swapper/011:38:000
228512458,14sleep20-21swapper/210:58:252
1856024310,15sleep10-21swapper/109:03:151
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional