You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-26 - 18:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Sun Jul 26, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8087276519,728sleep18135-21munin-node07:13:081
4236254214,511sleep34320-21memory08:43:253
24530239220,350sleep324578-21fschecks_count09:48:203
17379228811,265sleep217440-21cat11:08:242
2673227868,85sleep20-21swapper/208:38:232
20011226966,181sleep220035-21modprobe11:18:002
9697222664,85sleep29778-21switchtime10:43:292
23643222122,176sleep123785-21grep09:43:331
22155221213,132sleep222172-21latency_hist07:58:022
15596221110,18sleep20-21swapper/211:03:152
7691221042,115sleep27701-21modprobe07:13:012
12954221049,112sleep212989-21proc_pri10:53:512
27141220746,139sleep227229-21fschecks_time08:13:202
23105220015,141sleep20-21swapper/209:43:222
13303219764,114sleep213386-21latency_hist09:13:032
23671219425,150sleep0225-21systemd-journal08:03:010
4723219051,125sleep24726-21latency_hist12:13:022
4723219051,125sleep24726-21latency_hist12:13:022
1363821909,21sleep00-21swapper/009:13:210
12849219017,113sleep213001-21latency_hist07:28:032
9861217213,142sleep39918-21irqstats12:28:213
7989216613,138sleep28129-21sed08:53:372
7989216613,138sleep28129-21sed08:53:372
6825216148,73sleep26902-21cat12:18:232
7338215712,132sleep17363-21latency_hist10:38:031
32741215720,22sleep20-21swapper/211:58:132
1695021560,6sleep30-21swapper/307:38:333
22345215342,62sleep222384-21proc_pri11:23:292
1367321535,136sleep00-21swapper/010:57:240
11749215213,121sleep111824-21wc12:33:321
1112215114,123sleep21217-21latency_hist10:18:022
1112215114,123sleep21217-21latency_hist10:18:022
1925214621,106sleep32007-21switchtime10:18:323
1925214621,106sleep32007-21switchtime10:18:323
2490214523,98sleep32553-21fschecks_time08:38:203
17237214029,63sleep20-21swapper/209:23:322
12458213913,114sleep2225-21systemd-journal12:38:012
8749213819,11sleep20-21swapper/207:13:262
24456213727,75sleep2364-21snmpd11:30:072
15353213712,76sleep215395-21sed11:03:032
5242213617,104sleep05286-21ntpq07:03:270
24580213537,84sleep224708-21timerwakeupswit08:03:352
24580213537,84sleep224708-21timerwakeupswit08:03:352
1069221358,113sleep210706-21grep09:03:222
2972213460,25sleep20-21swapper/210:23:232
28711213331,87sleep228835-21timerandwakeup11:43:322
32448213227,54sleep26226-21cyclictest11:58:012
31331213214,101sleep031457-21sed10:08:320
899821318,15sleep00-21swapper/012:23:340
2797421258,64sleep228023-21ntpq09:58:282
25022212510,72sleep20-21swapper/211:33:182
7039212310,97sleep06529-21/usr/sbin/munin12:18:290
6486211946,24sleep26629-21grep08:48:492
6486211946,24sleep26629-21grep08:48:492
2237211934,50sleep26226-21cyclictest12:03:262
7551211653,43sleep20-21swapper/210:38:122
26598211615,83sleep326676-21/usr/sbin/munin09:53:333
27857211535,65sleep228006-21latency_hist11:43:032
5520211460,19sleep2364-21snmpd10:30:422
22325211412,21sleep00-21swapper/011:23:270
29353211324,74sleep329368-21apt11:48:013
3954211214,41sleep20-21swapper/208:43:162
18420211217,69sleep10-21swapper/109:28:241
31159211155,37sleep20-21swapper/210:08:302
31190211014,79sleep131257-21/usr/sbin/munin11:53:151
9361210914,79sleep39429-21readlink08:58:313
20322210810,45sleep220324-21apt-get11:18:132
16651210823,59sleep016670-21apt-get09:23:140
1618621088,43sleep20-21swapper/207:38:162
6972210715,11sleep20-21swapper/210:34:212
6674210613,72sleep06703-21/usr/sbin/munin07:08:230
30715210519,11sleep20-21swapper/211:50:382
11975210512,78sleep011994-21apt-get09:08:160
21724210311,77sleep321773-21ntpq09:38:273
20093210116,64sleep120219-21wc11:18:031
11141210040,41sleep20-21swapper/210:48:302
203542995,12sleep20-21swapper/207:50:402
1976529940,18sleep26226-21cyclictest09:33:132
951829724,57sleep19552-21/usr/sbin/munin10:43:241
1852329614,65sleep118675-21latency_hist11:13:031
1512729628,56sleep215120-21apt-get09:18:152
57529539,44sleep2585-21munin-run08:33:022
268329537,17sleep20-21swapper/210:23:032
268329537,17sleep20-21swapper/210:23:032
2444429534,15sleep20-21swapper/209:48:152
3271629416,63sleep332717-21sshd08:29:283
2639529410,71sleep226442-21ntpq09:53:282
1411729436,18sleep20-21swapper/209:13:312
111262948,22sleep00-21swapper/010:48:280
1064729422,48sleep110780-21unixbench_multi07:18:331
3019629333,15sleep20-21swapper/208:23:222
2882129331,50sleep228878-21if_err_eth008:18:212
1886629315,63sleep018953-21fw_packets11:13:220
1306029333,16sleep20-21swapper/207:28:152
1249629210,68sleep312497-21sshd07:24:293
794529012,61sleep37974-21/usr/sbin/munin10:38:263
2318929012,61sleep323341-21sed11:28:023
2733228922,20sleep10-21swapper/109:58:031
1479428914,59sleep014842-21cat07:33:220
1479428914,59sleep014842-21cat07:33:220
484828827,38sleep34901-21ls07:03:183
123628810,20sleep20-21swapper/208:33:272
2261728712,16sleep20-21swapper/207:58:232
895328624,28sleep00-21swapper/008:58:190
1948428613,60sleep2225-21systemd-journal09:33:012
3149928524,25sleep30-21swapper/308:28:033
2209128510,60sleep222236-21unixbench_singl09:38:342
2164128523,30sleep00-21swapper/011:23:030
902128420,12sleep20-21swapper/212:23:362
1045828427,15sleep20-21swapper/212:28:322
504828315,47sleep14938-21/usr/sbin/munin12:13:211
144328311,57sleep01535-21fschecks_time10:18:200
144328311,57sleep01535-21fschecks_time10:18:200
1380128321,32sleep30-21swapper/310:58:033
98962825,11sleep20-21swapper/208:59:512
891528213,54sleep38985-21sensors_temp07:13:313
830928216,44sleep08447-21grep10:38:350
73872829,12sleep00-21swapper/008:53:230
73872829,12sleep00-21swapper/008:53:230
2946328231,13sleep20-21swapper/210:03:252
2664328220,46sleep126745-21fschecks_time11:38:221
807128115,48sleep07802-21munin-node07:13:070
2595928122,44sleep20-21swapper/208:08:282
598328020,12sleep20-21swapper/212:15:002
2559928018,39sleep325688-21fschecks_time08:08:213
1985728017,41sleep119928-21fschecks_time09:33:201
112672808,60sleep211272-21systemd-run07:23:022
2899927915,44sleep129010-21/usr/sbin/munin08:18:241
2895127913,15sleep10-21swapper/110:03:041
2654927920,25sleep00-21swapper/011:38:170
183082795,11sleep30-21swapper/309:28:223
1731527922,26sleep30-21swapper/309:23:333
1708627914,49sleep017098-21vmstat07:38:350
1552727919,11sleep20-21swapper/207:33:362
1552727919,11sleep20-21swapper/207:33:362
1281527913,51sleep112847-21cat07:28:021
1206327929,31sleep312231-21sed10:53:033
368127730,24sleep30-21swapper/312:08:233
368127730,24sleep30-21swapper/312:08:233
2110027611,52sleep121162-21latency07:53:251
1003627617,43sleep310045-21df_inode07:18:193
177302759,16sleep20-21swapper/207:43:132
1169127519,38sleep011693-21cron09:08:000
2217427416,25sleep30-21swapper/307:58:033
129827419,38sleep11362-21ps08:33:301
341527312,19sleep00-21swapper/012:08:160
341527312,19sleep00-21swapper/012:08:160
3144727315,35sleep131473-21modprobe08:28:011
2825527324,26sleep00-21swapper/009:58:350
2766427324,23sleep10-21swapper/108:13:321
909627221,21sleep30-21swapper/310:43:103
1996327218,29sleep120095-21tail07:48:331
1277327211,47sleep012814-21systemd-run07:28:010
103327223,27sleep00-21swapper/010:18:000
103327223,27sleep00-21swapper/010:18:000
2351027111,47sleep023542-21tune2fs11:28:200
311632709,14sleep00-21swapper/011:53:100
2960227021,27sleep30-21swapper/310:03:303
2442627014,40sleep324447-21awk08:03:323
2442627014,40sleep324447-21awk08:03:323
1567727016,23sleep00-21swapper/009:18:340
350926920,32sleep03585-21/usr/sbin/munin10:23:370
245822690,8sleep00-21swapper/009:48:200
146852699,18sleep00-21swapper/010:58:330
535926822,22sleep30-21swapper/312:13:283
535626819,29sleep15560-21apt-config08:48:031
1498426811,43sleep314985-21cron10:59:003
1253626814,34sleep312608-21/usr/sbin/munin09:08:363
3010926724,26sleep10-21swapper/111:48:311
184226711,39sleep11861-21apt-get12:03:121
1820026710,33sleep018219-21apt-get09:28:140
1643726718,32sleep116499-21sed09:23:031
84662665,48sleep30-21swapper/312:23:253
64926615,18sleep00-21swapper/008:33:040
526626620,27sleep05244-21/usr/sbin/munin08:48:020
2255626616,33sleep122682-21/usr/sbin/munin11:23:321
2143226627,19sleep10-21swapper/109:38:211
1833826618,27sleep30-21swapper/307:43:303
1686026614,27sleep10-21swapper/109:23:231
1587026616,29sleep30-21swapper/307:38:013
1587026616,29sleep30-21swapper/307:38:013
1511826611,39sleep115214-21mailstats07:33:311
1511826611,39sleep115214-21mailstats07:33:311
1138326616,24sleep00-21swapper/012:33:230
552926521,19sleep20-21swapper/207:03:332
1904426511,41sleep019143-21latency_hist07:48:030
143526515,26sleep31512-21/usr/sbin/munin08:33:323
336526420,24sleep10-21swapper/110:23:341
3043626422,19sleep00-21swapper/010:08:030
2753226412,35sleep327591-21/usr/sbin/munin08:13:303
219562645,13sleep00-21swapper/007:56:230
88826319,25sleep1976-21switchtime11:58:321
817926311,33sleep10-21swapper/112:23:181
69532637,13sleep30-21swapper/310:34:063
302292636,12sleep30-21swapper/311:48:333
2804126312,18sleep10-21swapper/109:58:321
2780326314,34sleep327805-21cron11:43:003
205322635,12sleep00-21swapper/007:52:560
1871826315,31sleep318720-21/usr/sbin/munin11:13:133
163126318,22sleep00-21swapper/012:03:030
1216026317,32sleep112221-21idleruntime09:08:261
77526216,29sleep3800-21/usr/sbin/munin11:58:303
2505926213,21sleep00-21swapper/011:33:200
2287326216,28sleep122937-21processes07:58:301
1437226222,24sleep10-21swapper/107:33:031
40292619,40sleep30-21swapper/310:26:273
3142626113,33sleep331459-21/usr/sbin/munin11:53:213
2996826113,22sleep00-21swapper/008:23:030
2568526116,22sleep10-21swapper/111:33:341
2552126112,33sleep325588-21awk11:33:313
206852616,12sleep00-21swapper/009:33:350
1978826120,24sleep319880-21mailstats07:48:313
679026016,19sleep10-21swapper/112:18:221
613726022,21sleep30-21swapper/308:48:423
613726022,21sleep30-21swapper/308:48:423
57612606,11sleep10-21swapper/107:03:361
3124626014,30sleep331326-21switchtime10:08:313
2883726014,28sleep328886-21unin-run10:03:023
1096226022,20sleep00-21swapper/012:33:020
465325918,22sleep10-21swapper/110:28:221
320425915,19sleep30-21swapper/312:08:033
2486325914,30sleep024875-21vmstat08:03:370
2486325914,30sleep024875-21vmstat08:03:370
1753725921,21sleep10-21swapper/107:43:041
1691525914,27sleep016944-21munin-run11:08:010
1559725913,28sleep315664-21/usr/sbin/munin11:03:163
973725817,24sleep10-21swapper/112:28:191
751525817,27sleep30-21swapper/308:53:263
751525817,27sleep30-21swapper/308:53:263
680725814,28sleep36851-21sed07:08:263
425225815,26sleep14318-21/usr/sbin/munin12:08:331
425225815,26sleep14318-21/usr/sbin/munin12:08:331
3266325817,25sleep332672-21ntp_states10:13:273
3266325817,25sleep332672-21ntp_states10:13:273
3199925818,20sleep10-21swapper/108:28:251
2645525815,20sleep00-21swapper/009:53:300
2389325814,28sleep123956-21cut11:28:291
2142525816,24sleep021431-21diskstats09:38:200
2057225814,27sleep120601-21irqstats11:18:241
1312825816,26sleep313134-21df_inode07:28:163
3027525716,24sleep030165-21/usr/sbin/munin08:23:240
2833725714,17sleep10-21swapper/111:43:221
2681625723,17sleep10-21swapper/108:13:031
153725720,17sleep10-21swapper/110:18:211
153725720,17sleep10-21swapper/110:18:211
1487925712,29sleep114889-21munin-run09:18:021
1481225720,18sleep10-21swapper/110:58:341
72782567,19sleep10-21swapper/108:53:161
72782567,19sleep10-21swapper/108:53:161
638125618,20sleep10-21swapper/108:48:481
638125618,20sleep10-21swapper/108:48:481
481225616,25sleep04842-21awk10:28:260
2809325615,25sleep028089-21apt-get11:43:130
2574825619,23sleep125731-21latency_hist09:53:021
1744125614,26sleep317466-21irqstats11:08:253
757625512,25sleep17571-21apt-get10:38:151
3213725517,18sleep00-21swapper/008:28:280
247525515,24sleep12484-21df_inode08:38:181
1399325515,25sleep313999-21ntp_states09:13:283
996625412,24sleep09960-21apt-get07:18:160
781425420,20sleep30-21swapper/312:23:003
777325416,20sleep10-21swapper/107:13:041
685825413,26sleep06905-21users10:33:340
452925421,19sleep30-21swapper/310:28:203
2705425413,25sleep027049-21apt-get08:13:140
2550325414,24sleep025499-21apt-get08:08:140
21392547,25sleep00-21swapper/012:03:240
2033225414,21sleep30-21swapper/311:18:183
1617325417,17sleep10-21swapper/111:03:301
3095225315,16sleep00-21swapper/011:53:020
287325313,19sleep00-21swapper/008:38:290
2300925317,19sleep30-21swapper/307:58:323
1550025313,23sleep315547-21ntp_kernel_err09:18:293
1238625312,23sleep112423-21store10:53:051
1193325318,21sleep10-21swapper/110:53:011
1155525316,22sleep30-21swapper/312:33:263
910725213,24sleep09117-21apt-get10:43:110
2871225215,24sleep328719-21df_inode08:18:183
1040625213,21sleep30-21swapper/309:03:113
2298425118,19sleep00-21swapper/009:43:200
2369225014,17sleep30-21swapper/311:28:233
2110825014,15sleep30-21swapper/309:38:023
631724813,17sleep00-21swapper/008:48:470
631724813,17sleep00-21swapper/008:48:470
3222224811,22sleep132289-21df10:13:171
3222224811,22sleep132289-21df10:13:171
1083024812,17sleep10-21swapper/109:03:251
539624711,21sleep05419-21ntp_states12:13:290
1719724715,17sleep00-21swapper/011:08:180
101302475,30sleep10-21swapper/109:02:481
2333124610,19sleep323350-21ntp_states09:43:273
306672456,15sleep10-21swapper/110:08:121
2526724512,19sleep10-21swapper/108:08:031
2526724512,19sleep10-21swapper/108:08:031
1810124514,18sleep10-21swapper/107:43:241
1313624514,16sleep00-21swapper/007:28:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional