You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-19 - 00:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Nov 18, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
16703284090,587sleep116850-21grep12:04:251
10025268012,654sleep210026-21sshd10:00:262
26300260410,551sleep126418-21grep09:09:251
294642310104,185sleep129595-21grep09:19:271
16834225814,229sleep016873-21ntpq06:59:310
1966822559,232sleep119750-21systemctl12:14:221
688022466,230sleep1362-21snmpd09:53:271
18543224655,163sleep118678-21tail08:44:261
25189224311,215sleep225337-21sed12:33:542
3826222730,76sleep13857-21sshd07:59:161
26372221153,77sleep126375-21cron07:28:511
3290221040,146sleep10-21swapper/107:58:511
27276220711,108sleep10-21swapper/107:29:241
27276220711,108sleep10-21swapper/107:29:241
29159219019,150sleep00-21swapper/007:34:360
28067218722,145sleep328107-21fschecks_count10:59:143
22796218412,98sleep122819-21kernelversion08:59:161
22796218412,98sleep122819-21kernelversion08:59:161
716218315,110sleep1754-21latency07:49:141
7994218237,96sleep18084-21latency_hist11:38:531
18043217816,149sleep118067-21proc_pri12:09:221
2316217315,40sleep10-21swapper/109:38:531
2316217315,40sleep10-21swapper/109:38:531
2419221716,71sleep124228-21ntpq12:29:201
8330217051,66sleep18429-21fschecks_time11:39:141
24092216917,122sleep10-21swapper/109:04:081
825216531,120sleep1861-21ntpq11:14:181
31954215811,133sleep132034-21awk11:09:461
2448121578,111sleep124693-21latency_hist10:48:541
13023215784,19sleep10-21swapper/108:28:551
20852215611,48sleep10-21swapper/107:09:291
25769215417,113sleep10-21swapper/110:49:471
25111215228,99sleep224600-21apt-get07:24:082
21360215242,65sleep121495-21timerwakeupswit12:19:231
456721497,126sleep14601-21ps09:44:221
456721497,126sleep14601-21ps09:44:221
2346521499,98sleep123479-21cut07:19:511
17594214511,115sleep117676-21latency_hist08:43:531
3214621425,11sleep30-21swapper/309:29:223
14713214119,13sleep10-21swapper/110:15:271
11042213834,92sleep111061-21open_inodes10:04:211
2531213419,99sleep10-21swapper/109:39:051
25010212714,19sleep00-21swapper/010:49:340
12683212714,98sleep312755-21ls10:09:243
1034821270,7sleep30-21swapper/310:03:533
21544212435,43sleep121625-21switchtime08:54:231
22810212310,100sleep122926-21latency_hist10:43:541
5440212216,22sleep00-21swapper/009:48:530
5440212216,22sleep00-21swapper/009:48:530
17794212215,11sleep10-21swapper/110:25:121
6977212157,43sleep10-21swapper/108:09:291
6977212157,43sleep10-21swapper/108:09:291
25902212113,95sleep025917-21kernelversion09:09:160
7441211818,80sleep06934-21apt-get09:54:020
7441211818,80sleep06934-21apt-get09:54:020
27097211817,83sleep227255-21unixbench_multi10:54:252
27097211817,83sleep227255-21unixbench_multi10:54:252
11011211719,51sleep10-21swapper/111:48:261
1578021167,63sleep115779-21/usr/sbin/munin12:03:531
1278621156,95sleep112867-21latency_hist11:53:551
22827211311,89sleep022909-21cat07:18:530
16286211042,47sleep316308-21kernelversion12:04:153
30401210915,65sleep00-21swapper/009:24:140
1801521098,57sleep118060-21grep07:04:121
946521085,12sleep10-21swapper/108:18:051
3084321088,51sleep130930-21apt-config11:08:531
20318210816,31sleep020326-21needreboot07:09:200
200921088,85sleep02057-21awk11:19:100
200921088,85sleep02057-21awk11:19:100
19701210633,30sleep10-21swapper/110:33:521
16899210619,66sleep316939-21ps06:59:343
8392210534,21sleep10-21swapper/109:54:441
8392210534,21sleep10-21swapper/109:54:441
2954421048,44sleep10-21swapper/107:38:531
28192210416,40sleep10-21swapper/110:59:161
23718210414,74sleep323838-21sed10:44:243
12881210410,48sleep10-21swapper/110:09:261
6951210321,56sleep30-21swapper/311:34:153
25868210220,56sleep326015-21grep07:25:183
25115210214,69sleep025100-21apt-get07:24:100
22988210213,74sleep222994-21ntp_states08:59:212
22988210213,74sleep222994-21ntp_states08:59:212
3263210032,53sleep13266-21cron11:23:511
3263210032,53sleep13266-21cron11:23:511
1844121009,73sleep30-21swapper/307:04:223
1711929945,32sleep20-21swapper/206:59:372
77782988,16sleep20-21swapper/211:36:442
1662529817,21sleep00-21swapper/012:04:240
37622979,76sleep00-21swapper/009:43:120
319532978,43sleep132039-21ls07:44:221
2331629712,12sleep20-21swapper/210:44:152
212612977,43sleep121356-21sed10:38:531
1984329715,63sleep319141-21/usr/sbin/munin12:14:243
144342969,48sleep10-21swapper/108:33:521
1831829518,62sleep217817-21/usr/sbin/munin08:44:232
86632947,65sleep30-21swapper/311:39:203
52082946,13sleep00-21swapper/008:04:110
52082946,13sleep00-21swapper/008:04:110
3237029413,67sleep232434-21sendmail_mailqu09:29:292
1014629417,62sleep010156-21needreboot11:44:170
1271729318,62sleep012785-21sed11:53:540
1271729318,62sleep012785-21sed11:53:540
930229214,55sleep10-21swapper/109:59:161
2979629214,57sleep029846-21cpuspeed07:39:080
1130829215,23sleep10-21swapper/111:49:031
1348928918,48sleep213565-21wc11:54:252
1348928918,48sleep213565-21wc11:54:252
540528811,63sleep35436-21cut09:48:523
540528811,63sleep35436-21cut09:48:523
2291028845,27sleep123024-21cat07:18:541
2617728731,25sleep30-21swapper/310:53:533
2166428716,54sleep321665-21sshd12:19:543
170328720,46sleep21705-21cron07:53:522
881628611,62sleep08887-21sensors_temp08:14:290
2825828618,41sleep328218-21apt-get07:34:143
1311528610,60sleep012638-21apt-get08:28:590
879228518,51sleep38840-21latency_hist09:58:543
879228518,51sleep38840-21latency_hist09:58:543
644028525,15sleep20-21swapper/208:08:542
644028525,15sleep20-21swapper/208:08:542
3128828514,29sleep30-21swapper/311:09:033
2768828521,14sleep10-21swapper/110:58:521
2768828521,14sleep10-21swapper/110:58:521
1884528523,41sleep218867-21cron12:13:512
160352848,15sleep10-21swapper/108:38:531
604328210,52sleep36081-21/usr/sbin/munin09:49:173
522528221,12sleep10-21swapper/111:29:121
513228211,53sleep10-21swapper/108:04:081
513228211,53sleep10-21swapper/108:04:081
1068828219,47sleep210696-21vmstat11:44:272
429628119,20sleep30-21swapper/307:59:263
3261128112,48sleep10-21swapper/109:29:321
230412818,19sleep10-21swapper/112:24:281
230412818,19sleep10-21swapper/112:24:281
157182815,42sleep10-21swapper/110:19:191
1441528022,38sleep014432-21modprobe08:33:510
1266428017,49sleep212752-21apt-key08:28:522
1963127912,52sleep019667-21wc12:14:210
134222790,6sleep20-21swapper/208:29:362
375927819,12sleep10-21swapper/111:24:121
375927819,12sleep10-21swapper/111:24:121
312632785,40sleep10-21swapper/109:24:401
303527811,53sleep2362-21snmpd07:55:332
272272789,49sleep30-21swapper/309:14:083
2391027820,27sleep00-21swapper/012:29:150
2198927813,48sleep221996-21ntp_states07:14:192
1878527813,46sleep018827-21/usr/sbin/munin10:29:190
122027822,28sleep20-21swapper/209:34:182
122027822,28sleep20-21swapper/209:34:182
682327717,41sleep26885-21grep08:09:262
682327717,41sleep26885-21grep08:09:262
575627617,38sleep25839-21cut09:49:122
2998727619,41sleep3362-21snmpd11:04:193
2905327617,41sleep029116-21grep09:19:180
1599727617,45sleep316043-21grep10:19:253
3131127522,40sleep06928-21kworker/u8:411:09:040
256432759,46sleep20-21swapper/209:09:102
2275827517,44sleep222843-21cut12:24:252
2275827517,44sleep222843-21cut12:24:252
2466327426,22sleep00-21swapper/007:23:540
2121027418,34sleep30-21swapper/310:38:513
187622746,55sleep00-21swapper/008:44:290
11212748,50sleep01239-21timerwakeupswit11:14:250
308612735,57sleep10-21swapper/107:41:231
2858427312,46sleep228661-21grep10:59:252
2279127318,31sleep322811-21apt-get07:18:523
2133627315,40sleep320945-21/usr/sbin/munin08:54:173
2053827318,11sleep10-21swapper/108:52:251
1356627321,31sleep30-21swapper/311:54:263
1356627321,31sleep30-21swapper/311:54:263
98252727,15sleep10-21swapper/108:19:141
98252727,15sleep10-21swapper/108:19:141
334327223,24sleep00-21swapper/007:58:540
272327218,34sleep22778-21sort09:39:142
1815427214,45sleep318226-21wc12:09:243
1340627230,21sleep113488-21mailstats11:54:241
1340627230,21sleep113488-21mailstats11:54:241
1183827215,41sleep311880-21wc08:24:253
1686827114,27sleep00-21swapper/010:24:100
986827029,26sleep39761-21/usr/sbin/munin11:44:123
85892706,42sleep20-21swapper/208:14:232
636127016,36sleep36392-21modprobe11:33:503
446827026,18sleep20-21swapper/209:44:152
446827026,18sleep20-21swapper/209:44:152
2663927015,29sleep20-21swapper/207:29:052
2663927015,29sleep20-21swapper/207:29:052
2273927018,21sleep00-21swapper/008:59:150
2273927018,21sleep00-21swapper/008:59:150
1823527018,35sleep018356-21timerwakeupswit12:09:250
1463327014,28sleep30-21swapper/311:59:143
2456626915,34sleep224628-21sendmail_mailqu09:04:222
2385326920,27sleep30-21swapper/309:03:543
2385326920,27sleep30-21swapper/309:03:543
2069626916,31sleep00-21swapper/010:34:280
180792697,15sleep30-21swapper/310:28:503
2762526832,20sleep10-21swapper/109:14:181
2458126819,32sleep224600-21apt-get07:23:522
211226814,39sleep32148-21/usr/sbin/munin07:54:133
676826719,33sleep06827-21fschecks_count11:34:120
3155626714,29sleep20-21swapper/211:09:352
2390426722,21sleep30-21swapper/312:29:143
2318626718,32sleep322248-21/usr/sbin/munin12:24:303
2318626718,32sleep322248-21/usr/sbin/munin12:24:303
230532677,15sleep00-21swapper/010:44:090
1648826719,30sleep216552-21ps12:04:222
2534926621,22sleep30-21swapper/309:08:513
1606826615,34sleep216216-21sed08:38:542
843026512,32sleep20-21swapper/211:39:152
494626519,24sleep00-21swapper/008:03:550
2955726514,36sleep229706-21latency_hist07:38:542
2646726518,29sleep326492-21/usr/sbin/munin10:54:093
2646726518,29sleep326492-21/usr/sbin/munin10:54:093
207912658,14sleep00-21swapper/012:19:100
203732656,47sleep30-21swapper/308:50:183
281102640,13sleep00-21swapper/010:59:140
146526413,26sleep00-21swapper/009:34:240
146526413,26sleep00-21swapper/009:34:240
626926310,41sleep00-21swapper/011:32:380
313102637,14sleep30-21swapper/309:25:183
1837626322,20sleep20-21swapper/210:29:092
1530526318,23sleep30-21swapper/308:34:313
1170226320,28sleep211734-21awk11:49:162
1170226320,28sleep211734-21awk11:49:162
875526221,18sleep30-21swapper/308:14:283
777026216,20sleep20-21swapper/209:54:332
777026216,20sleep20-21swapper/209:54:332
532426212,34sleep35382-21/usr/sbin/munin08:04:193
532426212,34sleep35382-21/usr/sbin/munin08:04:193
381326217,25sleep20-21swapper/211:24:132
381326217,25sleep20-21swapper/211:24:132
2336626211,21sleep30-21swapper/307:19:373
2198726217,30sleep222064-21basename10:39:232
2162926221,22sleep20-21swapper/208:54:242
206002626,43sleep00-21swapper/008:53:130
1813026221,23sleep20-21swapper/210:28:532
1037526218,22sleep20-21swapper/208:19:362
1037526218,22sleep20-21swapper/208:19:362
2481326118,26sleep224325-21apt-get10:49:042
2038426114,33sleep220407-21ntp_states10:34:222
1931426115,23sleep20-21swapper/207:08:542
1798826119,21sleep00-21swapper/007:04:070
1538226115,22sleep00-21swapper/008:34:320
1338326115,30sleep013402-21proc_pri11:54:230
1338326115,30sleep013402-21proc_pri11:54:230
103022617,12sleep00-21swapper/008:19:340
103022617,12sleep00-21swapper/008:19:340
95226012,33sleep21030-21sendmail_mailtr11:14:232
81726015,24sleep30-21swapper/307:49:163
304012606,13sleep30-21swapper/307:39:233
1988226017,19sleep20-21swapper/208:49:242
1638126014,27sleep115903-21apt-get06:59:001
1578126015,19sleep20-21swapper/212:03:542
1501826018,20sleep00-21swapper/010:18:530
3171725914,22sleep30-21swapper/307:44:163
1855725922,23sleep30-21swapper/310:29:143
131182597,12sleep00-21swapper/010:10:220
1044625910,33sleep310541-21cat08:19:373
1044625910,33sleep310541-21cat08:19:373
3216125815,27sleep032254-21memory09:29:230
2291725816,20sleep00-21swapper/012:24:270
2291725816,20sleep00-21swapper/012:24:270
1318125817,24sleep30-21swapper/308:29:263
565225714,27sleep05647-21apt-get09:49:060
3060525723,18sleep20-21swapper/209:24:202
2158825721,20sleep00-21swapper/010:39:120
2131325716,25sleep221462-21ls07:13:542
1677825715,15sleep20-21swapper/208:39:222
1183925717,26sleep311868-21munin-run10:08:513
306125618,19sleep30-21swapper/309:39:233
2436025614,18sleep00-21swapper/009:04:160
1497425613,28sleep215012-21modprobe10:18:512
1417525619,19sleep30-21swapper/310:14:203
944925512,16sleep00-21swapper/009:59:190
47092556,12sleep20-21swapper/208:02:122
365525513,27sleep03758-21perl11:24:110
365525513,27sleep03758-21perl11:24:110
305222556,37sleep00-21swapper/011:05:250
2757625513,26sleep027578-21sshd10:57:250
2757625513,26sleep027578-21sshd10:57:250
2722925512,17sleep00-21swapper/009:14:090
2233225516,19sleep00-21swapper/008:58:550
2022725514,23sleep30-21swapper/307:09:193
200725513,25sleep32002-21apt-get11:19:093
200725513,25sleep32002-21apt-get11:19:093
1220025515,24sleep212207-21df_inode10:09:102
670125412,24sleep06676-21apt-get08:09:200
670125412,24sleep06676-21apt-get08:09:200
45625412,24sleep3451-21apt-get11:14:083
3164925413,25sleep231716-21memory07:44:152
3013825415,21sleep20-21swapper/211:04:222
191022549,24sleep30-21swapper/308:48:523
77232535,35sleep30-21swapper/308:10:573
77232535,35sleep30-21swapper/308:10:573
2720425314,17sleep00-21swapper/007:29:230
2720425314,17sleep00-21swapper/007:29:230
225025314,23sleep22277-21irqstats11:19:162
225025314,23sleep22277-21irqstats11:19:162
1547725314,23sleep015534-21irqstats10:19:140
153922536,36sleep00-21swapper/011:59:310
115292538,10sleep20-21swapper/210:04:502
109125310,17sleep30-21swapper/309:34:153
109125310,17sleep30-21swapper/309:34:153
1058225311,24sleep010576-21apt-get10:04:080
510225212,23sleep25099-21apt-get11:29:052
363725216,22sleep30-21swapper/311:24:093
363725216,22sleep30-21swapper/311:24:093
2714225213,24sleep327202-21/usr/sbin/munin07:29:223
2714225213,24sleep327202-21/usr/sbin/munin07:29:223
3000925115,20sleep20-21swapper/207:39:142
2563925110,24sleep325636-21apt-get09:09:073
2194325113,23sleep321954-21ntp_states10:39:203
2071825114,17sleep20-21swapper/212:19:082
2044925115,20sleep220475-21systemd-run12:18:522
1526725112,22sleep215261-21apt-get10:19:082
1754225013,21sleep317575-21munin-run08:43:513
165562508,17sleep00-21swapper/008:39:160
868924812,21sleep08695-21ntp_states11:39:210
3003824814,19sleep230081-21systemd-run09:23:522
2919624812,20sleep329233-21ntp_states09:19:213
1154224813,16sleep30-21swapper/311:49:133
1154224813,16sleep30-21swapper/311:49:133
2848724617,16sleep20-21swapper/209:18:512
3265024315,15sleep00-21swapper/007:48:530
6702419,16sleep00-21swapper/007:49:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional