You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-16 - 16:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sun Nov 16, 2025 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13362246356,4596sleep06899-21kworker/0:107:14:470
14495218187,1798sleep0195-21jbd2/mmcblk2p2-12:29:040
125572123012,1206sleep21703-21kworker/2:008:57:472
8001238324,228sleep00-21swapper/008:44:000
86432337143,171sleep00-21swapper/007:04:010
27164233711,213sleep024040-21kworker/u8:209:43:590
14870229713,270sleep114901-21memory12:29:261
1681922827,262sleep016829-21find10:54:100
134762269126,125sleep013637-21latency_hist10:44:010
1313225630,32sleep00-21swapper/010:04:170
14239221114,159sleep214244-21modprobe12:28:582
27000220453,29sleep00-21swapper/011:24:300
20322219710,37sleep019928-21/usr/sbin/munin11:04:230
21504219611,166sleep021555-21sort11:09:160
19945219418,163sleep120009-21awk09:19:461
2981219142,30sleep00-21swapper/010:09:220
9266218425,136sleep00-21swapper/007:04:250
3066217622,133sleep23088-21sendmail11:53:582
2253921768,154sleep022684-21latency_hist09:29:010
2253921768,154sleep022684-21latency_hist09:29:010
3067421739,16sleep00-21swapper/009:54:200
29318217115,137sleep00-21swapper/008:09:000
4937216615,124sleep00-21swapper/010:14:300
29828216612,24sleep029900-21latency11:34:230
2384221619,131sleep00-21swapper/007:49:300
2134621615,12sleep00-21swapper/007:42:240
2134621615,12sleep00-21swapper/007:42:240
16708216017,118sleep00-21swapper/009:09:350
2618021560,22sleep015302-1kworker/0:2H07:58:580
2618021560,22sleep015302-1kworker/0:2H07:58:580
23390215210,129sleep023512-21cat09:29:300
23390215210,129sleep023512-21cat09:29:300
832121497,104sleep00-21swapper/008:44:160
11436214715,69sleep311438-21grep12:19:183
31942214611,120sleep132096-21latency_hist09:59:011
119182140103,19sleep00-21swapper/012:19:310
5365213931,66sleep00-21swapper/008:34:230
3082521395,99sleep00-21swapper/008:11:020
23478213917,109sleep323523-21grep11:14:403
1892921376,15sleep00-21swapper/009:16:130
2815221356,96sleep00-21swapper/009:44:310
1020821359,67sleep00-21swapper/008:49:250
1506621339,21sleep00-21swapper/010:49:010
20760213210,81sleep020763-21munin-run09:24:000
483421318,55sleep04909-21systemctl08:33:590
21097213019,92sleep321141-21latency_hist09:24:043
5022212913,100sleep15125-21timerwakeupswit10:14:311
335221275,27sleep00-21swapper/011:54:040
932021245,85sleep00-21swapper/012:10:560
21196212419,86sleep120719-21apt-get09:24:101
21196212419,86sleep120719-21apt-get09:24:101
4745212311,98sleep24755-21needreboot10:14:242
29758212320,82sleep129273-21apt-get08:09:091
1614221238,69sleep00-21swapper/007:24:310
26635212282,25sleep326683-21if_eth011:24:213
11587212280,19sleep00-21swapper/008:54:220
96021208,97sleep01038-21mailstats08:19:300
96021208,97sleep01038-21mailstats08:19:300
19660212014,88sleep319700-21modprobe11:03:583
19272212021,82sleep119315-21sed09:19:011
12742212024,71sleep10-21swapper/112:24:011
633421199,75sleep0362-21snmpd12:02:320
1080521197,76sleep00-21swapper/012:14:390
23296211824,74sleep122788-21apt-get11:14:091
2203421179,74sleep022097-21latency07:44:220
1492921175,12sleep00-21swapper/007:19:590
1942321166,13sleep00-21swapper/011:00:520
10707211621,70sleep110771-21fschecks_time10:34:211
2488321158,64sleep00-21swapper/011:19:170
13798211511,89sleep1362-21snmpd10:44:181
10273211517,78sleep310303-21proc_pri08:49:283
5886211319,73sleep26042-21unixbench_multi11:59:442
2786821130,7sleep20-21swapper/208:04:012
3229321115,93sleep00-21swapper/011:43:020
1429021117,70sleep014288-21/usr/sbin/munin12:29:000
12519211115,81sleep112608-21fschecks_time07:14:191
23526211020,73sleep023572-21ls11:14:410
7072210911,16sleep20-21swapper/208:39:262
23918210916,78sleep123999-21wc07:49:311
13066210919,68sleep313098-21fschecks_count12:24:193
911321086,62sleep00-21swapper/010:29:200
911321086,62sleep00-21swapper/010:29:200
2819121078,57sleep00-21swapper/011:29:220
202521077,60sleep00-21swapper/008:24:140
1911921079,58sleep00-21swapper/007:34:290
11525210758,26sleep00-21swapper/007:09:460
3943210619,74sleep23966-21netstat08:29:252
23513210525,58sleep123663-21sed09:29:311
23513210525,58sleep123663-21sed09:29:311
1618421055,11sleep00-21swapper/010:49:340
31377210353,34sleep00-21swapper/008:14:130
1887721036,16sleep30-21swapper/307:34:243
3265721019,24sleep00-21swapper/011:44:160
22103210013,71sleep322181-21find11:09:313
1418321006,84sleep00-21swapper/009:00:390
1418321006,84sleep00-21swapper/009:00:390
478229721,53sleep34932-21cmp11:59:003
184272979,15sleep00-21swapper/007:34:000
1502229665,16sleep00-21swapper/009:04:270
985429516,22sleep39873-21apt-get12:14:103
74812958,15sleep00-21swapper/010:24:150
74812958,15sleep00-21swapper/010:24:150
2165729513,64sleep221690-21latency_hist07:44:032
2165729513,64sleep221690-21latency_hist07:44:032
518529418,61sleep35248-21cpuspeed11:59:163
1510529420,60sleep115121-21/usr/sbin/munin09:04:281
2562629316,58sleep125779-21sshd11:19:341
2454829311,69sleep224550-21sshd07:53:442
904129215,57sleep39158-21users08:44:323
802629232,28sleep10-21swapper/108:44:011
154329215,55sleep11681-21cat11:49:011
75729113,64sleep2835-21switchtime11:44:302
18222916,13sleep00-21swapper/011:49:160
1449829114,62sleep214509-21systemd09:04:002
1449829114,62sleep214509-21systemd09:04:002
1020629113,59sleep310313-21latency_hist07:09:003
2615529026,32sleep30-21swapper/307:58:583
2615529026,32sleep30-21swapper/307:58:583
1660329017,57sleep216721-21date10:54:002
1089029059,21sleep010921-21latency10:34:240
258442899,70sleep217973-21kworker/dying11:20:122
3255028814,57sleep132595-21/usr/sbin/munin09:59:261
2953228813,60sleep129533-21/usr/sbin/munin11:34:111
321432876,13sleep00-21swapper/009:59:120
277522878,68sleep027756-21munin-run08:04:000
2618828723,32sleep30-21swapper/311:24:003
71802868,11sleep00-21swapper/012:04:260
1600328621,36sleep316030-21munin-run09:08:583
2707128520,43sleep127199-21/usr/sbin/munin11:24:311
1286228522,30sleep30-21swapper/308:59:013
384028411,60sleep23847-21ntp_states11:54:272
3263428422,42sleep132661-21munin-run08:18:581
2900828438,24sleep00-21swapper/009:49:150
253432846,12sleep00-21swapper/009:35:310
2329828426,30sleep20-21swapper/211:14:102
165612846,21sleep10-21swapper/109:09:321
1586528420,48sleep315917-21ls12:34:003
69402836,12sleep00-21swapper/010:20:480
3105628322,45sleep231091-21awk08:13:592
1122928332,27sleep10-21swapper/107:09:271
2773028218,18sleep20-21swapper/211:29:002
2185628228,28sleep10-21swapper/107:44:191
1305128219,47sleep113060-21df_inode12:24:171
701128011,16sleep10-21swapper/112:04:231
3150328016,49sleep131543-21ntp_kernel_pll_11:39:241
2806128030,28sleep30-21swapper/309:44:303
212622797,25sleep30-21swapper/309:24:403
212622797,25sleep30-21swapper/309:24:403
417727815,47sleep34330-21cat10:14:013
270927820,37sleep22870-21grep08:24:322
1816427817,46sleep218221-21latency09:14:262
1098927711,51sleep20-21swapper/210:34:262
2948227613,48sleep129541-21sendmail_mailqu09:49:291
97182756,12sleep20-21swapper/210:29:342
97182756,12sleep20-21swapper/210:29:342
856827526,31sleep28602-21munin-run07:03:592
262927510,39sleep10-21swapper/108:24:311
2940027425,25sleep10-21swapper/108:09:011
226327424,25sleep10-21swapper/111:49:291
1491127417,28sleep20-21swapper/209:04:252
826027320,29sleep10-21swapper/112:09:111
723627226,28sleep30-21swapper/312:04:293
3267827212,45sleep332710-21date08:19:003
2160127229,29sleep121656-21wc07:44:021
2160127229,29sleep121656-21wc07:44:021
1342127218,40sleep113433-21modprobe10:43:581
3266527121,34sleep2284-21dbus-daemon08:18:592
1855627112,23sleep30-21swapper/309:14:393
1417827133,24sleep313691-21/usr/sbin/munin10:44:283
12462719,48sleep10-21swapper/110:04:141
3085727019,38sleep330946-21sed09:54:233
3026227021,32sleep330320-21basename09:53:593
1819827021,25sleep20-21swapper/210:59:012
1677327021,35sleep316809-21wc09:09:393
2510326819,20sleep10-21swapper/107:54:241
2510326819,20sleep10-21swapper/107:54:241
2325426825,18sleep20-21swapper/209:29:282
2325426825,18sleep20-21swapper/209:29:282
2276926817,34sleep222788-21apt-get11:13:582
2031026824,24sleep30-21swapper/307:39:203
2031026824,24sleep30-21swapper/307:39:203
1599726813,40sleep116004-21/usr/sbin/munin07:24:281
1144026726,21sleep30-21swapper/307:09:453
785126614,30sleep30-21swapper/310:24:263
785126614,30sleep30-21swapper/310:24:263
3241426626,23sleep20-21swapper/209:59:232
2992726616,22sleep20-21swapper/211:34:252
2705326617,35sleep127145-21cat07:59:331
1347026625,23sleep30-21swapper/308:59:333
1347026625,23sleep30-21swapper/308:59:333
74726519,28sleep2836-21/usr/sbin/munin08:19:242
74726519,28sleep2836-21/usr/sbin/munin08:19:242
405826513,38sleep14131-21systemd-run10:13:591
1374326523,17sleep20-21swapper/207:19:002
134526525,22sleep30-21swapper/310:04:203
3156026413,35sleep331566-21ntp_states11:39:263
2121126420,30sleep3284-21dbus-daemon11:08:593
1861326419,27sleep20-21swapper/210:59:212
1398426420,27sleep10-21swapper/107:19:161
840626321,20sleep20-21swapper/212:09:202
378226313,22sleep30-21swapper/308:29:223
2378126318,22sleep30-21swapper/307:49:293
1380326324,25sleep213850-21cut10:44:192
883226224,19sleep10-21swapper/108:44:301
66132627,12sleep10-21swapper/110:19:321
646326221,22sleep10-21swapper/108:39:001
597826215,31sleep26004-21df10:19:172
580626217,31sleep15885-21grep11:59:431
413226216,25sleep20-21swapper/210:14:002
2001026224,20sleep20-21swapper/209:19:472
645926115,21sleep30-21swapper/310:19:303
364726120,22sleep10-21swapper/111:54:221
1576326115,22sleep30-21swapper/310:49:293
152026118,25sleep21614-21memory10:04:232
83626018,26sleep3963-21sed11:44:313
2799826016,28sleep327993-21apt-get08:04:153
255332608,43sleep10-21swapper/109:37:581
2482026015,25sleep20-21swapper/209:34:282
182582608,40sleep10-21swapper/107:32:521
1205026024,21sleep20-21swapper/212:19:342
2193025915,23sleep20-21swapper/207:44:202
144802596,12sleep20-21swapper/207:19:292
1266825919,22sleep10-21swapper/108:58:591
1049325918,19sleep20-21swapper/212:14:292
299132587,11sleep30-21swapper/308:09:243
2830825815,24sleep128361-21users09:44:321
223072589,13sleep30-21swapper/307:44:293
1773425822,16sleep20-21swapper/207:29:342
1500925817,25sleep215070-21grep12:29:312
89092579,34sleep29040-21timerwakeupswit08:44:312
2957825715,24sleep229719-21sed08:09:022
1745825720,22sleep30-21swapper/307:29:273
980825615,24sleep19802-21apt-get08:49:141
561125620,21sleep25617-21ntp_states08:34:292
3047725618,23sleep3225-21systemd-journal11:34:383
2925525611,29sleep229278-21irqstats09:49:222
2795125612,26sleep20-21swapper/211:29:122
2415525614,26sleep324310-21latency_hist09:34:013
1742625615,25sleep316820-21/usr/sbin/munin10:54:293
1639125615,25sleep216438-21users07:24:342
1334725615,20sleep10-21swapper/108:59:311
1334725615,20sleep10-21swapper/108:59:311
103925622,17sleep10-21swapper/108:19:311
103925622,17sleep10-21swapper/108:19:311
862025518,23sleep38638-21munin-run07:04:003
67572550,10sleep30-21swapper/308:39:183
366125516,19sleep10-21swapper/108:29:201
3114025520,18sleep20-21swapper/209:54:302
2797625511,22sleep10-21swapper/111:29:151
227682555,38sleep10-21swapper/111:13:571
178425520,18sleep30-21swapper/308:24:013
178425520,18sleep30-21swapper/308:24:013
671325412,24sleep26731-21apt-get12:04:122
54202547,10sleep30-21swapper/310:16:113
2641225419,19sleep20-21swapper/209:39:292
2534125410,28sleep20-21swapper/207:54:292
2534125410,28sleep20-21swapper/207:54:292
198872546,16sleep20-21swapper/207:38:342
1967025415,26sleep119671-21sshd07:35:471
1924425410,16sleep10-21swapper/110:59:321
1815725416,18sleep10-21swapper/110:59:001
110442547,14sleep10-21swapper/112:17:431
1061925413,24sleep310614-21apt-get10:34:153
100982546,17sleep10-21swapper/110:31:271
100982546,17sleep10-21swapper/110:31:271
950525311,26sleep39537-21/usr/sbin/munin10:29:313
950525311,26sleep39537-21/usr/sbin/munin10:29:313
79162530,11sleep10-21swapper/110:24:271
79162530,11sleep10-21swapper/110:24:271
776925323,17sleep10-21swapper/106:59:261
2588925314,23sleep125901-21apt-get09:39:111
221225313,23sleep32218-21ntp_states11:49:273
202125315,22sleep32020-21apt-get08:24:133
1772025318,17sleep20-21swapper/209:14:012
1574525317,17sleep30-21swapper/307:24:223
1284425317,15sleep30-21swapper/310:39:323
120752536,11sleep20-21swapper/207:13:312
1048925311,20sleep20-21swapper/208:49:312
886825212,17sleep10-21swapper/107:04:101
37125215,15sleep30-21swapper/309:59:323
2126725211,24sleep221216-21apt-get09:24:432
2126725211,24sleep221216-21apt-get09:24:432
209742526,34sleep10-21swapper/111:05:541
1225325215,17sleep20-21swapper/210:39:192
561825116,21sleep35631-21open_files08:34:303
2475725119,18sleep30-21swapper/309:34:253
2285225114,21sleep322858-21df_inode09:29:163
209232515,15sleep20-21swapper/211:05:142
1526825116,17sleep10-21swapper/110:49:151
934325019,15sleep20-21swapper/207:04:272
2598525012,22sleep325992-21diskstats09:39:173
3244024913,17sleep20-21swapper/211:44:012
288032498,17sleep20-21swapper/209:49:012
828424811,21sleep38281-21apt-get12:09:133
815424818,16sleep20-21swapper/210:24:332
815424818,16sleep20-21swapper/210:24:332
2844624813,16sleep30-21swapper/311:29:293
2691524714,16sleep30-21swapper/307:59:313
1265124612,17sleep30-21swapper/307:14:213
1441124414,15sleep30-21swapper/307:19:263
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional