You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 17:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Thu Dec 04, 2025 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22866276616,732sleep022879-21/usr/sbin/munin10:13:270
22866276616,732sleep022879-21/usr/sbin/munin10:13:270
13025252514,497sleep213018-21apt-get09:43:122
13025252514,497sleep213018-21apt-get09:43:122
3140725208,496sleep230397-21/usr/sbin/munin10:39:172
17446247018,405sleep317593-21grep08:13:293
27811245017,417sleep027829-21/usr/sbin/munin08:48:100
27811245017,417sleep027829-21/usr/sbin/munin08:48:100
27980241517,380sleep228026-21install07:07:582
2417235629,287sleep32550-21latency_hist10:52:583
15053234331,211sleep315117-21sendmail_mailqu09:48:263
15053234331,211sleep315117-21sendmail_mailqu09:48:263
1060323038,244sleep310700-21sed09:34:213
17980230228,28sleep30-21swapper/309:58:203
141592264106,58sleep30-21swapper/309:46:203
141592264106,58sleep30-21swapper/309:46:203
965022600,94sleep326432-21cyclictest09:32:593
965022600,94sleep326432-21cyclictest09:32:593
25398225640,205sleep322178-1kworker/3:2H10:22:593
13325225579,55sleep30-21swapper/311:27:583
7890224742,90sleep38038-21latency_hist09:27:593
30835222280,90sleep330968-21apt-config08:57:593
15374221447,151sleep215388-21irqstats08:08:192
2984221375,18sleep30-21swapper/307:29:033
2984221375,18sleep30-21swapper/307:29:033
22487220651,102sleep322580-21grep10:13:183
22487220651,102sleep322580-21grep10:13:183
2576621968,176sleep05666-21snmpd12:07:520
23047218921,150sleep123127-21tune2fs11:58:161
145821819,157sleep31618-21latency_hist12:32:583
4064217612,148sleep14150-21ls09:13:271
1578217414,93sleep30-21swapper/309:07:563
1578217414,93sleep30-21swapper/309:07:563
7036217214,141sleep37047-21modprobe07:42:563
4813216711,140sleep24891-21find10:58:272
27924216542,41sleep30-21swapper/308:48:223
27924216542,41sleep30-21swapper/308:48:223
12386216512,140sleep012397-21needreboot07:58:230
12386216512,140sleep012397-21needreboot07:58:230
7400216021,118sleep10-21swapper/111:08:181
7400216021,118sleep10-21swapper/111:08:181
83821598,99sleep3977-21latency_hist10:47:583
27007215927,106sleep127139-21unixbench_multi12:08:411
16257215748,29sleep3910-21sendmail-mta11:36:183
19556215665,53sleep30-21swapper/308:22:573
247421508,24sleep30-21swapper/309:08:293
4894214951,77sleep35036-21unixbench_multi10:58:283
10875214910,124sleep30-21swapper/307:53:223
10875214910,124sleep30-21swapper/307:53:223
16384214742,89sleep3225-21systemd-journal08:12:563
29901214310,117sleep30-21swapper/312:18:253
26717214319,12sleep30-21swapper/310:27:193
632121368,114sleep36393-21latency_hist09:22:583
4379213516,38sleep30-21swapper/309:13:373
26359213350,71sleep326458-21fschecks_time08:43:173
26359213350,71sleep326458-21fschecks_time08:43:173
27268212920,44sleep30-21swapper/310:28:203
12655212949,56sleep30-21swapper/307:58:303
12655212949,56sleep30-21swapper/307:58:303
32245212818,14sleep30-21swapper/312:27:493
29982212845,67sleep330129-21cat07:12:593
1989212820,86sleep01997-21uniq09:08:170
32518212620,80sleep10-21swapper/112:28:081
25070212628,86sleep325113-21cat08:39:173
20898212616,56sleep30-21swapper/310:08:173
17798212619,88sleep017863-21fschecks_time09:58:170
20198212568,38sleep30-21swapper/308:23:223
1614221258,100sleep30-21swapper/309:53:073
13291212418,86sleep10-21swapper/109:43:191
13291212418,86sleep10-21swapper/109:43:191
22712212230,42sleep322721-21munin-run11:57:573
9118212110,12sleep00-21swapper/009:29:420
9118212110,12sleep00-21swapper/009:29:420
24714212123,79sleep024208-21apt-get08:38:090
28030212047,26sleep328042-21munin-node07:07:593
19654211920,82sleep019692-21ntpq10:03:210
19654211920,82sleep019692-21ntpq10:03:210
4567211824,38sleep30-21swapper/307:33:233
32253211822,72sleep132283-21idleruntime-cro12:27:551
31176211824,43sleep30-21swapper/308:58:073
2081321186,13sleep30-21swapper/311:49:263
19587211815,88sleep019721-21cut11:47:570
19587211815,88sleep019721-21cut11:47:570
594211643,60sleep31-21systemd07:22:573
20608211610,84sleep30-21swapper/310:08:013
20608211610,84sleep30-21swapper/310:08:013
1684211610,89sleep21759-21find10:48:292
7065211415,79sleep17082-21latency_hist11:07:581
26905211320,72sleep227029-21cat08:43:282
26905211320,72sleep227029-21cat08:43:282
29311211119,72sleep129445-21sed07:08:291
7418211012,82sleep07445-21fschecks_count07:43:170
11561211027,65sleep311646-21fschecks_time09:38:173
11561211027,65sleep311646-21fschecks_time09:38:173
14472210926,44sleep314487-21awk08:03:313
28220210816,62sleep10-21swapper/108:48:281
28220210816,62sleep10-21swapper/108:48:281
2163210619,36sleep30-21swapper/307:27:583
2163210619,36sleep30-21swapper/307:27:583
20427210413,78sleep020504-21grep11:48:260
25496210218,41sleep10-21swapper/110:23:091
992721005,49sleep30-21swapper/311:14:583
5848210054,23sleep30-21swapper/311:03:163
1410121006,57sleep314174-21cat11:28:273
613229918,59sleep05738-21/usr/sbin/munin11:03:210
548829923,56sleep05498-21modprobe07:37:560
3039529819,60sleep129909-21apt-get10:38:071
1036729810,73sleep010412-21ntp_kernel_pll_09:34:140
1255129514,59sleep012624-21ls11:23:270
107629510,72sleep00-21swapper/007:23:180
107629510,72sleep00-21swapper/007:23:180
1729229436,43sleep30-21swapper/311:38:283
1729229436,43sleep30-21swapper/311:38:283
432029317,60sleep24343-21fschecks_count07:33:182
265229317,42sleep20-21swapper/207:28:132
265229317,42sleep20-21swapper/207:28:132
641529214,64sleep26538-21timerwakeupswit07:38:272
1021929237,29sleep10-21swapper/107:52:591
273392910,8sleep10-21swapper/110:28:211
2627029113,59sleep30-21swapper/312:08:073
2622629018,51sleep026372-21grep10:23:280
93528916,55sleep01062-21timerwakeupswit09:03:290
93528916,55sleep01062-21timerwakeupswit09:03:290
819828811,61sleep28264-21fschecks_time09:28:182
819828811,61sleep28264-21fschecks_time09:28:182
56828816,42sleep2593-21kworker/u8:207:22:562
1861328811,61sleep118622-21/usr/sbin/munin08:18:191
1235928810,64sleep212371-21needreboot11:23:212
476928718,49sleep04739-21latency_hist09:17:580
295382870,6sleep00-21swapper/012:18:160
3165228618,46sleep031773-21cat12:23:300
2581228526,34sleep125838-21munin-run12:07:571
2634728435,31sleep126382-21munin-run07:02:561
2333228412,33sleep323372-21ntp_kernel_pll_08:33:233
1323328414,52sleep013290-21/usr/sbin/munin09:43:180
1323328414,52sleep013290-21/usr/sbin/munin09:43:180
2419528322,43sleep024245-21apt-config08:37:570
2303128218,37sleep223132-21grep08:33:182
83828128,33sleep2857-21apt-get07:23:082
83828128,33sleep2857-21apt-get07:23:082
52428120,16sleep20-21swapper/209:03:192
52428120,16sleep20-21swapper/209:03:192
3075828114,51sleep130896-21latency_hist12:22:591
265428124,41sleep30-21swapper/310:53:113
752528019,43sleep30-21swapper/311:08:203
752528019,43sleep30-21swapper/311:08:203
2989628015,38sleep229909-21apt-get10:37:562
233828027,34sleep02344-21modprobe10:52:550
108442807,57sleep310887-21ntpq11:18:223
313012796,55sleep35666-21snmpd07:15:463
313012796,55sleep35666-21snmpd07:15:463
2989527817,43sleep029928-21munin-run07:12:560
2108127721,28sleep20-21swapper/210:08:202
1957627720,43sleep30-21swapper/311:47:563
1957627720,43sleep30-21swapper/311:47:563
1177927723,25sleep00-21swapper/007:57:590
1177927723,25sleep00-21swapper/007:57:590
3173927622,15sleep30-21swapper/310:42:573
2913527618,11sleep30-21swapper/312:17:403
2209227619,33sleep122228-21sshd08:28:291
1583227627,30sleep00-21swapper/009:52:550
1583227627,30sleep00-21swapper/009:52:550
898127512,49sleep29064-21fschecks_time07:48:162
2162227545,16sleep30-21swapper/308:28:193
1980227522,33sleep119796-21apt-get08:23:121
1902327519,25sleep20-21swapper/210:02:592
1801327520,31sleep00-21swapper/008:17:580
1643927510,54sleep216510-21cat11:37:582
1176027517,41sleep011847-21grep11:22:590
842427415,11sleep30-21swapper/307:45:513
752927410,15sleep20-21swapper/207:43:192
2563727428,24sleep30-21swapper/306:58:263
2424627419,35sleep124326-21rm08:37:581
2404027418,43sleep30-21swapper/312:00:363
906727336,18sleep30-21swapper/307:48:173
3270827317,34sleep1391-21grep10:43:281
249282738,12sleep30-21swapper/312:03:233
2958927218,9sleep30-21swapper/310:33:573
2881027223,30sleep127870-21/usr/sbin/munin12:13:301
2683127215,43sleep126904-21wc08:43:271
2683127215,43sleep126904-21wc08:43:271
2375827223,19sleep20-21swapper/210:18:012
2375827223,19sleep20-21swapper/210:18:012
508527124,26sleep10-21swapper/109:18:161
3090027119,31sleep131056-21grep07:13:301
3090027119,31sleep131056-21grep07:13:301
225822717,10sleep20-21swapper/211:56:472
1109127115,39sleep111218-21/usr/sbin/munin11:18:281
863727016,39sleep18731-21cat07:47:591
1383427022,24sleep00-21swapper/008:03:200
309026911,46sleep13136-21ntp_kernel_pll_07:29:051
309026911,46sleep13136-21ntp_kernel_pll_07:29:051
3048926918,22sleep00-21swapper/007:13:210
3048926918,22sleep00-21swapper/007:13:210
303752697,15sleep20-21swapper/207:13:192
303752697,15sleep20-21swapper/207:13:192
3008126920,25sleep10-21swapper/108:54:011
1683226923,24sleep20-21swapper/209:53:262
1521326916,40sleep115284-21fschecks_time11:33:171
917226824,21sleep00-21swapper/011:13:220
552126822,26sleep00-21swapper/009:18:270
2681926820,22sleep20-21swapper/210:27:582
2676726814,37sleep026829-21sendmail_mailst08:43:260
2676726814,37sleep026829-21sendmail_mailst08:43:260
1660326813,39sleep116610-21/usr/sbin/munin09:53:211
1487126819,23sleep10-21swapper/108:07:571
392526715,34sleep03973-21systemd-run10:57:560
2827326717,24sleep00-21swapper/012:13:210
2443226714,21sleep00-21swapper/010:18:360
2275726721,29sleep10-21swapper/111:58:001
1646426711,43sleep116561-21cat08:12:581
1320526711,13sleep10-21swapper/108:02:001
1320526711,13sleep10-21swapper/108:02:001
3097926619,19sleep00-21swapper/008:58:000
2765726612,29sleep20-21swapper/212:12:582
2589126615,25sleep20-21swapper/212:07:582
2525926620,24sleep20-21swapper/210:22:582
1888326613,35sleep019011-21sed08:18:260
1548626615,20sleep00-21swapper/008:08:210
1199326615,21sleep20-21swapper/207:58:092
1199326615,21sleep20-21swapper/207:58:092
2704426515,36sleep127072-21nfs4_client07:03:201
1798826520,23sleep10-21swapper/111:42:581
1798826520,23sleep10-21swapper/111:42:581
1354926516,34sleep113615-21sort11:28:131
898826421,24sleep10-21swapper/111:13:191
633826413,36sleep16414-21switchtime07:38:261
629326418,22sleep20-21swapper/211:03:272
618126415,30sleep06219-21ntp_kernel_pll_07:38:200
2476726410,43sleep15666-21snmpd08:38:421
106326414,22sleep10-21swapper/109:03:301
106326414,22sleep10-21swapper/109:03:301
991826317,29sleep19912-21apt-get09:33:101
838126321,24sleep10-21swapper/109:28:211
838126321,24sleep10-21swapper/109:28:211
3259126316,29sleep032707-21sed10:43:270
325326315,34sleep13361-21latency_hist09:12:591
945926216,21sleep00-21swapper/007:48:260
547426229,20sleep0284-21dbus-daemon11:02:570
399326217,21sleep00-21swapper/007:32:590
399326217,21sleep00-21swapper/007:32:590
397626210,36sleep24079-21latency_hist10:57:582
313626218,19sleep20-21swapper/209:12:562
2571826219,15sleep00-21swapper/006:58:270
2143826221,22sleep00-21swapper/008:28:150
391926119,23sleep20-21swapper/209:13:242
3260926115,22sleep00-21swapper/007:18:300
3126026113,20sleep00-21swapper/010:39:150
3074026117,25sleep230834-21apt-config08:57:572
304582616,44sleep00-21swapper/008:55:050
286102618,12sleep10-21swapper/110:33:061
2659726114,32sleep026609-21cut12:08:320
252052615,12sleep20-21swapper/208:39:202
1982426118,25sleep119818-21apt-get11:48:111
1480726126,20sleep00-21swapper/011:32:560
664626018,18sleep20-21swapper/209:23:172
3121026013,27sleep10-21swapper/108:58:131
290232607,11sleep20-21swapper/212:16:132
171392607,13sleep00-21swapper/011:38:250
171392607,13sleep00-21swapper/011:38:250
1665826019,19sleep00-21swapper/009:53:240
112032609,12sleep10-21swapper/107:53:281
112032609,12sleep10-21swapper/107:53:281
10512606,36sleep20-21swapper/212:29:062
2759525914,27sleep027667-21/usr/sbin/munin10:28:290
1987325915,29sleep219981-21cat10:03:272
1987325915,29sleep219981-21cat10:03:272
739525818,25sleep07399-21diskstats11:08:170
739525818,25sleep07399-21diskstats11:08:170
3139625811,33sleep231442-21grep08:58:462
242225814,20sleep10-21swapper/107:28:001
242225814,20sleep10-21swapper/107:28:001
627025717,21sleep20-21swapper/209:22:572
472025719,18sleep10-21swapper/110:58:261
2824325716,22sleep20-21swapper/207:08:022
2342925714,19sleep20-21swapper/211:58:252
1692325711,31sleep216946-21irqstats11:38:192
1692325711,31sleep216946-21irqstats11:38:192
1638825710,20sleep00-21swapper/011:37:560
1283625712,23sleep10-21swapper/109:42:591
1283625712,23sleep10-21swapper/109:42:591
54025614,27sleep0547-21ntp_states12:28:230
2863525615,25sleep028630-21apt-get10:33:090
2774525621,18sleep20-21swapper/210:28:312
2521325620,18sleep20-21swapper/206:58:172
2156325615,21sleep20-21swapper/208:28:182
1933625617,23sleep119342-21diskstats10:03:141
1933625617,23sleep119342-21diskstats10:03:141
1862825616,21sleep20-21swapper/211:43:232
1862825616,21sleep20-21swapper/211:43:232
1193125614,23sleep20-21swapper/209:38:242
1193125614,23sleep20-21swapper/209:38:242
1026625615,22sleep20-21swapper/209:34:122
1015825615,16sleep20-21swapper/211:17:562
312325520,16sleep10-21swapper/110:53:251
2029125518,16sleep20-21swapper/211:48:242
1732225517,19sleep20-21swapper/208:13:282
1518525516,19sleep10-21swapper/109:48:281
1518525516,19sleep10-21swapper/109:48:281
1490225515,21sleep20-21swapper/208:07:582
3070425412,18sleep20-21swapper/212:22:562
1693225414,24sleep116974-21irqstats08:13:181
1144325417,22sleep211444-21sshd07:54:172
1144325417,22sleep211444-21sshd07:54:172
321172535,36sleep20-21swapper/212:26:112
2665025314,24sleep026669-21apt-get07:03:070
2296125313,23sleep022955-21apt-get11:58:110
2134925312,27sleep121421-21sensors_temp10:08:271
1578725315,23sleep215789-21sshd09:52:212
1578725315,23sleep215789-21sshd09:52:212
67972529,19sleep00-21swapper/009:23:190
431325214,22sleep14319-21diskstats07:33:171
1843325216,16sleep20-21swapper/208:18:162
1822425212,23sleep118219-21apt-get11:43:101
1822425212,23sleep118219-21apt-get11:43:101
1375425216,17sleep20-21swapper/211:28:182
1199825212,24sleep111992-21apt-get11:23:121
1147825211,16sleep00-21swapper/009:38:140
1147825211,16sleep00-21swapper/009:38:140
656025111,24sleep16554-21apt-get09:23:121
2808425112,23sleep228113-21irqstats08:48:252
2808425112,23sleep228113-21irqstats08:48:252
218182519,17sleep00-21swapper/011:53:220
1901725112,23sleep119022-21munin-run10:02:581
107825112,22sleep11073-21apt-get10:48:121
2374824916,15sleep10-21swapper/110:18:001
2374824916,15sleep10-21swapper/110:18:001
2486324816,16sleep10-21swapper/110:18:411
2167124813,19sleep021718-21users10:08:300
2006224812,22sleep019777-21/usr/sbin/munin08:23:190
39524712,20sleep2426-21users10:43:292
3217224710,20sleep132220-21ntp_states07:18:231
198032478,15sleep20-21swapper/208:23:132
1054324714,16sleep20-21swapper/211:18:162
748424612,18sleep20-21swapper/211:08:192
748424612,18sleep20-21swapper/211:08:192
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional