You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-08-26 - 23:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Tue Aug 26, 2025 12:43:54)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2093279311,764sleep12150-21/usr/sbin/munin12:34:151
21812251544,449sleep2362-21snmpd10:09:202
21812251544,449sleep2362-21snmpd10:09:202
21297235219,310sleep2225-21systemd-journal11:53:462
5779234952,194sleep25795-21latency_hist11:03:472
5779234952,194sleep25795-21latency_hist11:03:472
14811232326,284sleep214888-21mailstats08:04:192
14811232326,284sleep214888-21mailstats08:04:192
1079823217,302sleep210840-21fschecks_count11:19:052
2615523127,293sleep2195-21jbd2/mmcblk2p2-08:39:282
31266226268,171sleep231415-21latency_hist08:58:492
149292252162,66sleep20-21swapper/209:49:052
149292252162,66sleep20-21swapper/209:49:052
13056223686,90sleep213177-21latency_hist09:43:482
305592222125,68sleep20-21swapper/207:14:112
13900222110,148sleep213995-21fschecks_time11:29:072
16400221990,36sleep216374-21/usr/sbin/munin09:54:052
30822221244,122sleep230864-21ntp_kernel_err10:39:122
30822221244,122sleep230864-21ntp_kernel_err10:39:122
6527220569,25sleep20-21swapper/207:39:152
6527220569,25sleep20-21swapper/207:39:152
24980220247,142sleep2362-21snmpd12:04:082
25849219811,143sleep225931-21fschecks_time10:24:052
24373219721,156sleep224522-21unixbench_singl08:34:192
1149219613,86sleep21237-21latency_hist07:23:482
29652219234,18sleep20-21swapper/208:53:492
26849219019,103sleep20-21swapper/212:09:202
15957218725,104sleep20-21swapper/208:09:072
12898218718,130sleep213001-21timerandwakeup11:24:182
9477218461,57sleep20-21swapper/207:49:042
20641218314,108sleep20-21swapper/208:24:052
32729217815,114sleep20-21swapper/212:29:112
1589217310,142sleep20-21swapper/207:24:092
5319217151,25sleep20-21swapper/209:18:482
12111217135,21sleep20-21swapper/207:54:332
2447521679,145sleep224505-21sed12:03:472
28092216425,125sleep228196-21date08:48:482
647921619,141sleep0362-21snmpd09:19:510
22716216117,121sleep322755-21cut10:14:093
2961421538,104sleep229757-21sed10:34:232
9161215246,21sleep20-21swapper/207:48:482
29863215214,99sleep20-21swapper/212:19:142
1099215121,114sleep01132-21awk07:23:460
2010821500,7sleep00-21swapper/011:49:020
1308215013,121sleep0823-21apt-get10:48:550
31861214829,24sleep231877-21vmstat12:24:182
12273214815,115sleep311723-21/usr/sbin/munin09:39:163
8381214313,117sleep08501-21latency_hist09:28:480
1876321418,77sleep218852-21cat11:44:132
1454721416,58sleep2362-21snmpd09:48:402
758121379,24sleep20-21swapper/211:09:042
6846213512,20sleep20-21swapper/209:23:492
2936213579,38sleep20-21swapper/207:29:012
6030213399,18sleep20-21swapper/211:04:032
1310213010,105sleep2823-21apt-get10:48:572
4571212710,102sleep20-21swapper/207:34:062
19610212710,89sleep20-21swapper/208:19:172
19610212710,89sleep20-21swapper/208:19:172
16045212712,100sleep216189-21tail11:34:202
22387212611,22sleep20-21swapper/208:29:102
18584212312,72sleep218706-21cat09:59:182
18584212312,72sleep218706-21cat09:59:182
14411212317,90sleep014443-21tune2fs08:04:080
14411212317,90sleep014443-21tune2fs08:04:080
7852212116,85sleep37967-21users09:24:183
9866212013,25sleep09375-21/usr/sbin/munin07:49:150
1703211927,82sleep2362-21snmpd09:07:012
4150211850,52sleep24215-21sendmail-msp10:58:462
2629211811,63sleep20-21swapper/209:09:192
2629211811,63sleep20-21swapper/209:09:192
19623211813,86sleep319722-21fschecks_time10:04:083
2263021169,17sleep20-21swapper/210:14:042
30285211417,78sleep030365-21grep07:13:530
28092211412,60sleep20-21swapper/212:14:092
298621138,84sleep20-21swapper/212:38:482
32210211119,34sleep20-21swapper/210:44:082
29500211113,77sleep329539-21fschecks_count12:19:073
11517211026,23sleep30-21swapper/309:38:483
11517211026,23sleep30-21swapper/309:38:483
28257210915,77sleep328307-21latency_hist07:08:473
1096921097,13sleep20-21swapper/209:34:192
1096921097,13sleep20-21swapper/209:34:192
734421079,14sleep20-21swapper/209:24:092
14368210718,72sleep014444-21sed11:29:170
2775121069,38sleep20-21swapper/210:29:182
4250210514,73sleep14394-21cat07:33:491
1332821058,13sleep20-21swapper/207:59:182
1332821058,13sleep20-21swapper/207:59:182
862421047,35sleep20-21swapper/209:29:062
862421047,35sleep20-21swapper/209:29:062
18245210317,70sleep118277-21latency_hist11:43:471
32656210122,26sleep20-21swapper/209:02:412
12950210119,67sleep113012-21cat07:59:091
12950210119,67sleep113012-21cat07:59:091
933729921,15sleep20-21swapper/211:14:092
933729921,15sleep20-21swapper/211:14:092
1998229911,50sleep20-21swapper/210:04:152
224322986,32sleep20-21swapper/211:54:182
1327229827,37sleep00-21swapper/009:43:580
197829720,54sleep00-21swapper/009:08:490
2167029613,70sleep121725-21fschecks_count11:54:051
1896129514,28sleep00-21swapper/011:44:190
170329532,28sleep00-21swapper/012:34:060
1300329522,48sleep313146-21grep11:24:193
586329424,44sleep05915-21cat11:03:490
586329424,44sleep05915-21cat11:03:490
2831329411,66sleep028457-21cat07:08:480
173552947,51sleep20-21swapper/211:39:122
2844329321,53sleep128528-21fschecks_time08:49:081
252172936,30sleep20-21swapper/210:20:262
2026629313,66sleep3225-21systemd-journal08:23:463
2026629313,66sleep3225-21systemd-journal08:23:463
457529216,55sleep04658-21timerandwakeup09:14:420
422329113,60sleep33906-21/usr/sbin/munin09:14:333
3010229114,60sleep130176-21netstat07:13:501
2952929011,67sleep0362-21snmpd07:10:300
291702900,6sleep00-21swapper/012:18:460
2413329014,58sleep324189-21grep08:34:163
175729012,64sleep11365-21/usr/sbin/munin07:24:121
1376429012,63sleep113802-21processes09:44:151
294722896,30sleep20-21swapper/207:09:472
1483028913,63sleep3362-21snmpd11:30:373
349328818,34sleep30-21swapper/309:13:483
349328818,34sleep30-21swapper/309:13:483
1252828811,53sleep112467-21latency_hist07:58:481
119228811,62sleep11294-21sed09:04:221
3151828727,32sleep331584-21cat12:24:153
185952875,29sleep20-21swapper/208:17:312
3274828612,58sleep1305-21ntp_states07:19:311
2687828624,47sleep126964-21fschecks_time08:44:071
94028422,39sleep11124-21cat10:48:481
1389928411,57sleep313976-21grep09:44:173
456028118,31sleep10-21swapper/110:59:091
456028118,31sleep10-21swapper/110:59:091
2741728122,43sleep127477-21sendmail_mailqu07:04:181
2563628111,16sleep30-21swapper/308:39:123
923428028,32sleep39337-21timerandwakeup09:29:233
923428028,32sleep39337-21timerandwakeup09:29:233
2419628016,49sleep024262-21swap08:34:170
2110928016,45sleep121193-21sed08:24:161
158322800,6sleep30-21swapper/311:34:163
304032795,62sleep00-21swapper/012:20:130
2715127928,26sleep20-21swapper/207:04:112
764127718,36sleep17715-21cat09:24:161
1353527719,30sleep00-21swapper/007:59:200
1353527719,30sleep00-21swapper/007:59:200
270612769,12sleep30-21swapper/310:28:473
1980027617,46sleep119772-21latency_hist11:48:481
765727521,34sleep17338-21apt-get07:43:501
765727521,34sleep17338-21apt-get07:43:501
2054727518,16sleep10-21swapper/111:49:171
1742527515,46sleep317426-21/usr/sbin/munin08:14:053
3193127424,28sleep331944-21/usr/sbin/munin08:59:153
2900827424,27sleep30-21swapper/307:09:163
1051527423,21sleep10-21swapper/111:18:481
3171927315,40sleep131860-21/usr/sbin/munin12:24:171
288927315,37sleep13039-21unixbench_singl09:09:221
288927315,37sleep13039-21unixbench_singl09:09:221
114362737,14sleep30-21swapper/307:54:223
2369527112,38sleep323764-21/usr/sbin/munin11:59:163
2325727117,26sleep10-21swapper/111:59:061
2247227115,40sleep322503-21/usr/sbin/munin08:29:113
2793527011,44sleep328048-21timerandwakeup10:29:233
1766027017,27sleep10-21swapper/108:14:111
57982698,19sleep30-21swapper/311:03:483
57982698,19sleep30-21swapper/311:03:483
2895026912,39sleep328957-21df_inode10:34:083
2211726919,32sleep322188-21/usr/sbin/munin11:54:153
147526921,24sleep00-21swapper/007:24:070
32026823,23sleep30-21swapper/312:29:123
2238726817,29sleep30-21swapper/310:13:493
2238726817,29sleep30-21swapper/310:13:493
1774626819,21sleep10-21swapper/109:58:491
582126717,19sleep10-21swapper/109:19:121
1802026722,25sleep00-21swapper/009:59:040
1802026722,25sleep00-21swapper/009:59:040
1563226722,22sleep30-21swapper/308:08:483
1563226722,22sleep30-21swapper/308:08:483
848526617,30sleep38588-21/usr/sbin/munin07:44:333
315572667,48sleep00-21swapper/010:41:440
315572667,48sleep00-21swapper/010:41:440
134026613,36sleep11332-21apt-get10:49:001
3175326520,28sleep11-21systemd10:43:471
3175326520,28sleep11-21systemd10:43:471
2860126523,21sleep10-21swapper/112:14:201
241582659,16sleep00-21swapper/012:00:140
223226413,35sleep02257-21fschecks_count09:09:090
223226413,35sleep02257-21fschecks_count09:09:090
1047226412,37sleep010514-21latency_hist11:18:470
1047226412,37sleep010514-21latency_hist11:18:470
1039626414,23sleep30-21swapper/311:18:463
1039626414,23sleep30-21swapper/311:18:463
96082635,11sleep00-21swapper/009:30:170
96082635,11sleep00-21swapper/009:30:170
1444626317,20sleep10-21swapper/108:04:091
1444626317,20sleep10-21swapper/108:04:091
985626223,19sleep30-21swapper/307:49:143
819126217,28sleep10-21swapper/111:09:191
283152625,45sleep00-21swapper/008:48:580
2429826218,29sleep324334-21/usr/sbin/munin10:19:063
2309226215,29sleep023124-21proc_pri10:14:180
1183326216,30sleep111877-21grep09:39:061
2981626121,24sleep129390-21/usr/sbin/munin12:19:131
2471826111,30sleep1284-21dbus-daemon10:19:161
2392226123,22sleep10-21swapper/108:34:091
2055526116,28sleep020551-21apt-get08:23:590
1691426120,26sleep016933-21apt-get11:38:580
111822616,12sleep30-21swapper/311:19:143
1098326116,18sleep10-21swapper/111:19:081
987826018,24sleep19899-21basename09:33:451
987826018,24sleep19899-21basename09:33:451
883726014,29sleep38838-21cron11:13:453
656226016,23sleep30-21swapper/307:39:163
656226016,23sleep30-21swapper/307:39:163
41332607,44sleep00-21swapper/010:58:320
2203726015,29sleep022044-21ntp_states11:54:120
1529026020,20sleep10-21swapper/109:49:121
1529026020,20sleep10-21swapper/109:49:121
785225925,20sleep10-21swapper/107:44:031
622525925,18sleep30-21swapper/311:04:083
3045225913,28sleep330448-21apt-get10:39:003
3045225913,28sleep330448-21apt-get10:39:003
297302595,12sleep30-21swapper/308:53:503
2469625917,25sleep024715-21apt-get12:03:570
910625815,26sleep19124-21apt-get11:13:591
3244225815,27sleep332455-21ntp_states10:44:133
286125810,35sleep10-21swapper/110:53:531
2499325816,24sleep325086-21memory12:04:093
2218225813,24sleep10-21swapper/108:29:051
1797325817,23sleep30-21swapper/309:59:033
1797325817,23sleep30-21swapper/309:59:033
1581725814,21sleep10-21swapper/111:34:151
1352225818,18sleep10-21swapper/111:28:451
1274025816,27sleep312756-21awk07:58:593
1274025816,27sleep312756-21awk07:58:593
85425715,26sleep00-21swapper/010:48:470
812325712,25sleep00-21swapper/011:09:180
341625715,26sleep30-21swapper/307:29:153
2949425718,17sleep10-21swapper/110:34:221
2262925714,26sleep122624-21apt-get10:14:031
1713125719,19sleep00-21swapper/008:13:450
1645425717,21sleep10-21swapper/108:09:181
1587425719,20sleep30-21swapper/308:09:043
1444525713,25sleep10-21swapper/111:29:181
960825616,19sleep10-21swapper/107:49:071
32925618,17sleep10-21swapper/109:03:491
2779625610,18sleep30-21swapper/312:13:583
2732025614,25sleep127317-21apt-get10:29:051
2634625614,27sleep126352-21df_inode12:09:071
2550425617,18sleep00-21swapper/008:39:090
2107725613,26sleep021072-21apt-get10:09:010
2107725613,26sleep021072-21apt-get10:09:010
2044125611,22sleep10-21swapper/110:04:221
1923725616,23sleep10-21swapper/110:03:451
1923725616,23sleep10-21swapper/110:03:451
1742925610,31sleep017497-21df08:14:060
111212566,12sleep00-21swapper/011:19:110
660125512,26sleep00-21swapper/011:04:170
540425515,27sleep15405-21sshd07:34:211
452525516,16sleep00-21swapper/007:34:040
44312559,21sleep10-21swapper/109:14:391
3149025515,22sleep30-21swapper/307:15:463
3035025510,31sleep030356-21ntpq08:54:150
2914625514,28sleep029169-21kernelversion10:34:130
188625516,27sleep31968-21sed12:34:093
1184825514,25sleep011966-21timerwakeupswit07:54:310
113322555,38sleep00-21swapper/009:37:320
113322555,38sleep00-21swapper/009:37:320
785125412,25sleep07844-21apt-get07:44:020
57825413,25sleep3577-21apt-get09:04:033
3249825415,22sleep132516-21apt-get12:29:021
2891225418,18sleep30-21swapper/308:49:183
2678025413,24sleep326775-21apt-get08:44:013
2551525413,26sleep025525-21munin-run10:23:470
2520825413,18sleep10-21swapper/108:38:581
2001425416,21sleep00-21swapper/010:04:170
1900525415,23sleep119002-21apt-get08:18:591
1900525415,23sleep119002-21apt-get08:18:591
131225410,18sleep30-21swapper/310:48:583
1174725414,24sleep011742-21apt-get09:39:010
664225316,20sleep00-21swapper/007:39:170
664225316,20sleep00-21swapper/007:39:170
602925313,24sleep16020-21apt-get11:04:021
573025317,18sleep30-21swapper/309:19:103
300925313,16sleep00-21swapper/007:29:030
2627625315,23sleep30-21swapper/312:09:053
2169125319,17sleep10-21swapper/110:09:191
2169125319,17sleep10-21swapper/110:09:191
1849425318,15sleep30-21swapper/311:44:063
175452538,36sleep30-21swapper/309:57:213
1693825313,24sleep316933-21apt-get11:39:013
1589825312,23sleep015967-21sensors_temp11:34:180
1274925314,17sleep00-21swapper/011:24:160
3225725214,23sleep331760-21apt-get07:18:573
3146925214,21sleep031465-21apt-get08:59:000
2988725213,24sleep129885-21apt-get08:54:001
2732425212,13sleep00-21swapper/010:29:080
2576225214,23sleep025757-21apt-get10:23:590
182225213,23sleep31848-21ntp_states07:24:143
1491525212,23sleep014921-21df_inode09:49:030
1491525212,23sleep014921-21df_inode09:49:030
332125112,24sleep33353-21ntp_states10:54:193
580125012,18sleep30-21swapper/307:38:473
2705425012,23sleep027147-21memory08:44:100
2603225015,15sleep10-21swapper/110:24:081
2262025012,18sleep00-21swapper/008:29:160
1016825010,20sleep10-21swapper/109:34:021
1016825010,20sleep10-21swapper/109:34:021
2664324911,19sleep00-21swapper/012:09:140
2511924914,22sleep125155-21ntpq12:04:111
1947324913,17sleep00-21swapper/008:19:150
1947324913,17sleep00-21swapper/008:19:150
132824918,17sleep00-21swapper/012:33:450
3105124812,19sleep00-21swapper/012:24:040
2182224815,16sleep30-21swapper/308:28:463
200302487,16sleep30-21swapper/311:48:583
1773324714,19sleep00-21swapper/009:58:480
1482224710,15sleep30-21swapper/309:48:573
1482224710,15sleep30-21swapper/309:48:573
2897924310,18sleep128985-21ntp_states07:09:141
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional