You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-16 - 03:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Oct 16, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1195221208815,12055sleep26508-21kworker/2:222:31:092
1536028368,810sleep215603-21kworker/u8:320:56:332
12435256518,476sleep012436-21ntp_states20:46:180
894125268,481sleep28949-21df20:36:102
30524249123,453sleep030591-21/usr/sbin/munin23:31:020
27806241638,330sleep227847-21ntpq23:21:152
27806241638,330sleep227847-21ntpq23:21:152
143323810,33sleep20-21swapper/221:56:192
3205223730,10sleep10-21swapper/120:06:091
3205223730,10sleep10-21swapper/120:06:091
6935236027,320sleep26983-21awk22:16:072
2649123426,325sleep2362-21snmpd19:50:442
23467229576,202sleep223613-21unixbench_singl23:06:212
2104422889,260sleep221108-21mailstats19:31:182
2333228712,21sleep20-21swapper/222:01:092
18863226412,205sleep218931-21latency_hist21:10:512
27195224517,203sleep20-21swapper/219:51:452
7153224150,123sleep2311-21in:imuxsock00:00:472
13796223044,174sleep213885-21cat20:51:122
25638220513,148sleep225698-21latency_hist23:15:502
17732218416,23sleep10-21swapper/119:21:121
17732218416,23sleep10-21swapper/119:21:121
1708321799,158sleep217163-21cat22:46:192
1708321799,158sleep217163-21cat22:46:192
12378217741,119sleep211947-21/usr/sbin/munin20:46:162
15689216915,79sleep215745-21interrupts00:26:112
11896216956,71sleep20-21swapper/219:05:512
1758216810,147sleep3362-21snmpd21:58:153
7861216615,26sleep10-21swapper/100:01:191
11018216618,120sleep10-21swapper/120:41:181
28465216515,133sleep028527-21apt-key19:55:510
2709521597,140sleep227148-21cat21:36:102
2709521597,140sleep227148-21cat21:36:102
32407215811,132sleep032412-21ntpq20:06:170
32407215811,132sleep032412-21ntpq20:06:170
28690215813,58sleep20-21swapper/219:55:532
4973215531,53sleep20-21swapper/220:21:332
2417921528,132sleep224215-21ntpq21:26:172
31343214945,87sleep231300-21latency_hist21:50:512
13557214914,87sleep213764-21apt-config19:10:522
4306214349,52sleep20-21swapper/223:51:072
22823214344,59sleep222976-21awk19:36:232
12804214315,108sleep012954-21tail19:06:210
22451214247,54sleep20-21swapper/223:05:492
16102214118,105sleep216174-21fschecks_time21:01:062
1927421389,115sleep219331-21idleruntime21:11:112
1927421389,115sleep219331-21idleruntime21:11:112
10796213510,113sleep210887-21cut22:26:332
24365213123,55sleep20-21swapper/219:41:202
805321296,111sleep28130-21grep00:01:242
30406212956,16sleep20-21swapper/220:01:062
1724212974,39sleep20-21swapper/220:11:322
14298212822,85sleep313402-21apt-get19:11:013
14298212822,85sleep313402-21apt-get19:11:013
19317212612,99sleep219379-21modprobe22:55:482
19904212517,44sleep20-21swapper/222:56:122
15764212412,94sleep00-21swapper/000:26:130
28155212020,72sleep128295-21grep23:21:221
28155212020,72sleep128295-21grep23:21:221
10498212011,72sleep210569-21fschecks_time20:41:072
19381211617,76sleep318989-21/usr/sbin/munin19:26:153
15189211621,78sleep315205-21idleruntime-cro00:25:473
15189211621,78sleep315205-21idleruntime-cro00:25:473
29061211433,66sleep229117-21fschecks_count23:26:082
8957211318,71sleep00-21swapper/020:36:120
201821120,7sleep00-21swapper/022:00:500
26061211142,16sleep20-21swapper/223:16:102
26061211142,16sleep20-21swapper/223:16:102
17464211032,56sleep20-21swapper/219:21:052
17464211032,56sleep20-21swapper/219:21:052
29244210916,70sleep10-21swapper/119:56:371
27595210820,59sleep127728-21sed21:36:211
27595210820,59sleep127728-21sed21:36:211
8479210612,74sleep18528-21/usr/sbin/munin22:21:021
11285210653,36sleep20-21swapper/200:11:172
19192210515,70sleep119270-21fschecks_time21:11:101
19192210515,70sleep119270-21fschecks_time21:11:101
16592210415,74sleep316485-21/usr/sbin/munin22:46:083
16592210415,74sleep316485-21/usr/sbin/munin22:46:083
3041210224,61sleep13075-21/usr/sbin/munin20:16:191
3041210224,61sleep13075-21/usr/sbin/munin20:16:191
3775210133,35sleep00-21swapper/022:06:030
27455210122,62sleep30-21swapper/319:51:503
17174210125,57sleep20-21swapper/219:20:492
17174210125,57sleep20-21swapper/219:20:492
6724210035,24sleep20-21swapper/222:15:512
26665210028,57sleep226674-21munin-run21:35:492
510529942,45sleep25106-21munin-run22:10:502
3132629915,72sleep131342-21sed21:50:501
1050329927,46sleep310024-21apt-get19:01:003
700129843,43sleep21-21systemd20:30:492
38562986,36sleep20-21swapper/220:20:162
38562986,36sleep20-21swapper/220:20:162
1214129811,71sleep025574-21kworker/u8:400:15:510
157629614,65sleep01604-21nfs4_client23:41:160
1499629616,64sleep115213-21apt-config20:55:521
1049129638,44sleep20-21swapper/200:10:492
1558329522,53sleep315708-21timerwakeupswit22:41:193
1390529521,38sleep00-21swapper/000:21:000
1390529521,38sleep00-21swapper/000:21:000
1972129310,68sleep319750-21fschecks_count22:56:093
646429220,50sleep16609-21grep20:26:241
1481529219,58sleep314881-21apt-config20:55:503
1238729214,62sleep012466-21switchtime22:31:190
120129240,41sleep21197-21apt-get23:41:012
1027429218,55sleep00-21swapper/022:26:060
1827829120,46sleep018416-21unixbench_singl21:06:240
1185229114,59sleep111890-21cpufreq19:05:501
36329010,20sleep20-21swapper/221:55:482
2524028913,58sleep125307-21df19:46:071
190142899,33sleep20-21swapper/219:26:062
1288528932,39sleep20-21swapper/200:16:182
1288528932,39sleep20-21swapper/200:16:182
974128810,63sleep29752-21modprobe22:25:482
5422889,68sleep2362-21snmpd20:08:002
5422889,68sleep2362-21snmpd20:08:002
1801328820,38sleep00-21swapper/019:21:190
1801328820,38sleep00-21swapper/019:21:190
1702428829,16sleep20-21swapper/200:30:592
3254128732,14sleep20-21swapper/223:36:172
2349528734,14sleep20-21swapper/221:25:482
141352868,20sleep20-21swapper/219:10:562
141352868,20sleep20-21swapper/219:10:562
262392856,12sleep10-21swapper/123:16:131
262392856,12sleep10-21swapper/123:16:131
2555928517,49sleep025572-21modprobe23:15:480
2387428520,50sleep023906-21tune2fs21:26:110
2232628517,44sleep022340-21fschecks_count21:21:100
2117328518,16sleep20-21swapper/221:16:202
2117328518,16sleep20-21swapper/221:16:202
2027128526,24sleep00-21swapper/019:30:480
152028516,56sleep11527-21open_files20:11:231
146322859,15sleep20-21swapper/200:21:262
146322859,15sleep20-21swapper/200:21:262
1369428525,36sleep313838-21latency_hist00:20:513
1369428525,36sleep313838-21latency_hist00:20:513
704728416,50sleep37106-21cat20:30:503
44728415,52sleep0384-21latency_hist21:55:500
3034628313,45sleep1225-21systemd-journal23:30:511
132928318,42sleep31363-21cat20:11:163
3082820,8sleep326974-21kworker/3:123:36:213
1104928224,26sleep30-21swapper/300:11:103
106882828,64sleep00-21swapper/019:01:170
2853828129,16sleep20-21swapper/221:41:062
2853828129,16sleep20-21swapper/221:41:062
1832428124,38sleep018390-21cat22:51:100
174112810,6sleep10-21swapper/122:46:211
174112810,6sleep10-21swapper/122:46:211
1348428128,13sleep20-21swapper/222:36:052
411228019,17sleep00-21swapper/020:20:540
310702807,17sleep20-21swapper/223:31:172
3016528017,37sleep330289-21cat20:00:513
193322800,6sleep30-21swapper/321:11:113
193322800,6sleep30-21swapper/321:11:113
555927924,32sleep35626-40mandb23:55:483
1551227911,53sleep215581-21grep22:41:182
532127814,49sleep05338-21apt22:11:000
1729127817,25sleep00-21swapper/022:46:210
1729127817,25sleep00-21swapper/022:46:210
1928227720,31sleep119346-21cat19:26:131
1025827722,32sleep30-21swapper/322:26:013
979527646,16sleep00-21swapper/022:25:500
1224427620,30sleep30-21swapper/320:46:133
84442755,36sleep20-21swapper/220:34:292
38642755,12sleep00-21swapper/020:20:220
38642755,12sleep00-21swapper/020:20:220
1721727522,33sleep30-21swapper/319:20:503
1721727522,33sleep30-21swapper/319:20:503
1454627515,29sleep10-21swapper/100:21:251
1454627515,29sleep10-21swapper/100:21:251
962027414,42sleep19656-21ntpq00:06:281
2250127425,28sleep30-21swapper/323:05:503
1852627425,31sleep10-21swapper/100:35:501
975827323,25sleep30-21swapper/322:25:493
2822627323,25sleep30-21swapper/321:40:513
2822627323,25sleep30-21swapper/321:40:513
2751227315,26sleep00-21swapper/023:21:090
2751227315,26sleep00-21swapper/023:21:090
2428727313,30sleep10-21swapper/123:11:011
1764527317,28sleep30-21swapper/319:21:103
1764527317,28sleep30-21swapper/319:21:103
975427221,26sleep30-21swapper/300:06:343
2335427216,43sleep123466-21cat23:06:201
2119527211,43sleep021262-21df23:01:070
1420027212,32sleep00-21swapper/019:10:580
1420027212,32sleep00-21swapper/019:10:580
1411727222,28sleep113402-21apt-get19:10:551
1411727222,28sleep113402-21apt-get19:10:551
747027111,12sleep00-21swapper/022:16:210
2182027121,32sleep021853-21awk19:35:490
1724827121,20sleep00-21swapper/019:20:510
1724827121,20sleep00-21swapper/019:20:510
3108427014,33sleep131212-21/usr/sbin/munin20:01:211
3090927019,28sleep30-21swapper/320:01:193
2494527025,30sleep31-21systemd19:45:493
1364127025,25sleep00-21swapper/000:20:480
1364127025,25sleep00-21swapper/000:20:480
1281127013,39sleep112878-21/usr/sbin/munin20:46:231
701826921,31sleep17027-21/usr/sbin/munin22:16:101
298362697,53sleep00-21swapper/019:57:160
2532426913,22sleep00-21swapper/019:46:100
2140226915,40sleep121452-21if_eth023:01:121
1488326924,24sleep00-21swapper/020:55:510
282826811,14sleep20-21swapper/223:46:062
2695126813,41sleep326956-21sshd23:18:303
2695126813,41sleep326956-21sshd23:18:303
2428426820,31sleep124362-21find19:41:191
425426714,37sleep34260-21ntp_states22:06:193
2699426718,26sleep126509-21apt-get19:51:001
208922677,48sleep10-21swapper/123:00:391
1513826717,24sleep10-21swapper/122:41:091
720526622,21sleep00-21swapper/000:00:500
3214226621,30sleep30-21swapper/321:51:213
2010026620,19sleep00-21swapper/022:56:180
1816726616,25sleep30-21swapper/321:06:233
158492667,12sleep10-21swapper/100:26:141
113926624,25sleep00-21swapper/020:11:090
874226518,28sleep38761-21munin-run00:05:493
458726514,33sleep14626-21ntpq20:21:161
2298026513,36sleep023013-21irqstats23:06:110
2274726531,18sleep30-21swapper/321:21:213
2846426416,20sleep00-21swapper/021:41:040
2846426416,20sleep00-21swapper/021:41:040
220712645,12sleep10-21swapper/121:20:511
1878526418,27sleep118935-21unixbench_singl22:51:191
1598426419,29sleep315981-21apt-get21:00:583
157942649,39sleep015944-21cat21:00:510
157462647,12sleep30-21swapper/300:26:123
1206126417,31sleep112116-21munin-run00:15:481
983226321,25sleep00-21swapper/000:06:350
635826326,21sleep06462-21ls20:26:230
418426324,21sleep30-21swapper/320:21:013
1575226318,22sleep30-21swapper/321:00:503
3220426220,21sleep00-21swapper/023:36:110
3049626216,20sleep30-21swapper/323:30:573
286226218,15sleep10-21swapper/122:01:201
2650126218,23sleep00-21swapper/023:16:210
2650126218,23sleep00-21swapper/023:16:210
2588426215,33sleep325950-21wc21:31:183
1937526219,24sleep00-21swapper/021:11:130
1937526219,24sleep00-21swapper/021:11:130
1652426222,18sleep10-21swapper/119:16:201
1652426222,18sleep10-21swapper/119:16:201
145672626,11sleep10-21swapper/122:39:281
1094626217,26sleep011017-21sensors_temp20:41:170
595926118,23sleep30-21swapper/323:56:083
561626117,23sleep30-21swapper/322:11:113
2938426122,23sleep00-21swapper/023:26:140
285912617,13sleep30-21swapper/323:24:353
285912617,13sleep30-21swapper/323:24:353
1295626117,26sleep10-21swapper/119:06:221
832026017,22sleep30-21swapper/320:32:563
3001326018,26sleep330008-21apt-get21:46:043
2490226014,22sleep30-21swapper/323:11:193
248662604,11sleep10-21swapper/121:28:301
248632606,42sleep30-21swapper/321:28:283
2082926019,18sleep30-21swapper/319:31:113
1831826021,19sleep30-21swapper/300:35:483
1472426021,18sleep30-21swapper/322:40:513
883025916,26sleep09000-21wc00:05:500
773725918,23sleep17743-21ntp_states20:31:131
3165925917,29sleep031662-21cron20:05:480
2361525918,20sleep30-21swapper/323:06:223
1274725915,26sleep10-21swapper/100:16:131
1274725915,26sleep10-21swapper/100:16:131
728025818,22sleep00-21swapper/020:31:000
589025815,25sleep15883-21apt-get23:56:061
2924625823,20sleep10-21swapper/123:26:111
140925820,18sleep10-21swapper/123:41:131
92402579,18sleep10-21swapper/120:36:171
69842578,27sleep30-21swapper/322:16:083
467825718,22sleep00-21swapper/023:51:170
39312577,11sleep10-21swapper/123:49:441
3245925720,16sleep30-21swapper/320:06:203
3245925720,16sleep30-21swapper/320:06:203
26862576,14sleep00-21swapper/023:45:520
256982579,11sleep10-21swapper/121:31:121
2541725723,19sleep00-21swapper/021:31:070
2389525713,28sleep023947-21/usr/sbin/munin19:41:090
227225711,30sleep32319-21df22:01:063
1702725716,24sleep017021-21apt-get00:31:010
913925612,19sleep30-21swapper/320:36:153
2096125617,19sleep00-21swapper/021:16:130
2096125617,19sleep00-21swapper/021:16:130
1398625610,23sleep00-21swapper/022:36:200
3078525514,25sleep030915-21awk21:46:210
2254425512,28sleep022589-21ntp_offset19:36:170
2069125517,19sleep10-21swapper/121:16:061
2069125517,19sleep10-21swapper/121:16:061
131425512,20sleep10-21swapper/121:56:191
335625416,17sleep30-21swapper/320:16:333
335625416,17sleep30-21swapper/320:16:333
2846025414,23sleep128457-21apt-get21:41:011
2846025414,23sleep128457-21apt-get21:41:011
2260825416,19sleep30-21swapper/319:36:203
1596125416,20sleep10-21swapper/121:00:561
1041425414,17sleep30-21swapper/320:41:043
987425317,18sleep10-21swapper/122:25:511
379725313,23sleep13794-21apt-get22:06:041
283625312,20sleep30-21swapper/323:46:073
2124425315,19sleep30-21swapper/321:16:213
2124425315,19sleep30-21swapper/321:16:213
2055625313,23sleep120555-21apt-get19:31:011
1622025319,17sleep00-21swapper/022:45:480
153192524,37sleep10-21swapper/120:56:041
1182425212,24sleep311821-21apt-get22:31:023
533925113,22sleep15341-21apt-get22:11:011
1805325114,22sleep318049-21apt-get22:51:003
166225115,21sleep31668-21ntp_states23:41:193
480325012,19sleep10-21swapper/123:51:221
323622498,16sleep10-21swapper/121:51:231
2891924911,17sleep30-21swapper/321:41:153
2891924911,17sleep30-21swapper/321:41:153
2523724917,16sleep30-21swapper/319:46:053
248524915,19sleep02517-21irqstats22:01:110
125742499,16sleep30-21swapper/319:06:163
1624124711,21sleep016259-21irqstats21:01:080
2934524515,17sleep30-21swapper/323:26:133
221422449,18sleep10-21swapper/119:36:061
1210624416,15sleep10-21swapper/122:31:121
321752426,14sleep10-21swapper/123:36:081
2751724211,15sleep00-21swapper/021:36:200
2751724211,15sleep00-21swapper/021:36:200
1881923911,15sleep10-21swapper/121:10:501
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional