You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-11 - 13:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Fri Jul 11, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1175252722,488sleep31242-21/usr/sbin/munin22:59:213
26194235913,326sleep226286-21xargs22:34:322
2870823300,11sleep00-21swapper/022:44:150
168512277142,84sleep20-21swapper/223:49:232
22179226919,232sleep022344-21cat18:59:050
23843223612,205sleep223857-21grep20:44:292
30037222753,23sleep20-21swapper/222:49:022
14955222115,125sleep21-21systemd23:44:022
25687221153,22sleep20-21swapper/219:09:042
18392221016,66sleep218406-21fschecks_count23:54:192
759922047,183sleep27651-21fschecks_count19:54:222
22511218918,35sleep20-21swapper/222:24:152
22511218918,35sleep20-21swapper/222:24:152
2568521829,156sleep20-21swapper/220:49:342
4818217518,31sleep20-21swapper/219:44:332
99721699,16sleep20-21swapper/219:34:032
99721699,16sleep20-21swapper/219:34:032
114621699,147sleep21152-21cat22:59:162
156922167127,29sleep215752-21find23:44:342
1228621678,144sleep212338-21cat23:34:252
1228621678,144sleep212338-21cat23:34:252
29096215711,21sleep20-21swapper/219:19:212
2711215713,22sleep20-21swapper/221:19:302
1889521579,136sleep218941-21grep20:29:192
16139215517,21sleep20-21swapper/220:19:262
131821529,12sleep00-21swapper/022:59:420
31406214413,43sleep20-21swapper/221:09:042
967021438,121sleep29783-21timerandwakeup19:59:352
24261214223,92sleep324401-21tail20:44:393
16281214217,112sleep316292-21grep22:04:173
1460121408,119sleep214646-21ntpq20:14:292
11360213912,112sleep311330-21latency_hist21:49:033
27837213761,61sleep227985-21grep22:39:332
6679213610,110sleep26809-21ls21:34:042
1144521346,14sleep20-21swapper/223:29:402
789213162,52sleep2866-21fschecks_time21:14:212
2462721306,90sleep20-21swapper/200:14:182
2462721306,90sleep20-21swapper/200:14:182
38421295,17sleep20-21swapper/221:13:452
27126212928,24sleep20-21swapper/220:54:352
272721289,68sleep20-21swapper/223:04:122
3980212782,11sleep20-21swapper/219:43:162
3980212782,11sleep20-21swapper/219:43:162
3273621278,107sleep232749-21awk22:54:352
2333212716,92sleep12346-21fschecks_count21:19:211
2506721255,110sleep20-21swapper/222:30:492
2373021245,108sleep20-21swapper/218:59:552
2373021245,108sleep20-21swapper/218:59:552
27205212313,94sleep027232-21munin-run19:14:020
26917212113,91sleep127062-21cat22:39:041
26577212118,89sleep326635-21sendmail_mailqu00:19:323
26577212118,89sleep326635-21sendmail_mailqu00:19:323
2139121218,101sleep221392-21ntp_states22:19:302
2139121218,101sleep221392-21ntp_states22:19:302
31568212079,22sleep20-21swapper/219:25:232
13915211874,11sleep20-21swapper/220:13:482
28079211710,68sleep20-21swapper/219:14:332
27525211711,34sleep30-21swapper/322:39:263
27107211772,11sleep20-21swapper/200:20:482
27107211772,11sleep20-21swapper/200:20:482
6876211669,13sleep20-21swapper/223:16:042
2962621165,79sleep20-21swapper/221:02:302
24233211671,11sleep20-21swapper/200:13:372
19978211662,41sleep220047-21fschecks_time23:59:202
26325211511,61sleep20-21swapper/219:09:282
21731211310,88sleep021824-21cut00:04:270
26932211218,77sleep026925-21apt-get20:54:210
21944211269,10sleep20-21swapper/218:57:012
1511621110,6sleep10-21swapper/121:59:261
9410211010,59sleep20-21swapper/223:24:272
5395211067,13sleep20-21swapper/223:11:562
30375211016,25sleep00-21swapper/022:49:200
19781210952,43sleep219828-21ntpq22:14:302
19781210952,43sleep219828-21ntpq22:14:302
17401210947,52sleep20-21swapper/222:07:152
1324210910,25sleep20-21swapper/219:34:232
1324210910,25sleep20-21swapper/219:34:232
12143210919,71sleep312154-21modprobe20:09:013
1834621087,65sleep2370-21snmpd20:25:432
862521075,73sleep20-21swapper/223:23:352
2715521069,17sleep10-21swapper/122:39:161
26466210613,77sleep326548-21latency_hist20:54:033
22706210545,49sleep20-21swapper/200:08:532
19724210334,49sleep319884-21grep18:49:353
32420210214,31sleep00-21swapper/019:29:290
32420210214,31sleep00-21swapper/019:29:290
22447210212,76sleep322457-21ntpq20:39:303
18727210046,37sleep20-21swapper/222:09:372
18727210046,37sleep20-21swapper/222:09:372
302129915,60sleep32760-21/usr/sbin/munin19:39:253
302129915,60sleep32760-21/usr/sbin/munin19:39:253
2285929914,67sleep122883-21memory22:24:261
2285929914,67sleep122883-21memory22:24:261
1314929925,48sleep113143-21apt-get21:54:181
1314929925,48sleep113143-21apt-get21:54:181
953729816,62sleep09037-21/usr/sbin/munin19:59:330
914729813,67sleep39206-21fschecks_count19:59:233
1264329717,61sleep312160-21apt-get20:09:133
738729615,59sleep06893-21/usr/sbin/munin21:34:300
2090329642,37sleep20-21swapper/220:34:322
229492955,78sleep20-21swapper/220:40:052
2027229535,36sleep220332-21apt-config18:54:022
1413729557,10sleep20-21swapper/221:55:362
1413729557,10sleep20-21swapper/221:55:362
315952949,20sleep231597-21munin-run22:54:032
2755629415,61sleep327614-21cut19:14:223
2140229410,69sleep021423-21awk22:19:320
2140229410,69sleep021423-21awk22:19:320
571529224,52sleep05728-21latency_hist19:49:030
3068429212,59sleep130776-21perl19:24:211
886629119,50sleep38904-21ntp_kernel_pll_21:39:303
816429150,12sleep20-21swapper/221:38:522
641629143,31sleep20-21swapper/219:49:332
2043329125,40sleep020647-21latency_hist18:54:040
220762906,67sleep10-21swapper/100:04:361
131722907,69sleep00-21swapper/023:36:570
131722907,69sleep00-21swapper/023:36:570
1178129037,39sleep20-21swapper/221:49:222
3207428916,57sleep332108-21/usr/sbin/munin22:54:243
2743528912,60sleep127521-21latency_hist00:24:021
2743528912,60sleep127521-21latency_hist00:24:021
1159028919,49sleep011721-21grep20:04:500
1094928942,13sleep20-21swapper/221:44:382
379028843,12sleep20-21swapper/221:24:062
2268328832,27sleep30-21swapper/318:59:123
2268328832,27sleep30-21swapper/318:59:123
2093828824,51sleep320946-21cpu22:19:193
2093828824,51sleep320946-21cpu22:19:193
942228743,12sleep20-21swapper/221:40:092
2974128721,48sleep329783-21sendmail21:04:013
1469428612,53sleep214745-21sort21:59:152
959828512,52sleep19669-21/usr/sbin/munin19:59:341
512028422,46sleep15131-21unin-run21:29:021
111202847,15sleep20-21swapper/220:04:402
2749528313,54sleep327556-21switchtime20:54:443
563428217,49sleep15789-21latency_hist23:14:051
45328215,51sleep0434-21latency_hist21:14:030
263692820,7sleep30-21swapper/319:09:283
1916728219,46sleep119259-21/usr/sbin/munin20:29:261
1400828218,45sleep113957-21latency_hist20:14:041
983828122,42sleep19991-21latency_hist21:44:041
2580928116,49sleep025842-21idleruntime-cro00:19:000
2580928116,49sleep025842-21idleruntime-cro00:19:000
204452810,7sleep00-21swapper/023:59:290
1251128110,14sleep30-21swapper/323:34:303
1251128110,14sleep30-21swapper/323:34:303
2560928015,50sleep025673-21cpuspeed22:34:180
1423728022,29sleep00-21swapper/023:39:310
2489027921,26sleep00-21swapper/020:49:040
3004827825,28sleep30-21swapper/322:49:033
2498827829,24sleep10-21swapper/119:04:351
113127815,47sleep1657-21apt-get22:59:071
865827720,42sleep08668-21modprobe23:24:010
82282778,55sleep18298-21latency_hist21:39:031
614127713,45sleep06219-21memory23:14:240
1946427714,46sleep019538-21fschecks_time22:14:230
1946427714,46sleep019538-21fschecks_time22:14:230
965027619,41sleep39779-21/usr/sbin/munin23:24:333
1440327618,42sleep014409-21modprobe21:59:010
1440327618,42sleep014409-21modprobe21:59:010
97812759,16sleep00-21swapper/023:24:340
3131827514,14sleep00-21swapper/021:09:020
979927417,26sleep00-21swapper/021:44:030
70227420,31sleep10-21swapper/122:59:021
545727424,19sleep00-21swapper/021:29:210
1160027413,45sleep111623-21grep21:49:171
86927319,23sleep30-21swapper/321:14:223
2471227310,49sleep10-21swapper/120:47:231
1067427321,39sleep310738-21idleruntime23:29:253
691827220,40sleep36927-21cpu21:34:173
451527213,42sleep34281-21/usr/sbin/munin23:09:263
2005127224,25sleep320110-21if_eth023:59:213
1370827220,36sleep113781-21fschecks_time23:39:201
64927115,33sleep0700-21apt-config22:59:010
46827121,18sleep10-21swapper/121:14:041
2975427125,30sleep129895-21grep19:19:371
2628827116,40sleep326422-21sed22:34:333
2340327119,34sleep123447-21ntpq00:09:291
1530127118,25sleep00-21swapper/023:44:240
2910727020,27sleep029146-21cut19:19:230
1994127013,44sleep119974-21fschecks_count23:59:191
2335426916,22sleep10-21swapper/120:44:041
2194126914,26sleep30-21swapper/300:04:343
2113626922,27sleep30-21swapper/320:34:373
190212696,51sleep00-21swapper/022:13:100
190212696,51sleep00-21swapper/022:13:100
1875126912,40sleep118762-21ntp_states23:54:261
136202697,46sleep30-21swapper/323:39:143
611426814,39sleep36136-21uname23:14:233
2852426813,22sleep00-21swapper/020:59:170
1221226821,31sleep112306-21sed20:09:031
48992677,13sleep10-21swapper/119:44:361
3114926719,35sleep331210-21switchtime19:24:313
3033826716,25sleep30-21swapper/321:04:293
27322670,8sleep30-21swapper/321:19:303
1998626719,36sleep320082-21sed22:14:353
1998626719,36sleep320082-21sed22:14:353
174162677,12sleep00-21swapper/022:07:250
132826720,23sleep10-21swapper/121:14:321
1321826720,26sleep30-21swapper/321:54:203
1321826720,26sleep30-21swapper/321:54:203
31462668,12sleep10-21swapper/119:39:271
31462668,12sleep10-21swapper/119:39:271
3074326610,15sleep30-21swapper/322:49:273
2512626614,38sleep325187-21cat00:14:313
2512626614,38sleep325187-21cat00:14:313
249812666,12sleep00-21swapper/022:29:420
1743526620,22sleep30-21swapper/320:24:233
1478026613,35sleep314853-21grep21:59:203
1113626616,26sleep10-21swapper/123:29:351
947526515,19sleep10-21swapper/123:24:311
302226520,25sleep10-21swapper/123:04:241
2761626515,19sleep00-21swapper/019:14:230
2723326514,37sleep127316-21wc19:14:031
1830926516,34sleep318372-21/usr/sbin/munin23:54:163
1485826522,23sleep00-21swapper/021:59:210
766926414,33sleep37764-21memory23:19:243
2456026420,26sleep124640-21mailstats22:29:301
1737426416,26sleep117476-21/usr/sbin/munin23:49:341
1576926419,27sleep015764-21apt-get20:19:150
360726317,25sleep00-21swapper/021:24:040
3096126318,28sleep130970-21/usr/sbin/munin21:04:391
2234626321,22sleep10-21swapper/118:59:061
2234626321,22sleep10-21swapper/118:59:061
1582126317,29sleep315846-21/usr/sbin/munin20:19:193
642826222,19sleep30-21swapper/319:49:343
561326216,23sleep00-21swapper/023:14:040
2474926217,24sleep30-21swapper/319:04:293
110172629,42sleep00-21swapper/021:45:160
3264826118,20sleep30-21swapper/319:29:363
3264826118,20sleep30-21swapper/319:29:363
2730426116,29sleep10-21swapper/120:54:381
1762026114,31sleep117776-21latency_hist22:09:031
63182608,12sleep10-21swapper/123:14:291
3216826019,16sleep30-21swapper/321:09:383
2298026013,19sleep30-21swapper/300:09:143
1783026018,25sleep317826-21apt-get22:09:183
1783026018,25sleep317826-21apt-get22:09:183
636925919,27sleep10-21swapper/119:49:321
2855025914,29sleep128556-21diskstats20:59:181
272082598,38sleep00-21swapper/000:22:070
272082598,38sleep00-21swapper/000:22:070
2174825916,26sleep021757-21unin-run20:39:020
2064525915,19sleep10-21swapper/122:19:011
2064525915,19sleep10-21swapper/122:19:011
1818725912,20sleep10-21swapper/122:09:281
1818725912,20sleep10-21swapper/122:09:281
1781825913,34sleep017325-21/usr/sbin/munin20:24:320
1445825922,16sleep30-21swapper/320:14:263
1431125917,26sleep014312-21diskstats20:14:230
1420225915,27sleep114221-21apt-get20:14:161
2924125816,20sleep10-21swapper/122:44:331
2550825815,18sleep10-21swapper/120:49:291
2451525819,23sleep324521-21ntp_states22:29:283
2439625816,24sleep00-21swapper/000:14:040
2085625811,29sleep10-21swapper/120:34:311
170625817,19sleep30-21swapper/319:34:323
170625817,19sleep30-21swapper/319:34:323
198812578,38sleep00-21swapper/020:30:340
112262575,10sleep10-21swapper/121:47:561
436525615,25sleep34357-21apt-get19:44:223
260572567,10sleep00-21swapper/020:49:480
150625617,22sleep10-21swapper/119:34:261
150625617,22sleep10-21swapper/119:34:261
750525514,22sleep37508-21apt-get19:54:153
3190625514,23sleep00-21swapper/019:29:040
284012555,38sleep10-21swapper/119:14:481
2494825513,23sleep10-21swapper/100:14:251
2494825513,23sleep10-21swapper/100:14:251
22202558,12sleep00-21swapper/019:35:300
22202558,12sleep00-21swapper/019:35:300
2084225511,23sleep30-21swapper/318:54:193
594625414,26sleep36029-21sed21:29:323
415725417,23sleep04192-21memory21:24:250
390525415,23sleep33907-21diskstats21:24:203
36525414,20sleep10-21swapper/119:29:381
36525414,20sleep10-21swapper/119:29:381
3114425415,24sleep131191-21users22:49:331
2312725418,21sleep023239-21timerandwakeup22:24:330
2312725418,21sleep023239-21timerandwakeup22:24:330
1788425413,24sleep117325-21/usr/sbin/munin20:24:331
156392546,26sleep30-21swapper/323:44:313
125425417,19sleep00-21swapper/021:14:310
39742537,35sleep00-21swapper/023:08:300
289932539,17sleep30-21swapper/319:19:143
643725218,15sleep00-21swapper/019:49:350
323925210,17sleep00-21swapper/019:39:300
323925210,17sleep00-21swapper/019:39:300
288125213,17sleep30-21swapper/323:04:223
2405925215,23sleep0286-21dbus-daemon19:04:010
2405925215,23sleep0286-21dbus-daemon19:04:010
1880825213,18sleep00-21swapper/023:54:300
546525119,17sleep10-21swapper/121:29:221
2722925113,22sleep027235-21df_inode22:39:190
1206225116,20sleep112072-21apt-get23:34:161
1206225116,20sleep112072-21apt-get23:34:161
1187125115,14sleep00-21swapper/021:49:240
759325010,23sleep17598-21diskstats19:54:211
759124914,21sleep17653-21if_eth023:19:221
2125724918,16sleep10-21swapper/118:54:421
1788924914,19sleep317890-21sshd23:52:223
1397624912,17sleep00-21swapper/020:14:030
1106524912,20sleep00-21swapper/023:29:340
2332724813,19sleep00-21swapper/020:44:030
2093624811,21sleep120931-21apt-get22:19:171
2093624811,21sleep120931-21apt-get22:19:171
1802024812,19sleep00-21swapper/023:54:010
2563524711,14sleep00-21swapper/019:09:020
1585524717,15sleep10-21swapper/120:19:211
882424616,15sleep00-21swapper/019:59:040
1531124612,19sleep115397-21perl23:44:251
1290824614,16sleep30-21swapper/321:54:033
2275624514,15sleep30-21swapper/322:24:243
2275624514,15sleep30-21swapper/322:24:243
190312429,15sleep10-21swapper/118:49:211
189422419,16sleep318966-21df20:29:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional