You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-27 - 12:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Thu Nov 27, 2025 00:43:56)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30145281917,784sleep130167-21/usr/sbin/munin22:38:341
22522244719,414sleep122566-21ntpq22:13:471
4944241417,382sleep15004-21/usr/sbin/munin21:18:421
1047023858,337sleep310613-21latency_hist19:48:253
17172232315,295sleep317175-21cron21:58:223
822423115,267sleep322178-1kworker/3:2H21:28:463
18922230126,260sleep318975-21tune2fs20:14:393
24989224643,184sleep325024-21latency20:38:453
26743224172,150sleep326804-21sendmail_mailqu20:43:503
1195522408,194sleep312011-21sendmail19:53:223
10827223520,197sleep111038-21latency_hist21:38:251
10827223520,197sleep111038-21latency_hist21:38:251
23711222611,196sleep223779-21awk20:33:482
24477222336,70sleep30-21swapper/300:03:463
24477222336,70sleep30-21swapper/300:03:463
14084222225,183sleep314127-21latency_hist21:48:243
3220922059,83sleep332272-21mailstats22:43:533
3220922059,83sleep332272-21mailstats22:43:533
918220410,138sleep31023-21fschecks_time22:48:423
26362220018,164sleep226429-21sshd20:43:412
17870219610,173sleep05666-21snmpd20:09:140
17870219610,173sleep05666-21snmpd20:09:140
1798219412,148sleep31833-21ls21:08:423
31814217626,135sleep331958-21cat00:28:253
9322217211,103sleep39474-21unixbench_multi23:13:543
346521716,122sleep30-21swapper/321:13:433
7631216911,145sleep37728-21sed23:08:533
7361216997,53sleep30-21swapper/319:38:243
13052216735,21sleep30-21swapper/323:28:243
267521448,97sleep32823-21sed19:23:253
1576021415,11sleep30-21swapper/320:03:523
31811214017,106sleep131847-21fschecks_count22:43:431
31811214017,106sleep131847-21fschecks_count22:43:431
303214027,14sleep30-21swapper/319:13:563
133121389,22sleep30-21swapper/321:08:223
11286213832,17sleep30-21swapper/321:39:243
2032421357,74sleep320384-21latency_hist22:08:243
24089213414,106sleep124119-21ntp_kernel_pll_22:18:471
24089213414,106sleep124119-21ntp_kernel_pll_22:18:471
30727213319,21sleep00-21swapper/019:08:400
29776213253,57sleep329951-21apt-config22:38:243
3245521316,85sleep30-21swapper/300:28:503
2333921317,16sleep30-21swapper/323:58:533
5504212849,15sleep30-21swapper/323:03:363
5504212849,15sleep30-21swapper/323:03:363
1431721288,106sleep314382-21sendmail_mailst19:58:513
30716212722,15sleep30-21swapper/319:08:393
1413212726,14sleep30-21swapper/319:18:413
2909221268,105sleep329130-21ls00:18:443
2974921255,35sleep30-21swapper/319:05:493
21068212471,15sleep30-21swapper/322:08:523
2051321235,11sleep30-21swapper/323:50:003
2782121225,12sleep30-21swapper/322:30:203
2782121225,12sleep30-21swapper/322:30:203
1593212253,42sleep20-21swapper/219:18:442
1593212253,42sleep20-21swapper/219:18:442
21993212013,93sleep0225-21systemd-journal22:13:250
989521197,77sleep39929-21ps21:33:503
989521197,77sleep39929-21ps21:33:503
2635421195,11sleep30-21swapper/322:26:213
6391211816,46sleep30-21swapper/321:23:413
3717211811,43sleep30-21swapper/322:58:233
30597211821,29sleep00-21swapper/022:39:060
23519211821,37sleep30-21swapper/322:18:253
23519211821,37sleep30-21swapper/322:18:253
7871211610,48sleep37926-21latency19:38:443
2677721165,10sleep30-21swapper/300:10:163
287421149,18sleep30-21swapper/319:23:383
12763211415,79sleep212825-21/usr/sbin/munin19:53:542
27351211318,33sleep20-21swapper/200:13:382
2717621139,87sleep027075-21/usr/sbin/munin18:58:390
18208211359,38sleep30-21swapper/323:43:463
18208211359,38sleep30-21swapper/323:43:463
23908211017,74sleep024060-21sed20:33:500
1331821105,34sleep30-21swapper/319:55:283
752210917,77sleep2878-21unixbench_multi21:03:572
5737210922,70sleep30-21swapper/319:33:233
16644210921,72sleep116684-21sh20:08:221
32381210818,70sleep132410-21/usr/sbin/munin00:28:471
31142210722,62sleep131289-21grep00:23:551
30616210617,31sleep10-21swapper/119:08:331
27720210613,77sleep127325-21/usr/sbin/munin00:13:491
22956210446,20sleep323105-21sed20:33:243
521021036,32sleep30-21swapper/321:18:483
1747721037,41sleep317459-21apt-get21:58:463
29325210213,71sleep029362-21ntp_states00:18:490
21832210220,65sleep121856-21munin-run22:13:221
17419210216,61sleep30-21swapper/320:08:533
17419210216,61sleep30-21swapper/320:08:533
13776210211,28sleep313841-21sendmail_mailst23:28:523
5683210120,60sleep25716-21munin-run19:33:222
20866210115,68sleep120976-21cat23:53:261
14385210013,71sleep014452-21switchtime19:58:520
314152998,33sleep30-21swapper/320:58:493
2805829919,64sleep128089-21idleruntime-cro22:33:211
2805829919,64sleep128089-21idleruntime-cro22:33:211
232452997,33sleep30-21swapper/320:33:373
280422986,33sleep30-21swapper/320:48:433
306502975,65sleep30-21swapper/320:58:043
1890729616,64sleep118916-21diskstats20:14:371
223692956,62sleep30-21swapper/323:58:203
1532729521,63sleep30-21swapper/321:50:063
1532729521,63sleep30-21swapper/321:50:063
1403529514,67sleep214175-21unixbench_singl23:28:552
719929420,59sleep17266-21fw_packets23:08:431
213622946,30sleep30-21swapper/320:23:433
332629330,33sleep10-21swapper/121:13:411
2406929312,65sleep024087-21netstat22:18:460
2406929312,65sleep024087-21netstat22:18:460
1103929311,67sleep211166-21cmp21:38:262
1049529314,61sleep010562-21memory23:18:480
596829218,58sleep06010-21wc23:03:490
596829218,58sleep06010-21wc23:03:490
232172915,30sleep30-21swapper/322:15:513
159062916,28sleep30-21swapper/323:35:353
112532917,29sleep30-21swapper/323:20:493
301772908,67sleep330169-21apt-get22:38:403
40382896,28sleep30-21swapper/322:58:393
40382896,28sleep30-21swapper/322:58:393
385128916,60sleep03975-21timerwakeupswit21:13:520
224862885,12sleep00-21swapper/020:28:410
2469428722,25sleep00-21swapper/020:38:340
3177628624,43sleep231813-21latency_hist00:28:242
105228614,58sleep15666-21snmpd21:04:481
90482857,17sleep00-21swapper/023:13:470
3114528512,60sleep131195-21basename20:58:431
1690128510,58sleep328481-21cyclictest23:38:503
1641828512,59sleep216459-21grep23:38:372
2429628422,43sleep024375-21tune2fs00:03:420
2429628422,43sleep024375-21tune2fs00:03:420
2718328316,51sleep127219-21fschecks_count18:58:411
921928214,55sleep29226-21df_inode19:43:402
366628221,38sleep10-21swapper/122:58:211
3209328221,31sleep20-21swapper/222:43:482
3209328221,31sleep20-21swapper/222:43:482
200902828,63sleep00-21swapper/022:05:580
2899428121,30sleep10-21swapper/100:18:421
2734728120,45sleep30-21swapper/300:13:353
1927328111,17sleep10-21swapper/122:03:451
1865428116,38sleep118791-21grep23:43:571
1865428116,38sleep118791-21grep23:43:571
495828013,19sleep10-21swapper/119:28:531
3266028016,44sleep0322-21timerwakeupswit00:28:550
2226628013,48sleep022337-21latency_hist20:28:250
785427915,34sleep10-21swapper/121:28:351
2836027933,26sleep30-21swapper/319:03:223
205452799,17sleep00-21swapper/023:50:260
680927811,21sleep10-21swapper/123:08:231
680927811,21sleep10-21swapper/123:08:231
2950727815,43sleep129601-21fschecks_time20:53:421
2929627815,48sleep129405-21timerandwakeup19:03:531
1512527823,34sleep00-21swapper/023:33:420
681227724,27sleep10-21swapper/121:23:511
3250627717,45sleep232587-21mailstats00:28:532
2061127718,28sleep30-21swapper/320:22:423
1558727727,31sleep00-21swapper/021:53:220
1558727727,31sleep00-21swapper/021:53:220
223272766,60sleep00-21swapper/023:57:490
983327519,41sleep09869-21ntpq21:33:470
983327519,41sleep09869-21ntpq21:33:470
1314427510,50sleep112749-21/usr/sbin/munin21:43:461
63322747,15sleep10-21swapper/119:33:451
473127423,31sleep04750-21apt-get21:18:330
456927412,28sleep00-21swapper/019:28:440
404927413,42sleep04139-21fschecks_time22:58:410
404927413,42sleep04139-21fschecks_time22:58:410
3250527416,42sleep032579-21cat19:13:490
192242747,43sleep20-21swapper/223:48:222
192242747,43sleep20-21swapper/223:48:222
175502745,16sleep00-21swapper/023:41:420
1702027416,41sleep117030-21/usr/sbin/munin20:08:431
1702027416,41sleep117030-21/usr/sbin/munin20:08:431
297927313,26sleep00-21swapper/022:53:520
2707427311,45sleep20-21swapper/218:58:332
2582927320,33sleep025905-21wc22:23:550
2140927314,45sleep221447-21ps20:23:462
301027218,19sleep00-21swapper/019:23:430
2906727214,43sleep029222-21unixbench_multi22:33:530
1632127215,40sleep016367-21/usr/sbin/munin21:53:590
1648027125,27sleep10-21swapper/121:54:041
644327011,40sleep06517-21/usr/sbin/munin21:23:420
2998727016,38sleep029996-21munin-run19:08:240
2146627011,46sleep121483-21netstat23:53:461
1830327018,36sleep218439-21apt-config20:13:242
1830327018,36sleep218439-21apt-config20:13:242
1579427010,13sleep00-21swapper/020:03:530
1383327016,29sleep20-21swapper/219:58:382
2865326912,41sleep228686-21latency22:33:432
1532426919,24sleep20-21swapper/223:33:482
130626817,29sleep00-21swapper/022:48:510
360326724,25sleep20-21swapper/221:13:462
289802677,49sleep00-21swapper/020:51:590
255292676,14sleep10-21swapper/122:23:481
216282676,15sleep00-21swapper/022:10:430
1743926717,35sleep217447-21acpi21:58:332
1493626720,24sleep10-21swapper/121:48:551
1493626720,24sleep10-21swapper/121:48:551
1225026721,25sleep00-21swapper/019:53:350
1203526717,19sleep10-21swapper/119:53:241
75726611,41sleep1758-21sshd19:14:471
3065126616,23sleep20-21swapper/222:39:082
289026619,31sleep2225-21systemd-journal21:13:222
2877526620,27sleep20-21swapper/219:03:422
2156626624,23sleep20-21swapper/223:53:512
189026620,32sleep11915-21cut21:08:441
1877826612,36sleep018749-21apt-get20:13:510
138926616,36sleep11454-21sed22:48:521
1174126619,23sleep20-21swapper/223:23:382
683426512,30sleep20-21swapper/223:08:242
683426512,30sleep20-21swapper/223:08:242
3137826519,29sleep231521-21sed19:08:552
2964526520,30sleep129694-21dpkg22:38:221
207826518,29sleep22108-21munin-run22:53:212
1898826519,23sleep20-21swapper/222:03:342
1413026519,26sleep00-21swapper/021:48:250
690426423,22sleep20-21swapper/221:23:522
28682645,46sleep00-21swapper/021:13:030
2797926415,24sleep20-21swapper/220:48:422
2627926423,20sleep20-21swapper/200:09:022
2437826415,22sleep10-21swapper/100:03:441
2437826415,22sleep10-21swapper/100:03:441
1668626419,19sleep20-21swapper/220:08:232
1617826417,24sleep10-21swapper/123:38:231
118112645,11sleep00-21swapper/019:51:290
988826318,29sleep110028-21grep19:43:551
2979826319,26sleep20-21swapper/220:53:472
2688026311,23sleep10-21swapper/120:43:521
742526218,21sleep00-21swapper/019:38:250
714426218,23sleep00-21swapper/023:08:420
443026220,27sleep10-21swapper/122:58:501
443026220,27sleep10-21swapper/122:58:501
2365126215,31sleep123141-21/usr/sbin/munin20:33:471
147426219,23sleep00-21swapper/019:18:420
147426219,23sleep00-21swapper/019:18:420
2525626129,15sleep20-21swapper/222:23:412
1479026113,22sleep20-21swapper/221:48:522
1479026113,22sleep20-21swapper/221:48:522
755126020,19sleep20-21swapper/223:08:522
3228526015,29sleep232397-21cat21:03:242
3172326015,27sleep031716-21apt-get22:43:380
3172326015,27sleep031716-21apt-get22:43:380
3096226014,28sleep00-21swapper/020:58:380
2769826015,19sleep00-21swapper/022:28:560
2769826015,19sleep00-21swapper/022:28:560
2471626016,17sleep20-21swapper/200:03:532
2471626016,17sleep20-21swapper/200:03:532
1256926017,19sleep00-21swapper/021:43:250
863925922,20sleep20-21swapper/223:13:372
2801425911,31sleep028032-21/usr/sbin/munin00:13:560
2742525919,19sleep20-21swapper/222:28:542
2742525919,19sleep20-21swapper/222:28:542
1227925917,23sleep112346-21df19:53:401
1215725912,31sleep012195-21ntp_states23:23:490
1019525913,28sleep110264-21df23:18:411
3027925818,22sleep00-21swapper/000:23:250
2103425817,21sleep10-21swapper/120:23:371
184572585,17sleep00-21swapper/021:59:570
1722525821,17sleep20-21swapper/221:58:252
1300025823,18sleep10-21swapper/123:28:221
912425718,20sleep00-21swapper/019:43:340
2576225719,21sleep125757-21apt-get00:08:361
2312525713,18sleep00-21swapper/023:58:500
183225718,19sleep10-21swapper/119:18:511
183225718,19sleep10-21swapper/119:18:511
1119125714,27sleep110708-21apt-get21:38:331
797125619,22sleep10-21swapper/119:38:461
395325615,24sleep23949-21apt-get22:58:342
395325615,24sleep23949-21apt-get22:58:342
3271025617,20sleep20-21swapper/219:13:552
2888525614,25sleep228904-21apt-get00:18:352
239725610,17sleep20-21swapper/222:53:382
2071425615,20sleep00-21swapper/020:23:240
1927725617,14sleep00-21swapper/023:48:240
1927725617,14sleep00-21swapper/023:48:240
1805825611,22sleep10-21swapper/121:59:091
1329125614,25sleep013284-21apt-get23:28:370
988825516,23sleep29894-21ntp_states21:33:492
988825516,23sleep29894-21ntp_states21:33:492
890325511,16sleep20-21swapper/219:43:242
2292725513,20sleep20-21swapper/223:58:442
1641725512,25sleep116411-21apt-get23:38:361
1383225513,25sleep113826-21apt-get19:58:371
661025420,18sleep00-21swapper/019:33:530
2485425412,24sleep10-21swapper/120:38:431
2396025420,20sleep10-21swapper/100:03:231
914125318,20sleep0284-21dbus-daemon21:33:230
327125315,19sleep10-21swapper/119:23:481
2757725315,16sleep00-21swapper/020:48:240
1329225315,20sleep10-21swapper/123:28:381
1116925319,17sleep20-21swapper/219:48:532
51852528,16sleep20-21swapper/221:18:472
40402524,39sleep20-21swapper/219:27:082
2426225215,18sleep20-21swapper/222:18:522
2426225215,18sleep20-21swapper/222:18:522
1145025214,22sleep211476-21munin-run23:23:222
637225112,23sleep20-21swapper/219:33:462
2292425122,15sleep20-21swapper/220:33:232
1277125113,22sleep212767-21apt-get21:43:352
2567925014,20sleep025728-21modprobe00:08:270
231742504,32sleep20-21swapper/222:15:212
58224914,19sleep00-21swapper/021:03:550
2285424913,18sleep10-21swapper/120:33:221
2055924912,22sleep220555-21apt-get22:08:352
1875624910,24sleep218229-21apt-get20:13:372
2937624815,17sleep00-21swapper/020:53:330
277882477,15sleep10-21swapper/120:48:331
909324612,17sleep10-21swapper/123:13:481
782924616,16sleep20-21swapper/221:28:322
1991624211,19sleep20-21swapper/223:48:462
2066224112,12sleep20-21swapper/220:23:222
203952415,24sleep10-21swapper/120:19:581
249312396,15sleep20-21swapper/220:38:442
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional