You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 23:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Fri Nov 28, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
316082452016,4491sleep229986-21kworker/2:010:39:542
2451129407,921sleep324552-21ntpq10:18:433
11677285413,822sleep011792-21/usr/sbin/munin07:53:540
23079283233,785sleep323135-21unin-run08:33:183
23079283233,785sleep323135-21unin-run08:33:183
1827928245,806sleep318322-21ntpq09:58:443
3033627358,691sleep330369-21unin-run10:38:183
3033627358,691sleep330369-21unin-run10:38:183
671225626,543sleep36747-21unin-run09:23:183
17311232312,291sleep317315-21acpi11:38:303
23309228074,112sleep323450-21head10:13:463
23309228074,112sleep323450-21head10:13:463
1227322758,255sleep312362-21timerandwakeup09:38:493
1227322758,255sleep312362-21timerandwakeup09:38:493
26045223931,194sleep325364-21/usr/sbin/munin08:39:453
12920223822,199sleep212922-21cron09:43:162
12920223822,199sleep212922-21cron09:43:162
26429223539,116sleep326476-21latency_hist12:08:183
9069222930,187sleep39043-21latency_hist07:48:193
514222414,126sleep3654-21latency_hist09:03:193
929522209,81sleep30-21swapper/311:13:273
21205221896,73sleep30-21swapper/310:08:373
21205221896,73sleep30-21swapper/310:08:373
16273221215,178sleep316405-21unixbench_singl08:08:513
833220811,182sleep0860-21awk12:28:430
7393219811,169sleep30-21swapper/309:23:433
7393219811,169sleep30-21swapper/309:23:433
411021856,169sleep35666-21snmpd07:30:183
31920218010,85sleep30-21swapper/310:43:183
27638218015,92sleep30-21swapper/310:29:253
32714217021,134sleep0318-21wc10:43:480
20702216516,50sleep30-21swapper/311:48:403
16248215914,133sleep316282-21ps11:33:463
13995215066,16sleep30-21swapper/308:03:303
17007214911,75sleep317127-21timerwakeupswit09:53:553
20196214833,27sleep30-21swapper/311:48:203
2879721460,7sleep00-21swapper/010:33:170
11164214323,30sleep30-21swapper/307:53:443
1152213916,32sleep30-21swapper/310:48:193
32205213812,112sleep027028-21kworker/u8:307:18:190
25120213813,23sleep10-21swapper/112:03:321
30893213732,93sleep330930-21grep07:13:343
11371213716,97sleep30-21swapper/309:38:163
5345213516,99sleep35492-21unixbench_singl10:58:493
5345213516,99sleep35492-21unixbench_singl10:58:493
24224213334,84sleep324351-21cat11:58:503
19049213341,71sleep30-21swapper/308:18:403
749621329,110sleep37581-21latency_hist11:08:183
2064221325,115sleep00-21swapper/010:07:250
18890213115,17sleep00-21swapper/008:18:370
1245521307,30sleep30-21swapper/307:58:373
26920212826,17sleep30-21swapper/308:43:283
13542212730,80sleep312632-21/usr/sbin/munin11:24:213
9740212630,85sleep35666-21snmpd09:32:353
9740212630,85sleep35666-21snmpd09:32:353
26082212424,80sleep326126-21ntpq10:23:443
24127212010,82sleep30-21swapper/308:34:193
14518212030,22sleep30-21swapper/309:48:183
10050212016,81sleep30-21swapper/307:48:483
20126211927,55sleep320248-21timerwakeupswit10:03:493
6282211716,84sleep26375-21uniq11:03:372
29308211615,26sleep329359-21sort07:08:333
26480211611,88sleep00-21swapper/012:08:190
7489211532,17sleep30-21swapper/307:43:183
31565211528,73sleep331622-21if_eth012:23:413
31565211528,73sleep331622-21if_eth012:23:413
24376211412,84sleep024405-21irqstats10:18:400
1356821135,96sleep00-21swapper/008:01:330
1327211326,18sleep30-21swapper/307:23:193
18440211226,27sleep318566-21grep08:18:203
3242821116,93sleep00-21swapper/012:25:440
3242821116,93sleep00-21swapper/012:25:440
26754211114,25sleep30-21swapper/312:08:353
30234211011,86sleep35666-21snmpd08:53:383
28021211019,66sleep128146-21timerwakeupswit07:03:511
22659211032,44sleep322785-21timerwakeupswit11:53:493
2845210914,80sleep22877-21latency_hist07:28:192
28938210816,28sleep10-21swapper/107:08:231
17394210811,28sleep017407-21irqstats08:13:400
5920210629,34sleep35956-21munin-run07:38:173
30216210523,17sleep30-21swapper/312:18:403
28021210513,71sleep228129-21latency_hist12:13:182
146821059,73sleep31487-21apt-get10:48:273
113921058,83sleep30-21swapper/312:28:503
30448210425,32sleep20-21swapper/210:38:212
30448210425,32sleep20-21swapper/210:38:212
1999210313,72sleep11513-21/usr/sbin/munin07:23:461
15046210317,16sleep30-21swapper/309:48:463
15046210317,16sleep30-21swapper/309:48:463
12135210319,66sleep112200-21ps09:38:471
12135210319,66sleep112200-21ps09:38:471
9858210221,64sleep19922-21awk07:48:461
9284210211,74sleep29292-21/usr/sbin/munin07:48:262
3290210223,38sleep327105-21cyclictest10:53:403
3290210223,38sleep327105-21cyclictest10:53:403
2457721018,82sleep020999-1kworker/0:0H12:00:000
4752210019,60sleep04793-21fschecks_count07:33:380
17432975,12sleep30-21swapper/309:04:493
3071229615,59sleep230805-21timerandwakeup08:54:012
45029516,65sleep1473-21pluginstate07:18:451
249329510,17sleep30-21swapper/309:08:403
796329414,63sleep28002-21irqstats11:08:382
2391729417,56sleep223993-21latency08:34:102
439829323,31sleep10-21swapper/110:58:191
439829323,31sleep10-21swapper/110:58:191
364329211,66sleep13672-21latency_hist09:13:191
3082029210,57sleep030588-21/usr/sbin/munin10:38:390
2871729217,13sleep30-21swapper/312:13:443
2871729217,13sleep30-21swapper/312:13:443
2748529215,57sleep127468-21apt-get10:28:341
824629121,53sleep18247-21cron09:28:171
824629121,53sleep18247-21cron09:28:171
506729114,60sleep20-21swapper/207:33:442
2710629113,60sleep227312-21latency_hist10:28:192
2314529117,57sleep023278-21sed08:33:190
2314529117,57sleep023278-21sed08:33:190
409929013,61sleep00-21swapper/009:13:380
2419529021,50sleep00-21swapper/008:34:200
17502908,16sleep30-21swapper/307:23:403
277128919,48sleep22280-21/usr/sbin/munin09:08:472
202362898,22sleep320228-21apt-get08:23:333
1802128919,54sleep218107-21fw_packets09:58:392
1320928922,39sleep013205-21apt-get09:43:300
1958428813,48sleep018860-21/usr/sbin/munin11:43:480
155362888,17sleep30-21swapper/311:33:173
57992870,12sleep15769-21latency09:18:391
2986128712,60sleep229887-21sh12:18:312
1227028723,20sleep20-21swapper/211:23:192
2978928620,34sleep10-21swapper/112:18:261
3173228516,43sleep231766-21fschecks_count08:58:382
282342859,64sleep020999-1kworker/0:0H12:13:240
282342859,64sleep020999-1kworker/0:0H12:13:240
277032859,55sleep127838-21head08:43:511
890328421,10sleep30-21swapper/311:11:583
3008428324,37sleep130078-21apt-get08:53:321
2624828322,45sleep126270-21pluginstate06:59:071
2624828322,45sleep126270-21pluginstate06:59:071
1179928323,28sleep30-21swapper/311:19:003
746628212,52sleep27486-21awk07:43:172
551828210,59sleep05523-21/usr/sbin/munin09:18:350
445128220,42sleep24374-21latency_hist10:58:202
445128220,42sleep24374-21latency_hist10:58:202
2860628111,39sleep30-21swapper/307:08:173
97328026,34sleep10-21swapper/110:48:171
2413628020,41sleep224180-21/usr/sbin/munin10:18:342
776527913,34sleep20-21swapper/207:43:342
513227925,27sleep10-21swapper/107:33:471
2113727910,52sleep021282-21/usr/sbin/munin11:48:490
1999027932,23sleep10-21swapper/108:23:191
2593427829,27sleep10-21swapper/108:39:431
264372779,53sleep026580-21sed06:59:100
264372779,53sleep026580-21sed06:59:100
285112767,53sleep30-21swapper/308:48:333
101922760,7sleep00-21swapper/007:48:480
3215327523,31sleep20-21swapper/207:18:172
2905827522,23sleep10-21swapper/110:33:311
2905827522,23sleep10-21swapper/110:33:311
2154327512,47sleep221671-21ls08:28:202
1692727523,25sleep20-21swapper/209:53:542
1628627514,42sleep016360-21sendmail_mailtr11:33:470
2908327424,26sleep10-21swapper/108:48:471
2665227420,38sleep026669-21idleruntime-cro08:43:170
1496427417,28sleep10-21swapper/109:48:441
1496427417,28sleep10-21swapper/109:48:441
1063327413,46sleep210721-21latency_hist11:18:192
3183127314,39sleep231837-21/usr/sbin/munin12:23:482
3183127314,39sleep231837-21/usr/sbin/munin12:23:482
104227327,25sleep20-21swapper/212:28:492
829927216,14sleep00-21swapper/009:28:200
829927216,14sleep00-21swapper/009:28:200
561527217,28sleep20-21swapper/209:18:372
2168327220,21sleep10-21swapper/110:08:481
2168327220,21sleep10-21swapper/110:08:481
1688427212,45sleep216912-21sed08:13:182
1313427220,28sleep20-21swapper/207:58:522
1218027220,36sleep112269-21apt-key11:23:181
2967727114,11sleep00-21swapper/007:08:420
2234427121,27sleep00-21swapper/010:13:190
2234427121,27sleep00-21swapper/010:13:190
1003127118,35sleep19296-21/usr/sbin/munin11:13:491
397527011,47sleep24012-21fschecks_count09:13:362
2757627020,28sleep00-21swapper/008:43:500
1587127016,35sleep215895-21fschecks_count11:33:372
2932726917,24sleep00-21swapper/010:33:380
2932726917,24sleep00-21swapper/010:33:380
2536026921,21sleep30-21swapper/306:58:203
2536026921,21sleep30-21swapper/306:58:203
1331126914,25sleep10-21swapper/111:24:191
1305626913,23sleep10-21swapper/107:58:511
2991226822,27sleep20-21swapper/207:08:482
2849926816,26sleep20-21swapper/212:13:382
2849926816,26sleep20-21swapper/212:13:382
2576926817,28sleep10-21swapper/110:23:381
2414426817,25sleep20-21swapper/211:58:482
2304226820,27sleep20-21swapper/208:33:172
2304226820,27sleep20-21swapper/208:33:172
2257926810,44sleep10-21swapper/110:13:281
2257926810,44sleep10-21swapper/110:13:281
1429426824,21sleep20-21swapper/211:28:362
83926712,40sleep2913-21cut09:03:342
48832675,28sleep30-21swapper/309:14:543
2685326717,24sleep10-21swapper/112:08:371
2636826712,33sleep226381-21/usr/sbin/munin08:39:482
146426715,37sleep020999-1kworker/0:0H10:48:240
3112526622,21sleep00-21swapper/012:23:180
2146326623,25sleep20-21swapper/210:08:442
2146326623,25sleep20-21swapper/210:08:442
172426620,24sleep11753-21munin-run12:33:161
104126620,23sleep20-21swapper/210:48:182
3215226515,33sleep032238-21mailstats08:58:480
2742026519,21sleep00-21swapper/012:08:480
2694426514,36sleep027003-21/usr/sbin/munin10:28:170
1741426514,30sleep217420-21diskstats11:38:372
1649826514,28sleep00-21swapper/009:53:440
744626412,37sleep17482-21idleruntime-cro11:08:161
2081126416,21sleep00-21swapper/008:23:500
18202645,16sleep10-21swapper/109:05:471
1704526419,22sleep10-21swapper/111:38:161
2627126320,18sleep20-21swapper/206:59:082
2627126320,18sleep20-21swapper/206:59:082
2426326317,27sleep10-21swapper/108:34:211
2262726320,25sleep00-21swapper/010:13:310
2262726320,25sleep00-21swapper/010:13:310
2040526313,35sleep220423-21/usr/sbin/munin11:48:272
97442625,48sleep20-21swapper/209:32:382
97442625,48sleep20-21swapper/209:32:382
2396426212,20sleep10-21swapper/111:58:431
90022615,45sleep10-21swapper/107:48:051
752226115,25sleep00-21swapper/007:43:190
396626112,24sleep10-21swapper/109:13:351
3145526117,28sleep131465-21/usr/sbin/munin12:23:391
3145526117,28sleep131465-21/usr/sbin/munin12:23:391
3140326116,28sleep00-21swapper/008:58:190
2915626121,23sleep20-21swapper/208:48:482
275312619,40sleep20-21swapper/210:29:042
2335726122,17sleep20-21swapper/211:58:202
1779026122,20sleep10-21swapper/108:13:501
167372618,40sleep00-21swapper/008:12:190
471426011,28sleep14720-21diskstats10:58:351
471426011,28sleep14720-21diskstats10:58:351
368926013,25sleep00-21swapper/007:28:500
2900426013,23sleep00-21swapper/008:48:460
2703326015,20sleep20-21swapper/208:43:352
2247526013,28sleep00-21swapper/011:53:460
2215926012,29sleep10-21swapper/108:28:401
2215926012,29sleep10-21swapper/108:28:401
1654526023,20sleep10-21swapper/109:53:451
1591426017,26sleep00-21swapper/009:51:540
1591426017,26sleep00-21swapper/009:51:540
1554826014,30sleep015689-21latency_hist11:33:190
1058326016,25sleep00-21swapper/011:18:160
597025911,26sleep00-21swapper/011:03:190
597025911,26sleep00-21swapper/011:03:190
321252595,42sleep10-21swapper/107:18:001
2872925920,19sleep20-21swapper/207:08:202
214425917,19sleep00-21swapper/007:23:490
192732596,12sleep10-21swapper/111:43:411
1447125918,26sleep20-21swapper/209:48:172
3164225817,24sleep131636-21apt-get08:58:331
2873825815,28sleep128797-21/usr/sbin/munin12:13:461
2873825815,28sleep128797-21/usr/sbin/munin12:13:461
1944525815,26sleep119442-21apt-get10:03:301
1876725813,29sleep218779-21/usr/sbin/munin08:18:352
1859525818,24sleep118617-21/usr/sbin/munin11:43:171
1409225816,21sleep00-21swapper/008:03:350
791725715,22sleep10-21swapper/111:08:371
2440625716,26sleep124495-21load10:18:411
1816725712,24sleep00-21swapper/009:58:410
1142425715,19sleep20-21swapper/209:38:192
1970825616,26sleep219424-21/usr/sbin/munin10:03:402
1323625613,26sleep013310-21sensors_temp11:24:180
863825514,22sleep08666-21users07:43:520
238725513,24sleep02394-21diskstats09:08:370
216112556,11sleep10-21swapper/111:52:131
1744825510,17sleep00-21swapper/011:38:400
1681125517,17sleep10-21swapper/108:13:171
1525325516,18sleep20-21swapper/208:08:172
1468225519,18sleep10-21swapper/108:03:481
1146525520,16sleep20-21swapper/207:53:512
1032425517,23sleep110382-21irqstats09:33:381
74652546,16sleep10-21swapper/107:43:161
73602548,30sleep2195-21jbd2/mmcblk2p2-09:23:422
73602548,30sleep2195-21jbd2/mmcblk2p2-09:23:422
334925414,24sleep03391-21latency10:53:410
334925414,24sleep03391-21latency10:53:410
3263125412,22sleep20-21swapper/207:18:382
314425414,24sleep23155-21df_inode10:53:352
314425414,24sleep23155-21df_inode10:53:352
3118525421,17sleep10-21swapper/110:38:481
2069225416,21sleep120698-21ntp_states08:23:471
1759725417,20sleep20-21swapper/208:13:472
1534825410,28sleep20-21swapper/209:48:542
1534825410,28sleep20-21swapper/209:48:542
853725313,24sleep08532-21apt-get09:28:320
853725313,24sleep08532-21apt-get09:28:320
2319125315,20sleep20-21swapper/210:13:452
2319125315,20sleep20-21swapper/210:13:452
1093525312,19sleep00-21swapper/009:33:490
2732325213,24sleep027342-21apt-get07:03:290
1850925217,22sleep117874-21/usr/sbin/munin09:58:501
143182525,11sleep10-21swapper/109:46:181
12222528,32sleep00-21swapper/007:23:000
64392519,15sleep00-21swapper/011:03:390
423325112,23sleep24234-21sshd07:31:542
2561125012,15sleep20-21swapper/212:03:452
146725012,22sleep2985-21apt-get10:48:262
889824916,18sleep00-21swapper/011:11:550
164524917,17sleep10-21swapper/110:49:181
308624812,20sleep13083-21apt-get07:28:301
2242824811,19sleep122465-21ntp_states11:53:431
146552487,15sleep10-21swapper/111:28:431
1164524813,16sleep00-21swapper/009:38:300
1164524813,16sleep00-21swapper/009:38:300
1096724812,20sleep110975-21diskstats07:53:381
9672476,14sleep00-21swapper/009:03:360
1113024712,20sleep011143-21irqstats11:18:410
193112468,15sleep20-21swapper/211:43:422
131672468,15sleep20-21swapper/211:24:162
899724510,20sleep19003-21ntp_states09:28:451
899724510,20sleep19003-21ntp_states09:28:451
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional