You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-25 - 22:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Tue Mar 25, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
430629378,917sleep04317-21ntpq07:15:520
13375271717,681sleep313428-21/usr/sbin/munin11:30:443
2631269421,657sleep12641-21/usr/sbin/munin10:50:421
14875266912,638sleep214956-21/usr/sbin/munin07:50:462
14875266912,638sleep214956-21/usr/sbin/munin07:50:462
2130125918,569sleep021337-21ntpq12:00:450
18254255115,518sleep318271-21cron09:50:243
21476251020,473sleep121539-21/usr/sbin/munin12:00:501
233624659,448sleep323851-1kworker/3:0H07:11:023
14275245312,427sleep114276-21sshd09:32:141
4958239610,324sleep05008-21awk09:00:520
1026623168,296sleep010295-21grep09:20:460
2267722910,83sleep00-21swapper/008:15:460
1198822699,199sleep011594-21/usr/sbin/munin11:25:490
28474225197,68sleep00-21swapper/012:26:000
26211222826,191sleep0370-21snmpd10:19:360
989022098,138sleep09871-21latency_hist09:20:260
17253220834,115sleep00-21swapper/011:45:430
30864220512,164sleep214440-21kworker/u8:408:40:432
6299219913,173sleep36359-21basename09:05:473
1242219837,113sleep01385-21latency_hist08:50:260
1242219837,113sleep01385-21latency_hist08:50:260
1501621977,175sleep015047-21latency07:50:480
1501621977,175sleep015047-21latency07:50:480
22160219514,116sleep022364-21cat10:05:270
16570219415,132sleep016588-21uname07:55:480
1068218659,105sleep01114-21fschecks_count10:45:440
22697218117,147sleep022729-21fschecks_time10:06:000
22697218117,147sleep022729-21fschecks_time10:06:000
20527217518,135sleep320611-21latency_hist10:00:253
8160217016,137sleep18214-21latency11:10:481
26558216721,128sleep226552-21apt-get10:20:352
26558216721,128sleep226552-21apt-get10:20:352
5192216210,137sleep25301-21cat09:00:582
26961216148,92sleep00-21swapper/010:20:450
26961216148,92sleep00-21swapper/010:20:450
8531215862,81sleep08582-21sort07:30:410
29957215350,87sleep00-21swapper/010:30:460
5779215230,102sleep00-21swapper/007:20:520
2930215287,50sleep00-21swapper/010:50:490
5213214710,123sleep24727-21/usr/sbin/munin11:00:482
4539214720,110sleep14666-21unixbench_multi07:15:571
427021466,27sleep00-21swapper/010:57:510
1294921460,7sleep00-21swapper/007:45:260
1746214520,53sleep21774-21munin-run07:10:252
19600214159,64sleep019368-21/usr/sbin/munin08:05:450
19600214159,64sleep019368-21/usr/sbin/munin08:05:450
8212213814,102sleep00-21swapper/007:30:250
1839213620,37sleep02013-21cat07:10:270
3478213518,96sleep23518-21proc_pri08:55:532
31152213538,82sleep031193-21grep08:40:490
15445213510,66sleep00-21swapper/009:40:360
13769213336,80sleep013849-21xargs07:45:550
16156213212,101sleep015598-21/usr/sbin/munin11:40:550
957321319,111sleep3370-21snmpd09:16:543
15159212717,32sleep015179-21awk09:40:240
2543021228,50sleep025336-21cron12:15:260
2543021228,50sleep025336-21cron12:15:260
28984212015,90sleep00-21swapper/008:35:370
2317211920,19sleep10-21swapper/110:50:251
2896211816,21sleep00-21swapper/007:11:280
916521166,16sleep00-21swapper/009:14:530
916521166,16sleep00-21swapper/009:14:530
18706211528,18sleep00-21swapper/009:50:460
18706211528,18sleep00-21swapper/009:50:460
1830211510,24sleep00-21swapper/008:50:510
1830211510,24sleep00-21swapper/008:50:510
18276211535,18sleep018333-21systemd-run09:50:250
21477211323,72sleep30-21swapper/308:10:533
11703211225,65sleep311716-21fschecks_count11:25:443
13267211110,19sleep00-21swapper/009:30:390
23880211010,86sleep023790-21cron08:20:270
31965210931,19sleep031854-21/usr/sbin/munin10:40:390
28109210919,18sleep00-21swapper/010:25:380
28109210919,18sleep00-21swapper/010:25:380
21656210923,56sleep321804-21sed12:00:523
11696210912,17sleep20-21swapper/207:40:452
1102721095,92sleep00-21swapper/011:21:290
7522210723,60sleep27687-21dpkg09:10:272
29508210715,74sleep329595-21/usr/sbin/munin06:55:453
25709210714,75sleep025747-21fschecks_count08:25:440
31669210510,79sleep031819-21sed07:01:100
21554210425,12sleep00-21swapper/008:10:540
19157210414,75sleep219127-21latency_hist08:05:272
27409210311,75sleep027428-21apt-get08:30:360
22369210316,71sleep122502-21cat10:05:281
22369210316,71sleep122502-21cat10:05:281
1617121039,12sleep30-21swapper/307:55:273
10790210211,76sleep210935-21/usr/sbin/munin07:36:002
4822210111,71sleep10-21swapper/109:00:491
997729915,69sleep210044-21latency_hist09:20:282
837629910,28sleep00-21swapper/011:10:550
639129822,50sleep26423-21fschecks_count11:05:442
256492989,80sleep10-21swapper/112:16:161
256492989,80sleep10-21swapper/112:16:161
221102975,82sleep20-21swapper/212:04:182
73932967,67sleep00-21swapper/009:10:250
272329623,51sleep12746-21cut08:55:241
272329623,51sleep12746-21cut08:55:241
1950829617,59sleep218722-21/usr/sbin/munin11:50:582
1866329625,52sleep318703-21sort09:50:453
1866329625,52sleep318703-21sort09:50:453
235472956,79sleep00-21swapper/012:07:430
1638829415,61sleep116424-21/usr/sbin/munin07:55:451
920629322,51sleep39220-21idleruntime-cro09:15:243
920629322,51sleep39220-21idleruntime-cro09:15:243
60652939,69sleep06170-21latency_hist11:05:260
1138129315,23sleep20-21swapper/211:25:262
96512928,10sleep30-21swapper/307:33:433
2144029218,53sleep221476-21wc08:10:522
6472919,71sleep30-21swapper/310:44:503
44122915,11sleep10-21swapper/110:59:421
257682908,12sleep20-21swapper/212:17:462
257682908,12sleep20-21swapper/212:17:462
2047629020,51sleep120497-21latency_hist10:00:231
113629012,14sleep20-21swapper/208:50:242
113629012,14sleep20-21swapper/208:50:242
3124028914,54sleep131299-21/usr/sbin/munin07:00:471
234282894,11sleep10-21swapper/112:06:191
2121928912,62sleep121225-21ntp_states10:00:481
142292885,72sleep00-21swapper/011:32:310
2478428717,32sleep30-21swapper/310:11:163
2478428717,32sleep30-21swapper/310:11:163
2441028723,40sleep224365-21apt-get10:11:022
2441028723,40sleep224365-21apt-get10:11:022
1984928710,60sleep219856-21ntp_states08:05:512
1984928710,60sleep219856-21ntp_states08:05:512
119212877,16sleep011667-21/usr/sbin/munin09:25:470
2468128618,51sleep124783-21/usr/sbin/munin12:10:551
2468128618,51sleep124783-21/usr/sbin/munin12:10:551
2207028620,51sleep31-21systemd10:05:263
151172868,66sleep10-21swapper/109:39:531
767728510,60sleep07833-21date11:10:270
2454428522,45sleep324051-21/usr/sbin/munin12:10:533
2454428522,45sleep324051-21/usr/sbin/munin12:10:533
2405028517,45sleep20-21swapper/212:10:362
2405028517,45sleep20-21swapper/212:10:362
141382857,66sleep20-21swapper/211:31:202
244802847,14sleep00-21swapper/010:11:040
244802847,14sleep00-21swapper/010:11:040
38082836,67sleep30-21swapper/310:55:123
2809128313,53sleep228103-21apt-get10:25:362
2809128313,53sleep228103-21apt-get10:25:362
272742838,67sleep20-21swapper/212:22:142
272742838,67sleep20-21swapper/212:22:142
2602328318,19sleep30-21swapper/312:20:263
2602328318,19sleep30-21swapper/312:20:263
746028224,37sleep17483-21munin-run09:10:261
549628218,20sleep30-21swapper/307:20:473
2873028217,50sleep128745-21systemd-run08:35:261
3007328113,55sleep330075-21sshd08:38:143
2737728120,50sleep00-21swapper/012:23:330
2737728120,50sleep00-21swapper/012:23:330
2397928130,29sleep30-21swapper/310:10:273
2397928130,29sleep30-21swapper/310:10:273
1230328125,25sleep10-21swapper/111:25:581
828328018,46sleep18311-21latency_hist07:30:261
536628020,26sleep10-21swapper/111:00:511
34592807,11sleep00-21swapper/008:55:510
865627915,26sleep30-21swapper/311:10:583
2619927914,52sleep326246-21switchtime08:25:553
234762795,64sleep30-21swapper/308:16:263
204502795,65sleep20-21swapper/210:00:042
2632427826,27sleep126397-21date10:20:261
1760827824,28sleep20-21swapper/208:00:272
79142779,21sleep20-21swapper/209:10:312
79142779,21sleep20-21swapper/209:10:312
689827724,28sleep10-21swapper/111:05:561
2823327715,38sleep228267-21/usr/sbin/munin12:25:542
234027716,40sleep12297-21apt-get07:11:041
1855727722,40sleep318705-21unixbench_singl08:01:003
1810627711,53sleep118181-1kworker/1:0H11:46:161
2696127622,33sleep227032-21dpkg08:30:262
188932766,59sleep00-21swapper/008:02:430
183852765,63sleep20-21swapper/211:49:532
113172766,10sleep00-21swapper/007:40:140
313072755,11sleep00-21swapper/010:36:400
277427518,25sleep30-21swapper/308:55:263
277427518,25sleep30-21swapper/308:55:263
267892759,51sleep326866-21swap12:20:503
267892759,51sleep326866-21swap12:20:503
1076027516,42sleep310897-21timerwakeupswit09:20:573
5632749,48sleep10-21swapper/107:05:431
236027419,34sleep22516-21latency_hist10:50:262
198362746,11sleep00-21swapper/011:53:100
95842736,11sleep00-21swapper/011:17:490
516627320,24sleep30-21swapper/307:20:273
3100527316,30sleep30-21swapper/310:35:263
249932736,12sleep00-21swapper/012:11:020
249932736,12sleep00-21swapper/012:11:020
202862736,12sleep00-21swapper/009:57:560
152142736,11sleep00-21swapper/011:38:530
152142736,11sleep00-21swapper/011:38:530
3111727218,40sleep331188-21idleruntime07:00:453
3095527216,38sleep130983-21munin-run10:35:241
2969027221,30sleep20-21swapper/210:30:402
202792727,11sleep00-21swapper/011:55:400
1527427224,22sleep30-21swapper/307:50:553
1527427224,22sleep30-21swapper/307:50:553
2740127113,42sleep326900-21apt-get08:30:313
229462719,15sleep20-21swapper/212:05:512
530227017,35sleep35449-21unixbench_singl09:00:593
2164627019,33sleep10-21swapper/108:10:551
1838627022,25sleep10-21swapper/108:00:581
1562327014,30sleep20-21swapper/211:40:402
901126918,28sleep10-21swapper/107:30:551
711326914,38sleep37205-21users11:05:583
2568626922,22sleep10-21swapper/110:15:251
2568626922,22sleep10-21swapper/110:15:251
1483126921,22sleep10-21swapper/107:50:451
1483126921,22sleep10-21swapper/107:50:451
1169326922,24sleep30-21swapper/309:25:423
2609026820,25sleep10-21swapper/108:25:531
2252926818,30sleep222525-21apt-get08:15:402
1599026812,22sleep30-21swapper/311:40:493
3164626727,22sleep30-21swapper/310:40:263
2965526737,16sleep10-21swapper/106:55:471
2796226714,38sleep21-21systemd08:31:232
1997926711,13sleep10-21swapper/108:05:551
1997926711,13sleep10-21swapper/108:05:551
1728326714,25sleep20-21swapper/209:45:472
113342667,42sleep30-21swapper/307:40:253
310226516,20sleep10-21swapper/108:55:441
3074626520,21sleep10-21swapper/108:40:341
671426422,21sleep20-21swapper/207:25:272
1937426415,34sleep119507-21timerwakeupswit11:50:571
1069126415,18sleep20-21swapper/209:20:562
3208726318,24sleep20-21swapper/210:40:412
316326315,25sleep20-21swapper/210:50:552
2698726320,21sleep10-21swapper/112:20:521
2698726320,21sleep10-21swapper/112:20:521
229926317,28sleep21766-21apt-get07:10:372
852626221,20sleep20-21swapper/211:10:572
2993226224,19sleep00-21swapper/006:55:540
2838126217,20sleep10-21swapper/110:25:471
2838126217,20sleep10-21swapper/110:25:471
2085426217,19sleep20-21swapper/210:00:412
1442826216,28sleep114486-21awk07:50:241
1334926224,24sleep30-21swapper/309:30:423
3267726114,23sleep30-21swapper/307:05:333
2407826115,30sleep224074-21apt-get08:20:412
1538026120,19sleep30-21swapper/311:40:253
1538026120,19sleep30-21swapper/311:40:253
1465626119,24sleep114867-21modprobe11:35:261
1465626119,24sleep114867-21modprobe11:35:261
1337826112,31sleep313431-21/usr/sbin/munin07:45:463
128002616,10sleep20-21swapper/209:28:312
2905626012,22sleep20-21swapper/208:35:392
2900226018,26sleep129055-21df08:35:381
2787126014,31sleep127950-21latency_hist10:25:261
2787126014,31sleep127950-21latency_hist10:25:261
1874926013,32sleep318819-21df11:50:423
1495226015,22sleep214463-21apt-get11:35:362
1495226015,22sleep214463-21apt-get11:35:362
1353826017,27sleep213569-21/usr/sbin/munin07:45:492
103512608,39sleep110057-21/usr/sbin/munin07:35:511
72125914,27sleep1740-21munin-run10:45:251
55325917,20sleep30-21swapper/308:45:573
55325917,20sleep30-21swapper/308:45:573
2859325924,20sleep10-21swapper/112:26:011
202952598,41sleep10-21swapper/111:55:531
3086925818,21sleep30-21swapper/308:40:443
206925816,18sleep20-21swapper/208:50:572
206925816,18sleep20-21swapper/208:50:572
808825720,17sleep10-21swapper/109:10:581
808825720,17sleep10-21swapper/109:10:581
741725713,20sleep20-21swapper/207:25:552
2829825716,20sleep30-21swapper/310:25:453
2829825716,20sleep30-21swapper/310:25:453
2785825718,23sleep327854-21apt-get12:25:383
2274825715,18sleep10-21swapper/108:15:481
1327625714,22sleep10-21swapper/111:30:421
95082566,36sleep10-21swapper/111:16:501
437525614,24sleep20-21swapper/210:59:142
305232565,11sleep20-21swapper/206:58:542
2538425619,19sleep325518-21latency_hist08:25:263
1676125615,18sleep20-21swapper/209:45:272
1521125623,16sleep20-21swapper/209:40:262
1292125619,18sleep30-21swapper/307:45:263
1199425615,24sleep112025-21nfs_client07:40:521
562925514,16sleep10-21swapper/107:20:491
1046925512,26sleep110518-21ntp_states11:20:451
1029925513,21sleep10-21swapper/109:20:471
876525414,17sleep20-21swapper/207:30:472
744125414,18sleep30-21swapper/307:25:563
39752548,35sleep30-21swapper/308:56:353
386125412,15sleep20-21swapper/207:15:422
3233225416,23sleep132329-21apt-get08:45:391
3233225416,23sleep132329-21apt-get08:45:391
3142625414,20sleep10-21swapper/110:38:141
3097225414,23sleep230966-21apt-get07:00:402
1982325413,28sleep119824-21sshd09:55:151
1982325413,28sleep119824-21sshd09:55:151
1865525416,23sleep218662-21diskstats09:50:442
1865525416,23sleep218662-21diskstats09:50:442
1544325414,23sleep315438-21apt-get09:40:353
1208925421,18sleep20-21swapper/211:25:552
1716725312,24sleep317163-21apt-get11:45:383
207412529,25sleep20-21swapper/212:00:262
1744725214,22sleep117453-21ntp_states09:45:501
1214425217,20sleep112151-21ntp_states09:25:521
983325113,23sleep29888-21latency_hist11:20:262
98025011,22sleep3976-21apt-get10:45:393
2338525013,21sleep30-21swapper/312:05:563
265492499,16sleep30-21swapper/310:20:323
265492499,16sleep30-21swapper/310:20:323
2447924917,16sleep10-21swapper/108:20:521
1950824913,20sleep319598-21perl08:05:443
1950824913,20sleep319598-21perl08:05:443
168782499,14sleep10-21swapper/111:45:241
1015124913,19sleep30-21swapper/311:20:393
522824817,18sleep30-21swapper/311:00:493
2627224814,21sleep326303-21munin-run10:20:243
446724611,15sleep30-21swapper/311:00:243
227962469,21sleep222845-21irqstats10:06:022
227962469,21sleep222845-21irqstats10:06:022
1326224612,17sleep20-21swapper/209:30:372
1594024313,13sleep10-21swapper/109:40:501
1487024211,19sleep3370-21snmpd09:36:403
262592368,10sleep20-21swapper/210:20:152
313762358,19sleep20-21swapper/210:37:352
951523411,13sleep30-21swapper/311:16:563
2026723310,13sleep30-21swapper/311:55:313
99202328,13sleep10-21swapper/109:20:271
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional