You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 15:52

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Nov 15, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
8667248123,441sleep38684-21/usr/sbin/munin11:04:153
1871123370,6sleep20-21swapper/209:54:052
16096233611,310sleep116140-21ntpq09:44:281
5500223950,84sleep25649-21unixbench_multi09:09:452
31011223998,29sleep231047-21run-parts10:34:022
31011223998,29sleep231047-21run-parts10:34:022
2902223210,210sleep23043-21sed10:44:362
10482220710,184sleep025064-1kworker/0:2H11:09:160
2398922068,125sleep224064-21tr10:09:322
17048220611,141sleep2362-21snmpd08:04:332
23247220593,26sleep20-21swapper/211:49:192
9495219815,63sleep20-21swapper/207:39:382
9495219815,63sleep20-21swapper/207:39:382
12283219612,134sleep212282-21ntp_states07:49:312
6915219118,115sleep20-21swapper/210:59:052
1693218712,157sleep21759-21cpuspeed08:59:202
13928218340,104sleep20-21swapper/209:39:042
8479218213,153sleep18549-21latency_hist07:39:041
8479218213,153sleep18549-21latency_hist07:39:041
31114218246,116sleep2362-21snmpd07:09:022
10746217612,17sleep10-21swapper/111:09:381
16725217210,103sleep216699-21/usr/sbin/munin11:29:212
16725217210,103sleep216699-21/usr/sbin/munin11:29:212
6929216926,122sleep16936-21cat07:34:041
8817216810,135sleep00-21swapper/011:04:240
19908216813,136sleep219948-21latency08:14:262
6847216517,127sleep36908-21sendmail-msp07:34:023
2969221598,79sleep229745-21grep07:04:052
2969221598,79sleep229745-21grep07:04:052
9466215645,93sleep29476-21acpi09:24:142
20158215624,117sleep30-21swapper/309:59:033
738215529,19sleep20-21swapper/212:19:352
31092215474,66sleep231137-21fschecks_count08:49:232
22631215259,14sleep20-21swapper/210:04:352
15557215016,23sleep20-21swapper/211:24:402
15557215016,23sleep20-21swapper/211:24:402
31033214912,64sleep20-21swapper/212:14:182
20116214941,12sleep20-21swapper/209:59:022
3075214625,40sleep33155-21switchtime07:19:413
3075214625,40sleep33155-21switchtime07:19:413
19831214620,23sleep20-21swapper/211:39:142
8962214518,23sleep20-21swapper/211:04:262
2757321419,26sleep20-21swapper/212:03:482
2757321419,26sleep20-21swapper/212:03:482
24873213848,75sleep224978-21fschecks_time11:54:232
24873213848,75sleep224978-21fschecks_time11:54:232
27299213615,25sleep20-21swapper/210:19:372
1769213620,34sleep31812-21fschecks_count12:24:243
1156521367,37sleep20-21swapper/207:49:032
1156521367,37sleep20-21swapper/207:49:032
15247213528,34sleep2362-21snmpd09:41:422
14010213416,104sleep014119-21timerandwakeup07:54:340
19945213320,92sleep019957-21fschecks_count11:39:240
15457213381,16sleep20-21swapper/207:59:332
25069213231,32sleep20-21swapper/210:14:162
14822212914,96sleep014945-21cat09:39:340
756212559,19sleep20-21swapper/210:39:232
31747212518,16sleep20-21swapper/210:35:072
7651212331,70sleep27654-21cron09:19:022
2825521216,12sleep20-21swapper/212:04:292
27587212029,79sleep227588-21sshd08:38:492
27587212029,79sleep227588-21sshd08:38:492
6395211916,18sleep00-21swapper/009:14:190
284921197,11sleep20-21swapper/212:27:502
791721188,98sleep27923-21cat09:19:142
529621175,13sleep20-21swapper/207:29:002
529621175,13sleep20-21swapper/207:29:002
2822211710,93sleep12899-21find10:44:351
31563211520,79sleep131041-21apt-get10:34:171
2229121145,13sleep20-21swapper/208:22:012
1452211423,30sleep30-21swapper/308:59:043
1067211422,79sleep31042-21fschecks_count07:14:283
1067211422,79sleep31042-21fschecks_count07:14:283
13543211353,20sleep229613-21cyclictest11:19:072
13543211353,20sleep229613-21cyclictest11:19:072
31574211217,27sleep131610-21df_inode07:09:091
25951211218,24sleep20-21swapper/208:34:052
24773211112,16sleep00-21swapper/010:14:020
186021085,91sleep00-21swapper/007:14:450
186021085,91sleep00-21swapper/007:14:450
806521077,90sleep20-21swapper/210:59:462
3140221067,82sleep231529-21diskstats07:09:062
2285210617,76sleep12357-21date07:19:061
2285210617,76sleep12357-21date07:19:061
21717210636,29sleep222-21ksoftirqd/211:44:222
18502210621,44sleep20-21swapper/208:09:302
28786210514,71sleep228870-21xargs10:24:332
2372121055,10sleep20-21swapper/208:25:292
15472210512,77sleep115556-21/usr/sbin/munin11:24:391
15472210512,77sleep115556-21/usr/sbin/munin11:24:391
28289210418,67sleep328334-21/usr/sbin/munin12:04:303
4295210311,52sleep20-21swapper/210:49:402
4566210210,76sleep20-21swapper/209:09:042
213521025,45sleep20-21swapper/207:18:142
213521025,45sleep20-21swapper/207:18:142
10458210210,15sleep20-21swapper/211:09:142
2140821006,83sleep20-21swapper/210:00:442
2140821006,83sleep20-21swapper/210:00:442
39922987,50sleep20-21swapper/212:29:402
2077629814,70sleep120799-21netstat09:59:301
2077629814,70sleep120799-21netstat09:59:301
24942978,15sleep20-21swapper/207:19:192
24942978,15sleep20-21swapper/207:19:192
195652978,14sleep20-21swapper/211:39:012
195652978,14sleep20-21swapper/211:39:012
1471129642,19sleep214774-21sed07:59:042
2630329512,68sleep025807-21apt-get08:34:070
2630329512,68sleep025807-21apt-get08:34:070
1642829413,62sleep016573-21/usr/sbin/munin09:44:350
120722946,77sleep20-21swapper/209:31:042
1190129319,53sleep111918-21fschecks_count07:49:221
1795529212,64sleep018111-21unixbench_multi09:49:360
2018729111,56sleep020332-21grep09:59:040
2789729010,44sleep20-21swapper/208:39:182
2321628918,58sleep0419-21cron10:09:010
2219228913,58sleep322225-21irqstats10:04:243
300342888,41sleep20-21swapper/208:44:342
300342888,41sleep20-21swapper/208:44:342
1359628818,18sleep20-21swapper/211:19:152
1359628818,18sleep20-21swapper/211:19:152
2127528713,52sleep121351-21fschecks_time08:19:221
1301828711,16sleep10-21swapper/109:34:311
1157628730,33sleep30-21swapper/307:49:043
1157628730,33sleep30-21swapper/307:49:043
157162867,46sleep20-21swapper/209:44:172
2943428521,17sleep20-21swapper/212:09:172
8782845,13sleep00-21swapper/010:39:250
2328128410,14sleep00-21swapper/011:49:210
159862845,68sleep00-21swapper/008:01:080
2831528315,48sleep028401-21/usr/sbin/munin10:24:230
703928220,46sleep20-21swapper/207:34:172
218728218,47sleep31666-21/usr/sbin/munin08:59:333
526728126,37sleep35309-21run-parts10:54:013
1659028113,54sleep016685-21fschecks_time08:04:230
100532819,55sleep010208-21sed07:44:040
100532819,55sleep010208-21sed07:44:040
3023728014,51sleep130299-21/usr/sbin/munin10:29:331
3023728014,51sleep130299-21/usr/sbin/munin10:29:331
12132805,42sleep20-21swapper/208:56:372
530027910,55sleep05341-21systemd-run07:29:020
530027910,55sleep05341-21systemd-run07:29:020
2984527916,41sleep129410-21/usr/sbin/munin12:09:281
2509327911,54sleep00-21swapper/010:14:180
82262785,40sleep20-21swapper/207:36:272
82262785,40sleep20-21swapper/207:36:272
3160627713,31sleep10-21swapper/112:14:331
2918627724,27sleep00-21swapper/012:09:030
259292779,13sleep00-21swapper/011:57:320
259292779,13sleep00-21swapper/011:57:320
2313327717,37sleep323167-21/usr/sbin/munin08:24:293
2952527616,39sleep129684-21sort10:29:041
1182927611,50sleep311933-21latency_hist11:14:043
2453727518,23sleep30-21swapper/311:54:043
2035727516,42sleep320416-21/usr/sbin/munin11:39:513
775727431,20sleep00-21swapper/010:59:350
340427417,42sleep13466-21fw_packets09:04:231
2707427411,46sleep027098-21ntp_states10:19:310
1978727415,28sleep00-21swapper/008:14:240
1020627421,26sleep10-21swapper/111:09:051
690527326,29sleep10-21swapper/110:59:041
400727327,27sleep1225-21systemd-journal10:49:191
3049627317,34sleep030645-21grep10:29:360
3049627317,34sleep030645-21grep10:29:360
2253327321,26sleep10-21swapper/110:04:341
615527220,27sleep10-21swapper/109:14:041
472127213,44sleep14823-21timerwakeupswit07:24:431
472127213,44sleep14823-21timerwakeupswit07:24:431
3189327119,27sleep30-21swapper/307:09:233
1745627128,23sleep10-21swapper/109:49:241
820927019,33sleep18225-21/usr/sbin/munin09:19:251
42927017,26sleep30-21swapper/310:39:043
3110627013,41sleep031115-21/usr/sbin/munin12:14:200
2741927018,24sleep30-21swapper/310:19:383
113412706,13sleep10-21swapper/109:29:261
831126920,29sleep30-21swapper/309:19:273
3079326916,32sleep00-21swapper/012:14:030
1851226910,46sleep018549-21ps08:09:320
1806126915,30sleep10-21swapper/111:34:041
1806126915,30sleep10-21swapper/111:34:041
1164526915,28sleep00-21swapper/009:29:330
822826817,27sleep30-21swapper/311:01:483
501626814,35sleep05070-21/usr/sbin/munin09:09:270
334026816,28sleep30-21swapper/309:04:213
2848126823,18sleep10-21swapper/106:59:241
2848126823,18sleep10-21swapper/106:59:241
2640226818,23sleep10-21swapper/110:19:031
3266726720,32sleep132688-21fschecks_count12:19:211
300526719,24sleep10-21swapper/109:04:031
299372679,14sleep10-21swapper/107:04:201
2797326719,26sleep10-21swapper/112:04:241
1417526717,31sleep114243-21/usr/sbin/munin11:19:431
1417526717,31sleep114243-21/usr/sbin/munin11:19:431
1221426721,32sleep312257-21awk07:49:283
3113826620,27sleep331215-21/usr/sbin/munin08:49:243
1232526616,25sleep00-21swapper/007:49:330
995326517,28sleep30-21swapper/309:24:323
90792656,14sleep00-21swapper/009:22:270
546526521,28sleep35460-21apt-get07:29:173
546526521,28sleep35460-21apt-get07:29:173
2449826517,26sleep00-21swapper/008:29:140
24312658,13sleep00-21swapper/010:44:250
2399926519,28sleep1284-21dbus-daemon08:29:031
1850926521,25sleep00-21swapper/011:34:440
1850926521,25sleep00-21swapper/011:34:440
1681326513,40sleep00-21swapper/011:29:270
1681326513,40sleep00-21swapper/011:29:270
148726517,26sleep00-21swapper/008:59:050
396626413,31sleep04110-21/usr/sbin/munin09:04:340
387426418,20sleep00-21swapper/007:24:050
387426418,20sleep00-21swapper/007:24:050
1872426423,19sleep30-21swapper/308:09:353
1625526414,36sleep116281-21latency_hist08:04:041
3013326322,25sleep330258-21cat08:44:353
3013326322,25sleep330258-21cat08:44:353
2887226312,18sleep30-21swapper/310:24:343
1960626313,20sleep30-21swapper/311:39:033
1960626313,20sleep30-21swapper/311:39:033
1271826313,31sleep012827-21perl09:34:260
40626214,23sleep10-21swapper/108:54:301
2966726224,21sleep10-21swapper/108:44:251
2966726224,21sleep10-21swapper/108:44:251
2365726216,29sleep123658-21ntp_states11:49:301
2360026215,23sleep10-21swapper/110:09:231
2094726216,22sleep30-21swapper/308:19:043
103826218,20sleep30-21swapper/310:39:283
66502616,44sleep00-21swapper/010:56:240
325712615,11sleep00-21swapper/010:35:420
216462616,41sleep30-21swapper/311:44:193
1659826116,23sleep00-21swapper/011:29:060
1659826116,23sleep00-21swapper/011:29:060
67026016,28sleep1665-21apt-get10:39:181
43382605,12sleep10-21swapper/112:32:051
3116526019,23sleep00-21swapper/007:09:040
3104826014,23sleep30-21swapper/310:34:033
3104826014,23sleep30-21swapper/310:34:033
295662600,8sleep00-21swapper/008:44:230
295662600,8sleep00-21swapper/008:44:230
286926012,29sleep10-21swapper/107:19:341
286926012,29sleep10-21swapper/107:19:341
2586026015,30sleep125886-21munin-run08:34:041
2496226012,32sleep325013-21ntp_states08:29:503
2126126020,24sleep30-21swapper/308:19:203
1888326015,28sleep10-21swapper/111:34:531
1888326015,28sleep10-21swapper/111:34:531
135012606,12sleep00-21swapper/011:19:050
135012606,12sleep00-21swapper/011:19:050
1339026018,26sleep113387-21apt-get07:54:151
1262126012,28sleep312616-21apt-get09:34:203
937925915,25sleep19494-21/usr/sbin/munin07:39:371
937925915,25sleep19494-21/usr/sbin/munin07:39:371
528625919,22sleep30-21swapper/309:09:423
3140125918,25sleep30-21swapper/312:14:273
2541525919,21sleep10-21swapper/111:54:351
2541525919,21sleep10-21swapper/111:54:351
2517325913,18sleep30-21swapper/311:54:293
2517325913,18sleep30-21swapper/311:54:293
2164825916,26sleep021642-21apt-get11:44:200
1895625918,23sleep10-21swapper/108:09:371
1103825918,25sleep311034-21apt-get09:29:163
1026625917,26sleep010262-21apt-get07:44:160
1026625917,26sleep010262-21apt-get07:44:160
760625816,24sleep10-21swapper/110:59:321
2273625814,27sleep022731-21apt-get08:24:170
169862586,11sleep10-21swapper/108:04:321
1647625812,32sleep316493-21/usr/sbin/munin08:04:143
1417225821,22sleep10-21swapper/109:39:191
413425718,21sleep30-21swapper/312:29:413
321025716,22sleep00-21swapper/012:29:170
2118825714,26sleep021182-21apt-get08:19:170
189325715,21sleep10-21swapper/112:24:261
1726725715,24sleep317262-21apt-get09:49:183
1695225712,27sleep317010-21/usr/sbin/munin11:29:423
1695225712,27sleep317010-21/usr/sbin/munin11:29:423
1398025720,16sleep30-21swapper/309:39:053
3270325617,17sleep30-21swapper/312:19:233
2847925610,17sleep30-21swapper/308:39:353
2710725616,19sleep10-21swapper/108:34:481
2710725616,19sleep10-21swapper/108:34:481
2406525620,20sleep30-21swapper/310:09:333
2217125616,19sleep10-21swapper/111:45:181
1697825619,23sleep31-21systemd09:49:023
106512566,10sleep10-21swapper/109:27:441
559125514,16sleep30-21swapper/310:54:223
295225515,20sleep00-21swapper/009:04:020
2826625522,19sleep10-21swapper/108:39:291
2817225510,16sleep00-21swapper/008:39:280
1882425514,22sleep118842-21apt-get09:54:171
1495225516,23sleep314948-21apt-get07:59:173
1206625514,25sleep312062-21apt-get11:14:163
1206625514,25sleep312062-21apt-get11:14:163
893125413,24sleep38981-21/usr/sbin/munin07:39:273
893125413,24sleep38981-21/usr/sbin/munin07:39:273
704625415,24sleep07036-21apt-get07:34:190
2662025415,21sleep126639-21apt-get10:19:161
2509125414,24sleep325088-21apt-get10:14:173
2171825416,15sleep00-21swapper/010:04:050
2171825416,15sleep00-21swapper/010:04:050
2089625415,21sleep120928-21munin-run08:19:021
1985625412,26sleep119851-21apt-get11:39:171
74572534,12sleep30-21swapper/309:16:323
3217625316,20sleep00-21swapper/007:09:280
2952025315,18sleep30-21swapper/312:09:223
2677825313,22sleep326816-21ntp_states08:34:423
2677825313,22sleep326816-21ntp_states08:34:423
98925211,15sleep00-21swapper/012:19:370
397325214,21sleep03493-21apt-get10:49:100
2863925218,17sleep00-21swapper/006:59:270
2863925218,17sleep00-21swapper/006:59:270
2764525213,23sleep127783-21latency_hist12:04:031
2764525213,23sleep127783-21latency_hist12:04:031
2083825213,23sleep320861-21ntp_states09:59:323
2083825213,23sleep320861-21ntp_states09:59:323
3121725114,19sleep031275-21irqstats08:49:250
2904325112,18sleep30-21swapper/306:59:343
2904325112,18sleep30-21swapper/306:59:343
2630625113,22sleep326325-21apt-get11:59:133
2630625113,22sleep326325-21apt-get11:59:133
1050925117,20sleep110541-21irqstats07:44:261
1050925117,20sleep110541-21irqstats07:44:261
846925014,21sleep08478-21munin-run07:39:030
846925014,21sleep08478-21munin-run07:39:030
639825015,22sleep16467-21df09:14:201
22182509,15sleep30-21swapper/310:44:163
1643025014,22sleep316460-21munin-run11:29:013
1643025014,22sleep316460-21munin-run11:29:013
1369725012,22sleep013704-21diskstats11:19:220
1369725012,22sleep013704-21diskstats11:19:220
999424911,23sleep010067-21sendmail_mailtr09:24:330
16862499,16sleep00-21swapper/012:24:210
1347624917,17sleep30-21swapper/307:54:213
74152489,16sleep30-21swapper/307:34:303
74152489,16sleep30-21swapper/307:34:303
1206924810,20sleep112136-21df11:14:181
1206924810,20sleep112136-21df11:14:181
3127624712,21sleep10-21swapper/108:49:261
2984624611,20sleep329854-21apt-get07:04:143
2639624611,20sleep026407-21df_inode11:59:180
2639624611,20sleep026407-21df_inode11:59:180
137182467,15sleep30-21swapper/311:19:333
137182467,15sleep30-21swapper/311:19:333
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional