You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-28 - 16:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Sep 28, 2026 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23452269421,657sleep323525-21/usr/sbin/munin12:27:373
23452269421,657sleep323525-21/usr/sbin/munin12:27:373
139672481114,304sleep214005-21fschecks_count08:32:282
139672481114,304sleep214005-21fschecks_count08:32:282
27370239968,306sleep2231-21systemd-journal07:32:082
11659237015,332sleep00-21swapper/010:07:270
1190235094,210sleep21277-21latency_hist07:52:112
6423232518,292sleep16548-21sed11:32:401
5847226894,156sleep25937-21fschecks_time11:32:292
26185224761,159sleep226331-21unixbench_multi10:52:392
9534223033,20sleep20-21swapper/210:00:272
24326220958,23sleep20-21swapper/207:22:112
18393220150,57sleep218429-21apt-key08:47:102
29990218859,20sleep20-21swapper/209:22:312
24803218313,149sleep024879-21latency_hist09:07:110
27589218011,154sleep227667-21mailstats10:57:362
18465217915,46sleep20-21swapper/212:12:302
1518217811,127sleep21575-21grep07:52:312
1518217811,127sleep21575-21grep07:52:312
20121217713,142sleep220272-21latency_hist08:52:112
3126121678,121sleep2367-21snmpd09:27:202
177821668,141sleep2367-21snmpd11:17:392
2863521647,34sleep20-21swapper/211:02:202
26683216411,140sleep026771-21fschecks_time09:12:280
26683216411,140sleep026771-21fschecks_time09:12:280
14420216062,83sleep214576-21latency_hist10:17:102
14420216062,83sleep214576-21latency_hist10:17:102
29209215816,125sleep329228-21apt-get07:37:183
16298215693,45sleep20-21swapper/208:37:412
27934215517,90sleep20-21swapper/207:32:322
13738215536,61sleep20-21swapper/211:57:322
4709214810,93sleep20-21swapper/211:27:402
2266721457,100sleep222819-21sed08:57:402
26341214411,48sleep226358-21latency_hist09:12:102
26983214323,104sleep127017-21/usr/sbin/munin09:12:331
26983214323,104sleep127017-21/usr/sbin/munin09:12:331
11771214241,78sleep20-21swapper/210:07:292
2749213934,36sleep20-21swapper/211:22:272
2417721399,104sleep00-21swapper/010:47:290
6479213412,104sleep3231-21systemd-journal09:51:273
6479213412,104sleep3231-21systemd-journal09:51:273
24276213478,38sleep024296-21munin-run07:22:090
18358213411,91sleep018415-21idleruntime12:12:280
10566213421,61sleep210719-21latency_hist08:22:112
32084213088,24sleep20-21swapper/207:47:102
15644212920,96sleep20-21swapper/212:02:372
27412212715,94sleep027424-21latency_hist07:32:100
3089212613,97sleep23129-21grep07:57:282
3089212613,97sleep23129-21grep07:57:282
7993212429,80sleep28124-21sed11:37:382
2200821249,65sleep222100-21cat07:12:382
2522521208,65sleep20-21swapper/212:32:412
2522521208,65sleep20-21swapper/212:32:412
29998211920,73sleep330134-21apt-config11:07:103
27658211929,44sleep00-21swapper/007:32:220
30707211773,21sleep20-21swapper/211:07:382
30707211773,21sleep20-21swapper/211:07:382
19221211714,81sleep019358-21latency_hist07:07:110
18213211739,62sleep218281-21mailstats10:27:362
26361211612,20sleep30-21swapper/309:12:113
20378211517,83sleep120470-21/usr/sbin/munin12:17:421
611021148,93sleep20-21swapper/208:07:252
611021148,93sleep20-21swapper/208:07:252
23719211226,63sleep20-21swapper/210:47:102
24412211111,71sleep10-21swapper/110:47:341
16274211126,57sleep316366-21fschecks_time10:22:283
16274211126,57sleep316366-21fschecks_time10:22:283
29862211011,49sleep229991-21timerwakeupswit07:37:372
18758211016,79sleep118760-21/usr/sbin/munin08:47:171
323210830,48sleep1342-21apt-get09:32:191
23369210819,41sleep20-21swapper/212:27:362
23369210819,41sleep20-21swapper/212:27:362
2091121075,12sleep20-21swapper/212:20:212
19472210721,69sleep319496-21awk12:17:073
324421068,62sleep23309-21date09:42:092
1720321068,81sleep217279-21sed12:07:382
15467210615,74sleep015469-21/usr/sbin/munin12:02:310
32569210513,78sleep032595-21latency_hist09:32:100
2567121058,86sleep2367-21snmpd07:25:132
24644210521,64sleep124160-21apt-get12:32:201
24644210521,64sleep124160-21apt-get12:32:201
1947721058,16sleep20-21swapper/207:07:252
1350521055,10sleep20-21swapper/208:31:032
3459210324,33sleep00-21swapper/009:42:200
12430210337,56sleep212464-21ps11:52:352
12430210337,56sleep212464-21ps11:52:352
48121026,64sleep20-21swapper/211:13:192
17491210215,63sleep10-21swapper/110:27:111
17491210215,63sleep10-21swapper/110:27:111
21336210110,19sleep00-21swapper/012:22:210
14314210112,70sleep0314-21rs:main0
14314210112,70sleep0314-21rs:main0
13631210011,25sleep013640-21systemd-run08:32:100
43729910,72sleep0441-21diskstats09:32:260
420829910,21sleep00-21swapper/011:27:260
105072998,37sleep20-21swapper/211:47:272
2586429814,27sleep125875-21munin-run07:27:101
214812984,86sleep00-21swapper/008:55:280
1354329827,44sleep00-21swapper/011:57:220
867529715,67sleep38825-21latency_hist11:42:113
3254329716,65sleep332599-21/usr/sbin/munin07:47:313
2081029710,69sleep20-21swapper/208:52:352
1882729720,57sleep318778-21apt-get08:47:483
1500229715,64sleep014608-21/usr/sbin/munin10:17:330
1401029613,61sleep313856-21/usr/sbin/munin08:32:293
1401029613,61sleep313856-21/usr/sbin/munin08:32:293
896929521,23sleep30-21swapper/311:42:273
594129512,65sleep35999-21if_eth011:32:303
236729522,58sleep32401-21idleruntime-cro11:22:083
1787629512,71sleep00-21swapper/012:11:430
1168129515,65sleep311714-21awk11:52:083
107572958,40sleep20-21swapper/208:22:192
990629411,64sleep010036-21sed08:17:410
18002947,12sleep30-21swapper/307:52:363
18002947,12sleep30-21swapper/307:52:363
2742729311,67sleep127574-21latency_hist07:32:111
792629220,56sleep07963-21grep08:12:320
1606629213,62sleep016143-21grep08:37:390
54682918,60sleep10-21swapper/109:48:221
54682918,60sleep10-21swapper/109:48:221
2930229112,61sleep029414-21timerwakeupswit11:02:390
2277829117,21sleep00-21swapper/010:42:320
2133829131,49sleep221386-21awk12:22:232
885929013,64sleep18867-21open_files09:57:351
3117929014,60sleep131223-21awk07:42:331
1903128913,26sleep30-21swapper/310:32:093
2722528824,44sleep127289-21fw_packets10:57:271
367428713,18sleep10-21swapper/109:42:291
2601028714,58sleep326070-21latency_hist12:37:103
2601028714,58sleep326070-21latency_hist12:37:103
1748028714,58sleep017490-21munin-run10:27:100
1748028714,58sleep017490-21munin-run10:27:100
103222878,32sleep20-21swapper/210:02:332
658428634,40sleep26696-21latency_hist09:52:112
658428634,40sleep26696-21latency_hist09:52:112
1825928628,28sleep10-21swapper/107:02:321
13172868,14sleep00-21swapper/011:17:290
356728516,56sleep33576-21forks09:42:273
719028412,57sleep37219-21/usr/sbin/munin09:52:323
208922846,30sleep20-21swapper/210:37:252
1861228411,42sleep10-21swapper/112:12:331
1617728429,39sleep20-21swapper/210:22:232
1617728429,39sleep20-21swapper/210:22:232
550228311,57sleep35511-21munin-run11:32:093
279352839,20sleep20-21swapper/209:17:102
279352839,20sleep20-21swapper/209:17:102
1507228313,55sleep315073-21munin-node12:02:173
2147128219,43sleep021578-21cat10:37:390
442528116,39sleep20-21swapper/209:43:022
323272810,12sleep10-21swapper/107:47:201
2160828125,34sleep321634-21munin-run08:57:093
146042816,33sleep20-21swapper/210:17:182
81262805,30sleep20-21swapper/208:12:382
456228021,16sleep20-21swapper/208:02:242
456228021,16sleep20-21swapper/208:02:242
1807628018,44sleep218146-21if_eth007:02:292
1158628011,56sleep311655-21fschecks_time10:07:263
2658427910,56sleep326622-21ps07:27:413
2172827819,44sleep321766-21ntp_kernel_pll_12:22:323
191327818,31sleep10-21swapper/109:37:201
1513727820,41sleep115156-21sendmail08:37:081
1513727820,41sleep115156-21sendmail08:37:081
249312775,28sleep20-21swapper/210:48:282
239782779,46sleep20-21swapper/209:02:412
135902769,14sleep30-21swapper/308:32:093
2211927526,29sleep122143-21munin-run10:42:081
70782747,21sleep00-21swapper/008:08:280
70782747,21sleep00-21swapper/008:08:280
361027422,35sleep13691-21cat07:57:391
361027422,35sleep13691-21cat07:57:391
313927415,44sleep33201-21sendmail_mailst11:22:373
205092745,28sleep20-21swapper/210:36:402
1920127424,31sleep319218-21munin-run07:07:103
1907327421,22sleep00-21swapper/010:32:100
188352746,14sleep00-21swapper/008:47:530
34627317,40sleep20-21swapper/209:32:202
3209727316,36sleep00-21swapper/011:12:280
2584727311,50sleep0367-21snmpd12:36:020
2584727311,50sleep0367-21snmpd12:36:020
2019027316,42sleep00-21swapper/010:32:430
3087827217,33sleep00-21swapper/007:42:280
3029327220,27sleep10-21swapper/109:22:381
2104827216,36sleep321078-21munin-run12:22:083
307002717,10sleep30-21swapper/309:23:333
954227025,30sleep20-21swapper/211:42:382
650827019,25sleep30-21swapper/308:07:343
650827019,25sleep30-21swapper/308:07:343
2044327021,29sleep30-21swapper/307:08:273
1471727014,23sleep30-21swapper/310:17:283
2922326923,12sleep20-21swapper/209:19:282
214726920,29sleep30-21swapper/309:37:303
2989526820,27sleep129914-21apt-get11:07:081
2217026819,28sleep30-21swapper/310:42:113
1209126821,23sleep10-21swapper/108:27:101
1056026825,24sleep00-21swapper/010:02:390
913226718,26sleep10-21swapper/111:42:291
465026722,27sleep34730-21tune2fs08:02:293
465026722,27sleep34730-21tune2fs08:02:293
215002677,14sleep10-21swapper/107:12:251
1412226712,25sleep30-21swapper/311:57:413
951126617,29sleep30-21swapper/308:17:313
709626610,38sleep20-21swapper/209:52:312
703126614,33sleep07081-21/usr/sbin/munin09:52:290
504726614,22sleep00-21swapper/008:02:390
504726614,22sleep00-21swapper/008:02:390
248362668,15sleep30-21swapper/307:22:323
1772726620,25sleep10-21swapper/110:27:231
3205726517,23sleep30-21swapper/311:12:273
3163126515,29sleep10-21swapper/109:27:311
2940526517,27sleep00-21swapper/007:37:270
2309326515,22sleep10-21swapper/107:17:251
1965526513,40sleep319262-21/usr/sbin/munin10:32:343
2942726417,28sleep30-21swapper/309:22:083
2839026419,25sleep30-21swapper/311:02:093
866926317,30sleep08705-21cat09:57:300
318522637,11sleep30-21swapper/311:12:153
318522637,11sleep30-21swapper/311:12:153
2947826317,22sleep10-21swapper/109:22:101
1651826311,40sleep10-21swapper/108:39:341
1636026326,18sleep10-21swapper/112:07:081
1433426311,31sleep10-21swapper/111:57:431
359126210,17sleep10-21swapper/111:22:591
3058426211,35sleep030648-21idleruntime11:07:350
3058426211,35sleep030648-21idleruntime11:07:350
2841426214,25sleep00-21swapper/011:02:100
255202628,11sleep00-21swapper/007:23:180
1714926211,33sleep017215-21fw_packets08:42:280
48526113,19sleep00-21swapper/007:47:410
1985026124,20sleep00-21swapper/007:07:350
1331126115,32sleep313323-21/usr/sbin/munin10:12:313
1331126115,32sleep313323-21/usr/sbin/munin10:12:313
765626014,31sleep17657-21/usr/sbin/munin08:12:251
2793826017,24sleep30-21swapper/309:17:113
2793826017,24sleep30-21swapper/309:17:113
2288426015,23sleep10-21swapper/112:27:231
2288426015,23sleep10-21swapper/112:27:231
2073226019,25sleep120740-21modprobe07:12:081
168626016,19sleep10-21swapper/111:17:381
1305126015,27sleep10-21swapper/110:12:251
1305126015,27sleep10-21swapper/110:12:251
3218225917,26sleep132234-21/usr/sbin/munin11:12:291
2775325918,24sleep327844-21/usr/sbin/munin07:32:293
2315625915,23sleep00-21swapper/012:27:300
2315625915,23sleep00-21swapper/012:27:300
170125918,18sleep10-21swapper/107:52:341
170125918,18sleep10-21swapper/107:52:341
781925818,19sleep00-21swapper/011:37:350
38102587,10sleep00-21swapper/011:25:440
2278225818,22sleep30-21swapper/307:17:113
1976725811,18sleep00-21swapper/012:17:210
1275925810,20sleep10-21swapper/110:12:091
308542576,42sleep00-21swapper/009:25:320
3210325615,25sleep332251-21date07:47:113
307425612,29sleep03080-21df_inode07:57:260
307425612,29sleep03080-21df_inode07:57:260
2512025614,26sleep025124-21diskstats09:07:270
2249025611,19sleep30-21swapper/310:42:273
2200225616,24sleep122132-21sed12:22:391
1409325612,17sleep00-21swapper/008:32:300
1409325612,17sleep00-21swapper/008:32:300
1242325612,29sleep112429-21ntp_states11:52:341
1242325612,29sleep112429-21ntp_states11:52:341
1067125618,17sleep10-21swapper/110:02:401
937125510,25sleep10-21swapper/108:17:291
653325510,16sleep00-21swapper/009:52:090
653325510,16sleep00-21swapper/009:52:090
2765925515,26sleep127654-21apt-get07:32:231
2240425514,24sleep122401-21apt-get10:42:211
2120625521,20sleep00-21swapper/007:12:150
1936325514,25sleep119372-21diskstats10:32:281
1697825513,19sleep30-21swapper/308:42:193
1298425514,18sleep10-21swapper/108:27:431
117322559,22sleep10-21swapper/111:52:101
2814825415,19sleep00-21swapper/009:17:200
2601125412,25sleep026103-21mailstats10:52:370
2405925416,24sleep324153-21sed09:02:423
1078125413,25sleep010776-21apt-get08:22:210
723025314,20sleep10-21swapper/108:10:271
723025314,20sleep10-21swapper/108:10:271
2034725313,23sleep120341-21apt-get08:52:221
839525212,24sleep38391-21apt-get09:57:213
582825212,24sleep05834-21diskstats11:32:270
2684425214,23sleep126923-21latency_hist10:57:111
2603225212,29sleep10-21swapper/109:08:411
1680525213,15sleep30-21swapper/308:42:113
1512825215,20sleep10-21swapper/110:17:381
1002525215,22sleep310031-21df_inode10:02:263
41492515,35sleep30-21swapper/308:00:363
41492515,35sleep30-21swapper/308:00:363
2377325114,21sleep323918-21date10:47:113
1683425112,21sleep10-21swapper/112:07:281
168325113,22sleep31699-21munin-run09:37:103
1240325112,23sleep012422-21ntp_states11:52:330
1240325112,23sleep012422-21ntp_states11:52:330
927125012,22sleep09309-21ntpq11:42:320
789925017,17sleep10-21swapper/111:37:361
456125016,20sleep14556-21apt-get08:02:231
456125016,20sleep14556-21apt-get08:02:231
3133725014,19sleep30-21swapper/309:27:253
2230125015,23sleep30-21swapper/308:57:333
1710425014,20sleep317110-21ntp_states12:07:343
150325010,16sleep00-21swapper/007:52:290
150325010,16sleep00-21swapper/007:52:290
1196525012,22sleep311962-21apt-get11:52:203
1196525012,22sleep311962-21apt-get11:52:203
501224913,17sleep10-21swapper/111:27:431
2946824913,19sleep10-21swapper/107:37:281
238724912,19sleep02393-21ntp_states09:37:350
816024814,18sleep10-21swapper/109:57:111
182322485,34sleep10-21swapper/108:46:451
1096724814,17sleep00-21swapper/011:47:370
3033124710,20sleep129914-21apt-get11:07:121
3033124710,20sleep129914-21apt-get11:07:121
2560324710,19sleep325712-21perl10:52:283
242792476,13sleep10-21swapper/109:02:441
2324324712,14sleep10-21swapper/109:02:111
1204124613,14sleep30-21swapper/308:27:083
238712458,14sleep30-21swapper/307:17:403
900024211,18sleep30-21swapper/308:17:113
2234524110,15sleep00-21swapper/008:57:340
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional