You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-03 - 19:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Sun May 03, 2026 12:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30883217478,1725sleep024639-1kworker/0:1H11:55:310
30883217478,1725sleep024639-1kworker/0:1H11:55:310
28643213658,1343sleep328668-21df10:05:433
28643213658,1343sleep328668-21df10:05:433
2858529316,917sleep228144-1kworker/2:1H10:05:332
2858529316,917sleep228144-1kworker/2:1H10:05:332
20357279418,762sleep020435-21memory09:40:360
872277110,744sleep1886-21/usr/sbin/munin12:00:451
10835276320,725sleep210844-21/usr/sbin/munin09:10:382
2061126678,647sleep0191-21jbd2/mmcblk2p2-08:00:190
19069254999,433sleep319080-21apt-get07:55:253
154862354145,149sleep30-21swapper/307:45:153
29740232819,288sleep229744-21latency_hist08:30:162
1466231618,274sleep01583-21grep10:20:460
1466231618,274sleep01583-21grep10:20:460
20333229218,261sleep320351-21grep09:40:353
3057227117,60sleep32863-21latency_hist12:10:163
207592259142,91sleep30-21swapper/311:25:133
25723224859,150sleep30-21swapper/311:40:163
101072237105,24sleep30-21swapper/312:30:433
21531222664,31sleep321582-21processes11:25:423
4640221214,179sleep34643-21cron12:15:133
12309220943,142sleep30-21swapper/309:15:353
10958220682,47sleep311030-21mailstats09:10:433
1339220437,18sleep30-21swapper/312:05:153
10763220161,93sleep310801-21modprobe07:30:133
9837220050,102sleep325884-21systemd-journal10:50:133
22381219815,166sleep022526-21latency_hist11:30:150
23153219569,19sleep30-21swapper/308:06:233
10221219371,51sleep310307-21grep10:50:333
7724218712,143sleep30-21swapper/307:20:163
341621879,168sleep013441-21snmpd10:27:360
3132218728,79sleep13279-21grep08:45:491
3132218728,79sleep13279-21grep08:45:491
23108218069,91sleep322874-21cron09:50:163
3039521768,149sleep30-21swapper/310:10:353
2843321739,25sleep30-21swapper/308:25:283
25903217012,145sleep325982-21sort09:56:543
1312421647,18sleep30-21swapper/307:35:423
27880216325,122sleep327905-21awk11:45:373
2770216315,133sleep32839-21basename07:05:343
23997216057,89sleep324018-21apt-key11:35:153
9321215514,100sleep39361-21grep09:05:383
3074215511,103sleep33224-21unixbench_singl10:25:463
697221547,135sleep37009-21ps12:20:423
19006215013,12sleep30-21swapper/309:35:423
5542214911,75sleep30-21swapper/308:55:163
399021488,67sleep30-21swapper/308:50:273
3179921479,123sleep331836-21fschecks_count10:15:323
10181214224,99sleep210200-21modprobe09:10:132
8357214167,62sleep38392-21ntpq07:20:403
12576214025,98sleep112640-21cat09:15:411
12316213917,84sleep31-21systemd07:35:143
17545213712,87sleep317624-21cat07:50:373
17545213712,87sleep317624-21cat07:50:373
30848213418,100sleep130338-21apt-get11:55:241
30848213418,100sleep130338-21apt-get11:55:241
27592213317,23sleep20-21swapper/208:20:472
2622621318,89sleep325939-21/usr/sbin/munin11:40:363
2984212879,22sleep30-21swapper/308:45:473
2984212879,22sleep30-21swapper/308:45:473
25905212589,17sleep30-21swapper/308:15:443
14179212419,87sleep214234-21cpuspeed07:40:302
10939212317,81sleep211027-21wc12:35:142
15903212213,96sleep115981-21mailstats09:25:431
872212112,93sleep2883-21df_inode10:20:322
872212112,93sleep2883-21df_inode10:20:322
1249121218,16sleep30-21swapper/310:55:463
1249121218,16sleep30-21swapper/310:55:463
15264211811,17sleep30-21swapper/311:05:393
8309211610,27sleep30-21swapper/312:25:343
4617211513,24sleep34678-21cat07:10:223
28238211416,23sleep328383-21date08:25:163
17113211311,88sleep017220-21unixbench_singl11:10:450
14905211313,86sleep014943-21grep11:05:320
32195211210,26sleep332331-21date12:00:153
32195211210,26sleep332331-21date12:00:153
16984211213,18sleep30-21swapper/311:10:443
20138210813,82sleep220144-21df_inode09:40:302
7677210711,16sleep30-21swapper/310:40:443
3338210721,65sleep22862-21apt-get12:10:232
31238210716,78sleep331265-21modprobe08:35:133
29038210712,76sleep30-21swapper/311:50:253
13662210710,80sleep30-21swapper/311:00:383
8619210610,16sleep30-21swapper/309:05:133
17032210515,15sleep317112-21tune2fs09:30:333
1567321057,15sleep30-21swapper/309:25:373
1115210521,16sleep30-21swapper/308:40:423
1092421050,6sleep20-21swapper/207:30:152
24386210214,73sleep124407-21apt11:35:251
18005210123,58sleep018148-21sed07:50:460
18005210123,58sleep018148-21sed07:50:460
3911210012,71sleep30-21swapper/310:30:303
23511210017,57sleep223525-21grep09:50:472
1385429813,18sleep20-21swapper/209:20:312
2886529713,67sleep128918-21latency10:05:491
2886529713,67sleep128918-21latency10:05:491
56929617,24sleep20-21swapper/210:20:152
12352968,13sleep30-21swapper/310:20:413
12352968,13sleep30-21swapper/310:20:413
805429514,65sleep10-21swapper/112:25:281
73429528,30sleep20-21swapper/208:40:332
50262956,38sleep30-21swapper/312:15:293
2257129411,67sleep122588-21/usr/sbin/munin11:30:241
1231129418,58sleep212315-21cron07:35:132
452429213,61sleep24595-21sensors_temp08:50:432
214272927,12sleep30-21swapper/308:00:463
1906629214,64sleep30-21swapper/311:18:233
609029118,51sleep16099-21modprobe07:15:131
117082917,67sleep00-21swapper/010:55:270
117082917,67sleep00-21swapper/010:55:270
558329014,60sleep25671-21/usr/sbin/munin10:35:342
2437529011,64sleep223888-21apt-get11:35:182
216172909,64sleep30-21swapper/309:45:273
2817528913,59sleep028347-21cmp10:05:150
65972887,10sleep30-21swapper/310:38:213
325322886,11sleep30-21swapper/308:36:353
157228812,61sleep00-21swapper/012:05:240
45202878,12sleep20-21swapper/210:30:452
167282866,70sleep30-21swapper/307:46:573
804628524,40sleep18064-21fschecks_count07:20:341
277232856,10sleep30-21swapper/310:01:063
2411728517,54sleep324164-21ntpq08:10:393
2237628516,48sleep10-21swapper/109:45:461
58762845,11sleep10-21swapper/112:15:511
330428425,37sleep23446-21unixbench_multi07:05:452
871028314,49sleep08823-21/usr/sbin/munin12:25:440
3163428313,48sleep131534-21/usr/sbin/munin08:35:321
1986228318,18sleep00-21swapper/009:40:150
1061028213,55sleep010673-21sendmail_mailqu10:50:420
147922816,43sleep30-21swapper/309:21:133
296328016,48sleep23052-21apt-key12:10:152
2759028012,52sleep127596-21diskstats11:45:311
2131128016,50sleep221426-21sort08:00:452
2854427949,16sleep00-21swapper/010:05:230
2854427949,16sleep00-21swapper/010:05:230
374427810,18sleep20-21swapper/208:50:142
374427810,18sleep20-21swapper/208:50:142
2391927826,31sleep223955-21munin-run11:35:142
91827721,40sleep1955-21latency08:40:361
74402778,16sleep20-21swapper/209:00:322
3105327718,42sleep231060-21fschecks_time11:55:532
3105327718,42sleep231060-21fschecks_time11:55:532
289982770,9sleep20-21swapper/211:50:152
1770227717,37sleep111349-21kworker/1:111:15:141
1328127714,18sleep10-21swapper/107:35:451
410027621,28sleep10-21swapper/107:10:161
3208227621,31sleep00-21swapper/010:15:370
2523327621,35sleep225218-21apt-get09:55:322
128527516,34sleep01314-21munin-run12:05:120
2341827419,36sleep123395-21apt-get09:50:381
82852737,54sleep00-21swapper/009:00:540
471827324,32sleep10-21swapper/112:15:151
2053527318,30sleep10-21swapper/109:40:411
1721527310,46sleep217283-21latency09:30:362
158527319,27sleep10-21swapper/110:20:471
158527319,27sleep10-21swapper/110:20:471
1326727318,34sleep113261-21apt-get11:00:281
1081527318,22sleep10-21swapper/107:30:151
3224127219,27sleep10-21swapper/110:15:421
2643027223,30sleep10-21swapper/111:40:421
224627222,27sleep10-21swapper/112:05:461
2113727220,26sleep20-21swapper/211:25:332
1085927222,34sleep010883-21munin-run12:35:120
457627117,34sleep23897-21apt-get07:10:212
2295927117,37sleep223105-21cat09:50:152
2224927116,39sleep022375-21timerwakeupswit09:45:450
200152710,7sleep10-21swapper/109:40:151
125262719,16sleep00-21swapper/009:15:400
696527018,27sleep20-21swapper/212:20:412
2900727013,39sleep129097-21switchtime08:25:431
2861727016,23sleep00-21swapper/008:25:340
2715227016,33sleep10-21swapper/108:20:381
1852327021,30sleep20-21swapper/209:35:292
655626917,33sleep10-21swapper/107:15:331
3012626911,27sleep00-21swapper/010:10:280
265352697,49sleep00-21swapper/008:19:320
2605026911,46sleep026068-21fschecks_count11:40:330
1577926924,27sleep20-21swapper/211:06:232
521826825,23sleep00-21swapper/012:15:340
396426811,39sleep03985-21apt-get08:50:240
3031926819,25sleep20-21swapper/211:55:132
225426821,27sleep20-21swapper/207:04:442
225426821,27sleep20-21swapper/207:04:442
1543526817,28sleep10-21swapper/107:45:131
1067526811,25sleep10-21swapper/110:50:431
73892679,14sleep10-21swapper/109:00:311
436226718,26sleep10-21swapper/108:50:381
3056526716,27sleep10-21swapper/111:55:161
2569726715,20sleep10-21swapper/109:56:481
2257326714,38sleep222617-21ntpq08:05:372
2017626718,24sleep10-21swapper/111:20:431
1619126719,28sleep20-21swapper/207:45:432
1906026616,24sleep20-21swapper/207:55:242
983826414,23sleep00-21swapper/009:05:470
895426423,22sleep00-21swapper/010:45:370
444726417,18sleep00-21swapper/007:10:190
400926415,29sleep04099-21gpgv07:10:150
2659026418,30sleep126609-21munin-run08:20:141
2636426423,21sleep20-21swapper/211:40:392
251262648,33sleep00-21swapper/009:55:180
2166026413,38sleep121744-21expr11:25:441
1425626413,34sleep014262-21diskstats07:40:320
715826323,22sleep20-21swapper/209:00:162
3047826317,18sleep00-21swapper/011:55:150
1772726316,24sleep20-21swapper/211:15:152
844926216,26sleep20-21swapper/212:25:372
620626218,26sleep10-21swapper/108:56:031
120712625,11sleep00-21swapper/007:32:080
690726121,23sleep00-21swapper/012:20:390
412326115,20sleep10-21swapper/110:30:361
311582616,11sleep10-21swapper/110:12:151
289826115,22sleep10-21swapper/110:25:441
2520226111,24sleep00-21swapper/011:35:490
1996326116,26sleep219998-21grep11:20:402
1924326116,28sleep119252-21/usr/sbin/munin07:56:281
1828026121,19sleep00-21swapper/009:35:150
1171226115,27sleep111822-21/usr/sbin/munin07:30:441
2371926012,34sleep123734-21acpi08:10:251
2122026016,19sleep10-21swapper/108:00:441
2081026018,19sleep00-21swapper/011:25:140
1385226015,29sleep113846-21apt-get09:20:301
759225923,18sleep20-21swapper/210:40:432
752025918,19sleep10-21swapper/110:40:421
3055525919,22sleep10-21swapper/108:30:431
2339225912,26sleep00-21swapper/009:50:250
1631225912,29sleep016404-21/usr/sbin/munin07:45:450
1230125916,28sleep212302-21apt-get09:15:302
921725815,22sleep00-21swapper/007:25:140
3155925816,24sleep231553-21apt-get08:35:292
3037625822,19sleep00-21swapper/008:30:380
284602586,12sleep00-21swapper/011:46:100
1228725816,27sleep112370-21sed10:55:441
1228725816,27sleep112370-21sed10:55:441
977925710,24sleep00-21swapper/007:25:380
926825712,30sleep2281-21dbus-daemon07:25:162
713025714,26sleep07272-21awk07:15:450
3065225719,19sleep20-21swapper/208:30:442
272125716,18sleep00-21swapper/008:45:400
272125716,18sleep00-21swapper/008:45:400
2410725719,19sleep20-21swapper/208:10:382
960325613,27sleep19597-21apt-get12:30:291
336425611,24sleep00-21swapper/012:10:250
255152566,11sleep20-21swapper/208:15:342
2416725613,24sleep024199-21ntp_states08:10:400
1528925619,17sleep10-21swapper/111:05:411
1306025613,26sleep013084-21ntp_states07:35:390
1190825616,22sleep00-21swapper/009:15:160
674125520,21sleep10-21swapper/110:40:131
506825510,28sleep25163-21perl12:15:322
204925513,24sleep00-21swapper/008:45:160
1860425516,25sleep118690-21gpgv07:55:151
1860425516,25sleep118690-21gpgv07:55:151
1520425512,27sleep215355-21modprobe09:25:152
1323625511,26sleep00-21swapper/011:00:230
113225516,18sleep30-21swapper/307:00:343
113225516,18sleep30-21swapper/307:00:343
1111225516,19sleep10-21swapper/109:10:451
1012925513,24sleep210123-21apt-get10:50:282
1803625413,22sleep20-21swapper/211:15:312
1411925411,31sleep014147-21kernelversion09:20:370
76272535,10sleep10-21swapper/112:23:261
591925311,22sleep05943-21ntp_states10:35:400
3193825312,27sleep031982-21ntpq08:35:390
2216925313,19sleep10-21swapper/108:05:241
1698525316,24sleep11-21systemd07:50:141
1537925313,24sleep015391-21apt-get09:25:260
110025313,23sleep21097-21apt-get12:03:122
524625219,17sleep00-21swapper/010:35:150
2732325213,21sleep127393-21sensors_temp10:00:441
2724225213,23sleep227248-21ntp_states10:00:412
1924825210,25sleep019258-21munin-run11:20:140
1916825212,18sleep00-21swapper/007:56:230
712125110,23sleep07216-21perl10:40:320
2688125111,24sleep026876-21apt-get08:20:290
1735025114,21sleep217356-21df_inode07:50:312
1735025114,21sleep217356-21df_inode07:50:312
129825121,18sleep10-21swapper/107:00:361
129825121,18sleep10-21swapper/107:00:361
879525013,20sleep18852-21irqstats10:45:341
1925925017,16sleep10-21swapper/111:20:151
1297525012,22sleep213003-21munin-run11:00:132
1297525012,22sleep213003-21munin-run11:00:132
127212509,16sleep20-21swapper/207:35:332
903224910,17sleep20-21swapper/210:45:392
770924916,17sleep20-21swapper/207:20:152
3012324911,21sleep230120-21apt-get10:10:262
108424913,18sleep00-21swapper/007:00:310
108424913,18sleep00-21swapper/007:00:310
958924810,16sleep10-21swapper/109:05:441
2345824814,17sleep00-21swapper/011:30:480
1716524813,18sleep10-21swapper/109:30:351
297224715,16sleep20-21swapper/210:25:452
2219424710,18sleep00-21swapper/008:05:270
58882459,19sleep00-21swapper/008:55:330
132682436,14sleep20-21swapper/211:00:292
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional