You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-31 - 18:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Aug 31, 2026 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1312324446,428sleep1367-21snmpd10:35:301
1669424147,395sleep116703-21awk12:29:181
1669424147,395sleep116703-21awk12:29:181
422623938,14sleep10-21swapper/108:24:161
25745229685,187sleep125893-21grep11:14:221
5961226675,167sleep16024-21fschecks_time10:14:071
15992223476,16sleep10-21swapper/112:28:501
28558222653,119sleep128597-21ntpq07:59:131
15936221510,191sleep316082-21awk10:44:203
15936221510,191sleep316082-21awk10:44:203
3266221175,78sleep11-21systemd11:48:491
3266221175,78sleep11-21systemd11:48:491
1442422029,179sleep114445-21sed07:14:121
31378220110,30sleep10-21swapper/111:34:081
26137219823,160sleep126171-21latency09:34:111
1388521957,127sleep113945-21if_err_eth010:39:101
1388521957,127sleep113945-21if_err_eth010:39:101
24791219411,166sleep124847-21processes09:29:151
2908221788,149sleep10-21swapper/109:44:081
3764217517,140sleep13833-21df11:49:061
13540217514,73sleep113688-21latency_hist07:13:511
30355217211,146sleep230468-21cat08:04:192
2372121716,152sleep10-21swapper/109:24:511
841321707,17sleep10-21swapper/112:03:591
32276216715,132sleep10-21swapper/109:54:091
2062821668,146sleep120654-21proc_pri10:59:161
10650216418,125sleep210690-21sort10:29:102
10650216418,125sleep210690-21sort10:29:102
2776216261,19sleep10-21swapper/110:04:021
21723216249,54sleep121793-21sed07:38:501
14774215612,21sleep014866-21perl12:24:080
27163215368,19sleep10-21swapper/107:54:211
27163215368,19sleep10-21swapper/107:54:211
17262215256,31sleep117277-21acpi07:24:011
5904215056,21sleep10-21swapper/108:29:211
19305215070,17sleep10-21swapper/107:29:181
13703214814,107sleep00-21swapper/010:39:050
13703214814,107sleep00-21swapper/010:39:050
1924621405,12sleep10-21swapper/109:12:321
1778321389,71sleep117839-21true09:08:481
1889821336,69sleep10-21swapper/110:54:361
2363021320,8sleep30-21swapper/307:44:083
14992213052,60sleep315058-21/usr/sbin/munin07:14:223
16600212812,102sleep116755-21latency_hist10:48:501
1044321288,68sleep110487-21ntpq08:44:141
1044321288,68sleep110487-21ntpq08:44:141
527921275,61sleep10-21swapper/110:09:511
2218921255,71sleep10-21swapper/109:20:031
2218921255,71sleep10-21swapper/109:20:031
664621239,62sleep16583-21latency_hist11:58:501
18236212013,30sleep10-21swapper/112:34:151
9494211910,22sleep10-21swapper/110:24:161
7291211911,93sleep27338-21grep08:34:142
389211953,50sleep10-21swapper/111:39:001
26618211710,15sleep20-21swapper/211:18:592
23230211720,75sleep323268-21grep09:24:163
28187211613,19sleep00-21swapper/007:59:040
2795421157,94sleep128074-21date11:23:501
30361211310,81sleep10-21swapper/111:29:191
21117211117,71sleep021264-21head07:34:210
12092211130,50sleep012111-21apt-get10:34:000
853521106,13sleep10-21swapper/108:39:051
2055210916,77sleep32109-21grep11:44:073
2055210916,77sleep32109-21grep11:44:073
11799210922,69sleep211312-21apt-get08:49:002
22980210833,15sleep10-21swapper/111:08:491
22980210833,15sleep10-21swapper/111:08:491
18917210617,72sleep318924-21diskstats07:29:073
21640210527,51sleep321666-21cut07:38:483
397721037,17sleep30-21swapper/308:24:103
30115210213,68sleep330161-21/usr/sbin/munin11:29:133
737721019,21sleep10-21swapper/108:34:171
28756210119,65sleep028900-21cat09:43:510
18272210027,57sleep118435-21gpgv10:53:511
18272210027,57sleep118435-21gpgv10:53:511
1551829915,69sleep215577-21latency10:44:112
1551829915,69sleep215577-21latency10:44:112
746029866,14sleep10-21swapper/111:59:191
285672988,32sleep128594-21awk11:24:151
2566829820,60sleep225744-21timerandwakeup07:49:212
2566829820,60sleep225744-21timerandwakeup07:49:212
16712985,47sleep10-21swapper/108:15:521
2143129715,60sleep21-21systemd11:03:492
2974429611,16sleep20-21swapper/211:29:042
1232629617,26sleep00-21swapper/008:49:340
141922955,11sleep10-21swapper/112:21:111
885429415,58sleep08467-21/usr/sbin/munin08:39:130
62432946,12sleep20-21swapper/210:14:122
24442945,10sleep10-21swapper/110:03:131
221529421,55sleep10-21swapper/108:19:031
297382938,68sleep10-21swapper/108:04:021
1640529210,68sleep116528-21cat07:19:201
152442928,16sleep10-21swapper/110:44:021
1201029224,12sleep10-21swapper/112:14:171
53692915,12sleep30-21swapper/308:29:063
2009529118,57sleep020101-21modprobe07:33:490
1359429115,58sleep313106-21/usr/sbin/munin12:19:153
2791429010,68sleep127961-21processes09:39:161
1384529015,51sleep013940-21fschecks_time07:13:540
80692899,14sleep10-21swapper/110:19:201
2950128915,51sleep229468-21latency_hist08:03:502
200412898,63sleep10-21swapper/109:14:131
1541328813,16sleep10-21swapper/108:59:161
1541328813,16sleep10-21swapper/108:59:161
2413828756,16sleep10-21swapper/111:09:581
2413828756,16sleep10-21swapper/111:09:581
130628716,50sleep21370-21fschecks_time09:59:072
2559228610,61sleep10-21swapper/107:49:201
2559228610,61sleep10-21swapper/107:49:201
1786728613,57sleep017963-21sed07:24:200
106152868,63sleep110744-21ls12:09:201
656528519,41sleep36590-21munin-run11:58:483
247022856,50sleep10-21swapper/107:47:591
110152857,15sleep10-21swapper/110:29:171
110152857,15sleep10-21swapper/110:29:171
870728419,50sleep08721-21uname12:04:100
3267728410,22sleep132571-21latency_hist08:13:501
2256428417,45sleep022645-21find07:39:220
1561828411,51sleep015662-21ntpq10:44:130
1561828411,51sleep015662-21ntpq10:44:130
2476628312,57sleep224767-21cron07:48:482
1030628331,35sleep210322-21/usr/sbin/munin08:44:112
1030628331,35sleep210322-21/usr/sbin/munin08:44:112
219622827,18sleep121957-21apt-get07:39:021
453428110,56sleep34588-21interrupts10:09:093
3132928111,52sleep10-21swapper/109:49:331
3132928111,52sleep10-21swapper/109:49:331
2724228127,26sleep20-21swapper/207:54:222
2724228127,26sleep20-21swapper/207:54:222
2712628123,37sleep227151-21munin-run09:38:482
114328117,36sleep20-21swapper/208:14:362
512628010,58sleep05178-21cat08:28:510
2182628017,45sleep021928-21/usr/sbin/munin09:19:170
2182628017,45sleep021928-21/usr/sbin/munin09:19:170
2031928015,37sleep320483-21unixbench_multi09:14:193
131512807,15sleep10-21swapper/108:53:511
1415827917,21sleep30-21swapper/308:54:213
3026027814,47sleep20-21swapper/209:48:492
133852787,16sleep10-21swapper/108:54:031
2090227715,30sleep00-21swapper/009:18:480
1919827617,46sleep019292-21switchtime10:54:430
761927514,44sleep37437-21/usr/sbin/munin10:19:103
221802749,14sleep30-21swapper/307:39:133
1912827415,36sleep318558-21/usr/sbin/munin10:54:423
3197427325,25sleep30-21swapper/308:09:203
300127313,41sleep02191-21/usr/sbin/munin08:19:220
2211527321,29sleep20-21swapper/207:39:122
149682736,14sleep30-21swapper/308:59:063
149682736,14sleep30-21swapper/308:59:063
1464627322,35sleep214657-21modprobe08:58:482
100422738,43sleep00-21swapper/008:44:050
100422738,43sleep00-21swapper/008:44:050
88527211,44sleep2907-21/usr/sbin/munin11:39:182
2819827216,31sleep00-21swapper/011:24:060
231427223,29sleep30-21swapper/308:19:093
2023927217,27sleep20-21swapper/209:14:182
1704927215,15sleep30-21swapper/307:23:503
1369027221,29sleep213769-21grep07:13:522
817227115,37sleep08197-21munin-plugin-st08:38:480
2477227110,48sleep024816-21systemd-run11:13:480
2477227110,48sleep024816-21systemd-run11:13:480
221822716,48sleep00-21swapper/011:04:160
221822716,48sleep00-21swapper/011:04:160
1775327122,29sleep20-21swapper/207:24:182
452727012,22sleep20-21swapper/211:49:232
1594327018,29sleep00-21swapper/009:00:040
1594327018,29sleep00-21swapper/009:00:040
1070427030,22sleep30-21swapper/307:04:093
3061926910,46sleep030645-21users08:04:210
2938626916,35sleep328969-21/usr/sbin/munin09:44:143
2506326922,24sleep00-21swapper/011:14:040
2169326912,42sleep221752-21sendmail_mailqu09:19:152
2169326912,42sleep221752-21sendmail_mailqu09:19:152
1928626913,41sleep0231-21systemd-journal09:13:030
173592697,12sleep00-21swapper/012:32:080
173592697,12sleep00-21swapper/012:32:080
168726920,31sleep31751-21ps09:59:163
116522695,52sleep00-21swapper/007:05:550
614426819,28sleep00-21swapper/010:14:100
3263026812,38sleep332576-21latency_hist11:38:503
3218726819,25sleep00-21swapper/009:54:080
2955026818,28sleep00-21swapper/009:44:190
269626821,28sleep00-21swapper/011:44:200
269626821,28sleep00-21swapper/011:44:200
2554026812,38sleep025591-21/usr/sbin/munin07:49:190
2554026812,38sleep025591-21/usr/sbin/munin07:49:190
2438626815,38sleep224397-21df_inode09:29:042
175426828,26sleep01823-21sensors_temp09:59:170
1032226817,26sleep00-21swapper/010:28:510
574026719,19sleep00-21swapper/008:29:150
467626719,19sleep00-21swapper/011:49:250
2485326714,36sleep224798-21latency_hist11:13:502
1475826716,26sleep20-21swapper/212:24:062
877126618,26sleep00-21swapper/010:23:510
598426620,20sleep20-21swapper/211:54:192
3212726617,27sleep00-21swapper/008:09:210
2297926620,28sleep023035-21/usr/sbin/munin09:24:090
1855726617,31sleep218566-21/usr/sbin/munin10:54:002
1129726615,27sleep30-21swapper/308:48:483
1129726615,27sleep30-21swapper/308:48:483
8522659,18sleep00-21swapper/011:39:150
1431626518,28sleep30-21swapper/310:39:203
1431626518,28sleep30-21swapper/310:39:203
2797326418,23sleep30-21swapper/307:58:513
2789326412,37sleep327903-21ntpq09:39:143
230326421,24sleep20-21swapper/208:19:082
1620426422,17sleep20-21swapper/207:19:172
1424726424,21sleep20-21swapper/210:39:192
1424726424,21sleep20-21swapper/210:39:192
127442647,10sleep30-21swapper/312:17:543
33102637,12sleep30-21swapper/310:04:163
2750826311,37sleep027522-21vmstat07:54:240
2750826311,37sleep027522-21vmstat07:54:240
2492026321,22sleep30-21swapper/309:29:173
1825226313,31sleep218315-21find12:34:162
1348926315,32sleep213640-21cut10:38:502
98226216,20sleep20-21swapper/209:58:502
969926219,26sleep2289-21dbus-daemon12:08:492
688426210,26sleep20-21swapper/211:59:042
424726212,28sleep30-21swapper/311:49:203
286126219,21sleep00-21swapper/010:04:050
2720726218,24sleep00-21swapper/009:38:500
2306426212,19sleep20-21swapper/209:24:112
2257126212,33sleep222624-21users11:04:222
2257126212,33sleep222624-21users11:04:222
1750026211,34sleep017650-21/usr/sbin/munin10:49:220
1750026211,34sleep017650-21/usr/sbin/munin10:49:220
1709126215,26sleep00-21swapper/007:23:510
141492629,40sleep00-21swapper/012:20:370
127512628,45sleep00-21swapper/012:17:590
956126119,23sleep20-21swapper/210:24:172
49326114,19sleep30-21swapper/311:39:063
432426111,35sleep00-21swapper/010:09:020
2372126117,29sleep023771-21/usr/sbin/munin07:44:100
1859926114,31sleep318740-21sed07:28:513
1472626118,20sleep00-21swapper/008:58:500
894526018,20sleep30-21swapper/312:04:173
2401926012,33sleep224025-21ntp_states11:09:552
2368526019,24sleep30-21swapper/311:09:483
2368526019,24sleep30-21swapper/311:09:483
1069226020,21sleep10-21swapper/107:04:071
728125915,26sleep07320-21ntpq11:59:140
513025913,31sleep25227-21users10:09:202
2877225915,28sleep328854-21switchtime11:24:213
1377625910,34sleep213898-21timerwakeupswit12:19:182
962925814,27sleep39758-21wc10:24:183
3105325819,20sleep00-21swapper/011:33:500
2971425816,25sleep329733-21apt-get08:03:593
269242585,15sleep30-21swapper/309:36:113
2661325820,19sleep30-21swapper/307:54:063
2148025819,26sleep321543-21latency_hist11:03:503
213102588,16sleep20-21swapper/207:34:312
2082225821,18sleep00-21swapper/010:59:190
627725715,22sleep30-21swapper/310:14:133
3128925716,24sleep231283-21apt-get08:09:032
1777325714,26sleep017767-21apt-get12:34:020
755025624,17sleep20-21swapper/210:19:092
3257725616,23sleep232629-21munin-run11:38:492
3111725619,18sleep30-21swapper/309:49:313
3111725619,18sleep30-21swapper/309:49:313
196725611,30sleep01976-21munin-run08:18:500
189825615,24sleep31964-21modprobe08:18:493
187312568,38sleep30-21swapper/312:34:533
1725525615,25sleep317261-21ntp_states10:49:173
1725525615,25sleep317261-21ntp_states10:49:173
1061525610,18sleep20-21swapper/207:04:042
698925510,19sleep30-21swapper/308:34:073
2574525515,16sleep30-21swapper/307:49:223
2574525515,16sleep30-21swapper/307:49:223
2440425512,27sleep024458-21fschecks_count09:29:060
1660525515,25sleep316609-21diskstats09:04:063
872925415,18sleep20-21swapper/208:39:112
849025414,21sleep30-21swapper/308:39:033
3064025417,19sleep00-21swapper/009:49:080
3064025417,19sleep00-21swapper/009:49:080
2873525416,22sleep328809-21grep07:59:193
2664225413,24sleep026637-21apt-get11:19:010
1904925417,19sleep20-21swapper/207:29:102
116082545,10sleep30-21swapper/310:31:043
116082545,10sleep30-21swapper/310:31:043
1074625416,16sleep20-21swapper/212:09:212
2869325316,16sleep20-21swapper/211:24:202
2682925318,16sleep30-21swapper/311:19:083
1833425314,22sleep318357-21irqstats09:09:113
423625218,15sleep20-21swapper/208:24:172
2110625210,30sleep30-21swapper/309:18:513
2110625210,30sleep30-21swapper/309:18:513
196525212,23sleep21959-21apt-get11:44:022
196525212,23sleep21959-21apt-get11:44:022
1630525212,22sleep316314-21df_inode12:29:083
1630525212,22sleep316314-21df_inode12:29:083
1086625216,22sleep310894-21users12:09:223
582425113,22sleep25830-21ntp_states08:29:172
3156125111,26sleep031619-21sed11:34:110
2505825113,21sleep225055-21apt-get11:14:022
1703025114,15sleep20-21swapper/209:04:172
1476825111,21sleep30-21swapper/312:24:073
1242425115,21sleep312473-21irqstats07:09:093
2632525013,20sleep326350-21munin-run11:18:483
2199125017,16sleep30-21swapper/311:04:103
2199125017,16sleep30-21swapper/311:04:103
1723025012,22sleep217254-21ntp_states10:49:162
1723025012,22sleep217254-21ntp_states10:49:162
1493825010,23sleep214957-21apt-get08:59:012
1493825010,23sleep214957-21apt-get08:59:012
1265324914,20sleep00-21swapper/007:09:140
121972499,16sleep20-21swapper/207:09:002
537824812,20sleep05387-21df_inode11:54:050
1289224811,17sleep20-21swapper/212:18:502
1239224816,17sleep30-21swapper/310:34:113
1063524811,22sleep010641-21df_inode10:29:080
1063524811,22sleep010641-21df_inode10:29:080
815124713,12sleep30-21swapper/312:03:493
1661124714,13sleep00-21swapper/009:04:070
2870524610,18sleep228730-21munin-run09:43:492
1621424611,19sleep20-21swapper/212:28:592
1621424611,19sleep20-21swapper/212:28:592
121352468,16sleep30-21swapper/308:49:293
1054524613,16sleep00-21swapper/012:09:190
326462459,22sleep332707-21switchtime09:54:193
181562459,14sleep20-21swapper/209:09:062
3062224410,15sleep20-21swapper/209:49:062
3062224410,15sleep20-21swapper/209:49:062
27722408,15sleep20-21swapper/210:03:592
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional