You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-02 - 04:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat May 02, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
293682104714,1015sleep028431-21/usr/sbin/munin21:35:510
293682104714,1015sleep028431-21/usr/sbin/munin21:35:510
26910268838,638sleep326959-21ntpq23:10:433
1700125219,495sleep017037-21ntpq00:20:420
14169248672,282sleep30-21swapper/322:30:383
4017241855,168sleep30-21swapper/322:00:333
1077023538,300sleep310806-21ntpq00:00:503
1251723259,278sleep312564-21grep22:25:333
1251723259,278sleep312564-21grep22:25:333
19037231323,223sleep319180-21unixbench_multi19:20:563
19037231323,223sleep319180-21unixbench_multi19:20:563
13904227830,189sleep31-21systemd20:50:203
26530227318,237sleep226574-21ntpq19:45:462
26530227318,237sleep226574-21ntpq19:45:462
167162269115,124sleep30-21swapper/300:20:373
51982246109,120sleep30-21swapper/323:45:283
8058224254,171sleep38195-21unixbench_multi22:11:073
10504224149,92sleep310453-21latency_hist22:20:203
160252236169,48sleep30-21swapper/322:35:433
160252236169,48sleep30-21swapper/322:35:433
2395122167,198sleep313441-21snmpd19:36:023
2395122167,198sleep313441-21snmpd19:36:023
3841221349,120sleep33915-21switchtime20:15:523
3841221349,120sleep33915-21switchtime20:15:523
20320221323,174sleep020407-21find00:30:470
3140221115,145sleep33270-21grep21:55:493
10632219816,165sleep010850-21latency_hist22:20:210
14574219361,115sleep314629-21fschecks_time20:51:173
7884218812,161sleep17956-21awk22:11:051
20083218821,64sleep020227-21cat22:50:210
20083218821,64sleep020227-21cat22:50:210
25514218010,64sleep30-21swapper/323:05:473
3166021788,148sleep00-21swapper/023:26:190
16965217412,146sleep016928-21latency_hist22:40:200
16965217412,146sleep016928-21latency_hist22:40:200
2012921708,126sleep320170-21ntpq00:30:423
5839216710,38sleep30-21swapper/322:05:423
5839216710,38sleep30-21swapper/322:05:423
18830216537,112sleep30-21swapper/322:45:363
18830216537,112sleep30-21swapper/322:45:363
138721618,105sleep31467-21readlink21:50:483
138721618,105sleep31467-21readlink21:50:483
14228216019,113sleep30-21swapper/300:10:523
14111215917,129sleep214227-21sed00:10:512
29428215614,125sleep329518-21latency_hist23:20:203
23617215611,130sleep223681-21if_eth023:00:372
7980215513,127sleep28054-21find22:11:062
20409215223,94sleep20-21swapper/221:10:212
12460215113,78sleep30-21swapper/300:05:473
1664521488,125sleep316717-21date19:15:233
1664521488,125sleep316717-21date19:15:233
3081521469,124sleep330891-21sed20:00:213
3081214414,109sleep30-21swapper/323:36:233
1927021427,11sleep20-21swapper/219:20:592
1927021427,11sleep20-21swapper/219:20:592
17300214112,63sleep30-21swapper/321:00:213
15281214112,92sleep315331-21if_err_eth000:15:403
23894214019,100sleep123931-21proc_pri23:00:441
12312214020,20sleep30-21swapper/320:45:193
12312214020,20sleep30-21swapper/320:45:193
9341213913,106sleep30-21swapper/320:35:223
2048421389,111sleep30-21swapper/322:50:413
1957921369,88sleep319645-21mailstats21:05:463
8597213149,19sleep30-21swapper/323:55:333
1548821319,107sleep00-21swapper/019:10:440
1548821319,107sleep00-21swapper/019:10:440
8380212826,55sleep30-21swapper/323:55:213
8380212826,55sleep30-21swapper/323:55:213
3107621268,65sleep331205-21apt-config23:25:213
25034212413,97sleep125035-21sshd21:25:141
1562212414,71sleep31605-21latency_hist20:10:303
1562212414,71sleep31605-21latency_hist20:10:303
18275212147,31sleep318367-21fschecks_time00:25:373
18275212147,31sleep318367-21fschecks_time00:25:373
625211912,91sleep0634-21diskstats23:30:350
3197121195,66sleep30-21swapper/321:46:043
18374211819,82sleep00-21swapper/000:25:390
18374211819,82sleep00-21swapper/000:25:390
13205211516,80sleep213231-21munin-plugin-st00:10:182
2208821117,48sleep30-21swapper/322:55:413
32526211017,74sleep232675-21/usr/sbin/munin21:46:152
17632210825,57sleep017723-21fschecks_time21:00:380
4978210650,17sleep30-21swapper/323:45:203
16772210517,56sleep30-21swapper/319:15:273
24488210410,59sleep313441-21snmpd23:02:403
22890210312,77sleep013441-21snmpd22:57:040
21915210319,67sleep221942-21idleruntime-cro21:15:182
5963210112,19sleep10-21swapper/122:05:441
5963210112,19sleep10-21swapper/122:05:441
11792210120,24sleep30-21swapper/319:00:423
11792210120,24sleep30-21swapper/319:00:423
14393210020,61sleep214391-21apt-get20:50:312
2855029925,56sleep228645-21fschecks_time21:35:372
2855029925,56sleep228645-21fschecks_time21:35:372
2816829920,64sleep128250-21cut19:50:501
1558829917,66sleep215621-21ntp_kernel_err19:10:462
1558829917,66sleep215621-21ntp_kernel_err19:10:462
1387029918,61sleep213903-21basename20:50:192
230229711,69sleep32392-21timerandwakeup20:10:513
1972829721,53sleep219797-21df00:30:332
1606829717,56sleep016123-21fschecks_count20:55:380
212092966,52sleep321266-21sed19:30:213
1472929522,56sleep014794-21latency_hist19:10:230
1466929532,26sleep314727-21latency_hist19:10:223
1293929525,48sleep112998-21systemd-run19:05:201
1293929525,48sleep112998-21systemd-run19:05:201
3215129416,20sleep30-21swapper/320:03:163
123429420,54sleep21380-21unixbench_multi23:30:492
314972937,44sleep30-21swapper/323:26:023
1170029313,65sleep111835-21latency_hist00:05:211
2629629218,54sleep126449-21latency_hist23:10:211
462429114,63sleep00-21swapper/023:41:210
2714929132,17sleep30-21swapper/321:30:413
1821029128,23sleep318333-21cat21:00:503
1188229124,29sleep00-21swapper/019:00:430
1188229124,29sleep00-21swapper/019:00:430
867929028,33sleep00-21swapper/020:31:030
3136729018,54sleep031472-21sed21:45:200
2967229028,43sleep129725-21fschecks_time19:56:031
1574129019,46sleep215828-21fschecks_time22:35:372
1574129019,46sleep215828-21fschecks_time22:35:372
1566429025,43sleep115684-21idleruntime-cro20:55:181
1374129013,63sleep113778-21ps19:06:041
2732828815,52sleep227433-21users23:10:502
5162865,12sleep30-21swapper/323:30:263
2760728515,43sleep027750-21grep21:30:500
270528537,29sleep20-21swapper/223:35:502
2525128513,24sleep00-21swapper/023:05:400
1802528416,51sleep118111-21sendmail_mailtr21:00:471
156228410,58sleep1859-21/usr/sbin/munin21:50:501
156228410,58sleep1859-21/usr/sbin/munin21:50:501
1452928412,55sleep20-21swapper/222:30:452
1044028419,39sleep20-21swapper/222:20:192
784528317,27sleep00-21swapper/022:11:040
3007128334,33sleep30-21swapper/321:40:353
2234228314,42sleep222384-21wc22:55:482
181928316,53sleep11899-21fschecks_time20:10:411
147228314,48sleep0859-21/usr/sbin/munin21:50:490
147228314,48sleep0859-21/usr/sbin/munin21:50:490
1309828318,17sleep10-21swapper/122:25:471
1309828318,17sleep10-21swapper/122:25:471
763928223,42sleep17667-21munin-plugin-st20:30:191
763928223,42sleep17667-21munin-plugin-st20:30:191
147028222,37sleep21546-21apt-get20:10:292
147028222,37sleep21546-21apt-get20:10:292
2231028131,35sleep30-21swapper/321:15:353
2157228129,27sleep20-21swapper/219:30:402
1105528123,12sleep30-21swapper/322:20:493
1105528123,12sleep30-21swapper/322:20:493
28862800,6sleep00-21swapper/023:35:510
266422807,59sleep326756-21latency_hist21:30:213
1385528023,26sleep20-21swapper/222:30:202
1385528023,26sleep20-21swapper/222:30:202
951427919,32sleep10-21swapper/120:35:401
3105327913,52sleep031059-21cpu20:00:340
2400127930,34sleep30-21swapper/321:20:403
372927824,32sleep00-21swapper/022:00:180
34027820,29sleep20-21swapper/223:30:212
2845227821,14sleep30-21swapper/321:35:303
2845227821,14sleep30-21swapper/321:35:303
2820727819,43sleep128263-21tune2fs23:15:381
1482527814,46sleep014863-21ntp_kernel_err20:51:220
1220627816,27sleep00-21swapper/000:05:400
61127719,37sleep132577-21/usr/sbin/munin20:05:481
61127719,37sleep132577-21/usr/sbin/munin20:05:481
490627727,35sleep30-21swapper/320:20:363
3117527714,47sleep231198-21fschecks_count20:00:402
2063627719,13sleep30-21swapper/321:10:313
974127617,46sleep19825-21mailstats22:16:431
267922769,52sleep326873-21cat19:45:563
267922769,52sleep326873-21cat19:45:563
2506127611,46sleep30-21swapper/319:41:073
2506127611,46sleep30-21swapper/319:41:073
2191427619,12sleep30-21swapper/319:30:493
1890527628,29sleep10-21swapper/119:20:501
1890527628,29sleep10-21swapper/119:20:501
1395627615,19sleep00-21swapper/020:50:210
697027512,48sleep26493-21apt-get23:50:232
697027512,48sleep26493-21apt-get23:50:232
3089327522,24sleep00-21swapper/020:00:220
2331127515,39sleep023396-21/usr/sbin/munin19:35:430
2331127515,39sleep023396-21/usr/sbin/munin19:35:430
22262750,11sleep20-21swapper/220:10:492
1171727518,35sleep20-21swapper/220:40:542
1171727518,35sleep20-21swapper/220:40:542
1166527523,25sleep10-21swapper/122:21:031
1166527523,25sleep10-21swapper/122:21:031
964227430,10sleep30-21swapper/320:35:503
732527430,11sleep30-21swapper/320:26:173
732527430,11sleep30-21swapper/320:26:173
46422740,6sleep20-21swapper/222:00:482
45342748,53sleep20-21swapper/220:20:152
45342748,53sleep20-21swapper/220:20:152
2881827428,11sleep30-21swapper/319:53:473
1519227425,23sleep20-21swapper/200:15:392
1290727417,24sleep00-21swapper/022:25:420
1290727417,24sleep00-21swapper/022:25:420
997027322,28sleep10-21swapper/100:00:201
82127313,41sleep1825-21/usr/sbin/munin23:30:401
805027330,21sleep10-21swapper/120:30:391
394827315,40sleep14042-21memory23:40:411
3227827313,43sleep132285-21/usr/sbin/munin21:46:101
2710727323,27sleep00-21swapper/023:10:480
1828227329,12sleep30-21swapper/322:43:013
1334527316,41sleep213348-21entropy19:05:412
91342728,11sleep00-21swapper/023:55:510
2659127212,18sleep10-21swapper/121:30:191
2426427212,43sleep224327-21/usr/sbin/munin21:20:472
135227223,33sleep21384-21ps21:50:472
135227223,33sleep21384-21ps21:50:472
456427028,24sleep10-21swapper/122:00:481
328127017,29sleep10-21swapper/120:15:331
328127017,29sleep10-21swapper/120:15:331
377926915,37sleep23869-21latency_hist22:00:202
418126823,25sleep00-21swapper/022:00:380
3060826824,21sleep10-21swapper/121:40:491
1815326824,26sleep20-21swapper/200:25:282
1815326824,26sleep20-21swapper/200:25:282
1465026817,29sleep10-21swapper/122:30:481
1161726819,30sleep211534-21apt-get19:00:332
1161726819,30sleep211534-21apt-get19:00:332
3004526720,23sleep20-21swapper/223:20:442
266826616,26sleep10-21swapper/121:55:391
2641426620,25sleep00-21swapper/019:45:440
2641426620,25sleep00-21swapper/019:45:440
2065126618,25sleep20-21swapper/222:50:442
183192666,10sleep10-21swapper/122:43:281
650026513,37sleep26562-21perl20:25:402
650026513,37sleep26562-21perl20:25:402
2957526513,23sleep20-21swapper/219:55:402
1741326519,23sleep20-21swapper/222:40:382
1730426522,26sleep20-21swapper/219:15:502
1509326515,21sleep10-21swapper/119:10:281
1509326515,21sleep10-21swapper/119:10:281
762626416,29sleep07759-21grep23:50:500
762626416,29sleep07759-21grep23:50:500
288882649,11sleep00-21swapper/019:54:410
2478426415,31sleep00-21swapper/019:41:000
2478426415,31sleep00-21swapper/019:41:000
2354326419,20sleep00-21swapper/021:20:210
2349626413,32sleep123556-21/usr/sbin/munin19:35:481
2349626413,32sleep123556-21/usr/sbin/munin19:35:481
549226319,29sleep25524-21/usr/sbin/munin23:45:392
493426323,24sleep21-21systemd23:45:192
2535026315,34sleep225387-21sed23:05:422
2115626318,22sleep10-21swapper/121:10:491
103672636,11sleep20-21swapper/200:00:332
827026213,33sleep18331-40mandb23:55:191
827026213,33sleep18331-40mandb23:55:191
2735626217,28sleep127421-21grep21:30:471
2351826218,20sleep00-21swapper/023:00:350
1252526222,25sleep012555-21apt-get20:45:280
2966826113,29sleep029662-21apt-get23:20:330
2199626114,27sleep00-21swapper/021:15:200
2000426115,27sleep120097-21memory00:30:401
1964826119,27sleep019722-21ls21:05:470
707526015,30sleep27269-21date22:10:212
707526015,30sleep27269-21date22:10:212
686926013,29sleep06963-21systemd-run22:10:190
686926013,29sleep06963-21systemd-run22:10:190
2975926021,19sleep10-21swapper/121:40:201
2975926021,19sleep10-21swapper/121:40:201
2795026025,20sleep20-21swapper/219:50:432
2596926015,25sleep20-21swapper/219:45:222
2596926015,25sleep20-21swapper/219:45:222
2510326015,28sleep025256-21wc21:25:210
179952605,12sleep10-21swapper/119:19:291
1168926013,32sleep011699-21munin-run00:05:200
1692525915,20sleep00-21swapper/019:15:400
2604125819,20sleep00-21swapper/021:25:500
2273725823,19sleep20-21swapper/221:15:472
1717625815,25sleep017195-21apt-get22:40:280
1406325814,26sleep10-21swapper/120:50:221
3037925724,17sleep10-21swapper/123:20:511
2286425715,26sleep00-21swapper/019:35:230
2057125714,20sleep10-21swapper/122:50:431
178282576,10sleep20-21swapper/200:24:312
1517125716,24sleep10-21swapper/100:15:381
139802577,13sleep00-21swapper/000:10:490
972425618,19sleep20-21swapper/220:36:012
55812567,14sleep00-21swapper/020:20:570
480525613,23sleep14824-21apt-get20:20:301
328025612,27sleep03275-21apt-get20:15:320
328025612,27sleep03275-21apt-get20:15:320
251025614,28sleep12554-21ntpq23:35:451
229672565,10sleep10-21swapper/122:58:011
2011325611,22sleep00-21swapper/019:25:420
19722567,14sleep00-21swapper/020:10:440
2612525514,24sleep126119-21apt-get19:45:331
2612525514,24sleep126119-21apt-get19:45:331
2540525517,18sleep20-21swapper/221:25:362
1996525515,17sleep10-21swapper/119:25:341
1915025515,20sleep20-21swapper/222:45:432
1915025515,20sleep20-21swapper/222:45:432
1835025514,25sleep018343-21apt-get19:20:330
1835025514,25sleep018343-21apt-get19:20:330
1112725513,18sleep10-21swapper/120:40:371
1112725513,18sleep10-21swapper/120:40:371
792725413,25sleep27921-21apt-get20:30:322
3150025413,24sleep131449-21apt-get23:26:041
2172125414,21sleep10-21swapper/119:30:431
2001125418,20sleep10-21swapper/122:50:181
2001125418,20sleep10-21swapper/122:50:181
1909725412,24sleep119091-21apt-get21:05:321
1811325414,18sleep20-21swapper/221:00:482
1093825415,22sleep00-21swapper/022:20:290
1093825415,22sleep00-21swapper/022:20:290
3126125312,21sleep20-21swapper/221:45:182
2848925314,21sleep228528-21ntpq23:15:432
2156025314,15sleep00-21swapper/022:55:180
1685825311,23sleep116900-21irqstats00:20:391
1641425313,18sleep00-21swapper/000:20:220
1235225314,23sleep012365-21munin-run20:45:200
1193825312,25sleep111931-21apt-get00:05:331
998925212,24sleep09995-21ntp_states20:36:070
2728725213,21sleep227332-21ntp_states21:30:442
1948925217,18sleep10-21swapper/100:30:201
1948925217,18sleep10-21swapper/100:30:201
510325114,21sleep24806-21/usr/sbin/munin20:20:432
3181725112,23sleep031779-21apt-get21:45:530
1391625113,22sleep113951-21ntp_states00:10:461
1004325113,19sleep20-21swapper/200:00:212
239722505,14sleep20-21swapper/219:36:182
239722505,14sleep20-21swapper/219:36:182
2002225015,16sleep20-21swapper/219:25:402
1145125013,17sleep20-21swapper/222:21:002
1145125013,17sleep20-21swapper/222:21:002
522624911,20sleep05294-21df23:45:330
3104924913,20sleep131046-21apt-get20:00:311
2719624813,20sleep127327-21sed23:10:491
2514424816,16sleep10-21swapper/119:41:091
2514424816,16sleep10-21swapper/119:41:091
165242488,16sleep20-21swapper/220:55:482
1597424811,20sleep115976-21apt-get20:55:311
681224710,22sleep06836-21ntp_states20:25:460
681224710,22sleep06836-21ntp_states20:25:460
2206724712,14sleep10-21swapper/121:15:211
1407224712,18sleep00-21swapper/022:30:310
132324714,17sleep00-21swapper/020:09:140
132324714,17sleep00-21swapper/020:09:140
648024612,16sleep10-21swapper/123:50:181
2811824414,18sleep00-21swapper/023:15:330
156532377,15sleep10-21swapper/122:35:321
156532377,15sleep10-21swapper/122:35:321
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional