You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-05 - 01:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Fri Sep 04, 2026 12:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
308282194020,1909sleep125989-21kworker/1:207:19:071
308282194020,1909sleep125989-21kworker/1:207:19:071
203052116952,1101sleep120435-21/usr/sbin/munin11:54:061
203052116952,1101sleep120435-21/usr/sbin/munin11:54:061
23841288010,854sleep025411-1kworker/0:2H10:23:560
15255261111,586sleep315266-21ntpq08:13:593
17171252313,494sleep017176-21ntp_states08:20:010
189424567,441sleep318867-1kworker/3:1H09:14:133
265623709,349sleep12681-21awk10:58:571
10904227898,110sleep10-21swapper/109:43:371
25887227011,189sleep126041-21latency_hist12:13:361
25887227011,189sleep126041-21latency_hist12:13:361
65462256101,103sleep16584-21fschecks_count09:28:521
65462256101,103sleep16584-21fschecks_count09:28:521
23799224310,220sleep123819-21apt-get10:23:451
32045223093,36sleep132063-21idleruntime-cro12:33:331
19016222068,45sleep10-21swapper/108:24:061
3006622187,197sleep130112-21grep10:43:511
2051221210,135sleep12186-21cat10:58:361
4072221027,149sleep14111-21ntpq07:39:011
9210220977,31sleep19287-21switchtime07:54:591
9210220977,31sleep19287-21switchtime07:54:591
16200220920,173sleep116358-21sed08:18:371
31248220447,95sleep10-21swapper/109:03:571
31248220447,95sleep10-21swapper/109:03:571
27781219922,121sleep127863-21cut12:18:561
27781219922,121sleep127863-21cut12:18:561
10157218418,97sleep10-21swapper/107:58:471
9854217816,149sleep09912-21sed09:38:560
7857217816,76sleep10-21swapper/111:14:021
24856217710,153sleep224864-21ntpq08:44:012
1463021749,152sleep0367-21snmpd11:38:070
389921709,101sleep10-21swapper/109:19:031
26323217038,23sleep10-21swapper/108:48:561
26323217038,23sleep10-21swapper/108:48:561
2317217015,118sleep12355-21ps09:14:591
12389216931,123sleep112413-21modprobe09:48:331
12389216931,123sleep112413-21modprobe09:48:331
12286216933,31sleep112360-21switchtime08:04:041
30497216740,107sleep130518-21latency_hist12:28:331
549521608,139sleep15117-21apt-get11:08:371
549521608,139sleep15117-21apt-get11:08:371
10425215010,126sleep210136-21/usr/sbin/munin07:58:562
26733214510,120sleep126845-21timerwakeupswit12:14:041
26733214510,120sleep126845-21timerwakeupswit12:14:041
26647214245,74sleep10-21swapper/110:33:341
23004214150,23sleep10-21swapper/112:03:551
23004214150,23sleep10-21swapper/112:03:551
27031213788,25sleep10-21swapper/110:33:541
12955213662,56sleep112660-21/usr/sbin/munin09:48:581
12955213662,56sleep112660-21/usr/sbin/munin09:48:581
13666213216,103sleep113682-21kernelversion11:33:561
18165212925,88sleep318182-21uniq11:48:563
18479212860,55sleep118520-21ntpq11:49:021
464212732,19sleep10-21swapper/110:53:361
23087212611,98sleep023092-21/usr/sbin/munin12:03:580
23087212611,98sleep023092-21/usr/sbin/munin12:03:580
26850212442,19sleep10-21swapper/107:09:021
24662212410,93sleep10-21swapper/108:43:561
15506212330,22sleep10-21swapper/109:58:341
15506212330,22sleep10-21swapper/109:58:341
8851212216,86sleep18909-21fschecks_count11:18:551
22959212111,94sleep322479-21apt-get12:03:413
22959212111,94sleep322479-21apt-get12:03:413
20215211819,23sleep10-21swapper/110:13:361
463211617,75sleep1529-21sendmail_mailqu09:09:021
24882211612,17sleep00-21swapper/007:03:530
24882211612,17sleep00-21swapper/007:03:530
23849211627,60sleep323819-21apt-get10:24:033
1794211520,80sleep10-21swapper/107:33:071
1794211520,80sleep10-21swapper/107:33:071
28491211445,17sleep10-21swapper/107:13:541
28491211445,17sleep10-21swapper/107:13:541
1484211427,28sleep00-21swapper/009:13:370
16605211356,20sleep116624-21apt-get08:18:451
5604211220,72sleep25117-21apt-get11:08:462
5604211220,72sleep25117-21apt-get11:08:462
5362211248,17sleep10-21swapper/109:23:591
4305211216,75sleep03818-21/usr/sbin/munin11:04:000
29807211252,22sleep129875-21systemctl08:59:011
29807211252,22sleep129875-21systemctl08:59:011
2137821086,9sleep10-21swapper/110:14:121
19995210813,24sleep20-21swapper/211:53:592
19995210813,24sleep20-21swapper/211:53:592
7257210747,17sleep10-21swapper/107:49:021
3251621067,10sleep10-21swapper/107:25:491
27371210528,51sleep227397-21apt12:18:332
27371210528,51sleep227397-21apt12:18:332
18447210332,43sleep30-21swapper/308:23:563
45021028,70sleep10-21swapper/112:34:021
12770210018,63sleep212863-21grep09:48:552
12770210018,63sleep212863-21grep09:48:552
183192996,57sleep10-21swapper/110:04:441
379029812,73sleep03863-21fschecks_time07:38:560
1652529826,57sleep10-21swapper/111:43:451
934829714,68sleep19497-21latency_hist09:38:361
33029712,32sleep30-21swapper/312:34:003
524829511,16sleep10-21swapper/107:43:491
50392956,10sleep10-21swapper/111:07:441
2325329413,17sleep10-21swapper/108:38:591
2251929411,59sleep10-21swapper/112:03:351
2251929411,59sleep10-21swapper/112:03:351
2796429211,68sleep028010-21sed12:19:000
2796429211,68sleep028010-21sed12:19:000
346229110,18sleep10-21swapper/107:38:361
3033629113,63sleep030419-21sed10:43:580
2979129111,67sleep029872-21switchtime12:24:040
289132908,17sleep128951-21ntpq10:38:591
2612829022,46sleep326095-21latencyplot07:08:353
2612829022,46sleep326095-21latencyplot07:08:353
2476829020,52sleep324799-21/usr/sbin/munin08:43:583
2174229027,31sleep20-21swapper/208:34:042
65228913,59sleep3659-21diskstats07:28:543
65228913,59sleep3659-21diskstats07:28:543
3209628914,60sleep332186-21latency_hist12:33:353
3198728921,55sleep332033-21/usr/sbin/munin10:49:023
2260828929,29sleep20-21swapper/212:03:362
2260828929,29sleep20-21swapper/212:03:362
1533528917,19sleep20-21swapper/208:14:022
531628717,50sleep04889-21/usr/sbin/munin09:23:580
2394028614,50sleep223947-21forks10:24:102
2353328625,29sleep30-21swapper/308:39:053
197202867,14sleep10-21swapper/108:28:461
163082867,17sleep116384-21systemctl09:59:031
278322858,59sleep10-21swapper/108:53:531
278322858,59sleep10-21swapper/108:53:531
1789228512,54sleep20-21swapper/208:23:342
1652928513,58sleep016561-21cpu11:43:480
2165428417,34sleep30-21swapper/311:59:013
2165428417,34sleep30-21swapper/311:59:013
519628312,51sleep24889-21/usr/sbin/munin09:23:562
462128318,44sleep04623-21cron09:23:340
2940928326,32sleep00-21swapper/008:58:520
2940928326,32sleep00-21swapper/008:58:520
2517428334,26sleep20-21swapper/207:03:592
2517428334,26sleep20-21swapper/207:03:592
1866928330,29sleep20-21swapper/210:08:362
2224328210,59sleep0367-21snmpd12:00:390
2224328210,59sleep0367-21snmpd12:00:390
1181028216,48sleep311966-21grep09:44:043
1181028216,48sleep311966-21grep09:44:043
2666028115,49sleep026676-21/usr/sbin/munin07:08:570
1233028114,50sleep312362-21/usr/sbin/munin11:29:063
1233028114,50sleep312362-21/usr/sbin/munin11:29:063
1954128015,48sleep019690-21date08:28:370
150592806,13sleep10-21swapper/111:38:531
800427918,36sleep28001-21apt-get09:33:452
13862797,16sleep30-21swapper/309:13:363
1084127922,37sleep210870-21grep09:43:342
614727812,51sleep26157-21sshd07:44:072
232012785,12sleep10-21swapper/110:22:161
2253727846,13sleep10-21swapper/108:38:351
183327813,50sleep01875-21systemd07:33:350
183327813,50sleep01875-21systemd07:33:350
1548627828,28sleep00-21swapper/009:58:330
1548627828,28sleep00-21swapper/009:58:330
813627719,39sleep38277-21latency_hist07:53:363
3184927710,53sleep031887-21uname10:48:590
2548827721,38sleep125655-21unixbench_singl07:04:061
2548827721,38sleep125655-21unixbench_singl07:04:061
2187027719,38sleep021959-21awk08:34:060
128092777,11sleep10-21swapper/111:29:321
128092777,11sleep10-21swapper/111:29:321
115352776,12sleep10-21swapper/111:28:071
115352776,12sleep10-21swapper/111:28:071
3096427612,22sleep00-21swapper/009:03:520
3096427612,22sleep00-21swapper/009:03:520
2990027616,25sleep30-21swapper/307:18:453
2990027616,25sleep30-21swapper/307:18:453
144372766,10sleep10-21swapper/108:12:311
2969027519,26sleep30-21swapper/307:18:363
2969027519,26sleep30-21swapper/307:18:363
261627514,46sleep32699-21awk07:34:023
1817827521,31sleep218175-21apt-get08:23:462
2750327412,16sleep00-21swapper/008:53:350
2750327412,16sleep00-21swapper/008:53:350
2481227312,45sleep024899-21latency12:08:550
2481227312,45sleep024899-21latency12:08:550
2127727311,32sleep30-21swapper/308:33:483
204527316,39sleep02097-21acpi07:33:430
2021227315,40sleep020301-21threads11:54:050
2021227315,40sleep020301-21threads11:54:050
1182727329,32sleep025411-1kworker/0:2H11:28:380
1182727329,32sleep025411-1kworker/0:2H11:28:380
3124827213,34sleep030761-21/usr/sbin/munin12:29:000
122027210,48sleep20-21swapper/209:13:112
363027123,28sleep20-21swapper/209:18:562
2827427114,44sleep228332-21cat10:38:352
240232717,10sleep20-21swapper/212:05:342
240232717,10sleep20-21swapper/212:05:342
204592718,16sleep00-21swapper/010:13:510
195927113,44sleep01963-21fschecks_count09:14:380
108127115,41sleep21175-21mailstats07:29:052
108127115,41sleep21175-21mailstats07:29:052
844627020,33sleep08455-21modprobe11:18:330
775727021,22sleep00-21swapper/009:33:350
775727021,22sleep00-21swapper/009:33:350
495727019,34sleep24972-21modprobe07:43:342
1937527021,25sleep30-21swapper/310:09:063
1340727010,17sleep30-21swapper/311:33:503
3123626912,42sleep231325-21latency_hist07:23:362
3123626912,42sleep231325-21latency_hist07:23:362
2580426921,30sleep3307-21systemd-logind12:13:343
2580426921,30sleep3307-21systemd-logind12:13:343
1909926915,37sleep019162-21/usr/sbin/munin10:08:590
685826821,31sleep36903-21ntpq09:28:583
685826821,31sleep36903-21ntpq09:28:583
563526811,27sleep30-21swapper/311:08:513
563526811,27sleep30-21swapper/311:08:513
2627926817,27sleep20-21swapper/208:48:532
2627926817,27sleep20-21swapper/208:48:532
2343226819,21sleep00-21swapper/010:23:360
1711926818,26sleep20-21swapper/210:03:362
1405926819,30sleep014164-21users08:09:070
551326721,30sleep30-21swapper/307:43:553
53526713,39sleep2651-21timerwakeupswit12:34:032
1718126716,25sleep30-21swapper/311:44:033
1544326711,44sleep315478-21grep11:39:033
903426621,19sleep00-21swapper/011:18:580
877526616,22sleep30-21swapper/307:54:493
877526616,22sleep30-21swapper/307:54:493
852526614,29sleep20-21swapper/211:18:352
2818826617,20sleep30-21swapper/312:19:073
2818826617,20sleep30-21swapper/312:19:073
263962669,11sleep20-21swapper/207:08:512
226826619,26sleep20-21swapper/207:33:532
1506726612,11sleep20-21swapper/211:38:542
1236326624,19sleep20-21swapper/208:04:052
1012726610,43sleep310198-21awk09:39:033
2043126527,21sleep30-21swapper/310:13:463
1556426512,40sleep215637-21latency_hist09:58:352
1556426512,40sleep215637-21latency_hist09:58:352
1408826516,28sleep214219-21grep11:34:052
828226421,19sleep00-21swapper/007:53:370
809126416,28sleep00-21swapper/009:33:510
174726416,30sleep21748-21munin-node09:13:432
1489426425,19sleep00-21swapper/008:13:520
55552630,13sleep00-21swapper/007:43:550
294912639,41sleep0364-21ntpd10:39:550
2924026321,20sleep20-21swapper/212:23:502
1588826318,22sleep30-21swapper/309:58:533
1062626316,23sleep00-21swapper/007:59:020
510326218,19sleep20-21swapper/211:08:342
3220326213,34sleep332276-21cat09:08:353
3220326213,34sleep332276-21cat09:08:353
3161626215,24sleep20-21swapper/210:48:532
3118626215,24sleep00-21swapper/007:23:340
3118626215,24sleep00-21swapper/007:23:340
2939126212,31sleep329457-21/usr/sbin/munin12:23:553
2411226215,20sleep00-21swapper/008:43:340
2214326215,22sleep30-21swapper/310:18:553
1110626218,27sleep011125-21apt-get09:43:440
1110626218,27sleep011125-21apt-get09:43:440
99426119,23sleep31078-21memory10:53:583
79426119,27sleep2816-21fschecks_count10:53:532
431426017,17sleep30-21swapper/307:39:073
333926017,27sleep33341-21/usr/sbin/munin09:18:453
296402607,14sleep20-21swapper/210:41:502
2584026018,24sleep325875-21grep10:29:003
2096426019,23sleep30-21swapper/311:58:363
2096426019,23sleep30-21swapper/311:58:363
1959026015,25sleep20-21swapper/210:09:092
1498126016,23sleep00-21swapper/011:38:480
3174825921,20sleep00-21swapper/007:23:580
2823125918,24sleep20-21swapper/208:54:022
2823125918,24sleep20-21swapper/208:54:022
268325919,18sleep20-21swapper/210:58:582
2609225916,26sleep326089-21apt-get12:13:463
2609225916,26sleep326089-21apt-get12:13:463
2025725918,23sleep20-21swapper/208:29:032
658825816,20sleep20-21swapper/209:28:532
658825816,20sleep20-21swapper/209:28:532
516425813,16sleep30-21swapper/311:08:353
318962586,11sleep20-21swapper/212:31:372
2860025822,19sleep30-21swapper/310:38:533
2072225819,19sleep20-21swapper/210:13:572
1180025815,25sleep311895-21perl08:03:543
2798125716,21sleep30-21swapper/308:53:553
2798125716,21sleep30-21swapper/308:53:553
2470825721,23sleep20-21swapper/212:08:532
2470825721,23sleep20-21swapper/212:08:532
1778925722,16sleep20-21swapper/211:48:342
1007425721,20sleep20-21swapper/211:23:362
2889325618,19sleep20-21swapper/207:14:032
2889325618,19sleep20-21swapper/207:14:032
2537625613,25sleep225370-21apt-get10:28:472
128652568,11sleep30-21swapper/309:48:563
128652568,11sleep30-21swapper/309:48:563
876325513,25sleep38759-21apt-get11:18:483
808725514,26sleep28135-21sort07:53:352
528125518,23sleep35313-21nfs_client09:23:573
3233625510,18sleep00-21swapper/012:33:490
3003425516,21sleep20-21swapper/210:43:442
2908625515,25sleep029195-21sed08:58:360
2908625515,25sleep029195-21sed08:58:360
2619525514,25sleep326183-21apt-get08:48:503
2619525514,25sleep326183-21apt-get08:48:503
1981025514,16sleep30-21swapper/308:28:523
1886225520,18sleep00-21swapper/008:24:050
1369425512,27sleep213717-21ntp_states08:09:012
1209125510,24sleep00-21swapper/008:03:590
1024325514,27sleep310267-21cut07:58:533
855225413,25sleep28071-21apt-get07:53:432
855225413,25sleep28071-21apt-get07:53:432
2910725415,18sleep00-21swapper/007:14:050
2910725415,18sleep00-21swapper/007:14:050
2693925413,25sleep026929-21apt-get10:33:490
2480125418,19sleep00-21swapper/008:43:590
2330225418,22sleep223328-21munin-run10:23:342
228725413,25sleep02281-21apt-get10:58:470
1678925413,24sleep316796-21diskstats08:19:523
1099425418,18sleep20-21swapper/211:24:042
1099425418,18sleep20-21swapper/211:24:042
882625312,21sleep00-21swapper/007:54:500
882625312,21sleep00-21swapper/007:54:500
3061725314,26sleep30-21swapper/310:44:053
2959225313,17sleep30-21swapper/308:58:553
2959225313,17sleep30-21swapper/308:58:553
2646325312,25sleep326470-21df_inode07:08:523
253325314,21sleep30-21swapper/310:58:553
1756525314,18sleep30-21swapper/310:03:533
104222538,18sleep30-21swapper/311:23:533
104222538,18sleep30-21swapper/311:23:533
2751325215,17sleep30-21swapper/310:34:043
1422725211,23sleep314222-21apt-get09:53:483
1422725211,23sleep314222-21apt-get09:53:483
1120725211,23sleep211214-21diskstats09:43:502
1120725211,23sleep211214-21diskstats09:43:502
56432519,25sleep00-21swapper/011:08:570
56432519,25sleep00-21swapper/011:08:570
3154425113,22sleep231551-21df_inode07:23:522
2665525114,21sleep026732-21cat12:14:030
2665525114,21sleep026732-21cat12:14:030
182982515,34sleep00-21swapper/010:04:270
1815825114,21sleep218164-21diskstats11:48:552
1157525112,23sleep011621-21unin-run11:28:340
1157525112,23sleep011621-21unin-run11:28:340
2939325012,22sleep229402-21df_inode08:58:502
2939325012,22sleep229402-21df_inode08:58:502
1854225015,18sleep00-21swapper/011:49:050
131072508,15sleep00-21swapper/009:49:010
131072508,15sleep00-21swapper/009:49:010
844724912,22sleep38458-21ntp_states09:33:583
1232224910,20sleep212328-21ntp_states11:29:052
1232224910,20sleep212328-21ntp_states11:29:052
319162489,21sleep30-21swapper/307:24:013
2895024810,16sleep20-21swapper/212:23:332
2895024810,16sleep20-21swapper/212:23:332
70624710,21sleep0702-21apt-get10:53:470
1578124711,20sleep015799-21apt-get09:58:460
1323824710,15sleep30-21swapper/308:08:463
724424615,20sleep37250-21df_inode11:13:483
3137424512,17sleep230856-21/usr/sbin/munin09:04:032
3137424512,17sleep230856-21/usr/sbin/munin09:04:032
2556324512,16sleep00-21swapper/010:28:530
2285224413,19sleep20-21swapper/208:38:482
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional