You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-01 - 06:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Sep 01, 2026 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23734277023,728sleep323777-21/usr/sbin/munin22:44:153
23734277023,728sleep323777-21/usr/sbin/munin22:44:153
18970275541,562sleep219100-21grep00:14:172
18970275541,562sleep219100-21grep00:14:172
83712386104,268sleep08398-21users21:54:220
8006237519,234sleep28024-21modprobe20:13:462
3735236547,184sleep23799-21fschecks_time19:59:092
295412350113,203sleep229048-21/usr/sbin/munin21:19:152
301032341129,197sleep230182-21mailstats23:04:162
301032341129,197sleep230182-21mailstats23:04:162
25482233838,283sleep125566-21memory19:24:121
25482233838,283sleep125566-21memory19:24:121
6042228988,183sleep26149-21fschecks_time21:49:072
2319322559,206sleep223232-21ntpq20:59:122
27490224418,160sleep227633-21unixbench_singl19:29:262
27490224418,160sleep227633-21unixbench_singl19:29:262
22066224312,216sleep122069-21http19:14:021
14334224329,199sleep214448-21timerandwakeup22:14:182
31506223334,181sleep231600-21fschecks_time19:44:072
2617223110,204sleep02684-21awk19:54:190
2617223110,204sleep02684-21awk19:54:190
5283221918,184sleep20-21swapper/223:29:072
29551221018,67sleep20-21swapper/219:38:462
29551221018,67sleep20-21swapper/219:38:462
12667220045,141sleep212678-21sh23:53:462
12667220045,141sleep212678-21sh23:53:462
16351219914,167sleep216475-21latency_hist20:38:502
8523219313,167sleep18574-21cat23:39:071
2438218514,123sleep20-21swapper/219:54:122
2438218514,123sleep20-21swapper/219:54:122
1650021817,163sleep216537-21cat00:08:472
1650021817,163sleep216537-21cat00:08:472
2186721789,130sleep221888-21fschecks_count22:39:052
2186721789,130sleep221888-21fschecks_count22:39:052
13432217439,120sleep221425-21munin-node20:28:572
2123521727,148sleep221386-21latency_hist00:23:492
5708217110,64sleep20-21swapper/220:04:182
9747216914,131sleep29911-21unixbench_multi21:59:162
27673216310,138sleep00-21swapper/021:14:080
9846216040,54sleep20-21swapper/220:18:472
983321609,106sleep20-21swapper/223:43:572
983321609,106sleep20-21swapper/223:43:572
3642214923,78sleep20-21swapper/223:24:022
21583214717,115sleep021646-21idleruntime00:24:080
714121449,42sleep20-21swapper/223:34:122
18109214411,119sleep218120-21acpi20:43:582
1667121396,13sleep20-21swapper/220:39:062
1667121396,13sleep20-21swapper/220:39:062
18072213618,95sleep318218-21latency_hist00:13:493
2757321359,91sleep227582-21forks21:14:062
388621327,114sleep2367-21snmpd21:40:592
2774121329,81sleep227808-21modprobe22:58:472
2774121329,81sleep227808-21modprobe22:58:472
2450621328,84sleep20-21swapper/200:33:592
141121325,32sleep20-21swapper/223:17:122
16439213130,91sleep2367-21snmpd22:22:092
16439213130,91sleep2367-21snmpd22:22:092
1968021298,81sleep20-21swapper/220:48:592
1693721289,37sleep20-21swapper/200:09:042
5701212718,96sleep05740-21latency_hist21:48:480
25698212710,102sleep225833-21cat21:08:492
21823212610,65sleep20-21swapper/220:54:202
18928212611,78sleep218969-21latency22:29:162
18928212611,78sleep218969-21latency22:29:162
31896212515,84sleep332040-21grep23:09:183
18745212519,86sleep318799-21grep22:29:133
18745212519,86sleep318799-21grep22:29:133
8027212320,24sleep30-21swapper/320:13:473
24198212228,59sleep224267-21cat19:19:162
2353421227,76sleep222959-21/usr/sbin/munin00:29:192
10407212210,94sleep110474-21awk23:44:161
10407212210,94sleep110474-21awk23:44:161
20953212116,21sleep10-21swapper/120:53:461
2903821205,29sleep20-21swapper/223:00:042
2704721185,29sleep20-21swapper/221:11:522
3091421175,29sleep20-21swapper/219:41:002
2076211614,25sleep02174-21users21:34:190
16661211623,76sleep016811-21apt-key22:23:490
16661211623,76sleep016811-21apt-key22:23:490
1337921165,30sleep20-21swapper/222:13:052
28381211513,86sleep328418-21/usr/sbin/munin19:34:073
28381211513,86sleep328418-21/usr/sbin/munin19:34:073
8898211418,44sleep20-21swapper/223:39:162
9572211318,77sleep09587-21ls23:43:460
31179211220,70sleep031225-21cat19:43:490
2618821129,81sleep20-21swapper/222:53:462
4104211027,60sleep34119-21apt21:43:473
28031211031,54sleep028050-21apt-get22:58:570
17163211027,70sleep217085-21apt-get22:24:522
17163211027,70sleep217085-21apt-get22:24:522
25008210912,83sleep125071-21fschecks_time22:49:091
516210615,75sleep00-21swapper/019:49:020
516210615,75sleep00-21swapper/019:49:020
9469210416,71sleep09514-21ntp_offset21:59:100
8196210415,70sleep18404-21latency_hist20:13:491
25163210413,70sleep325252-21/usr/sbin/munin22:49:153
5654210316,72sleep35699-21systemd-run21:48:473
29715210314,72sleep129809-21fschecks_time23:04:071
29715210314,72sleep129809-21fschecks_time23:04:071
263621025,82sleep00-21swapper/023:20:170
10830210215,68sleep010872-21proc_pri20:19:180
4075210114,71sleep04107-21ntp_states19:59:190
565829914,70sleep05665-21sh20:04:160
212629916,70sleep32134-21wc23:19:273
1123329917,69sleep011316-21wc22:04:190
451129841,17sleep20-21swapper/221:44:062
2364629814,20sleep30-21swapper/319:19:013
2073629846,16sleep20-21swapper/219:09:192
225292968,49sleep222564-21grep19:14:162
2042229620,56sleep120480-21/usr/sbin/munin22:34:111
2042229620,56sleep120480-21/usr/sbin/munin22:34:111
3026029516,63sleep030378-21sed23:04:180
3026029516,63sleep030378-21sed23:04:180
2719429511,67sleep327199-21kworker/u8:321:13:473
1997329514,58sleep319853-21/usr/sbin/munin00:19:063
181332959,74sleep0367-21snmpd20:44:000
204129420,28sleep10-21swapper/119:53:591
204129420,28sleep10-21swapper/119:53:591
1461929421,56sleep20-21swapper/223:59:022
1461929421,56sleep20-21swapper/223:59:022
3115429312,66sleep331223-21sensors_temp21:24:173
2402629316,64sleep124066-21ntp_kernel_pll_19:19:111
1242429335,42sleep20-21swapper/220:24:162
20822927,17sleep20-21swapper/223:18:572
2520229112,38sleep219876-21cyclictest19:24:052
2520229112,38sleep219876-21cyclictest19:24:052
2080629110,64sleep20-21swapper/222:34:202
2080629110,64sleep20-21swapper/222:34:202
79129011,65sleep2864-21latency19:49:122
79129011,65sleep2864-21latency19:49:122
3143629033,42sleep20-21swapper/223:09:082
2291029011,65sleep122978-21fschecks_time20:59:071
2178429018,50sleep321425-21munin-node19:13:563
1584829025,45sleep115280-21/usr/sbin/munin22:19:181
1584829025,45sleep115280-21/usr/sbin/munin22:19:181
1033529020,31sleep20-21swapper/220:18:592
103129032,17sleep21093-21latency_hist21:33:482
1785428811,61sleep017973-21timerwakeupswit22:25:180
1785428811,61sleep017973-21timerwakeupswit22:25:180
248452877,34sleep20-21swapper/222:48:582
248452877,34sleep20-21swapper/222:48:582
2439128724,41sleep024532-21grep19:19:180
1454828724,37sleep014542-21apt-get23:59:000
1454828724,37sleep014542-21apt-get23:59:000
671128613,53sleep06729-21/usr/sbin/munin23:33:570
62628615,54sleep0629-21/usr/sbin/munin21:29:220
856128517,53sleep38567-21cpu20:14:203
495128515,52sleep15098-21latency_hist23:28:501
2161828511,52sleep00-21swapper/020:54:140
849528419,31sleep20-21swapper/221:55:292
2170428420,28sleep20-21swapper/200:24:102
106502837,60sleep20-21swapper/222:04:052
980528220,40sleep19828-21munin-run20:18:461
2650728216,50sleep026576-21sed21:09:160
2063028212,51sleep020769-21awk00:19:190
2010528213,21sleep30-21swapper/319:09:043
147122828,63sleep00-21swapper/022:15:210
12692817,34sleep20-21swapper/221:34:012
296032790,6sleep30-21swapper/321:19:153
2484027914,49sleep124877-21ps21:04:161
1253827925,35sleep012552-21needreboot22:09:110
1162027919,28sleep10-21swapper/120:23:471
1104327913,51sleep311172-21unixbench_multi20:19:213
319202786,33sleep20-21swapper/221:28:482
2241227817,48sleep022528-21timerandwakeup22:39:310
2241227817,48sleep022528-21timerandwakeup22:39:310
1823427816,16sleep30-21swapper/320:44:083
1188527817,41sleep011904-21apt-get20:23:580
2661627724,31sleep10-21swapper/122:54:181
2661627724,31sleep10-21swapper/122:54:181
1984327724,33sleep00-21swapper/019:08:490
3217327521,27sleep10-21swapper/121:29:031
2821627512,40sleep328361-21unixbench_singl21:14:193
145492757,47sleep10-21swapper/123:59:011
145492757,47sleep10-21swapper/123:59:011
123692750,7sleep10-21swapper/122:09:071
2737027414,41sleep127466-21timerandwakeup19:29:251
2737027414,41sleep127466-21timerandwakeup19:29:251
170982748,14sleep10-21swapper/122:24:021
170982748,14sleep10-21swapper/122:24:021
2967927320,28sleep00-21swapper/021:19:170
1627127317,38sleep316276-21modprobe20:38:473
1510827318,37sleep115251-21latency_hist22:18:491
1060327321,31sleep310747-21sed23:44:183
1060327321,31sleep310747-21sed23:44:183
1295927220,35sleep20-21swapper/223:54:022
410827111,14sleep10-21swapper/119:59:201
343927127,19sleep10-21swapper/123:23:491
2937627122,26sleep30-21swapper/323:03:493
2913227112,35sleep028268-21/usr/sbin/munin19:34:200
2913227112,35sleep028268-21/usr/sbin/munin19:34:200
2706827018,28sleep30-21swapper/322:54:283
2706827018,28sleep30-21swapper/322:54:283
2142627020,22sleep10-21swapper/119:13:501
1830127021,31sleep118479-21modprobe19:03:501
1746127029,23sleep10-21swapper/100:09:151
1024627013,34sleep00-21swapper/023:44:110
1024627013,34sleep00-21swapper/023:44:110
1782026921,22sleep00-21swapper/000:11:280
807726820,23sleep30-21swapper/323:38:493
2467826812,38sleep024744-21memory21:04:110
2437326818,32sleep324368-21apt-get21:04:023
1317226814,36sleep013229-21irqstats23:54:080
94526719,25sleep30-21swapper/323:14:183
63426719,32sleep00-21swapper/023:14:110
3070826713,35sleep130595-21/usr/sbin/munin21:24:061
415626618,22sleep10-21swapper/121:43:491
3038926618,17sleep10-21swapper/121:23:501
2985926613,35sleep329888-21irqstats23:04:093
2985926613,35sleep329888-21irqstats23:04:093
113426619,25sleep30-21swapper/319:49:213
113426619,25sleep30-21swapper/319:49:213
2342826517,20sleep00-21swapper/020:59:190
147026513,30sleep10-21swapper/121:34:081
69852647,12sleep10-21swapper/121:50:591
3061926416,30sleep030613-21apt-get21:24:010
2258226421,23sleep30-21swapper/320:58:493
1955626415,31sleep319584-21modprobe00:18:463
1955626415,31sleep319584-21modprobe00:18:463
123872649,14sleep10-21swapper/120:24:151
2881326311,14sleep10-21swapper/119:34:171
2881326311,14sleep10-21swapper/119:34:171
2487026319,27sleep124864-21apt-get22:49:011
2487026319,27sleep124864-21apt-get22:49:011
2008726311,13sleep30-21swapper/320:49:123
1836926319,23sleep10-21swapper/100:14:031
1836926319,23sleep10-21swapper/100:14:031
1660326321,21sleep30-21swapper/322:23:483
1660326321,21sleep30-21swapper/322:23:483
1131826318,24sleep111472-21grep22:04:201
1042426319,19sleep30-21swapper/322:03:493
649826213,33sleep36576-21latency_hist23:33:483
1926126211,38sleep00-21swapper/000:15:030
1926126211,38sleep00-21swapper/000:15:030
1519926220,22sleep00-21swapper/020:34:070
1070226213,32sleep310711-21diskstats22:04:063
688926123,18sleep10-21swapper/123:34:071
542826118,19sleep00-21swapper/023:29:090
2372926123,17sleep00-21swapper/000:29:210
1346526120,21sleep10-21swapper/120:29:041
1323026119,25sleep10-21swapper/123:54:091
1177926114,32sleep311807-21nfs4_client23:49:103
1177926114,32sleep311807-21nfs4_client23:49:103
3014626019,23sleep30-21swapper/319:39:123
2888126016,21sleep10-21swapper/121:18:491
277392605,43sleep00-21swapper/022:58:450
277392605,43sleep00-21swapper/022:58:450
2146626016,26sleep00-21swapper/022:38:460
2146626016,26sleep00-21swapper/022:38:460
1429126016,25sleep314306-21munin-run23:58:473
1343126013,26sleep30-21swapper/322:13:463
412525918,24sleep04154-21systemd-run21:43:480
2504025912,30sleep025121-21mailstats00:34:170
2152125915,22sleep30-21swapper/300:24:073
2066625918,26sleep320672-21ntp_states22:34:173
2066625918,26sleep320672-21ntp_states22:34:173
1977525917,27sleep31-21systemd19:08:483
177525916,26sleep11-21systemd19:53:471
177525916,26sleep11-21systemd19:53:471
1328125917,22sleep013423-21kworker/u8:220:28:500
874225820,20sleep00-21swapper/020:14:290
69092587,41sleep00-21swapper/021:50:020
574325816,25sleep15885-21sed21:48:491
3220625816,24sleep332320-21users19:44:203
2528125814,28sleep025285-21diskstats19:24:070
2528125814,28sleep025285-21diskstats19:24:070
2452725818,24sleep324524-21apt-get00:34:003
2423925814,22sleep10-21swapper/100:33:461
206525818,19sleep30-21swapper/319:54:013
206525818,19sleep30-21swapper/319:54:013
118092589,33sleep0367-21snmpd23:49:110
118092589,33sleep0367-21snmpd23:49:110
689925719,20sleep00-21swapper/020:09:080
32525714,24sleep1332-21df_inode23:14:041
2730125713,27sleep027369-21/usr/sbin/munin19:29:240
2730125713,27sleep027369-21/usr/sbin/munin19:29:240
2252925716,18sleep10-21swapper/122:39:321
2252925716,18sleep10-21swapper/122:39:321
2119325715,21sleep10-21swapper/100:23:481
1882025715,25sleep218886-21memory19:04:112
1775325721,16sleep30-21swapper/322:25:173
1775325721,16sleep30-21swapper/322:25:173
1503625715,20sleep30-21swapper/323:59:163
1503625715,20sleep30-21swapper/323:59:163
875125614,18sleep10-21swapper/121:58:471
759225617,22sleep37678-21fschecks_time21:54:083
3272925618,16sleep00-21swapper/019:48:490
3141725616,25sleep131410-21apt-get19:44:021
2359725616,26sleep023610-21uname22:44:110
2359725616,26sleep023610-21uname22:44:110
1501025616,24sleep115005-21apt-get20:34:001
2307325517,18sleep30-21swapper/300:29:073
3138625414,22sleep131435-21irqstats23:09:071
2569325418,15sleep30-21swapper/319:24:183
2569325418,15sleep30-21swapper/319:24:183
2334025411,26sleep323426-21mailstats20:59:173
46712534,11sleep30-21swapper/323:25:533
2639125313,23sleep126397-21ntp_states21:09:131
1789725314,18sleep10-21swapper/120:43:491
1789725314,18sleep10-21swapper/120:43:491
1673725313,18sleep00-21swapper/020:39:080
1673725313,18sleep00-21swapper/020:39:080
645725214,23sleep36486-21munin-run20:08:473
644825213,23sleep16475-21cron23:33:461
36112527,32sleep00-21swapper/023:23:560
3122825215,21sleep131369-21latency_hist19:43:501
2199625210,18sleep30-21swapper/322:39:103
2199625210,18sleep30-21swapper/322:39:103
2010025213,20sleep00-21swapper/019:09:010
1862425211,17sleep10-21swapper/122:28:591
1862425211,17sleep10-21swapper/122:28:591
1659025212,23sleep316581-21apt-get20:39:033
1659025212,23sleep316581-21apt-get20:39:033
1645225217,21sleep11-21systemd00:08:461
1645225217,21sleep11-21systemd00:08:461
933525116,17sleep10-21swapper/121:59:071
2887025112,22sleep128961-21users22:59:191
2232925116,19sleep00-21swapper/019:14:100
1968425111,24sleep019678-21apt-get20:49:010
1653825116,20sleep316684-21sed00:08:483
1653825116,20sleep316684-21sed00:08:483
1322325114,24sleep313278-21sed20:28:493
1149225112,22sleep111495-21diskstats23:49:041
1149225112,22sleep111495-21diskstats23:49:041
527425012,23sleep15281-21df_inode20:04:041
2497725011,23sleep124988-21ntp_states00:34:141
1991525012,17sleep00-21swapper/022:33:460
1991525012,17sleep00-21swapper/022:33:460
95122498,25sleep30-21swapper/323:43:013
364224917,17sleep30-21swapper/319:59:023
261702496,32sleep00-21swapper/022:53:320
2133224913,16sleep10-21swapper/120:54:071
1608224917,16sleep00-21swapper/000:04:010
1608224917,16sleep00-21swapper/000:04:010
2763524813,18sleep30-21swapper/319:29:273
2763524813,18sleep30-21swapper/319:29:273
1445024811,24sleep314585-21unixbench_singl22:14:193
127024813,21sleep31261-21apt-get21:34:023
1373124714,15sleep30-21swapper/320:29:103
52124612,16sleep30-21swapper/321:29:213
3153624614,16sleep00-21swapper/023:09:110
216472467,14sleep10-21swapper/100:24:091
1813424411,18sleep118129-21apt-get20:44:011
643824313,15sleep30-21swapper/321:49:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional