You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-11 - 05:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Dec 11, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
180012239313,2368sleep314821-21kworker/3:221:41:583
194802100013,968sleep319515-21memory23:28:163
194802100013,968sleep319515-21memory23:28:163
6341277113,741sleep06347-21/usr/sbin/munin00:28:030
2277525599,538sleep17679-21kworker/u8:423:38:201
27987242319,385sleep328019-21/usr/sbin/munin22:12:563
807923879,341sleep28167-21cut22:52:532
26851232113,266sleep226886-21idleruntime-cro20:27:332
15536230314,273sleep015567-21latency_hist23:17:350
1861122990,14sleep231861-21cyclictest21:42:572
7207228019,244sleep37226-21latency_hist00:32:343
11284227083,91sleep211430-21unixbench_singl21:18:062
21514225169,139sleep221615-21timerandwakeup20:08:072
3261224036,186sleep23327-21awk00:18:032
454622360,7sleep10-21swapper/120:58:211
32176222751,160sleep232218-21fschecks_count00:07:532
10224222412,147sleep210368-21tail22:58:062
1575122128,192sleep215753-21acpi23:17:452
28980220915,177sleep029011-21idleruntime-cro22:17:330
25584220323,166sleep325585-21sshd22:04:153
5628219980,60sleep20-21swapper/221:02:352
28775219615,165sleep228909-21latency_hist23:57:362
3053219251,81sleep20-21swapper/222:37:342
3053219251,81sleep20-21swapper/222:37:342
2965021838,159sleep229699-21ntpq22:18:002
2536218133,133sleep22621-21cat00:17:352
5343217029,130sleep25666-21snmpd00:24:292
1792216418,86sleep20-21swapper/219:07:532
25927216332,90sleep226077-21latency_hist22:07:382
21956216227,80sleep20-21swapper/221:53:042
31581216027,40sleep20-21swapper/220:42:362
31581216027,40sleep20-21swapper/220:42:362
15026215920,11sleep20-21swapper/219:47:592
12576215719,79sleep212602-21munin-plugin-st19:42:342
12576215719,79sleep212602-21munin-plugin-st19:42:342
17795215210,51sleep20-21swapper/223:23:022
644521518,106sleep26535-21cat21:03:042
25545214527,98sleep1225-21systemd-journal23:47:331
20015214328,101sleep220084-21fschecks_time21:47:552
32501214231,53sleep232532-21/usr/sbin/munin19:03:202
32501214231,53sleep232532-21/usr/sbin/munin19:03:202
11987214214,115sleep212000-21vmstat23:03:072
26311214114,109sleep126370-21/usr/sbin/munin23:48:011
26311214114,109sleep126370-21/usr/sbin/munin23:48:011
2549921415,11sleep00-21swapper/022:03:110
16514213924,89sleep316655-21grep23:18:063
9380213832,83sleep20-21swapper/221:13:012
9380213832,83sleep20-21swapper/221:13:012
24103213814,40sleep324114-21df_inode20:17:533
2434213119,84sleep32567-21unixbench_singl19:08:063
1117213022,95sleep21158-21sort20:47:572
28994212939,73sleep20-21swapper/220:32:592
28994212939,73sleep20-21swapper/220:32:592
3813212833,80sleep21-21systemd20:57:342
3813212833,80sleep21-21systemd20:57:342
8724212766,40sleep20-21swapper/219:28:042
1908321246,76sleep20-21swapper/223:27:392
1908321246,76sleep20-21swapper/223:27:392
27219212316,84sleep027310-21cat23:52:370
27219212316,84sleep027310-21cat23:52:370
26309212310,69sleep226344-21idleruntime22:07:542
386721229,64sleep20-21swapper/222:38:152
2273212210,97sleep02210-21latency_hist20:52:350
23828212130,19sleep20-21swapper/220:17:362
31753212021,20sleep20-21swapper/219:02:352
19091212022,72sleep019110-21apt-get23:27:450
19091212022,72sleep019110-21apt-get23:27:450
4194211931,67sleep20-21swapper/219:13:222
4194211931,67sleep20-21swapper/219:13:222
9973211747,17sleep20-21swapper/219:32:572
4299211518,22sleep30-21swapper/300:22:453
30962211414,24sleep130985-21ntp_states00:03:001
22950211418,72sleep10-21swapper/120:13:031
5317211310,87sleep15331-21awk22:43:031
24135211028,21sleep224234-21grep20:17:562
21083211028,71sleep221104-21tune2fs23:33:082
13437211046,16sleep20-21swapper/219:43:432
13676210919,67sleep113766-21mailstats19:43:501
988210815,78sleep01116-21latency_hist00:12:360
4850210812,45sleep20-21swapper/222:42:472
18786210826,26sleep30-21swapper/323:27:363
4807210717,75sleep04843-21wc20:58:290
31837210325,24sleep25666-21snmpd22:24:222
15701210317,22sleep00-21swapper/021:33:020
14998210311,75sleep115050-21latency_hist21:32:351
6487210120,66sleep36490-21/usr/sbin/munin22:47:533
4323210116,68sleep34454-21uptime19:13:233
4323210116,68sleep34454-21uptime19:13:233
2394521015,62sleep20-21swapper/223:41:562
1039921016,75sleep00-21swapper/019:33:070
8797210013,73sleep38883-21sort19:28:053
3163229938,39sleep031667-21munin-run19:02:330
2940629921,14sleep20-21swapper/223:58:042
1934729923,13sleep20-21swapper/220:02:502
387429833,32sleep34006-21apt-key20:57:353
387429833,32sleep34006-21apt-key20:57:353
3039129811,73sleep330386-21fschecks_count20:37:563
3039129811,73sleep330386-21fschecks_count20:37:563
133272980,7sleep10-21swapper/123:08:061
3192629719,54sleep332096-21apt-config19:02:363
2605329736,45sleep20-21swapper/223:47:542
2605329736,45sleep20-21swapper/223:47:542
1755729715,69sleep117613-21sed19:57:361
2142829618,63sleep321492-21processes23:33:173
1422529620,15sleep20-21swapper/223:12:482
2310629421,14sleep20-21swapper/221:57:522
1505129410,65sleep20-21swapper/221:32:362
1210929420,56sleep20-21swapper/221:22:512
19812939,15sleep00-21swapper/022:33:330
19812939,15sleep00-21swapper/022:33:330
2305229212,27sleep323065-21irqstats23:38:423
2757329121,11sleep20-21swapper/220:28:022
2757329121,11sleep20-21swapper/220:28:022
2312729114,57sleep023222-21fschecks_time21:57:540
554629017,12sleep20-21swapper/219:18:022
377729017,56sleep13812-21modprobe20:57:331
377729017,56sleep13812-21modprobe20:57:331
118929014,58sleep21208-21apt-get22:32:332
1085129015,51sleep310937-21latency_hist23:02:353
2535028922,24sleep10-21swapper/120:22:361
197128922,45sleep32113-21grep00:13:083
173728923,45sleep11734-21apt-get22:32:451
173728923,45sleep11734-21apt-get22:32:451
130222889,45sleep213065-21ntpq23:08:002
584428713,24sleep30-21swapper/321:02:443
327428725,32sleep10-21swapper/122:37:461
243632878,15sleep20-21swapper/223:42:532
1628728613,54sleep316312-21/usr/sbin/munin19:52:533
1188128622,29sleep30-21swapper/321:22:353
78702858,14sleep20-21swapper/221:08:002
78702858,14sleep20-21swapper/221:08:002
1109228512,58sleep111157-21cpuspeed23:02:501
88852840,7sleep00-21swapper/019:28:050
3167528423,42sleep131752-21apt-key19:02:341
3061628425,47sleep20-21swapper/200:02:532
176142848,41sleep217757-21modprobe19:57:372
1744628414,54sleep117480-21fschecks_time23:22:531
1195728411,58sleep112068-21timerwakeupswit19:38:091
1195728411,58sleep112068-21timerwakeupswit19:38:091
546328317,46sleep04829-21/usr/sbin/munin22:43:060
2464928314,54sleep024657-21/usr/sbin/munin23:43:000
2102828312,58sleep021041-21fschecks_count23:32:540
1591728314,53sleep015982-21modprobe19:52:340
147828313,54sleep01497-21irqstats00:12:560
66422827,15sleep20-21swapper/219:22:512
277628219,35sleep00-21swapper/000:17:490
688428111,54sleep16947-21find22:48:041
352028116,48sleep13667-21awk00:18:061
2903128121,16sleep20-21swapper/222:17:352
280452800,6sleep10-21swapper/120:28:081
280452800,6sleep10-21swapper/120:28:081
180182805,64sleep20-21swapper/221:42:092
789927923,34sleep07918-21munin-run19:27:330
3251027926,16sleep20-21swapper/220:43:082
3251027926,16sleep20-21swapper/220:43:082
3099527917,39sleep331015-21/usr/sbin/munin00:03:023
2869027923,34sleep328704-21apt23:57:323
2740227916,44sleep127467-21modprobe22:12:331
1588127912,50sleep315980-21timerwakeupswit21:33:063
90627815,20sleep10-21swapper/100:12:331
1621127820,15sleep20-21swapper/219:52:482
1037327816,19sleep10-21swapper/121:17:371
640127721,39sleep20-21swapper/222:47:482
55682778,50sleep00-21swapper/019:18:040
3238527715,46sleep032171-21/usr/sbin/munin19:03:180
3238527715,46sleep032171-21/usr/sbin/munin19:03:180
2202427722,34sleep30-21swapper/321:53:053
1106527721,29sleep00-21swapper/023:02:450
2610327616,40sleep326162-21/usr/sbin/munin23:47:553
2610327616,40sleep326162-21/usr/sbin/munin23:47:553
235252767,48sleep10-21swapper/121:58:031
563527529,19sleep10-21swapper/100:27:351
3237627520,40sleep332380-21fschecks_time19:03:173
3237627520,40sleep332380-21fschecks_time19:03:173
3180027517,39sleep031801-21/usr/sbin/munin20:42:460
3180027517,39sleep031801-21/usr/sbin/munin20:42:460
298672750,7sleep10-21swapper/123:58:101
2227127420,25sleep10-21swapper/123:37:351
444627317,36sleep04515-21fschecks_time00:22:540
2216527323,29sleep00-21swapper/020:12:330
1505127315,33sleep314633-21/usr/sbin/munin19:48:003
3263427211,45sleep132679-21/usr/sbin/munin19:03:231
3263427211,45sleep132679-21/usr/sbin/munin19:03:231
940827122,29sleep30-21swapper/321:13:023
940827122,29sleep30-21swapper/321:13:023
99442708,16sleep00-21swapper/022:58:000
4252709,12sleep30-21swapper/322:28:063
2276027016,21sleep00-21swapper/020:12:570
2054027017,36sleep120650-21timerwakeupswit21:48:061
3241426921,26sleep10-21swapper/120:43:071
3241426921,26sleep10-21swapper/120:43:071
2370026921,26sleep30-21swapper/321:58:053
1911226913,22sleep10-21swapper/123:27:461
1911226913,22sleep10-21swapper/123:27:461
95572689,15sleep00-21swapper/021:13:070
95572689,15sleep00-21swapper/021:13:070
52732689,13sleep30-21swapper/322:43:013
290626820,27sleep00-21swapper/020:53:000
290626820,27sleep00-21swapper/020:53:000
2461526817,24sleep00-21swapper/020:18:060
156692686,16sleep20-21swapper/221:33:002
142626814,26sleep10-21swapper/122:32:371
1363026823,27sleep30-21swapper/319:43:483
102302685,41sleep10-21swapper/119:33:041
798126717,29sleep17975-21apt-get22:52:481
71212676,29sleep20-21swapper/200:32:212
167872678,16sleep00-21swapper/021:37:470
1374626714,22sleep00-21swapper/021:27:530
314626615,26sleep10-21swapper/119:12:361
2589426616,35sleep125926-21cut22:07:371
2960226519,32sleep329726-21sed23:58:093
273882655,48sleep30-21swapper/322:12:233
153032659,45sleep30-21swapper/323:15:123
1373526511,40sleep313744-21/usr/sbin/munin21:27:523
686726419,21sleep00-21swapper/022:48:030
640126416,34sleep36512-21latency_hist19:22:363
3052926413,35sleep030525-21apt-get00:02:470
2972826422,19sleep00-21swapper/023:58:100
285326412,29sleep30-21swapper/300:17:523
2608026419,21sleep30-21swapper/320:23:063
1995626414,25sleep10-21swapper/120:03:051
1730026418,24sleep00-21swapper/023:22:440
835926313,29sleep30-21swapper/322:52:583
623826318,28sleep00-21swapper/021:02:580
3229826311,35sleep332330-21/usr/sbin/munin20:43:053
3229826311,35sleep332330-21/usr/sbin/munin20:43:053
2115326319,27sleep021154-21cron21:52:330
93426217,19sleep10-21swapper/120:47:521
715926216,32sleep37237-21grep19:23:043
3121626217,24sleep00-21swapper/022:22:580
2489726217,26sleep30-21swapper/323:43:073
157492624,16sleep00-21swapper/023:17:430
2113726116,26sleep121171-21irqstats20:07:581
202892618,14sleep30-21swapper/321:48:003
1965826121,22sleep10-21swapper/121:47:371
134326120,18sleep30-21swapper/320:48:023
1309826112,19sleep00-21swapper/023:08:020
126152617,11sleep10-21swapper/121:23:061
792226015,28sleep17948-21munin-run19:27:351
129626014,28sleep01425-21gpgv22:32:360
1240426015,25sleep30-21swapper/323:07:363
909625918,19sleep10-21swapper/121:12:561
3024325913,31sleep030244-21/usr/sbin/munin20:37:450
3024325913,31sleep030244-21/usr/sbin/munin20:37:450
1842825921,23sleep30-21swapper/321:42:553
1657525918,24sleep10-21swapper/121:37:361
1079225912,32sleep010533-21/usr/sbin/munin21:17:560
1002625915,26sleep310039-21irqstats19:32:583
566725815,19sleep30-21swapper/321:02:363
3214225819,18sleep10-21swapper/122:27:361
2791525814,26sleep028044-21/usr/sbin/munin20:28:080
2791525814,26sleep028044-21/usr/sbin/munin20:28:080
2779025818,25sleep027791-21memory23:52:570
228812589,14sleep00-21swapper/023:38:390
226932585,44sleep00-21swapper/021:56:570
1260725814,26sleep31-21systemd19:42:353
1260725814,26sleep31-21systemd19:42:353
1117225819,24sleep111283-21timerandwakeup21:18:051
3118725711,29sleep331213-21netstat22:22:573
3006425719,18sleep30-21swapper/320:37:363
3006425719,18sleep30-21swapper/320:37:363
2893625714,25sleep128993-21irqstats20:32:581
2893625714,25sleep128993-21irqstats20:32:581
2147225715,24sleep10-21swapper/121:52:501
195292577,13sleep30-21swapper/320:02:553
187152575,40sleep00-21swapper/019:58:110
1080125716,26sleep110839-21modprobe23:02:331
741925618,22sleep17414-21apt-get21:07:471
741925618,22sleep17414-21apt-get21:07:471
68992565,11sleep10-21swapper/100:29:331
606825621,19sleep10-21swapper/121:02:551
394025611,19sleep30-21swapper/322:38:163
1140125615,24sleep011405-21diskstats19:37:550
1140125615,24sleep011405-21diskstats19:37:550
869025520,20sleep30-21swapper/321:12:343
869025520,20sleep30-21swapper/321:12:343
822925511,17sleep00-21swapper/022:52:550
2991625516,20sleep00-21swapper/022:18:060
1278525520,17sleep30-21swapper/321:23:083
1264025514,24sleep312636-21apt-get23:07:483
36602545,36sleep10-21swapper/119:12:561
36602545,36sleep10-21swapper/119:12:561
1882525415,22sleep00-21swapper/021:43:040
1807425410,27sleep318166-21memory19:57:583
1474425415,23sleep114804-21fschecks_time19:47:541
742425315,18sleep00-21swapper/021:07:500
742425315,18sleep00-21swapper/021:07:500
3027025313,20sleep10-21swapper/120:37:501
3027025313,20sleep10-21swapper/120:37:501
2919225321,19sleep00-21swapper/020:33:050
2919225321,19sleep00-21swapper/020:33:050
2616425319,18sleep00-21swapper/020:23:070
1976525311,17sleep00-21swapper/020:03:000
1739525316,19sleep10-21swapper/121:38:041
215125215,22sleep32166-21needreboot22:33:363
215125215,22sleep32166-21needreboot22:33:363
1779425213,24sleep117789-21apt-get19:57:481
531025112,23sleep15361-21irqstats19:17:571
329925113,22sleep03293-21apt-get22:37:490
3209125113,22sleep332159-21df00:07:503
3058125115,23sleep10-21swapper/122:22:351
303262517,32sleep30-21swapper/322:19:543
2763025117,17sleep10-21swapper/123:52:541
2621625113,23sleep026217-21diskstats22:07:520
1602225111,21sleep116048-21irqstats23:17:561
351325012,21sleep03531-21apt-get19:12:470
351325012,21sleep03531-21apt-get19:12:470
1556925012,17sleep10-21swapper/123:17:361
826424912,22sleep18271-21diskstats19:27:511
2741724910,15sleep30-21swapper/320:27:593
2741724910,15sleep30-21swapper/320:27:593
2136224914,21sleep321383-21awk20:08:043
2273424811,21sleep322758-21irqstats20:12:563
1225824811,17sleep00-21swapper/021:22:570
2577424715,18sleep10-21swapper/120:22:561
305824610,20sleep03077-21/usr/sbin/munin19:12:350
1623524410,17sleep00-21swapper/019:52:500
149824412,19sleep11592-21memory00:12:571
1309424410,18sleep00-21swapper/019:42:440
277212439,18sleep127791-21df22:12:501
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional