You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-10-20 - 15:44

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Oct 20, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
98412530312,5278sleep22143-21kworker/2:008:17:582
168992161613,1586sleep315988-21kworker/3:112:06:043
6630213296,1312sleep13658-21kworker/1:108:06:591
20512238527,335sleep320550-21fschecks_count07:10:563
1703623039,280sleep11-21systemd10:25:331
30308225766,172sleep130311-21cron09:25:331
2758422558,235sleep127629-21ntpq07:31:131
23826225245,197sleep1362-21snmpd09:02:311
11638223917,205sleep011671-21latency_hist08:25:350
256692230100,32sleep125710-21latency_hist09:10:351
28759222939,175sleep1225-21systemd-journal09:20:321
8373222614,171sleep18422-21wc11:40:471
8373222614,171sleep18422-21wc11:40:471
4184222336,174sleep14237-21fschecks_count08:00:531
22380222151,26sleep10-21swapper/107:15:591
25076221316,175sleep125270-21apt-key07:25:361
5231221039,155sleep15314-21fschecks_time11:30:541
5231221039,155sleep15314-21fschecks_time11:30:541
28900220810,100sleep10-21swapper/111:01:381
10102220452,89sleep110253-21latency_hist08:20:361
17929219011,164sleep218007-21find10:26:062
30687218910,161sleep130711-21netstat07:40:591
6113217614,142sleep10-21swapper/109:50:321
1060121748,107sleep10-21swapper/108:20:561
15346216711,143sleep315421-21find12:01:033
23944216342,108sleep123985-21ntpq10:46:001
32235216251,23sleep10-21swapper/107:45:591
233621579,110sleep12490-21sed07:55:361
26449215416,28sleep10-21swapper/109:11:031
26449215416,28sleep10-21swapper/109:11:031
799021536,65sleep10-21swapper/109:55:491
3180215245,26sleep10-21swapper/107:56:051
20347215219,66sleep220486-21grep08:51:052
15004215213,92sleep10-21swapper/108:35:461
2472321507,105sleep10-21swapper/107:24:231
31837214711,96sleep131919-21mailstats11:11:031
31837214711,96sleep131919-21mailstats11:11:031
2123214735,89sleep22164-21fschecks_count11:20:532
2123214735,89sleep22164-21fschecks_count11:20:532
8554214526,25sleep10-21swapper/111:41:271
28525214529,103sleep328555-21sed07:35:353
21005214513,117sleep020944-21latency_hist08:55:360
3430214316,23sleep10-21swapper/109:40:551
3430214316,23sleep10-21swapper/109:40:551
20913214349,83sleep1362-21snmpd08:55:241
6036214215,26sleep20-21swapper/211:31:072
6036214215,26sleep20-21swapper/211:31:072
12098214010,44sleep112170-21mailstats11:51:041
25441213914,109sleep325584-21cut12:35:353
31249213844,27sleep10-21swapper/109:26:011
31249213844,27sleep10-21swapper/109:26:011
19037213655,18sleep10-21swapper/107:06:051
28854213442,33sleep128904-21fschecks_count07:35:541
1417221268,103sleep114320-21awk08:31:061
12399212515,97sleep012453-21sort10:10:340
292521235,13sleep10-21swapper/109:39:211
27953212213,87sleep028104-21unixbench_singl07:31:200
2306212252,55sleep12348-21latency11:20:561
2306212252,55sleep12348-21latency11:20:561
20822212212,65sleep10-21swapper/107:11:021
1474321216,15sleep10-21swapper/112:00:451
1648212018,80sleep31811-21unixbench_multi07:51:063
14789212089,15sleep10-21swapper/110:16:041
2763121180,8sleep20-21swapper/207:31:132
11555211882,18sleep10-21swapper/110:06:021
2501621175,11sleep10-21swapper/112:31:171
1760621159,60sleep10-21swapper/112:10:341
1372221158,17sleep10-21swapper/111:56:051
21686211326,76sleep1362-21snmpd10:40:021
2161121139,92sleep121636-21awk08:55:571
2669421127,16sleep10-21swapper/110:55:491
19121211111,84sleep219242-21ls07:06:062
7203211024,45sleep10-21swapper/109:51:051
1757621109,66sleep117598-21irqstats10:25:571
18219210920,67sleep317821-21/usr/sbin/munin12:11:003
1085021078,85sleep10-21swapper/110:05:341
766210611,16sleep20-21swapper/209:31:272
2107521067,48sleep10-21swapper/112:20:511
2107521067,48sleep10-21swapper/112:20:511
10544210648,43sleep10-21swapper/111:46:041
3019521057,86sleep130231-21ntpq11:05:591
3019521057,86sleep130231-21ntpq11:05:591
27591210511,80sleep327604-21vmstat10:56:073
64921047,64sleep1765-21sed09:31:261
1922621027,84sleep1362-21snmpd08:48:381
17844210249,41sleep117839-21apt-get12:10:461
12317210249,12sleep10-21swapper/108:26:011
9856210117,62sleep39900-21latency_hist11:45:373
12911210120,25sleep30-21swapper/310:10:573
1998521008,37sleep10-21swapper/112:16:031
1998521008,37sleep10-21swapper/112:16:031
202242999,78sleep120226-21munin-run10:35:341
1892529915,69sleep318949-21proc_pri07:06:033
2314829818,60sleep223231-21wc12:26:052
2314829818,60sleep223231-21wc12:26:052
468529712,20sleep00-21swapper/008:01:040
292132967,11sleep30-21swapper/311:01:443
2415329618,62sleep324306-21latency_hist09:05:363
2258829512,44sleep10-21swapper/110:41:031
180522950,11sleep00-21swapper/012:10:560
1513029515,62sleep015142-21vmstat10:16:070
1684929424,52sleep216857-21/usr/sbin/munin07:00:462
1158829410,25sleep20-21swapper/210:06:032
3235629223,51sleep331875-21apt-get09:30:433
126402926,46sleep10-21swapper/110:10:471
1431829118,55sleep214405-21fschecks_time10:15:532
2520228912,64sleep325221-21grep10:50:543
2520228912,64sleep325221-21grep10:50:543
1978528922,41sleep019982-21tune2fs07:10:370
251892886,46sleep10-21swapper/110:50:521
251892886,46sleep10-21swapper/110:50:521
2307528831,14sleep10-21swapper/112:26:041
2307528831,14sleep10-21swapper/112:26:041
1819628823,32sleep30-21swapper/308:45:543
925128722,41sleep0362-21snmpd10:00:320
3269928728,43sleep032745-21latency_hist11:15:340
3269928728,43sleep032745-21latency_hist11:15:340
302772879,66sleep00-21swapper/009:25:070
1316528718,49sleep313166-21/usr/sbin/munin11:55:463
1255828712,59sleep212682-21sed08:26:062
351728518,50sleep23283-21/usr/sbin/munin09:40:562
351728518,50sleep23283-21/usr/sbin/munin09:40:562
2639028512,57sleep026448-21/usr/sbin/munin09:11:020
2639028512,57sleep026448-21/usr/sbin/munin09:11:020
858728414,53sleep38674-21sort09:56:033
2695928422,15sleep00-21swapper/010:55:560
172428423,41sleep21218-21apt-get09:35:432
16252845,44sleep10-21swapper/111:18:591
16252845,44sleep10-21swapper/111:18:591
150012848,17sleep30-21swapper/308:35:443
181162838,15sleep00-21swapper/008:45:500
3250128214,53sleep032623-21timerwakeupswit07:46:060
2680328224,30sleep00-21swapper/007:30:360
2718028113,49sleep227249-21modprobe09:15:322
2718028113,49sleep227249-21modprobe09:15:322
1355928116,28sleep00-21swapper/011:56:000
181422809,15sleep00-21swapper/010:26:080
768527917,40sleep22143-21kworker/2:008:11:042
1592327918,40sleep215752-21/usr/sbin/munin10:20:532
84312785,63sleep10-21swapper/108:15:071
639727826,32sleep36421-21awk11:35:323
639727826,32sleep36421-21awk11:35:323
2570127815,29sleep20-21swapper/210:51:062
2570127815,29sleep20-21swapper/210:51:062
2510927821,34sleep225257-21unixbench_singl09:06:072
243727815,38sleep00-21swapper/011:20:590
243727815,38sleep00-21swapper/011:20:590
2015927812,30sleep30-21swapper/312:16:053
2015927812,30sleep30-21swapper/312:16:053
1664327813,49sleep316728-21fschecks_time08:40:533
158902787,13sleep20-21swapper/208:36:032
616427724,29sleep30-21swapper/309:50:343
2665927725,31sleep226713-21apt-config07:30:342
2320027719,29sleep20-21swapper/209:00:592
110427717,45sleep01175-21fw_packets07:50:550
741227616,41sleep07540-21sed11:36:050
741227616,41sleep07540-21sed11:36:050
446527528,32sleep24496-21/usr/sbin/munin08:00:582
363627522,15sleep10-21swapper/111:25:491
363627522,15sleep10-21swapper/111:25:491
3064427513,46sleep330669-21/usr/sbin/munin09:25:473
3064427513,46sleep330669-21/usr/sbin/munin09:25:473
3071327415,28sleep20-21swapper/207:41:002
1937427422,34sleep219398-21latency_hist08:50:332
1032727415,36sleep310324-21apt-get08:20:463
2491527320,22sleep00-21swapper/009:06:050
2485127319,23sleep30-21swapper/309:06:043
2392927321,29sleep00-21swapper/010:45:590
2189727324,31sleep00-21swapper/008:56:040
2881227221,25sleep00-21swapper/009:20:350
2671527216,43sleep326801-21apt-config07:30:353
2054927217,40sleep320585-21fschecks_count10:35:523
175742725,12sleep10-21swapper/108:42:241
2903727127,26sleep00-21swapper/007:35:570
1440627114,40sleep314443-21basename10:15:543
867127013,26sleep00-21swapper/011:41:510
627227015,41sleep36358-21ls08:06:163
398427018,36sleep04102-21timerwakeupswit09:41:060
398427018,36sleep04102-21timerwakeupswit09:41:060
2445627020,27sleep00-21swapper/012:30:560
2250027013,35sleep0369-21sshd10:41:020
2022727017,35sleep220312-21cat10:35:352
1897127025,24sleep00-21swapper/010:30:530
1729027014,39sleep017327-21ntp_states07:01:000
1008227016,15sleep00-21swapper/008:20:350
3132526921,31sleep331327-21/usr/sbin/munin11:10:453
3132526921,31sleep331327-21/usr/sbin/munin11:10:453
3037426914,15sleep20-21swapper/211:06:042
3037426914,15sleep20-21swapper/211:06:042
2834426922,34sleep028435-21fschecks_time11:00:530
1397626923,28sleep314042-21/usr/sbin/munin08:31:043
1167126926,22sleep30-21swapper/310:06:043
2963026813,14sleep20-21swapper/209:21:062
411926721,23sleep20-21swapper/211:26:022
411926721,23sleep20-21swapper/211:26:022
385726721,22sleep30-21swapper/308:00:353
301526713,38sleep33049-21awk09:40:333
2238126715,31sleep321991-21/usr/sbin/munin10:40:573
2145126719,21sleep20-21swapper/212:21:012
2145126719,21sleep20-21swapper/212:21:012
20392677,16sleep30-21swapper/311:20:503
20392677,16sleep30-21swapper/311:20:503
2028726720,21sleep20-21swapper/207:10:432
1604526719,26sleep00-21swapper/010:20:550
929226616,37sleep39368-21systemctl08:16:063
3192026625,25sleep20-21swapper/211:11:042
3192026625,25sleep20-21swapper/211:11:042
271512664,10sleep30-21swapper/309:15:093
271512664,10sleep30-21swapper/309:15:093
2430426614,29sleep00-21swapper/007:21:060
2383826618,21sleep30-21swapper/310:45:583
1820026614,37sleep318280-21sed07:05:353
2348326519,23sleep00-21swapper/009:01:050
169272658,12sleep00-21swapper/008:40:580
575626416,26sleep00-21swapper/011:31:050
575626416,26sleep00-21swapper/011:31:050
48726415,32sleep3554-21/usr/sbin/munin11:15:503
48726415,32sleep3554-21/usr/sbin/munin11:15:503
3007126410,36sleep330165-21memory11:05:573
3007126410,36sleep330165-21memory11:05:573
2228626418,20sleep20-21swapper/210:40:562
1325226425,24sleep00-21swapper/010:11:060
385526317,25sleep30-21swapper/311:25:553
385526317,25sleep30-21swapper/311:25:553
2706026314,22sleep20-21swapper/210:55:582
2251826322,27sleep222608-21grep07:16:042
2017926314,20sleep00-21swapper/007:10:410
671626218,27sleep26710-21apt-get11:35:472
671626218,27sleep26710-21apt-get11:35:472
322512629,12sleep20-21swapper/207:46:002
2851126210,40sleep3362-21snmpd09:17:193
2481726216,27sleep224844-21munin-run07:25:332
183122626,13sleep20-21swapper/212:11:032
1294926220,24sleep313074-21latency_hist11:55:343
109322620,8sleep20-21swapper/210:05:342
105182625,43sleep00-21swapper/010:01:540
3097126113,29sleep031089-21/usr/sbin/munin07:41:060
204626117,20sleep00-21swapper/009:36:100
1468826121,20sleep00-21swapper/008:35:330
1361826114,19sleep20-21swapper/208:30:542
120026116,31sleep01218-21apt-get09:35:330
686126022,25sleep0369-21sshd09:50:590
2971826028,15sleep30-21swapper/309:21:073
1725826010,10sleep20-21swapper/212:06:482
1626526013,26sleep00-21swapper/008:40:330
1542526013,30sleep015554-21/usr/sbin/munin12:01:040
1498326011,32sleep215015-21irqstats12:00:542
1319026015,17sleep30-21swapper/308:30:353
449725920,22sleep30-21swapper/308:00:593
3056925913,19sleep20-21swapper/209:25:422
3056925913,19sleep20-21swapper/209:25:422
290725913,28sleep20-21swapper/207:55:582
2485525920,25sleep324895-21munin-run07:25:343
2097125921,22sleep221055-21mailstats10:36:032
1989325911,33sleep219899-21ntp_states12:16:002
1989325911,33sleep219899-21ntp_states12:16:002
1390925914,21sleep20-21swapper/210:15:322
1345925918,20sleep20-21swapper/211:55:582
104662598,10sleep30-21swapper/310:01:163
2201425815,21sleep00-21swapper/007:15:490
2201225818,24sleep322008-21apt-get07:15:483
209002587,15sleep30-21swapper/308:55:143
640025714,26sleep36395-21apt-get09:50:463
3007525715,20sleep20-21swapper/207:40:352
2254325718,19sleep30-21swapper/312:25:503
2254325718,19sleep30-21swapper/312:25:503
2181925714,26sleep30-21swapper/308:56:033
1948025711,31sleep319563-21find10:31:043
1042625719,20sleep20-21swapper/208:20:532
719525614,26sleep37191-21apt-get08:10:493
551825611,21sleep00-21swapper/009:46:070
2660625614,25sleep026647-21cat10:55:360
2114425615,19sleep00-21swapper/010:36:060
2074225618,19sleep00-21swapper/012:20:340
2074225618,19sleep00-21swapper/012:20:340
1161325614,25sleep211609-21apt-get11:50:492
773325515,21sleep27743-21munin-run09:55:342
375525516,18sleep30-21swapper/309:41:013
375525516,18sleep30-21swapper/309:41:013
3211125514,23sleep332140-21irqstats07:45:573
3089225517,17sleep30-21swapper/307:41:053
2140725513,23sleep021443-21ntp_states12:20:590
2140725513,23sleep021443-21ntp_states12:20:590
1682625514,18sleep20-21swapper/208:40:562
96325414,24sleep2956-21apt-get07:50:492
76542549,19sleep00-21swapper/008:11:020
732225420,18sleep30-21swapper/311:36:043
732225420,18sleep30-21swapper/311:36:043
506625415,20sleep20-21swapper/209:45:572
2768725414,21sleep20-21swapper/209:15:572
1750025415,18sleep10-21swapper/107:01:051
1675525415,20sleep00-21swapper/012:06:010
1069825417,20sleep30-21swapper/311:46:063
3084225311,17sleep00-21swapper/009:25:520
3084225311,17sleep00-21swapper/009:25:520
222042538,32sleep30-21swapper/312:24:573
222042538,32sleep30-21swapper/312:24:573
569525218,19sleep20-21swapper/208:05:512
356225213,19sleep00-21swapper/011:25:440
356225213,19sleep00-21swapper/011:25:440
2916625220,17sleep30-21swapper/307:36:003
2875625222,16sleep20-21swapper/211:01:352
254025213,22sleep02535-21apt-get07:55:470
2380825213,22sleep223837-21irqstats10:45:572
1208825216,23sleep00-21swapper/008:25:560
105202517,30sleep00-21swapper/011:46:030
896425014,22sleep29008-21ntpq11:41:572
48425014,21sleep0480-21apt-get11:15:470
48425014,21sleep0480-21apt-get11:15:470
2522425011,19sleep025323-21perl10:50:550
2451825015,22sleep224541-21ntp_states12:30:582
2254025014,21sleep022536-21apt-get12:25:470
2254025014,21sleep022536-21apt-get12:25:470
1005825011,23sleep210052-21apt-get11:45:482
573024915,17sleep00-21swapper/008:05:530
1743324915,14sleep30-21swapper/310:25:553
298022488,14sleep00-21swapper/011:05:490
298022488,14sleep00-21swapper/011:05:490
1198024812,16sleep00-21swapper/011:50:590
2255524412,14sleep20-21swapper/209:00:342
1577424410,18sleep30-21swapper/310:20:463
4602398,14sleep20-21swapper/211:15:452
4602398,14sleep20-21swapper/211:15:452
80542376,12sleep20-21swapper/209:55:502
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional