You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-02-13 - 07:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Wed Feb 12, 2025 12:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
37862667910,6656sleep032488-21kworker/0:108:02:000
30763224887,2465sleep04079-1kworker/0:2H09:28:040
124292155723,1510sleep312395-21/usr/sbin/munin12:22:553
7634269320,655sleep07636-21sshd10:03:000
13492262910,603sleep313561-21/usr/sbin/munin12:23:203
11414248025,437sleep211541-21/usr/sbin/munin10:13:192
536024690,11sleep20-21swapper/209:52:522
212123880,12sleep10-21swapper/107:53:081
7351224069,117sleep17381-21munin-run10:02:481
1748223842,50sleep10-21swapper/109:42:181
17747222837,45sleep10-21swapper/108:43:241
12513222415,193sleep312532-21grep08:28:323
20733220513,146sleep10-21swapper/107:07:561
2578821950,62sleep112-21rcu_sched09:08:461
2578821950,62sleep112-21rcu_sched09:08:461
31833219342,107sleep10-21swapper/107:43:241
15522218659,30sleep10-21swapper/108:38:071
25310217743,88sleep125392-21mailstats07:23:141
28683217446,113sleep128719-21idleruntime-cro09:22:481
28683217446,113sleep128719-21idleruntime-cro09:22:481
25508217211,143sleep10-21swapper/111:08:131
6005216811,123sleep10-21swapper/108:08:071
23393216842,54sleep10-21swapper/109:03:021
28678216460,89sleep128784-21latency_hist11:22:501
29510216334,114sleep129666-21latency_hist07:37:511
12549216132,90sleep112633-21grep08:28:341
13107215915,74sleep10-21swapper/112:23:141
8030215623,79sleep18160-21unixbench_multi12:03:311
8030215623,79sleep18160-21unixbench_multi12:03:311
4199215617,120sleep30-21swapper/309:48:343
542821529,131sleep0370-21snmpd08:05:070
30866215243,94sleep130911-21grep11:28:151
30866215243,94sleep130911-21grep11:28:151
15843214514,117sleep315884-21latency_hist10:32:503
12488214238,90sleep112564-21latency_hist12:22:571
1485214016,103sleep031727-21kworker/u8:211:42:470
1485214016,103sleep031727-21kworker/u8:211:42:470
26859213818,105sleep326939-21sendmail_mailtr07:28:203
3720213417,24sleep10-21swapper/111:48:131
26636213474,15sleep10-21swapper/107:28:141
1329721339,111sleep113341-21memory10:23:231
28147212917,96sleep227636-21apt-get07:33:002
32707212716,89sleep1194-21jbd2/mmcblk2p2-09:33:141
32707212716,89sleep1194-21jbd2/mmcblk2p2-09:33:141
18480212710,99sleep10-21swapper/110:43:061
27323212617,84sleep10-21swapper/109:13:201
27323212617,84sleep10-21swapper/109:13:201
19978212453,30sleep120001-21modprobe08:52:481
19696212437,67sleep119807-21timerandwakeup07:03:201
19696212437,67sleep119807-21timerandwakeup07:03:201
10734212234,74sleep110755-21kernelversion12:13:121
16911212126,21sleep10-21swapper/110:33:211
2277212040,62sleep12370-21mailstats11:43:131
25174211519,80sleep324688-21apt-get09:08:013
25174211519,80sleep324688-21apt-get09:08:013
21981211517,76sleep122020-21fschecks_time08:58:141
21981211517,76sleep122020-21fschecks_time08:58:141
17904211522,79sleep117946-21ntp_kernel_pll_06:58:151
18030211212,85sleep218003-21latency_hist10:42:502
21549211136,65sleep1370-21snmpd08:57:371
21549211136,65sleep1370-21snmpd08:57:371
17983211129,70sleep11-21systemd10:42:491
12370211030,59sleep012395-21munin-run12:22:480
809721087,54sleep18137-21ps10:03:131
450210811,30sleep3453-21/usr/sbin/munin07:48:073
30735210810,81sleep230748-21/usr/sbin/munin11:28:122
30735210810,81sleep230748-21/usr/sbin/munin11:28:122
2475821086,91sleep00-21swapper/011:06:340
2030210820,66sleep02118-21tune2fs07:53:080
14330210826,57sleep014361-21sort12:28:090
5421210723,43sleep00-21swapper/009:53:030
26829210712,40sleep10-21swapper/111:13:131
15279210741,14sleep10-21swapper/108:37:511
458021069,80sleep04650-21systemd-run11:52:490
32117210614,76sleep032126-21diskstats11:33:070
32117210614,76sleep032126-21diskstats11:33:070
21407210612,64sleep30-21swapper/310:53:053
6296210510,79sleep16312-21munin-run11:57:501
5514210520,62sleep35641-21/usr/sbin/munin11:53:203
21429210517,66sleep321011-21/usr/sbin/munin07:08:143
31465210469,14sleep10-21swapper/109:28:251
609210210,77sleep2638-21irqstats07:48:102
6386210127,48sleep26178-21apt-get11:57:512
28137210113,74sleep027636-21apt-get07:32:540
5154210017,58sleep10-21swapper/109:52:521
875929922,62sleep28804-21latency_hist12:07:522
875929922,62sleep28804-21latency_hist12:07:522
467629838,40sleep10-21swapper/108:03:351
727429719,64sleep07345-21latency_hist08:12:510
155042978,21sleep20-21swapper/208:38:042
375829611,17sleep20-21swapper/211:48:142
2272129611,67sleep122845-21latency_hist10:57:511
973729516,61sleep09863-21/usr/sbin/munin08:18:200
2641029525,51sleep026421-21modprobe11:12:480
207229431,22sleep118847-21cyclictest09:42:591
207229431,22sleep118847-21cyclictest09:42:591
1548929416,36sleep10-21swapper/110:29:171
1186729436,40sleep10-21swapper/108:27:491
1186729436,40sleep10-21swapper/108:27:491
946029330,42sleep09524-21cat10:08:120
759929310,35sleep10-21swapper/108:13:081
3072929320,52sleep330246-21apt-get09:27:593
2999829311,16sleep20-21swapper/207:38:132
2449329331,22sleep10-21swapper/111:03:201
2040229332,17sleep10-21swapper/110:48:181
3204329230,17sleep10-21swapper/111:33:041
3204329230,17sleep10-21swapper/111:33:041
233302915,74sleep07200-21kworker/u8:109:02:530
233302915,74sleep07200-21kworker/u8:109:02:530
155232910,8sleep30-21swapper/312:32:473
97272907,30sleep10-21swapper/110:08:221
1677629017,58sleep316819-21true08:42:493
1418029032,40sleep10-21swapper/112:27:581
2809128913,64sleep328129-21latency_hist09:17:503
2809128913,64sleep328129-21latency_hist09:17:503
2326728933,16sleep10-21swapper/107:13:211
1909128912,62sleep219110-21cpu07:03:042
1909128912,62sleep219110-21cpu07:03:042
69452886,19sleep10-21swapper/112:01:341
69452886,19sleep10-21swapper/112:01:341
3012428712,28sleep00-21swapper/007:38:150
2646928718,44sleep226503-21munin-run09:12:482
2646928718,44sleep226503-21munin-run09:12:482
1893728717,55sleep219048-21timerwakeupswit10:43:192
117672877,33sleep10-21swapper/110:14:271
754328617,18sleep30-21swapper/312:03:213
754328617,18sleep30-21swapper/312:03:213
95622859,63sleep29642-21mailstats08:18:182
3241428522,41sleep232310-21/usr/sbin/munin09:33:072
3241428522,41sleep232310-21/usr/sbin/munin09:33:072
302352839,57sleep130297-21systemd-run09:27:491
2256428313,33sleep20-21swapper/208:58:312
2256428313,33sleep20-21swapper/208:58:312
1123328316,45sleep310745-21/usr/sbin/munin10:13:163
2813328120,32sleep00-21swapper/009:17:510
1222628118,48sleep012329-21cat08:27:520
436728012,51sleep23864-21apt-get08:02:592
391028013,21sleep10-21swapper/108:02:501
299842807,30sleep10-21swapper/107:38:121
3136627915,46sleep331461-21perl07:43:143
216027911,51sleep22230-21fschecks_time09:43:042
216027911,51sleep22230-21fschecks_time09:43:042
968227811,15sleep30-21swapper/312:08:343
968227811,15sleep30-21swapper/312:08:343
2636327818,40sleep026358-21apt-get07:28:050
1739427826,29sleep30-21swapper/310:37:513
3133127716,46sleep031356-21/usr/sbin/munin07:43:110
2943627721,39sleep3227-21systemd-journal07:37:493
279222779,19sleep00-21swapper/011:17:000
1274227713,45sleep10-21swapper/110:22:511
3023127620,38sleep030234-21cron09:27:480
99192756,29sleep10-21swapper/112:09:491
99192756,29sleep10-21swapper/112:09:491
3046827516,41sleep030536-21df11:28:060
3046827516,41sleep030536-21df11:28:060
2682627524,31sleep20-21swapper/207:28:192
2096427520,26sleep20-21swapper/208:53:262
2096427520,26sleep20-21swapper/208:53:262
153627518,38sleep21626-21latency_hist11:42:502
153627518,38sleep21626-21latency_hist11:42:502
69862747,55sleep00-21swapper/012:02:050
69862747,55sleep00-21swapper/012:02:050
3040027425,29sleep330604-21apt-config09:27:513
233082748,29sleep123311-21cron11:02:481
1874527415,39sleep318760-21fschecks_count08:48:063
1426827413,35sleep00-21swapper/010:27:500
549627322,37sleep10-21swapper/109:53:081
39727313,44sleep10-21swapper/107:48:051
1334227322,37sleep213373-21nfs_client10:23:242
3209727220,28sleep20-21swapper/209:32:502
237502728,15sleep00-21swapper/009:03:170
2137727212,33sleep20-21swapper/210:53:002
2098927219,27sleep30-21swapper/308:53:273
2098927219,27sleep30-21swapper/308:53:273
1459427219,32sleep214688-21fschecks_time10:28:092
1334127218,30sleep213490-21unixbench_multi12:23:192
986427011,39sleep10-21swapper/108:18:211
2584927011,46sleep225850-21sshd07:25:002
1454627023,22sleep20-21swapper/208:33:202
1407027015,29sleep00-21swapper/008:33:090
126352708,12sleep20-21swapper/208:28:352
2108526919,23sleep30-21swapper/310:52:483
1987926920,26sleep00-21swapper/010:48:050
720226816,35sleep37272-21unin-run08:12:503
3011126820,36sleep10-21swapper/111:27:021
2953526821,24sleep229642-21cat11:23:282
245062685,32sleep10-21swapper/107:22:371
2399126816,27sleep30-21swapper/309:03:373
234122688,17sleep20-21swapper/209:03:032
2763826722,24sleep20-21swapper/211:13:272
1842426725,24sleep00-21swapper/008:47:510
142726719,37sleep10-21swapper/111:42:041
142726719,37sleep10-21swapper/111:42:041
1172626714,39sleep011738-21sh12:17:470
162932669,12sleep20-21swapper/210:33:092
1588726617,25sleep00-21swapper/010:32:510
1546526618,32sleep215522-21idleruntime-cro12:32:472
1522826614,37sleep315263-21awk08:37:493
141082668,44sleep04587-21kworker/u8:312:27:530
955626515,28sleep20-21swapper/212:08:332
955626515,28sleep20-21swapper/212:08:332
666126518,22sleep20-21swapper/211:57:592
666126518,22sleep20-21swapper/211:57:592
3030026517,27sleep20-21swapper/209:27:502
2972726513,35sleep329790-21cpuspeed07:38:063
2386926514,34sleep323958-21/usr/sbin/munin11:03:103
1876226520,27sleep318792-21munin-run07:02:493
434226418,25sleep30-21swapper/308:02:563
277602648,12sleep20-21swapper/209:14:092
277602648,12sleep20-21swapper/209:14:092
2563726414,32sleep025175-21/usr/sbin/munin09:08:420
2563726414,32sleep025175-21/usr/sbin/munin09:08:420
2310826412,38sleep323177-21readlink07:13:193
2208326414,25sleep00-21swapper/007:10:010
1676526412,31sleep016907-21grep10:33:200
1116326418,29sleep011201-21/usr/sbin/munin10:13:130
400926317,28sleep24215-21cat08:02:512
2607126328,22sleep0286-21dbus-daemon07:27:490
2335926315,32sleep323424-21latency_hist11:02:503
1903726319,18sleep00-21swapper/008:48:110
772126223,23sleep30-21swapper/308:13:103
5942629,16sleep20-21swapper/211:34:342
5942629,16sleep20-21swapper/211:34:342
590026218,20sleep20-21swapper/208:08:002
3179726219,28sleep331849-21latency_hist11:32:493
3179726219,28sleep331849-21latency_hist11:32:493
237326212,38sleep22451-21xargs11:43:142
1875626214,21sleep00-21swapper/010:43:170
1682126219,22sleep00-21swapper/008:42:500
1129326219,20sleep30-21swapper/308:23:233
1129326219,20sleep30-21swapper/308:23:233
326226117,17sleep30-21swapper/307:57:523
2548026118,30sleep325562-21timerwakeupswit07:23:163
222362618,16sleep20-21swapper/207:11:572
181102616,13sleep20-21swapper/206:58:202
300362606,12sleep30-21swapper/311:26:033
1256526024,18sleep20-21swapper/212:22:582
2383325922,19sleep20-21swapper/211:03:092
1945825915,20sleep30-21swapper/307:03:143
1945825915,20sleep30-21swapper/307:03:143
188825916,25sleep01975-21tune2fs11:43:040
1748425915,18sleep00-21swapper/006:58:020
9852586,15sleep00-21swapper/009:35:430
9852586,15sleep00-21swapper/009:35:430
89725816,18sleep30-21swapper/311:37:503
89725816,18sleep30-21swapper/311:37:503
808825823,21sleep08182-21switchtime08:13:190
2532425813,26sleep20-21swapper/211:08:102
185625816,20sleep30-21swapper/311:43:033
390925712,26sleep34000-21/usr/sbin/munin11:48:203
338825712,30sleep03465-21cut09:47:500
338825712,30sleep03465-21cut09:47:500
2182725716,25sleep321845-21apt08:57:593
2182725716,25sleep321845-21apt08:57:593
1893025719,22sleep20-21swapper/208:48:092
1753325722,19sleep20-21swapper/208:43:212
763125613,26sleep27627-21apt-get10:02:582
620825618,18sleep00-21swapper/011:57:490
705025515,23sleep00-21swapper/009:59:000
544425517,22sleep25440-21apt-get09:53:052
505225519,21sleep05153-21latency_hist09:52:500
2457225512,18sleep30-21swapper/307:22:513
2308525510,30sleep223107-21pluginstate07:13:182
1950525512,29sleep019542-21ntp_states07:03:150
1950525512,29sleep019542-21ntp_states07:03:150
194225514,25sleep21936-21apt-get07:53:022
1145325514,21sleep00-21swapper/008:23:240
1145325514,21sleep00-21swapper/008:23:240
1061325512,25sleep210607-21apt-get08:23:062
1061325512,25sleep210607-21apt-get08:23:062
916925414,18sleep00-21swapper/012:08:230
916925414,18sleep00-21swapper/012:08:230
2331225416,24sleep21-21systemd11:02:492
2259525411,28sleep022600-21sort07:12:590
1053425417,19sleep20-21swapper/210:12:512
906025312,25sleep39054-21apt-get08:18:023
877125315,22sleep2286-21dbus-daemon08:17:492
557525315,21sleep30-21swapper/309:53:183
36125312,28sleep0396-21cpu07:48:040
1127725313,16sleep20-21swapper/212:13:342
206525213,23sleep32067-21apt-get09:42:563
206525213,23sleep32067-21apt-get09:42:563
1337425218,17sleep30-21swapper/310:23:253
3253125117,19sleep20-21swapper/207:47:492
1271425116,18sleep00-21swapper/010:22:500
1059325112,22sleep310679-21perl12:13:103
864525014,16sleep00-21swapper/012:07:500
864525014,16sleep00-21swapper/012:07:500
2872125013,19sleep20-21swapper/209:22:492
2872125013,19sleep20-21swapper/209:22:492
2029825014,19sleep00-21swapper/008:53:040
2029825014,19sleep00-21swapper/008:53:040
1714125012,17sleep30-21swapper/308:43:093
790424916,15sleep30-21swapper/310:03:073
2910724910,17sleep00-21swapper/011:23:180
282722495,34sleep00-21swapper/011:18:130
176802495,35sleep00-21swapper/010:38:570
940924812,20sleep39458-21irqstats10:08:113
2538124814,22sleep325121-21/usr/sbin/munin11:08:113
2191024815,16sleep00-21swapper/010:53:180
1823824811,22sleep318246-21acpi10:42:553
178924812,19sleep30-21swapper/309:42:483
123812488,22sleep39703-1kworker/3:0H10:19:053
85024713,19sleep3851-21sshd09:34:003
85024713,19sleep3851-21sshd09:34:003
1982824612,17sleep20-21swapper/210:48:032
267362459,17sleep30-21swapper/311:13:073
65422436,13sleep30-21swapper/311:57:523
65422436,13sleep30-21swapper/311:57:523
2862624312,15sleep30-21swapper/311:22:483
276442439,16sleep00-21swapper/009:13:230
2873424111,13sleep30-21swapper/309:22:503
2873424111,13sleep30-21swapper/309:22:503
183124013,14sleep00-21swapper/009:42:490
1795723810,18sleep315553-1kworker/dying10:42:313
121632389,15sleep20-21swapper/210:17:512
73882367,15sleep20-21swapper/210:02:492
244952347,15sleep00-21swapper/007:22:280
73922337,13sleep30-21swapper/310:02:503
284092335,12sleep20-21swapper/211:20:012
244492337,16sleep20-21swapper/207:21:542
12682334,11sleep20-21swapper/209:37:512
229492328,12sleep00-21swapper/010:58:100
12652325,11sleep30-21swapper/311:40:003
12652325,11sleep30-21swapper/311:40:003
284122317,12sleep00-21swapper/009:19:200
284122317,12sleep00-21swapper/009:19:200
35002297,11sleep30-21swapper/307:58:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional