You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Fri May 02, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13990210208,1000sleep314026-21ntpq00:03:373
20205290310,882sleep022372-1kworker/0:0H00:23:410
9251269721,659sleep29286-21/usr/sbin/munin20:23:382
7771252013,493sleep17777-21ntpq20:18:401
5528233622,289sleep05613-21latency_hist20:13:140
13045229710,271sleep113089-21ntpq22:18:381
13045229710,271sleep113089-21ntpq22:18:381
18980227121,229sleep119004-21fschecks_count22:38:331
11778222832,181sleep3286-21dbus-daemon23:58:123
1769022166,199sleep3370-21snmpd00:14:193
18622221214,184sleep018720-21sed19:08:440
2210122089,31sleep30-21swapper/322:48:323
26586220628,97sleep326794-21sed23:03:143
11927219111,124sleep30-21swapper/320:33:123
2339218918,59sleep30-21swapper/321:43:443
6051218517,150sleep36117-21processes23:38:453
8381218220,145sleep18428-40ogrotate23:48:121
21404217915,57sleep30-21swapper/300:28:133
28619217714,143sleep328645-21wc23:08:273
28619217714,143sleep328645-21wc23:08:273
1457721710,15sleep314259-21cyclictest18:58:323
24505216611,143sleep329726-1kworker/3:1H21:13:153
12137216310,51sleep312225-21fschecks_time23:58:323
11533216311,55sleep30-21swapper/322:13:403
31744216121,16sleep30-21swapper/319:53:133
31744216121,16sleep30-21swapper/319:53:133
273821619,115sleep32844-21tune2fs20:03:353
3158021589,134sleep331684-21date23:18:133
12760215818,114sleep30-21swapper/322:18:323
12760215818,114sleep30-21swapper/322:18:323
16802215530,110sleep316829-21netstat19:03:383
16656214912,19sleep30-21swapper/320:48:143
24248214812,27sleep30-21swapper/322:53:453
14622214812,46sleep30-21swapper/322:23:373
2178321478,116sleep30-21swapper/319:18:443
30895214512,118sleep330936-21latency_hist21:33:143
26235214450,39sleep30-21swapper/321:18:323
26235214450,39sleep30-21swapper/321:18:323
460221437,124sleep34646-21ntpq20:08:403
20817214311,52sleep320840-21munin-run19:18:123
29125213911,109sleep329258-21latency_hist21:28:133
29057213936,14sleep30-21swapper/323:09:203
29057213936,14sleep30-21swapper/323:09:203
15851213911,114sleep315973-21timerwakeupswit00:08:463
2314521379,112sleep323118-21/usr/sbin/munin21:08:343
2265121377,15sleep30-21swapper/319:23:253
5769213338,55sleep30-21swapper/320:13:293
7382213228,16sleep30-21swapper/320:18:303
28248213220,95sleep128339-21latency_hist23:08:131
28248213220,95sleep128339-21latency_hist23:08:131
17470213217,95sleep30-21swapper/320:48:473
17470213217,95sleep30-21swapper/320:48:473
10136213110,106sleep310212-21sed22:08:433
30146212328,82sleep30-21swapper/319:48:133
2052621236,30sleep30-21swapper/322:43:323
19102212313,20sleep30-21swapper/322:38:353
32436212211,29sleep332541-21cat23:18:453
1057721207,15sleep00-21swapper/020:28:160
9109211923,14sleep30-21swapper/320:23:353
8917211910,18sleep30-21swapper/323:48:223
27735211922,16sleep30-21swapper/319:38:393
799211711,88sleep012-21rcu_sched19:58:110
4910211723,38sleep30-21swapper/321:53:293
4245211634,46sleep30-21swapper/323:33:363
3357211618,57sleep30-21swapper/321:48:283
32397211614,85sleep1286-21dbus-daemon21:38:121
313211611,88sleep30-21swapper/321:38:353
3020321149,47sleep030276-21latency_hist19:48:140
14213211317,12sleep30-21swapper/320:38:413
2940211219,15sleep30-21swapper/323:28:393
21745211213,18sleep30-21swapper/321:03:333
32268211116,15sleep30-21swapper/319:53:383
1953521119,88sleep319564-21munin-run00:23:123
1844821107,17sleep30-21swapper/320:53:283
1844821107,17sleep30-21swapper/320:53:283
3114421096,73sleep30-21swapper/321:33:303
30361210940,47sleep230393-21fschecks_count23:13:312
16049210917,13sleep30-21swapper/322:28:343
3222210817,66sleep03356-21grep23:28:440
20669210736,54sleep320792-21wc20:58:493
765721065,27sleep30-21swapper/322:02:333
22654210621,47sleep10-21swapper/119:23:271
1827621066,66sleep30-21swapper/322:33:593
1827621066,66sleep30-21swapper/322:33:593
12301210611,56sleep30-21swapper/320:33:363
20219210518,73sleep220233-21cut00:23:512
21825210317,69sleep121919-21memory21:03:351
30447210112,27sleep130514-21df19:48:311
30447210112,27sleep130514-21df19:48:311
2585121016,14sleep30-21swapper/319:33:343
2431721016,13sleep30-21swapper/319:28:383
61342978,12sleep00-21swapper/020:13:380
3012329724,49sleep230142-21apt19:48:122
1682129712,72sleep016873-21cut00:13:310
3041929611,70sleep030438-21apt-get19:48:260
3041929611,70sleep030438-21apt-get19:48:260
2238529512,68sleep022412-21awk22:48:380
1976529514,66sleep019868-21cut20:58:130
1976529514,66sleep019868-21cut20:58:130
1780929516,62sleep117847-21grep22:33:401
1780929516,62sleep117847-21grep22:33:401
91229413,18sleep00-21swapper/023:23:270
1180829416,63sleep011883-21latency_hist23:58:130
67229313,65sleep2744-21latency_hist23:23:132
2515029313,58sleep025184-21ps21:13:440
133032939,71sleep213347-21sh00:03:112
83672925,11sleep30-21swapper/323:48:023
182262926,47sleep30-21swapper/319:08:343
20612905,10sleep30-21swapper/319:59:313
384628912,59sleep13930-21/usr/sbin/munin21:48:421
1970028915,58sleep219747-21modprobe20:58:112
1970028915,58sleep219747-21modprobe20:58:112
19482895,11sleep30-21swapper/323:25:203
91652886,27sleep30-21swapper/322:06:533
215028811,54sleep10-21swapper/121:43:371
29132870,6sleep20-21swapper/223:28:372
1644228718,44sleep016472-21cut00:13:110
2786528621,29sleep20-21swapper/221:23:352
709628513,53sleep07097-21/usr/sbin/munin23:43:220
183928516,46sleep01990-21sed19:58:480
2372128416,17sleep10-21swapper/121:08:481
1357928412,57sleep213598-21find18:53:522
775828311,22sleep20-21swapper/222:03:132
276592830,8sleep10-21swapper/119:38:361
2748628310,59sleep227542-21idleruntime19:38:342
1341728320,45sleep013561-21unixbench_singl22:18:450
1341728320,45sleep013561-21unixbench_singl22:18:450
1222428112,41sleep20-21swapper/220:33:312
2902927921,36sleep10-21swapper/121:28:111
2660627913,49sleep126212-21/usr/sbin/munin21:18:421
2660627913,49sleep126212-21/usr/sbin/munin21:18:421
194592788,13sleep00-21swapper/000:22:120
1249927816,43sleep2227-21systemd-journal22:18:152
1249927816,43sleep2227-21systemd-journal22:18:152
2859327721,33sleep228590-21apt-get23:08:262
2859327721,33sleep228590-21apt-get23:08:262
2356927712,51sleep123570-21sendmail-mta22:53:271
1385427711,45sleep013866-21fschecks_count20:38:340
90927621,34sleep2906-21apt-get23:23:252
2598527630,28sleep10-21swapper/121:18:131
2520627617,21sleep10-21swapper/122:58:321
1011127614,38sleep010138-21munin-run20:28:120
1367227517,39sleep013679-21/usr/sbin/munin00:03:300
585227415,37sleep00-21swapper/023:38:380
402027423,27sleep10-21swapper/120:08:141
1578927411,29sleep20-21swapper/220:43:452
625327319,29sleep10-21swapper/120:13:431
372527316,26sleep00-21swapper/021:48:370
1088827318,42sleep110983-21latency_hist22:13:131
997427216,15sleep20-21swapper/222:08:382
534027211,49sleep05419-21cat23:38:130
860727126,27sleep20-21swapper/220:23:132
68427121,29sleep20-21swapper/221:38:422
626627115,38sleep26403-21grep23:38:482
537527115,18sleep20-21swapper/221:53:442
2587627118,29sleep20-21swapper/222:58:452
145002716,13sleep10-21swapper/122:23:351
323072706,41sleep10-21swapper/123:18:431
1690127017,34sleep116953-21sort20:48:331
1690127017,34sleep116953-21sort20:48:331
1342327014,36sleep013512-21mailstats18:53:500
1140027020,25sleep00-21swapper/022:13:370
966926918,33sleep09559-21/usr/sbin/munin22:08:320
432026918,29sleep24410-21grep20:08:352
2274126922,28sleep20-21swapper/219:23:322
1865326921,27sleep20-21swapper/222:38:142
1865326921,27sleep20-21swapper/222:38:142
1769626921,23sleep10-21swapper/119:08:121
1568426917,33sleep115765-21/usr/sbin/munin00:08:441
2164826823,25sleep20-21swapper/221:03:322
1810126822,23sleep10-21swapper/119:08:321
1559526811,15sleep00-21swapper/020:43:390
1130626812,23sleep20-21swapper/223:53:462
1122426820,28sleep20-21swapper/220:28:552
836926724,26sleep20-21swapper/222:03:372
168022677,46sleep20-21swapper/222:29:542
1664226717,30sleep016690-21if_eth019:03:350
2291726615,28sleep10-21swapper/122:48:471
2157126611,29sleep00-21swapper/019:18:410
2064926619,26sleep10-21swapper/122:43:341
1458326623,26sleep014614-21tune2fs18:58:340
1338226631,19sleep30-21swapper/318:53:493
1231726615,32sleep112354-21cut20:33:381
235422658,12sleep00-21swapper/021:08:430
757926415,24sleep00-21swapper/020:18:350
2463326417,29sleep224670-21/usr/sbin/munin19:28:452
1687426417,30sleep116944-21/usr/sbin/munin00:13:321
1541226415,32sleep115490-21grep20:43:361
1432426424,18sleep20-21swapper/222:23:322
43102635,11sleep10-21swapper/120:08:341
3081126311,35sleep130881-21sensors23:13:431
3010726315,29sleep10-21swapper/121:29:081
2958126317,26sleep00-21swapper/021:28:260
2273026315,30sleep022629-21/usr/sbin/munin19:23:300
1729526318,26sleep20-21swapper/220:48:422
1729526318,26sleep20-21swapper/220:48:422
1588626314,29sleep015905-21fschecks_count22:28:310
91192626,11sleep10-21swapper/122:06:191
62226216,30sleep0653-21latency_hist23:23:110
2926326216,27sleep029453-21apt-config21:28:140
2637226212,27sleep00-21swapper/023:03:110
2322926216,26sleep20-21swapper/221:08:372
2177126215,28sleep021813-21latency_hist22:48:130
1875826213,32sleep10-21swapper/100:18:411
1026026218,21sleep10-21swapper/123:53:131
827026116,29sleep08329-21/usr/sbin/munin22:03:350
642726117,18sleep00-21swapper/021:58:220
2024426115,27sleep020392-21grep19:13:440
1576626119,19sleep20-21swapper/200:08:452
117126112,23sleep20-21swapper/219:58:342
966926014,30sleep09792-21sed23:49:120
60052606,15sleep00-21swapper/021:56:080
528726018,23sleep20-21swapper/223:38:112
2948926017,25sleep129613-21sed19:43:461
2759226019,26sleep027656-21cat19:38:360
253826018,27sleep12563-21/usr/sbin/munin23:28:291
2110726017,26sleep121104-21apt-get19:18:261
1421226017,21sleep10-21swapper/100:03:431
1081926014,19sleep10-21swapper/120:28:461
2687525913,24sleep00-21swapper/023:03:240
2687525913,24sleep00-21swapper/023:03:240
261425912,26sleep20-21swapper/220:03:262
2132525919,24sleep18388-21kworker/u8:300:28:101
2028325918,19sleep10-21swapper/120:58:391
1996525917,19sleep20-21swapper/219:13:382
195592595,17sleep10-21swapper/120:56:201
195592595,17sleep10-21swapper/120:56:201
1754425915,19sleep00-21swapper/020:48:480
1754425915,19sleep00-21swapper/020:48:480
1502925915,29sleep115049-21modprobe20:43:121
1493225910,33sleep214946-21munin-run00:08:122
1473325919,20sleep00-21swapper/022:23:420
1118125913,18sleep10-21swapper/123:53:451
758525811,26sleep20-21swapper/223:43:412
74325813,31sleep1286-21dbus-daemon21:38:451
403725812,20sleep10-21swapper/123:33:301
252592584,12sleep00-21swapper/019:30:160
2001225823,16sleep20-21swapper/220:58:312
1457025811,32sleep214576-21df_inode18:58:322
1257825818,18sleep10-21swapper/123:58:421
100602585,11sleep20-21swapper/223:51:162
45352574,10sleep20-21swapper/221:52:062
2878425715,25sleep228590-21apt-get23:09:112
2878425715,25sleep228590-21apt-get23:09:112
2592725717,24sleep20-21swapper/219:33:372
1981425713,24sleep219550-21apt-get00:23:152
1680525713,20sleep20-21swapper/200:13:292
398125614,20sleep00-21swapper/020:08:130
266992567,12sleep00-21swapper/021:18:450
266992567,12sleep00-21swapper/021:18:450
2244725615,16sleep20-21swapper/222:48:402
1889325615,24sleep018887-21apt-get22:38:280
1802225614,24sleep0370-21snmpd22:33:430
1802225614,24sleep0370-21snmpd22:33:430
1416725615,19sleep20-21swapper/220:38:402
132482568,38sleep00-21swapper/000:02:290
491325518,21sleep10-21swapper/121:53:311
1818825512,18sleep20-21swapper/219:08:332
3221025414,23sleep232267-21irqstats19:53:372
2394125412,23sleep023973-21awk22:53:380
1705525415,23sleep117086-21munin-run22:33:121
1017625415,24sleep110201-21munin-run20:28:131
217825312,25sleep22215-21sed21:43:382
1335225314,22sleep113374-21ntp_states18:53:471
122182539,17sleep00-21swapper/020:33:290
576725213,23sleep25763-21apt-get20:13:282
3236825215,18sleep10-21swapper/119:53:401
2946925213,16sleep00-21swapper/023:09:300
2946925213,16sleep00-21swapper/023:09:300
2635625210,25sleep026425-21/usr/sbin/munin19:33:480
156052525,17sleep10-21swapper/119:01:211
64982515,12sleep10-21swapper/123:38:511
321682515,10sleep20-21swapper/221:35:152
2631025112,23sleep226319-21diskstats21:18:352
2631025112,23sleep226319-21diskstats21:18:352
2476825114,18sleep20-21swapper/221:13:342
2384025117,15sleep20-21swapper/222:53:362
644925014,22sleep16446-21apt-get21:58:231
2003025016,17sleep10-21swapper/119:13:411
1777225015,21sleep217819-21wc19:08:132
122225013,14sleep10-21swapper/123:23:361
1181425010,20sleep10-21swapper/122:13:461
2490924912,16sleep10-21swapper/119:28:521
2236124913,15sleep20-21swapper/219:23:112
205524912,20sleep02149-21memory21:43:360
1900624911,21sleep219101-21perl22:38:342
127924914,17sleep10-21swapper/119:58:371
3003524812,15sleep10-21swapper/123:13:141
3003524812,15sleep10-21swapper/123:13:141
2962724810,20sleep20-21swapper/221:28:532
2575424814,17sleep10-21swapper/119:33:281
1197424813,15sleep20-21swapper/222:13:472
2087824712,16sleep20-21swapper/222:43:392
2039024712,19sleep00-21swapper/020:58:420
2211724614,16sleep00-21swapper/021:03:420
3097224510,16sleep20-21swapper/219:48:442
3097224510,16sleep20-21swapper/219:48:442
3136124410,14sleep00-21swapper/021:33:430
307512439,13sleep00-21swapper/023:13:420
327082429,16sleep032763-21df21:38:320
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional