You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-02-18 - 23:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Feb 18, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32605225599,2481sleep25599-1kworker/2:1H07:17:462
25914252917,496sleep125922-21ntpq06:57:561
23503233519,235sleep223651-21unixbench_singl10:28:022
1344229919,251sleep21406-21fschecks_time09:07:502
18167225962,172sleep218320-21unixbench_singl12:08:032
1626223212,204sleep31670-21ntpq09:07:553
31935222313,193sleep332085-21latency_hist09:02:303
3851221693,33sleep23951-21latency_hist11:17:302
22876221326,163sleep023020-21unixbench_singl08:28:000
22876221326,163sleep023020-21unixbench_singl08:28:000
28088221120,171sleep228116-21basename08:47:282
15767221030,158sleep215819-21fschecks_count08:07:482
6077220867,72sleep26097-21modprobe07:37:282
30622220426,33sleep20-21swapper/208:52:572
17950220425,156sleep018116-21sed08:12:580
27503220013,171sleep327537-21proc_pri07:02:583
48602188148,23sleep20-21swapper/211:22:422
48602188148,23sleep20-21swapper/211:22:422
28085218879,22sleep20-21swapper/207:04:462
26785217313,144sleep026738-21latency_hist07:02:300
2860721699,145sleep228259-21kworker/2:007:07:332
13798216873,18sleep20-21swapper/209:52:472
31912216616,137sleep231933-21latency_hist09:02:292
11908216558,91sleep212066-21date09:47:302
32746216430,121sleep2374-21awk09:02:542
2690621629,141sleep226931-21awk10:42:472
3889215911,107sleep24029-21sed07:27:592
16077215711,99sleep216144-21sort12:03:152
3048215324,112sleep13101-21interrupts09:12:501
10658215317,117sleep310748-21cut09:42:433
5205214915,84sleep20-21swapper/209:22:442
2303521445,12sleep30-21swapper/310:27:513
18550214410,58sleep218595-21unin-run08:17:282
2199221380,8sleep30-21swapper/308:27:303
7007213110,109sleep07067-21basename09:28:070
7007213110,109sleep07067-21basename09:28:070
32143212864,48sleep232262-21latency_hist07:17:302
921221248,16sleep20-21swapper/207:47:292
21911212312,96sleep221991-21latency_hist08:27:302
30784212274,25sleep00-21swapper/007:12:390
2775821204,11sleep00-21swapper/010:46:560
532121187,89sleep00-21swapper/011:22:520
532121187,89sleep00-21swapper/011:22:520
2575321189,17sleep20-21swapper/208:37:552
966521177,99sleep2496-21sendmail-mta11:37:402
30988211612,87sleep131040-21/usr/sbin/munin07:13:011
6364211514,79sleep36520-21latency_hist09:27:293
30976211517,85sleep331009-21latency_hist10:57:293
9170211313,85sleep19211-21systemd-run07:47:281
10290211218,79sleep0227-21systemd-journal09:42:270
6806211117,75sleep16803-21apt-get09:27:391
6806211117,75sleep16803-21apt-get09:27:391
4142211013,81sleep04171-21latency_hist09:17:290
20335210855,42sleep220343-21munin-run10:17:282
12350210814,16sleep20-21swapper/207:57:302
23133210716,18sleep20-21swapper/212:27:532
3006210615,76sleep13072-21latency_hist07:27:301
10341210610,83sleep210445-21date09:42:292
7823210412,78sleep27888-21lockfile-remove11:32:272
23730210315,60sleep123705-21/usr/sbin/munin08:32:451
11122210313,72sleep111161-21fschecks_count07:52:491
30276210249,38sleep20-21swapper/210:53:162
20286210048,21sleep220064-21kworker/2:008:22:092
696429915,69sleep36988-21netstat11:28:123
473829814,16sleep20-21swapper/209:22:282
473829814,16sleep20-21swapper/209:22:282
2252229812,16sleep20-21swapper/212:27:292
2075729854,11sleep20-21swapper/210:20:112
1256329820,34sleep012713-21sed11:47:590
2813929717,62sleep028232-21latency_hist08:47:300
684129611,67sleep20-21swapper/209:28:012
684129611,67sleep20-21swapper/209:28:012
839629567,13sleep20-21swapper/209:32:452
7102958,14sleep20-21swapper/211:07:292
7102958,14sleep20-21swapper/211:07:292
2377529513,13sleep20-21swapper/208:32:472
2348829511,17sleep20-21swapper/208:32:302
2348829511,17sleep20-21swapper/208:32:302
2173029455,19sleep20-21swapper/210:22:572
2028629410,65sleep120316-21cut12:17:271
191072948,17sleep20-21swapper/212:12:442
134562948,15sleep10-21swapper/111:52:441
134562948,15sleep10-21swapper/111:52:441
96672938,15sleep20-21swapper/207:47:492
151522939,73sleep2370-21snmpd08:04:252
142932930,7sleep00-21swapper/009:52:560
636729228,12sleep20-21swapper/207:37:442
2131229212,68sleep021364-21cut10:22:470
178929251,26sleep21851-21grep07:22:512
248222918,72sleep224882-21find10:32:542
1212729132,15sleep20-21swapper/209:47:422
1209929114,59sleep20-21swapper/211:47:492
295002907,19sleep2370-21snmpd08:50:472
176182908,17sleep217669-21basename10:07:482
176182908,17sleep217669-21basename10:07:482
159002908,15sleep20-21swapper/210:02:392
129682898,18sleep213016-21ntpq07:57:552
454328713,57sleep24701-21latency_hist11:22:302
405628710,61sleep20-21swapper/209:16:592
3261528732,27sleep032653-21grep09:02:520
2251928721,47sleep122560-21ntp_kernel_err08:27:531
2251928721,47sleep122560-21ntp_kernel_err08:27:531
82232868,13sleep20-21swapper/211:32:472
1352428612,58sleep20-21swapper/211:52:452
1352428612,58sleep20-21swapper/211:52:452
381628514,51sleep13218-21/usr/sbin/munin07:27:581
312542847,13sleep20-21swapper/210:58:212
1720228415,56sleep017257-21latency_hist12:07:290
3164128311,57sleep031795-21cat11:02:300
199412830,7sleep00-21swapper/010:13:020
155452836,10sleep20-21swapper/210:01:382
11452835,11sleep20-21swapper/211:07:492
488828213,56sleep04895-21df_inode07:32:460
2186128215,47sleep021870-21modprobe08:27:280
1082428230,13sleep20-21swapper/211:42:522
2855528122,39sleep128605-21fschecks_time07:07:321
2785228125,29sleep10-21swapper/110:47:301
2711028110,52sleep00-21swapper/007:02:470
1550328111,49sleep115633-21apt-key12:02:291
1550328111,49sleep115633-21apt-key12:02:291
1497128121,40sleep314994-21modprobe09:57:283
44527913,51sleep3459-21grep07:18:163
285462786,56sleep20-21swapper/210:47:552
259142785,11sleep20-21swapper/210:38:112
2582427827,22sleep20-21swapper/206:57:532
2314227817,15sleep10-21swapper/110:27:531
205512786,12sleep20-21swapper/212:17:402
1805827719,38sleep118164-21cat12:08:021
151022766,11sleep20-21swapper/211:59:072
151022766,11sleep20-21swapper/211:59:072
1295427617,24sleep10-21swapper/107:57:541
1163927611,50sleep311744-21cat11:47:303
790327516,28sleep00-21swapper/011:32:290
864527421,21sleep30-21swapper/307:43:063
3047327424,22sleep30-21swapper/307:12:303
292427420,34sleep32943-21fschecks_count09:12:483
207127422,33sleep32110-21ntpq07:22:563
538927320,26sleep10-21swapper/111:22:531
538927320,26sleep10-21swapper/111:22:531
2809327317,40sleep30-21swapper/310:47:443
2409727322,29sleep10-21swapper/112:32:291
2157627312,46sleep021610-21ntp_kernel_err12:22:470
1626627314,40sleep316295-21/usr/sbin/munin10:02:483
1384327318,30sleep30-21swapper/308:02:283
1342927317,42sleep013437-21acpi11:52:390
1342927317,42sleep013437-21acpi11:52:390
373027216,27sleep00-21swapper/007:27:570
3157127224,30sleep131590-21apt11:02:271
1009127216,24sleep00-21swapper/011:42:300
529427116,37sleep05388-21/usr/sbin/munin09:22:460
295927115,34sleep03005-21unin-run07:27:290
1852727119,28sleep10-21swapper/108:17:271
1156827114,37sleep311010-21/usr/sbin/munin07:52:593
943327013,35sleep09437-21find07:47:390
85327018,25sleep00-21swapper/007:18:250
704727021,26sleep10-21swapper/107:37:591
603827021,30sleep30-21swapper/311:27:273
603827021,30sleep30-21swapper/311:27:273
3086927012,41sleep330892-21/usr/sbin/munin07:12:473
2585427021,22sleep30-21swapper/306:57:543
1871627022,29sleep118749-21munin-run10:12:271
1871627022,29sleep118749-21munin-run10:12:271
1345527015,42sleep113497-21latency_hist09:52:291
528526915,41sleep35341-21processes07:32:573
3194326917,20sleep00-21swapper/011:02:420
3194326917,20sleep00-21swapper/011:02:420
3025526915,37sleep030284-21munin-run07:12:270
290452696,16sleep00-21swapper/010:48:340
2402926921,31sleep024065-21modprobe10:32:280
1860426926,22sleep30-21swapper/308:17:293
171312696,10sleep00-21swapper/010:07:050
628726820,31sleep16310-21munin-run09:27:271
2970626811,40sleep329744-21latency_hist08:52:303
2942926814,39sleep129565-21cat10:52:301
2419426821,27sleep30-21swapper/308:33:003
3242626712,38sleep332519-21memory09:02:473
152726716,27sleep10-21swapper/109:07:531
1403126722,22sleep30-21swapper/311:52:593
1403126722,22sleep30-21swapper/311:52:593
899226616,34sleep09072-21find09:32:590
81482667,13sleep00-21swapper/011:32:450
2099126616,27sleep30-21swapper/310:22:303
2036526614,20sleep30-21swapper/308:22:303
1646626613,24sleep30-21swapper/312:03:243
1566326620,27sleep10-21swapper/110:02:301
767526514,20sleep00-21swapper/007:42:310
684226513,38sleep36843-21cpu09:28:023
684226513,38sleep36843-21cpu09:28:023
3183226519,23sleep10-21swapper/111:02:351
3183226519,23sleep10-21swapper/111:02:351
1911526513,32sleep319205-21/usr/sbin/munin12:12:453
1703226519,19sleep00-21swapper/008:12:300
1440626518,19sleep10-21swapper/108:02:501
1321326532,18sleep30-21swapper/311:52:303
1015026520,23sleep30-21swapper/307:48:073
823026418,31sleep38263-21tune2fs11:32:483
717026416,25sleep10-21swapper/111:28:181
298802647,12sleep10-21swapper/110:52:511
2535826420,27sleep10-21swapper/108:37:461
2228926411,39sleep122290-21sshd10:24:581
2084326413,36sleep120877-21irqstats08:22:521
1649226416,30sleep116575-21find10:02:551
1502526411,28sleep10-21swapper/109:57:301
1158626422,20sleep10-21swapper/111:47:271
2495826312,39sleep025075-21cat10:32:560
233712638,15sleep00-21swapper/012:27:590
1338326321,27sleep313403-21sendmail09:52:273
907526211,19sleep10-21swapper/109:33:001
323126219,21sleep30-21swapper/311:13:033
2628426218,30sleep026270-21/usr/sbin/munin10:42:280
1437126213,31sleep014405-21irqstats08:02:490
1057526211,24sleep00-21swapper/009:42:400
2442326114,32sleep324429-21diskstats10:32:433
2372926116,27sleep023723-21apt-get08:32:440
1879826117,25sleep30-21swapper/310:12:293
1879826117,25sleep30-21swapper/310:12:293
1665126118,25sleep116803-21sed12:03:261
1541026113,21sleep00-21swapper/012:02:280
1541026113,21sleep00-21swapper/012:02:280
12462615,12sleep10-21swapper/107:20:241
1224826123,20sleep10-21swapper/111:47:521
1188026118,19sleep10-21swapper/109:47:291
1772126016,24sleep00-21swapper/010:07:500
1772126016,24sleep00-21swapper/010:07:500
863325920,19sleep10-21swapper/111:32:591
2432125913,19sleep10-21swapper/110:32:371
208432595,32sleep30-21swapper/312:21:253
1912925913,27sleep319221-21perl10:12:493
1786425911,17sleep00-21swapper/012:07:570
2120725812,22sleep30-21swapper/312:22:383
1792625817,24sleep317984-21/usr/sbin/munin12:08:003
1568125815,25sleep315748-21df08:07:443
997425711,26sleep10-21swapper/107:48:051
839825718,17sleep00-21swapper/007:43:030
2899125719,19sleep10-21swapper/108:47:591
1461825721,16sleep30-21swapper/308:02:563
1222425722,18sleep00-21swapper/009:47:480
100172576,11sleep10-21swapper/111:42:111
636325614,25sleep36360-21apt-get07:37:413
508625617,23sleep15106-21sed07:32:511
3101125614,19sleep00-21swapper/010:57:300
2691925615,25sleep10-21swapper/108:42:511
216732566,29sleep30-21swapper/310:22:543
156725615,19sleep00-21swapper/011:07:590
132925620,22sleep10-21swapper/111:07:521
2999525513,24sleep030022-21df08:52:430
1905525514,25sleep019049-21apt-get08:17:430
1576025511,30sleep115766-21diskstats08:07:471
132925513,17sleep00-21swapper/009:07:480
95925413,25sleep3955-21apt-get11:07:423
2659225418,16sleep00-21swapper/008:42:290
2657125419,16sleep30-21swapper/308:42:283
192072546,12sleep00-21swapper/012:12:460
889825315,14sleep30-21swapper/309:32:583
73625313,19sleep30-21swapper/311:07:303
73625313,19sleep30-21swapper/311:07:303
3265525319,19sleep10-21swapper/109:02:531
2974825312,20sleep00-21swapper/008:52:310
2737725318,17sleep30-21swapper/308:43:013
258912536,11sleep10-21swapper/110:37:531
2316825313,22sleep323216-21ntp_states12:27:543
209092535,37sleep10-21swapper/112:22:151
113042536,11sleep00-21swapper/007:52:520
305725211,17sleep10-21swapper/111:13:011
3037925212,25sleep130401-21ntp_states08:52:521
209182525,38sleep00-21swapper/012:22:220
190022525,10sleep10-21swapper/110:12:311
1258925212,23sleep012583-21apt-get07:57:440
482425118,17sleep10-21swapper/109:22:311
452925114,20sleep10-21swapper/111:22:291
2215025111,23sleep322146-21apt-get08:27:433
2215025111,23sleep322146-21apt-get08:27:433
155212516,33sleep30-21swapper/310:01:193
1032225012,22sleep010316-21apt-get11:42:380
311382496,33sleep30-21swapper/308:56:293
2712024911,21sleep127126-21diskstats07:02:481
2127824911,22sleep121287-21df_inode12:22:401
208442497,33sleep10-21swapper/110:21:191
409124817,16sleep30-21swapper/311:17:433
2657424811,22sleep326601-21df10:42:403
1947024815,18sleep10-21swapper/108:18:041
1033024813,20sleep110340-21munin-run09:42:281
479224713,16sleep00-21swapper/009:22:300
479224713,16sleep00-21swapper/009:22:300
469624716,19sleep10-21swapper/109:21:581
292624710,15sleep00-21swapper/011:12:560
322024610,19sleep33237-21apt-get07:27:403
150802465,33sleep30-21swapper/311:58:523
150802465,33sleep30-21swapper/311:58:523
3185724511,20sleep131895-21munin-run09:02:281
316132458,14sleep30-21swapper/311:02:293
1007924411,19sleep310088-21/usr/sbin/munin11:42:293
46462416,25sleep30-21swapper/309:21:203
46462416,25sleep30-21swapper/309:21:203
40672418,16sleep00-21swapper/011:17:400
317642409,17sleep0370-21snmpd09:01:180
1565324010,14sleep00-21swapper/010:02:290
204452356,12sleep00-21swapper/010:17:300
259502348,14sleep30-21swapper/310:38:403
100712317,14sleep30-21swapper/309:39:403
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional