You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-24 - 13:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Mon Nov 24, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5922158711,1563sleep229816-21kworker/2:019:06:452
11125298212,953sleep011163-21memory19:38:580
541624470,5sleep20-21swapper/221:03:552
541624470,5sleep20-21swapper/221:03:552
2104024047,362sleep121065-21awk21:53:581
1219230317,270sleep21240-21awk00:18:322
5056229813,270sleep35066-21ntpq00:28:593
150322727,252sleep0362-21snmpd20:51:430
74772265127,120sleep17522-21sshd22:53:551
6712225412,13sleep00-21swapper/021:08:510
9246224916,217sleep39305-21find22:59:023
15553224777,54sleep10-21swapper/119:53:531
31278224645,73sleep10-21swapper/120:43:481
19461224669,52sleep10-21swapper/121:48:581
26857224037,139sleep1362-21snmpd23:54:321
26857224037,139sleep1362-21snmpd23:54:321
27470223248,65sleep10-21swapper/123:58:481
27470223248,65sleep10-21swapper/123:58:481
19694223294,54sleep10-21swapper/120:06:451
282052216144,14sleep10-21swapper/122:18:331
3098622048,21sleep10-21swapper/120:43:321
903521929,49sleep19123-21sed22:58:561
11577219163,58sleep111290-21/usr/sbin/munin21:23:541
2857221880,10sleep028571-21sed22:18:360
6971218617,155sleep07030-21latency_hist22:53:330
13133218123,135sleep013150-21sh23:13:320
5351217628,135sleep15414-21latency21:03:551
5351217628,135sleep15414-21latency21:03:551
19760217537,62sleep119829-21fschecks_time23:33:531
10555217518,141sleep210581-21latency_hist19:38:352
6413216819,44sleep06558-21grep21:08:340
6413216819,44sleep06558-21grep21:08:340
6391216651,66sleep10-21swapper/119:23:571
32676216411,91sleep132712-21latency00:13:541
522921590,29sleep35233-21fschecks_time21:03:523
522921590,29sleep35233-21fschecks_time21:03:523
12962215944,53sleep10-21swapper/119:44:081
2252721589,134sleep122603-21cat20:14:201
320421528,130sleep13253-21basename19:13:551
1521215216,118sleep21592-21fschecks_time19:08:512
921821498,129sleep19239-21apt-get19:33:461
106121496,11sleep30-21swapper/322:33:503
819421469,121sleep18211-21apt21:13:441
7294214519,110sleep27370-21cut22:53:522
2165421449,76sleep121850-21cut20:13:361
1295521427,121sleep113037-21cut21:28:551
22981214114,21sleep10-21swapper/123:43:541
22227214030,64sleep122202-21/usr/sbin/munin21:58:471
31560213729,69sleep131532-21latency_hist22:28:351
21686213717,105sleep321710-21proc_pri23:38:593
4733213610,17sleep10-21swapper/100:28:521
784213530,67sleep1344-21/usr/sbin/munin20:48:571
25331213429,66sleep10-21swapper/122:08:521
30546213236,79sleep230603-21/usr/sbin/munin18:58:542
660721318,87sleep16627-21dpkg21:08:441
5391213128,55sleep10-21swapper/122:48:341
1790321298,110sleep117914-21uname21:43:571
4280212614,34sleep04416-21uptime20:59:060
4280212614,34sleep04416-21uptime20:59:060
144521257,83sleep11479-21ps22:34:001
12104212310,75sleep112220-21latency_hist19:43:351
13154212111,68sleep11-21systemd23:13:331
17650212010,64sleep10-21swapper/119:59:061
8925211823,75sleep08942-21sh19:33:330
17250211764,38sleep10-21swapper/121:43:321
3891211643,20sleep10-21swapper/120:58:581
3891211643,20sleep10-21swapper/120:58:581
8438211511,62sleep10-21swapper/119:29:061
9499211010,81sleep00-21swapper/019:34:050
24255210916,74sleep024313-21/usr/sbin/munin22:04:030
16998210933,51sleep017017-21apt-get19:58:450
13892210920,74sleep10-21swapper/119:48:481
12199210813,78sleep312229-21memory23:08:553
11057210742,52sleep111077-21munin-run21:23:341
25655210614,78sleep325754-21sed20:24:043
2405021067,15sleep20-21swapper/222:03:572
7031210512,16sleep10-21swapper/122:53:351
16994210341,47sleep10-21swapper/123:23:591
30012210240,47sleep129988-21latency_hist22:23:341
20504210228,56sleep30-21swapper/320:09:013
8211210111,72sleep28280-21awk19:29:032
25583210117,70sleep225618-21awk23:53:322
22598210115,71sleep222734-21modprobe23:43:352
2236210147,15sleep10-21swapper/120:53:581
6120210016,67sleep26157-21grep19:23:502
23502210043,45sleep123493-21apt-get20:18:491
14518210013,72sleep014527-21diskstats21:33:500
147532996,14sleep10-21swapper/123:18:341
1514929814,67sleep21-21systemd19:53:342
148229811,71sleep21558-21sendmail_mailtr22:34:012
321712968,73sleep132299-21ls00:13:341
1935129618,66sleep10-21swapper/123:33:281
321372957,75sleep132192-21interrupts19:03:581
1905729515,60sleep219121-21/usr/sbin/munin20:04:042
2932129420,16sleep10-21swapper/100:04:341
689129321,49sleep26908-21idleruntime-cro22:53:312
2837929310,23sleep30-21swapper/322:18:363
52152929,62sleep00-21swapper/022:46:550
27302924,12sleep10-21swapper/100:23:031
27302924,12sleep10-21swapper/100:23:031
2098429214,58sleep221098-21latency_hist23:38:342
1615829213,64sleep216214-21sort21:38:532
1339829215,17sleep30-21swapper/321:29:053
302312916,12sleep20-21swapper/200:05:032
2776229115,65sleep10-21swapper/120:32:171
2776229115,65sleep10-21swapper/120:32:171
105262918,12sleep20-21swapper/221:19:062
912429012,57sleep28739-21/usr/sbin/munin22:58:572
2192629021,53sleep321945-21apt20:13:463
310528920,46sleep33121-21sort19:13:533
2129728921,14sleep10-21swapper/123:38:491
360528818,52sleep33496-21/usr/sbin/munin20:58:523
360528818,52sleep33496-21/usr/sbin/munin20:58:523
798028716,57sleep38021-21latency_hist21:13:343
793728719,48sleep37642-21/usr/sbin/munin19:28:553
2871328719,15sleep10-21swapper/120:34:031
2871328719,15sleep10-21swapper/120:34:031
1751128716,54sleep317576-21awk19:59:043
553128612,26sleep25532-21sshd19:19:462
350628638,34sleep10-21swapper/100:24:001
2482528534,36sleep10-21swapper/123:49:031
2234728511,62sleep322395-21fschecks_time21:58:533
2067228519,33sleep00-21swapper/021:53:490
1471528519,13sleep10-21swapper/121:33:551
134752858,60sleep013560-21expr21:29:060
1140128526,41sleep211427-21fschecks_count21:23:512
318072848,16sleep20-21swapper/222:28:482
103142847,61sleep10-21swapper/123:03:481
3123228322,41sleep131245-21/usr/sbin/munin18:59:061
282628325,34sleep02911-21cat00:23:340
282628325,34sleep02911-21cat00:23:340
270028320,39sleep32761-21modprobe19:13:333
2655328314,17sleep20-21swapper/223:54:052
2655328314,17sleep20-21swapper/223:54:052
250462837,15sleep10-21swapper/120:23:451
244728320,44sleep32508-21/usr/sbin/munin20:54:043
3188128231,36sleep10-21swapper/122:28:511
2503628211,58sleep125039-21cron22:08:321
197028210,19sleep10-21swapper/119:09:031
3259528125,29sleep30-21swapper/320:48:353
323912818,57sleep032431-21ps19:04:050
2572228120,42sleep225759-21/usr/sbin/munin22:09:022
1385528121,38sleep313431-21/usr/sbin/munin23:14:043
490028012,52sleep04978-21memory00:28:550
3226928016,27sleep30-21swapper/322:29:013
2285828021,43sleep322883-21fschecks_count23:43:523
2172528015,47sleep2369-21sshd21:55:492
587727923,27sleep30-21swapper/319:23:343
2785927912,50sleep01-21systemd20:33:330
2785927912,50sleep01-21systemd20:33:330
1444527919,45sleep214480-21cpu21:33:482
1102727921,43sleep011056-21latency_hist21:23:330
551427833,13sleep10-21swapper/119:19:331
373927819,36sleep23865-21/usr/sbin/munin00:24:052
1365227812,51sleep013761-21latency_hist19:48:350
505427722,25sleep00-21swapper/019:19:030
2287127720,23sleep00-21swapper/020:14:230
1700627613,17sleep20-21swapper/223:24:002
1561527613,50sleep315729-21timerandwakeup23:19:053
95727516,39sleep0965-21/usr/sbin/munin22:33:430
27482755,13sleep10-21swapper/122:38:531
619327415,28sleep00-21swapper/019:23:520
308282748,57sleep00-21swapper/020:41:290
2713627418,35sleep30-21swapper/320:29:043
2713627418,35sleep30-21swapper/320:29:043
25302747,17sleep30-21swapper/322:38:473
2609427317,31sleep30-21swapper/323:53:533
2609427317,31sleep30-21swapper/323:53:533
2353927315,43sleep023668-21latency_hist22:03:340
1401527320,32sleep014061-21fschecks_time19:48:550
1049627326,20sleep30-21swapper/323:03:543
996027217,27sleep30-21swapper/321:18:553
761727212,41sleep07662-21ntp_states22:53:580
2121927220,25sleep00-21swapper/023:38:440
82727119,26sleep30-21swapper/319:08:363
3107827116,32sleep030313-21/usr/sbin/munin18:59:050
2957627110,15sleep30-21swapper/300:04:523
1880227118,37sleep018837-21awk21:48:320
1843727117,41sleep218448-21vmstat21:44:062
2920527011,29sleep30-21swapper/322:19:163
2883727015,32sleep20-21swapper/222:19:072
1303827016,32sleep20-21swapper/221:28:562
106627018,31sleep31216-21sed20:49:033
2636026917,32sleep226448-21latency_hist20:28:352
2432526920,31sleep324317-21apt-get23:48:493
1731426915,37sleep317458-21latency_hist21:43:343
2837326830,19sleep00-21swapper/023:59:110
2837326830,19sleep00-21swapper/023:59:110
2469426816,33sleep024723-21/usr/sbin/munin23:48:580
229112688,13sleep20-21swapper/220:14:462
1917926815,27sleep00-21swapper/021:48:520
1912226812,40sleep319189-21/usr/sbin/munin20:04:053
1546926817,31sleep20-21swapper/219:53:502
2124526716,28sleep30-21swapper/321:54:043
1058226720,26sleep30-21swapper/319:38:373
3070726620,21sleep00-21swapper/000:08:360
2879726621,22sleep20-21swapper/200:03:342
2879726621,22sleep20-21swapper/200:03:342
2865326621,23sleep00-21swapper/020:34:020
2865326621,23sleep00-21swapper/020:34:020
2759726618,25sleep30-21swapper/323:58:583
2759726618,25sleep30-21swapper/323:58:583
2156226611,41sleep021652-21dpkg20:13:350
191026615,29sleep00-21swapper/019:09:020
649726520,30sleep36545-21ntpq19:23:593
3142526515,34sleep20-21swapper/200:09:072
3025026512,24sleep20-21swapper/222:23:472
159872655,12sleep10-21swapper/123:20:041
1106426515,34sleep311123-21/usr/sbin/munin19:38:573
1994326415,21sleep00-21swapper/023:33:560
1168326412,31sleep30-21swapper/321:23:563
3181026317,20sleep00-21swapper/022:28:500
1786126320,19sleep00-21swapper/023:28:350
1678626315,32sleep30-21swapper/319:58:353
122742636,10sleep00-21swapper/021:24:420
3125826217,28sleep231270-21apt-get20:43:452
2960726217,29sleep029699-21/usr/sbin/munin00:04:530
2943626215,32sleep229459-21munin-run20:38:322
2943626215,32sleep229459-21munin-run20:38:322
429226016,26sleep24349-21irqstats22:43:542
428026015,25sleep24308-21munin-run19:18:332
328126022,17sleep20-21swapper/220:58:352
264732600,9sleep00-21swapper/023:54:020
264732600,9sleep00-21swapper/023:54:020
2058926011,18sleep20-21swapper/220:09:042
9932597,11sleep30-21swapper/300:15:423
899725917,22sleep09032-21irqstats22:58:550
867725914,29sleep38684-21ntp_states21:14:033
3173625910,22sleep30-21swapper/319:03:353
2630925910,24sleep00-21swapper/020:28:330
2350625913,23sleep00-21swapper/020:18:520
234662598,38sleep30-21swapper/320:18:383
2227725919,25sleep222301-21df_inode21:58:482
1919225918,21sleep00-21swapper/020:04:060
1881125923,20sleep00-21swapper/023:29:030
1747625910,37sleep30-21swapper/323:24:183
848425817,25sleep20-21swapper/221:13:582
2826425819,22sleep30-21swapper/320:33:533
2826425819,22sleep30-21swapper/320:33:533
263622587,38sleep00-21swapper/022:10:370
1462625820,20sleep30-21swapper/321:33:533
757825717,25sleep37612-21ntp_kernel_err22:53:573
3110425715,26sleep331151-21users22:24:053
3081725711,33sleep330818-21cpu00:08:523
2351825715,21sleep00-21swapper/023:44:050
2026025718,24sleep30-21swapper/323:34:113
1625925717,18sleep20-21swapper/223:23:322
1573325722,19sleep00-21swapper/023:19:060
1390825715,26sleep213915-21ntp_states23:14:072
114362575,18sleep00-21swapper/023:06:460
629525616,18sleep30-21swapper/322:49:043
2972925616,24sleep229720-21apt-get20:38:482
2446025615,23sleep324548-21sed22:04:063
1957925616,23sleep219662-21mailstats21:49:032
1837225623,16sleep20-21swapper/220:03:352
1212925616,18sleep20-21swapper/223:08:542
435025513,19sleep30-21swapper/322:43:553
332225513,17sleep00-21swapper/019:13:580
2690825516,23sleep226905-21apt-get22:13:462
1870525513,22sleep318808-21timerandwakeup23:29:023
1554625513,17sleep00-21swapper/019:53:520
306325414,23sleep03058-21apt-get00:23:470
2667025416,20sleep20-21swapper/220:28:512
2667025416,20sleep20-21swapper/220:28:512
252925413,23sleep22523-21apt-get22:38:462
190932549,19sleep30-21swapper/321:48:473
151225412,25sleep31506-21apt-get00:18:493
151225412,25sleep31506-21apt-get00:18:493
1416425417,21sleep314252-21sed19:48:583
1028925413,24sleep210308-21apt-get23:03:452
794925312,17sleep20-21swapper/221:13:332
323025316,22sleep03258-21munin-run20:58:330
300625311,18sleep00-21swapper/022:39:000
2590325318,23sleep325991-21switchtime22:09:073
407525214,18sleep20-21swapper/220:59:042
407525214,18sleep20-21swapper/220:59:042
3009425217,16sleep30-21swapper/320:38:583
3032325112,14sleep00-21swapper/022:23:500
2480225114,20sleep20-21swapper/220:23:342
143822515,34sleep00-21swapper/023:14:210
2870325012,15sleep00-21swapper/022:18:530
2130525012,22sleep221315-21/usr/sbin/munin23:38:502
127225010,18sleep00-21swapper/000:18:340
1850824912,21sleep218550-21ntp_states23:28:572
1599324911,21sleep015985-21apt-get21:38:470
1321624914,20sleep313249-21users19:44:103
511624714,15sleep20-21swapper/200:29:022
26722475,12sleep20-21swapper/200:22:192
26722475,12sleep20-21swapper/200:22:192
386624611,16sleep30-21swapper/300:24:063
3153524612,18sleep00-21swapper/020:43:550
903224513,14sleep00-21swapper/021:14:080
4622459,22sleep2467-21diskstats20:48:502
1270924515,16sleep20-21swapper/219:44:012
1121024512,15sleep20-21swapper/219:39:002
1754124311,16sleep30-21swapper/321:43:473
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional