You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-05 - 04:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Sep 05, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
17618264016,606sleep017704-21/usr/sbin/munin23:03:340
17618264016,606sleep017704-21/usr/sbin/munin23:03:340
6428244627,407sleep26470-21ntpq20:43:372
14120236937,185sleep214244-21latency_hist22:53:142
32540236510,314sleep232609-21latency_hist22:08:142
6532226043,196sleep16561-21grep19:03:341
6532226043,196sleep16561-21grep19:03:341
3135622489,228sleep3362-21snmpd23:47:003
4568223378,37sleep24644-21switchtime00:03:462
4568223378,37sleep24644-21switchtime00:03:462
15882223115,202sleep115932-21unin-run19:33:131
15882223115,202sleep115932-21unin-run19:33:131
28706221046,57sleep20-21swapper/221:53:452
2957822099,180sleep229613-21grep20:13:422
2957822099,180sleep229613-21grep20:13:422
3231220114,166sleep23357-21latency_hist22:18:152
3231220114,166sleep23357-21latency_hist22:18:152
13293218812,153sleep20-21swapper/219:23:412
13293218812,153sleep20-21swapper/219:23:412
10509218721,21sleep20-21swapper/200:23:362
23768218416,149sleep2225-21systemd-journal19:56:312
19594217914,149sleep019660-21swap23:08:460
2763921788,145sleep329982-1kworker/3:2H20:08:313
2399521789,20sleep27757-21cyclictest23:23:342
13533217616,142sleep213535-21cron21:08:112
718021759,153sleep27225-21fschecks_count00:13:352
21387217511,149sleep221495-21latency_hist21:33:142
19209217523,125sleep219363-21unixbench_multi21:23:462
5124217421,131sleep05215-21fschecks_time22:23:380
21703217312,17sleep00-21swapper/021:33:320
23486217010,148sleep223556-21sed23:23:132
1164821667,34sleep20-21swapper/219:18:362
1164821667,34sleep20-21swapper/219:18:362
2215121638,136sleep00-21swapper/023:18:230
24521216110,112sleep224674-21latency_hist21:43:152
24521216110,112sleep224674-21latency_hist21:43:152
575021589,136sleep21-21systemd20:43:132
20854215510,123sleep220833-21latency_hist19:48:152
1720221559,99sleep217204-21cron23:03:112
14024215325,103sleep014157-21sed00:33:470
31809214711,14sleep20-21swapper/220:21:172
31809214711,14sleep20-21swapper/220:21:172
839521458,104sleep28439-21ntpq19:08:352
839521458,104sleep28439-21ntpq19:08:352
32221417,32sleep20-21swapper/220:23:432
32221417,32sleep20-21swapper/220:23:432
9783213910,114sleep29903-21if_eth019:13:242
12569213916,68sleep212693-21latency_hist22:48:142
404621386,12sleep10-21swapper/100:03:331
404621386,12sleep10-21swapper/100:03:331
17529213812,112sleep217615-21latency_hist19:38:142
17529213812,112sleep217615-21latency_hist19:38:142
6186213211,38sleep20-21swapper/200:08:462
6186213211,38sleep20-21swapper/200:08:462
510221328,36sleep20-21swapper/222:23:352
11071213017,77sleep00-21swapper/022:43:150
2776121285,28sleep20-21swapper/221:52:402
7888212623,86sleep37937-21grep22:33:133
989421259,43sleep20-21swapper/222:38:332
84421257,41sleep20-21swapper/223:53:292
84421257,41sleep20-21swapper/223:53:292
2756021257,39sleep20-21swapper/220:08:262
1161421249,37sleep20-21swapper/220:58:472
26392212314,91sleep326521-21grep20:03:423
2628821235,28sleep20-21swapper/223:30:032
26585212212,96sleep126650-21cat23:33:131
823321207,36sleep20-21swapper/220:48:462
39221207,37sleep20-21swapper/222:08:302
1809321199,95sleep20-21swapper/223:03:442
1809321199,95sleep20-21swapper/223:03:442
854021177,34sleep20-21swapper/222:33:412
26510211719,76sleep126025-21apt-get21:48:231
1827621158,92sleep20-21swapper/221:23:132
4451211419,43sleep20-21swapper/222:19:402
32582211413,28sleep032621-21memory20:23:370
32582211413,28sleep032621-21memory20:23:370
20993211437,42sleep20-21swapper/223:13:372
9512211322,78sleep29560-21sed20:53:382
2502321139,34sleep20-21swapper/221:43:352
3261121120,6sleep30-21swapper/322:08:143
9056211113,24sleep30-21swapper/300:18:403
2863921108,16sleep20-21swapper/223:38:272
2863921108,16sleep20-21swapper/223:38:272
24217211036,39sleep10-21swapper/119:58:281
24217211036,39sleep10-21swapper/119:58:281
1288021105,12sleep20-21swapper/200:31:072
3003821097,16sleep20-21swapper/221:58:372
24833210811,16sleep20-21swapper/219:58:452
24833210811,16sleep20-21swapper/219:58:452
3627210633,40sleep20-21swapper/200:02:172
3238321058,82sleep232513-21sed23:48:442
10398210520,68sleep010443-21basename20:58:120
2203621046,44sleep20-21swapper/219:49:122
8453210310,79sleep08461-21ntpq19:08:370
8453210310,79sleep08461-21ntpq19:08:370
1904921038,45sleep20-21swapper/223:08:322
17966210322,61sleep017450-21apt-get19:38:280
1706921039,12sleep00-21swapper/023:01:320
16869210320,58sleep317001-21tail19:33:553
16869210320,58sleep317001-21tail19:33:553
21716210146,40sleep221604-21/usr/sbin/munin21:33:342
15687210119,63sleep115704-21grep22:58:141
15687210119,63sleep115704-21grep22:58:141
475121007,16sleep00-21swapper/020:38:390
29944210014,67sleep00-21swapper/023:43:130
29944210014,67sleep00-21swapper/023:43:130
13743210015,58sleep10-21swapper/119:23:521
13743210015,58sleep10-21swapper/119:23:521
205642999,76sleep00-21swapper/023:13:220
149929816,70sleep01569-21idleruntime20:28:340
13502988,49sleep20-21swapper/220:28:282
1264429811,69sleep012683-21ntpq21:03:370
179412977,16sleep217450-21apt-get19:38:182
150902978,77sleep215091-21ntp_states19:28:462
150902978,77sleep215091-21ntp_states19:28:462
939029619,19sleep10-21swapper/122:38:121
3098929614,61sleep331114-21cat22:03:143
153892967,34sleep20-21swapper/221:13:282
77202958,12sleep10-21swapper/122:31:461
302642956,34sleep20-21swapper/223:43:322
2739929520,44sleep20-21swapper/223:33:432
138122957,39sleep213831-21apt-get21:08:242
77982949,68sleep20-21swapper/219:08:152
77982949,68sleep20-21swapper/219:08:152
685029412,66sleep16980-21cat00:13:131
685029412,66sleep16980-21cat00:13:131
257712947,34sleep20-21swapper/220:03:252
2590429311,60sleep326035-21timerwakeupswit23:28:433
411529218,41sleep20-21swapper/220:37:362
961229118,41sleep20-21swapper/200:18:562
846229115,57sleep18500-21/usr/sbin/munin19:08:381
846229115,57sleep18500-21/usr/sbin/munin19:08:381
770329118,40sleep20-21swapper/222:31:352
2977929111,63sleep129902-21cat20:13:451
2977929111,63sleep129902-21cat20:13:451
2472529112,31sleep324744-21apt-get21:43:243
1550129118,40sleep20-21swapper/222:56:452
1550129118,40sleep20-21swapper/222:56:452
134172917,33sleep20-21swapper/200:33:332
2328929017,59sleep023393-21timerwakeupswit19:53:460
3019928913,56sleep130263-21/usr/sbin/munin23:43:311
2580628912,61sleep125845-21df20:03:281
127892897,71sleep00-21swapper/000:29:560
766828830,37sleep37706-21munin-run19:08:123
766828830,37sleep37706-21munin-run19:08:123
2782328814,58sleep327852-21munin-run21:53:133
3251428627,33sleep30-21swapper/323:48:453
795728519,47sleep37556-21/usr/sbin/munin20:48:393
2080028530,34sleep30-21swapper/319:48:123
1102128513,53sleep311068-21grep22:43:143
542428318,51sleep15461-21sed22:23:431
2818828321,42sleep328279-21cut21:53:333
2108228318,50sleep3362-21snmpd19:48:243
3116528221,26sleep00-21swapper/020:18:410
3116528221,26sleep00-21swapper/020:18:410
2896428212,52sleep328996-21/usr/sbin/munin23:39:173
2896428212,52sleep328996-21/usr/sbin/munin23:39:173
203682820,7sleep10-21swapper/119:43:581
1627428219,47sleep116278-21entropy19:33:411
1627428219,47sleep116278-21entropy19:33:411
3229828127,39sleep10-21swapper/123:48:431
2824328125,30sleep10-21swapper/123:38:141
877728024,24sleep00-21swapper/022:33:470
1206228011,49sleep311762-21/usr/sbin/munin00:28:373
563127918,47sleep35708-21grep22:23:493
2815427914,46sleep028242-21gpgv23:38:130
210227923,36sleep12165-40mandb23:58:121
210227923,36sleep12165-40mandb23:58:121
1289027815,42sleep312249-21/usr/sbin/munin21:03:453
1909627612,49sleep319121-21sh23:08:333
177792769,15sleep30-21swapper/321:18:463
1154927617,37sleep011595-21cat00:28:140
79922749,45sleep30-21swapper/319:08:243
79922749,45sleep30-21swapper/319:08:243
294827421,24sleep10-21swapper/123:59:131
2529627421,26sleep10-21swapper/121:43:421
1779527414,45sleep317862-21memory23:03:373
1779527414,45sleep317862-21memory23:03:373
3178727318,36sleep031816-21df23:48:300
2030027321,33sleep120314-21apt23:13:111
1964627316,25sleep00-21swapper/019:43:450
1467627315,36sleep314789-21latency_hist00:38:143
1080327329,24sleep00-21swapper/020:58:330
358027217,14sleep30-21swapper/320:33:453
1748327216,39sleep017553-21/usr/sbin/munin21:18:430
1743227226,29sleep017442-21modprobe19:38:120
1743227226,29sleep017442-21modprobe19:38:120
1725527217,33sleep117381-21grep23:03:141
309382718,46sleep10-21swapper/122:03:121
2157027118,31sleep021634-21find19:48:420
139332717,13sleep10-21swapper/122:51:321
2451327022,23sleep10-21swapper/121:43:141
2451327022,23sleep10-21swapper/121:43:141
2357427018,28sleep30-21swapper/321:38:403
2357427018,28sleep30-21swapper/321:38:403
2085927024,18sleep30-21swapper/323:13:343
104962709,16sleep00-21swapper/019:13:540
923726923,25sleep10-21swapper/120:53:331
306592695,52sleep00-21swapper/021:59:400
1335726916,20sleep00-21swapper/022:48:440
708426817,29sleep06381-21/usr/sbin/munin19:03:450
708426817,29sleep06381-21/usr/sbin/munin19:03:450
3131326811,41sleep031316-21diskstats22:03:310
3103226816,22sleep30-21swapper/320:18:363
3103226816,22sleep30-21swapper/320:18:363
1539026818,34sleep315431-21cpu21:13:293
1411226815,21sleep30-21swapper/321:08:373
73042677,13sleep00-21swapper/000:13:370
2969126719,28sleep00-21swapper/020:13:440
2969126719,28sleep00-21swapper/020:13:440
2553726715,29sleep00-21swapper/020:03:150
2553726715,29sleep00-21swapper/020:03:150
1857526721,29sleep118665-21grep19:39:431
601026616,25sleep30-21swapper/300:08:433
601026616,25sleep30-21swapper/300:08:433
2864626618,31sleep128630-21apt-get23:38:301
2864626618,31sleep128630-21apt-get23:38:301
2785426617,25sleep00-21swapper/021:53:140
142322669,12sleep10-21swapper/121:08:391
573926521,28sleep15749-21modprobe20:43:121
415126519,29sleep34277-21/usr/sbin/munin22:18:483
2301426521,26sleep123054-21grep19:53:391
1889626521,25sleep30-21swapper/321:23:383
1881026518,31sleep118956-21cat23:08:151
1881026518,31sleep118956-21cat23:08:151
150562657,49sleep00-21swapper/021:12:370
1032626523,22sleep30-21swapper/300:23:333
2339726416,26sleep00-21swapper/021:38:370
2339726416,26sleep00-21swapper/021:38:370
2261526417,33sleep322632-21apt19:53:253
2177726422,23sleep30-21swapper/321:33:353
2015626415,34sleep320159-21sshd21:28:313
1146926419,22sleep10-21swapper/120:58:461
1126226425,21sleep30-21swapper/322:43:303
1034526413,37sleep310380-21ntp_kernel_pll_19:13:493
697426316,29sleep07018-21/usr/sbin/munin22:28:380
2951126317,22sleep30-21swapper/320:13:393
2951126317,22sleep30-21swapper/320:13:393
91526212,32sleep0369-21sshd23:53:310
91526212,32sleep0369-21sshd23:53:310
2431626216,27sleep30-21swapper/319:58:343
2431626216,27sleep30-21swapper/319:58:343
1481226218,27sleep30-21swapper/322:53:403
1481226218,27sleep30-21swapper/322:53:403
94126112,22sleep30-21swapper/323:53:343
94126112,22sleep30-21swapper/323:53:343
779826115,26sleep00-21swapper/020:48:360
734026116,21sleep30-21swapper/320:48:143
2381026117,25sleep323900-21tune2fs23:23:313
2262326119,24sleep30-21swapper/323:18:383
2133626111,38sleep021366-21munin-run21:33:120
1085626113,26sleep10-21swapper/122:41:471
947026020,18sleep30-21swapper/322:38:143
769226010,20sleep10-21swapper/100:13:461
2808426015,31sleep328110-21munin-run23:38:123
2297226018,24sleep123077-21users23:18:441
1905726014,23sleep00-21swapper/021:23:440
1074526014,32sleep110839-21grep00:23:431
728925911,29sleep10-21swapper/120:48:121
554125919,24sleep05536-21apt-get00:08:280
554125919,24sleep05536-21apt-get00:08:280
409025913,27sleep04178-21/usr/sbin/munin00:03:360
409025913,27sleep04178-21/usr/sbin/munin00:03:360
2577425918,24sleep00-21swapper/020:03:270
2493525916,18sleep00-21swapper/021:43:330
1056325920,20sleep10-21swapper/119:13:551
43125811,29sleep1536-21perl22:08:331
424125817,23sleep00-21swapper/020:38:140
3193025813,18sleep10-21swapper/122:03:451
2764725815,27sleep027661-21/usr/sbin/munin20:08:320
2462125815,19sleep00-21swapper/023:23:440
2177125810,16sleep10-21swapper/123:17:031
1500325819,22sleep00-21swapper/022:53:450
1500325819,22sleep00-21swapper/022:53:450
1334725816,28sleep113414-21df00:33:321
579025716,19sleep30-21swapper/320:43:143
253925717,19sleep30-21swapper/322:13:443
253925717,19sleep30-21swapper/322:13:443
1879725713,20sleep10-21swapper/121:23:361
1593425715,25sleep315929-21apt-get22:58:283
1283525717,24sleep312884-21df22:48:313
1159825714,27sleep111747-21latency_hist00:28:151
1133425711,18sleep10-21swapper/122:43:331
956125613,24sleep39593-21ntp_states20:53:393
247312565,40sleep10-21swapper/123:24:581
1852425618,17sleep30-21swapper/319:39:413
1624725613,18sleep10-21swapper/121:13:481
1513625614,18sleep00-21swapper/019:28:480
1513625614,18sleep00-21swapper/019:28:480
1283325617,19sleep00-21swapper/019:23:150
1283325617,19sleep00-21swapper/019:23:150
3028425516,16sleep00-21swapper/023:43:340
265125517,24sleep32661-21munin-run20:33:133
264962555,38sleep00-21swapper/023:32:400
1823625517,24sleep118257-21munin-run21:23:121
135012556,11sleep10-21swapper/121:07:461
1196325511,26sleep111982-21ntp_states19:18:441
1196325511,26sleep111982-21ntp_states19:18:441
3252225415,18sleep30-21swapper/320:23:363
3252225415,18sleep30-21swapper/320:23:363
3227625415,23sleep132321-21df20:23:301
3227625415,23sleep132321-21df20:23:301
2755125413,23sleep127554-21apt-get20:08:251
2171125416,22sleep121715-21diskstats21:33:331
1676825416,16sleep00-21swapper/021:18:150
1471125414,25sleep314722-21awk19:28:343
1471125414,25sleep314722-21awk19:28:343
816325311,25sleep18232-21/usr/sbin/munin20:48:451
2809125313,24sleep128086-21apt-get21:53:271
1162825317,22sleep30-21swapper/320:58:483
1016525313,24sleep010171-21ntp_states22:38:390
869025213,23sleep18684-21apt-get00:18:301
708425213,23sleep37086-21apt-get00:13:283
347025212,23sleep03463-21apt-get22:18:290
340925215,21sleep1284-21dbus-daemon20:33:431
3070025211,25sleep130695-21apt-get20:18:261
3070025211,25sleep130695-21apt-get20:18:261
2087825214,19sleep10-21swapper/121:28:451
2943225119,17sleep00-21swapper/021:58:150
134925111,22sleep11344-21apt-get20:28:271
999725013,17sleep00-21swapper/000:23:130
315125011,22sleep00-21swapper/022:18:130
315125011,22sleep00-21swapper/022:18:130
478224915,20sleep14899-21cat22:23:141
29332499,17sleep00-21swapper/023:59:120
146624914,17sleep00-21swapper/022:11:390
1307624911,19sleep00-21swapper/019:23:310
1307624911,19sleep00-21swapper/019:23:310
31424811,21sleep3307-21apt-get22:08:273
194924812,21sleep11955-21df_inode22:13:291
194924812,21sleep11955-21df_inode22:13:291
1165124810,15sleep30-21swapper/319:18:373
1165124810,15sleep30-21swapper/319:18:373
670624713,21sleep06005-21/usr/sbin/munin20:43:430
2132924716,15sleep10-21swapper/119:48:351
1315124512,17sleep30-21swapper/319:23:343
1315124512,17sleep30-21swapper/319:23:343
46452448,15sleep30-21swapper/300:03:473
46452448,15sleep30-21swapper/300:03:473
243624210,16sleep30-21swapper/323:58:303
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional