You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-30 - 03:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Nov 30, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27477271119,674sleep127590-21/usr/sbin/munin20:58:491
27477271119,674sleep127590-21/usr/sbin/munin20:58:491
21006263024,594sleep221012-21ntpq00:03:402
990826016,557sleep29935-21ntpq20:03:412
990826016,557sleep29935-21ntpq20:03:412
22111242347,340sleep222166-21tune2fs20:43:352
9132236510,295sleep29134-21cpu23:28:272
16133230212,276sleep116179-21ntpq20:23:391
16133230212,276sleep116179-21ntpq20:23:391
3091322927,274sleep25666-21snmpd21:11:132
14601225550,165sleep214640-21ntp_kernel_pll_20:18:392
43632250135,71sleep20-21swapper/219:48:142
43632250135,71sleep20-21swapper/219:48:142
1195223343,67sleep21300-21fschecks_time23:03:322
23893223131,179sleep223928-21cron22:33:112
16240222416,87sleep20-21swapper/223:48:582
16240222416,87sleep20-21swapper/223:48:582
5877222022,68sleep20-21swapper/223:18:342
24816221643,125sleep224880-21mailstats19:08:442
24816221643,125sleep224880-21mailstats19:08:442
17103220918,174sleep017177-21systemd-run23:53:120
17103220918,174sleep017177-21systemd-run23:53:120
11157220567,55sleep20-21swapper/220:08:332
21288219025,145sleep321433-21unixbench_singl00:03:453
29598217215,96sleep229741-21grep22:48:462
22614217018,133sleep122726-21timerandwakeup20:43:461
28711216643,13sleep20-21swapper/221:03:422
12653216246,55sleep20-21swapper/220:13:302
19193215785,53sleep20-21swapper/222:18:132
19193215785,53sleep20-21swapper/222:18:132
9216215612,132sleep19222-21cat23:28:301
1365121556,13sleep20-21swapper/221:58:432
1365121556,13sleep20-21swapper/221:58:432
2479421549,132sleep224888-21cat22:33:482
29046215175,18sleep20-21swapper/200:28:402
25763214323,53sleep20-21swapper/222:38:282
16833214329,104sleep216273-21/usr/sbin/munin22:08:442
29577214213,114sleep329638-21switchtime19:23:483
19790214250,77sleep219803-21awk20:33:542
2731521417,16sleep20-21swapper/222:43:292
2764821399,58sleep20-21swapper/219:18:342
18869213810,117sleep05666-21snmpd22:14:440
18869213810,117sleep05666-21snmpd22:14:440
1981721370,8sleep00-21swapper/023:58:440
3170021365,12sleep20-21swapper/219:31:342
17305213642,53sleep20-21swapper/220:28:282
29557213520,85sleep10-21swapper/121:08:141
29508213531,92sleep229576-21systemctl19:23:472
27408213515,99sleep20-21swapper/200:23:392
11235213411,109sleep211310-21munin-run21:53:132
11235213411,109sleep211310-21munin-run21:53:132
7570213315,96sleep37075-21apt-get23:23:233
7570213315,96sleep37075-21apt-get23:23:233
26142213218,16sleep20-21swapper/220:55:022
16004213214,105sleep21-21systemd22:08:122
16004213214,105sleep21-21systemd22:08:122
31510213112,102sleep231591-21fschecks_time21:13:362
7451213014,91sleep37585-21unixbench_singl21:38:493
7451213014,91sleep37585-21unixbench_singl21:38:493
7020213019,59sleep20-21swapper/221:38:402
7020213019,59sleep20-21swapper/221:38:402
54621305,14sleep20-21swapper/222:59:522
523721308,86sleep25280-21tr19:49:112
3112213018,22sleep00-21swapper/023:08:430
1450221287,103sleep014287-1kworker/0:0H23:43:390
372921265,12sleep20-21swapper/223:10:502
1516212448,39sleep21584-21latency19:38:372
28012212314,97sleep328079-21awk19:18:433
24803212345,67sleep224856-21modprobe20:53:122
1926721208,52sleep219323-21latency23:58:352
24314211750,24sleep20-21swapper/200:13:382
17516211615,88sleep317519-21df_inode23:53:353
4792211119,75sleep34315-21apt-get19:48:233
31227211119,69sleep131348-21wc19:28:461
5250211014,29sleep00-21swapper/021:33:330
2379821100,6sleep00-21swapper/000:13:140
30258210913,81sleep030281-21/usr/sbin/munin21:08:410
20980210743,33sleep20-21swapper/222:23:262
5624210613,72sleep35631-21/usr/sbin/munin21:33:443
11289210514,78sleep011358-21switchtime23:33:460
11289210514,78sleep011358-21switchtime23:33:460
21515210432,45sleep121675-21unixbench_singl18:58:591
19404210416,64sleep30-21swapper/322:18:223
4049210369,18sleep20-21swapper/221:28:422
31770210312,51sleep20-21swapper/222:58:152
25008210214,73sleep125090-21sed19:08:471
25008210214,73sleep125090-21sed19:08:471
24331210225,46sleep024325-21apt-get19:08:300
24331210225,46sleep024325-21apt-get19:08:300
24285210218,62sleep024426-21unixbench_singl20:48:460
1668621025,13sleep20-21swapper/220:24:042
1668621025,13sleep20-21swapper/220:24:042
524421015,62sleep20-21swapper/223:14:582
11177210111,78sleep211203-21proc_pri23:33:442
11177210111,78sleep211203-21proc_pri23:33:442
8892210010,75sleep08955-21cut23:28:130
8892210010,75sleep08955-21cut23:28:130
2601829940,40sleep20-21swapper/219:13:332
815629840,39sleep20-21swapper/223:24:192
815629840,39sleep20-21swapper/223:24:192
749329830,11sleep20-21swapper/219:55:572
2803429610,17sleep20-21swapper/221:03:142
2803429610,17sleep20-21swapper/221:03:142
126402965,58sleep20-21swapper/221:54:512
2117629528,43sleep220765-21/usr/sbin/munin18:58:512
1450629525,48sleep014592-21cat22:03:130
1450629525,48sleep014592-21cat22:03:130
486729416,60sleep04924-21munin-run21:33:120
1861829430,11sleep20-21swapper/223:57:022
1327329412,31sleep113282-21diskstats21:58:341
1327329412,31sleep113282-21diskstats21:58:341
1006029416,60sleep39490-21/usr/sbin/munin20:03:463
1006029416,60sleep39490-21/usr/sbin/munin20:03:463
2237929331,16sleep20-21swapper/219:03:272
2237929331,16sleep20-21swapper/219:03:272
263929117,20sleep30-21swapper/319:43:143
2686629022,23sleep10-21swapper/100:23:141
208952909,71sleep220904-21awk20:38:422
1354629022,36sleep20-21swapper/223:40:302
1354629022,36sleep20-21swapper/223:40:302
267472898,67sleep226749-21cron00:23:112
2637128912,61sleep326463-21cat22:38:463
123922899,15sleep00-21swapper/023:38:180
123922899,15sleep00-21swapper/023:38:180
498828824,48sleep15043-21tune2fs19:49:061
1107628810,64sleep011147-21gpgv21:53:110
1107628810,64sleep011147-21gpgv21:53:110
3258928723,49sleep20-21swapper/219:33:402
3258928723,49sleep20-21swapper/219:33:402
22192877,68sleep00-21swapper/023:06:280
1864028721,50sleep318648-21grep20:33:143
1224828722,50sleep312277-21sensors_temp21:54:113
109528721,40sleep31092-21apt-get23:03:253
1009428716,52sleep310148-21latency21:48:383
1009428716,52sleep310148-21latency21:48:383
83892866,15sleep20-21swapper/221:43:362
83892866,15sleep20-21swapper/221:43:362
187328623,44sleep01930-21latency_hist21:23:150
187328623,44sleep01930-21latency_hist21:23:150
952128513,57sleep39552-21ntpq23:28:363
2577928510,56sleep125857-21/usr/sbin/munin00:18:371
237442850,8sleep10-21swapper/119:08:141
237442850,8sleep10-21swapper/119:08:141
2257728530,14sleep20-21swapper/200:08:492
179528526,31sleep30-21swapper/321:23:143
179528526,31sleep30-21swapper/321:23:143
1419628529,38sleep20-21swapper/223:43:332
100272859,12sleep10-21swapper/121:48:361
100272859,12sleep10-21swapper/121:48:361
2834128417,48sleep328372-21latency_hist00:28:133
2043428413,53sleep120436-21http20:38:291
195112848,15sleep20-21swapper/222:18:302
1258128417,51sleep112586-21diskstats23:38:311
1258128417,51sleep112586-21diskstats23:38:311
202272830,10sleep30-21swapper/320:38:153
482528222,30sleep00-21swapper/019:48:280
272028214,50sleep32860-21sed21:23:463
272028214,50sleep32860-21sed21:23:463
1940228218,47sleep119450-21/usr/sbin/munin23:58:381
4732817,13sleep20-21swapper/221:18:242
4732817,13sleep20-21swapper/221:18:242
423828122,29sleep10-21swapper/123:13:281
2403028117,26sleep10-21swapper/120:48:431
63022805,63sleep20-21swapper/221:36:382
1757728020,45sleep117592-21fschecks_count23:53:451
606027914,50sleep16110-21sed23:18:371
1979927915,46sleep019405-21/usr/sbin/munin22:18:360
553927818,41sleep15580-21grep21:33:421
484327817,44sleep34863-21cron21:33:113
1968027821,32sleep30-21swapper/323:58:443
301042779,13sleep00-21swapper/022:52:560
1201327724,33sleep312066-21munin-run23:38:133
1201327724,33sleep312066-21munin-run23:38:133
212052760,8sleep30-21swapper/318:58:513
1592027612,46sleep30-21swapper/323:48:313
1592027612,46sleep30-21swapper/323:48:313
1428627616,46sleep314297-21forks20:18:323
961827513,47sleep39676-21sort21:48:223
961827513,47sleep39676-21sort21:48:223
1935727518,36sleep10-21swapper/120:33:491
1088227521,27sleep30-21swapper/320:08:143
269872748,53sleep00-21swapper/019:15:490
238482748,17sleep00-21swapper/022:32:370
1730927423,26sleep10-21swapper/120:28:321
2736327320,24sleep00-21swapper/020:58:480
2736327320,24sleep00-21swapper/020:58:480
2585927313,39sleep326000-21sed20:53:463
2087527318,31sleep00-21swapper/018:58:330
3114527216,33sleep00-21swapper/019:28:450
3043627216,28sleep30-21swapper/321:08:443
2749727215,42sleep327559-21cat22:43:343
273527217,41sleep32760-21grep23:08:333
1638427217,32sleep016477-21fschecks_time22:08:340
1077027219,28sleep00-21swapper/020:08:110
1077027219,28sleep00-21swapper/020:08:110
952427120,28sleep10-21swapper/121:48:121
952427120,28sleep10-21swapper/121:48:121
3118627119,33sleep131328-21grep22:53:461
2916427136,18sleep10-21swapper/100:28:431
713227019,30sleep30-21swapper/319:53:483
45227025,26sleep10-21swapper/122:58:461
304927014,34sleep10-21swapper/123:08:391
3041727016,22sleep00-21swapper/022:53:240
1757827014,42sleep017636-21sed20:28:390
1934726910,45sleep019356-21ntpq20:33:480
1637026921,32sleep016465-21timerandwakeup20:23:460
1637026921,32sleep016465-21timerandwakeup20:23:460
3214926818,19sleep00-21swapper/019:33:300
3214926818,19sleep00-21swapper/019:33:300
2112326820,20sleep10-21swapper/100:03:431
1473326829,20sleep10-21swapper/123:43:441
2558026716,22sleep00-21swapper/022:38:140
1328526716,36sleep313329-21grep21:58:363
1328526716,36sleep313329-21grep21:58:363
126162677,14sleep10-21swapper/121:54:341
2992426614,35sleep329866-21latency_hist00:33:133
1610426617,25sleep00-21swapper/022:08:140
847026519,21sleep00-21swapper/019:58:460
847026519,21sleep00-21swapper/019:58:460
808526519,12sleep20-21swapper/219:58:362
808526519,12sleep20-21swapper/219:58:362
32122656,27sleep20-21swapper/221:27:092
32122656,27sleep20-21swapper/221:27:092
3101726519,23sleep30-21swapper/322:53:443
2965126516,32sleep029778-21cat19:23:490
2739926518,21sleep10-21swapper/122:43:321
2722726524,19sleep30-21swapper/319:18:143
2397426413,23sleep10-21swapper/122:33:141
1877826414,32sleep318788-21/usr/sbin/munin22:13:463
1877826414,32sleep318788-21/usr/sbin/munin22:13:463
1390926418,25sleep00-21swapper/020:18:130
1288926415,37sleep312911-21netstat23:38:543
1288926415,37sleep312911-21netstat23:38:543
967726317,21sleep00-21swapper/023:28:410
814626316,26sleep00-21swapper/021:43:220
814626316,26sleep00-21swapper/021:43:220
432626318,26sleep10-21swapper/119:48:131
432626318,26sleep10-21swapper/119:48:131
2872626316,21sleep00-21swapper/019:23:120
250426316,33sleep12550-21gpgv19:43:121
2236826315,26sleep30-21swapper/320:43:403
1647926322,25sleep116273-21/usr/sbin/munin22:08:351
699026214,21sleep10-21swapper/121:38:391
699026214,21sleep10-21swapper/121:38:391
3246326214,30sleep00-21swapper/022:58:410
2364026217,19sleep30-21swapper/319:08:133
2364026217,19sleep30-21swapper/319:08:133
110292626,11sleep00-21swapper/020:08:190
704926113,33sleep07110-21apt-config23:23:120
3200526111,33sleep131359-21/usr/sbin/munin21:13:471
3019526120,25sleep10-21swapper/122:53:131
2837526110,28sleep028517-21sort00:28:140
2782226112,31sleep127834-21/usr/sbin/munin19:18:371
2428526117,27sleep30-21swapper/322:33:333
2418026121,19sleep30-21swapper/319:08:213
2418026121,19sleep30-21swapper/319:08:213
229826115,23sleep10-21swapper/121:23:361
229826115,23sleep10-21swapper/121:23:361
1583726114,30sleep315842-21/usr/sbin/munin20:23:333
1583726114,30sleep315842-21/usr/sbin/munin20:23:333
1023326112,24sleep10-21swapper/120:03:481
1023326112,24sleep10-21swapper/120:03:481
804426017,25sleep38048-21diskstats19:58:333
804426017,25sleep38048-21diskstats19:58:333
2675526020,22sleep30-21swapper/300:23:123
231382606,13sleep10-21swapper/100:09:001
2225426020,24sleep3225-21systemd-journal22:28:123
2148126012,24sleep10-21swapper/122:23:411
2061826017,26sleep320671-21if_err_eth020:38:353
204932605,41sleep10-21swapper/122:20:431
95325914,29sleep1982-21cut19:38:111
95325914,29sleep1982-21cut19:38:111
615925916,28sleep36193-21/usr/sbin/munin23:18:393
2639225918,24sleep10-21swapper/120:58:121
2524825916,29sleep125353-21cat00:18:131
223512597,41sleep00-21swapper/019:03:210
223512597,41sleep00-21swapper/019:03:210
1511325916,26sleep315157-21ntpq22:03:363
1511325916,26sleep315157-21ntpq22:03:363
93425814,26sleep1950-21ntp_states21:18:481
93425814,26sleep1950-21ntp_states21:18:481
856425815,24sleep10-21swapper/123:24:431
856425815,24sleep10-21swapper/123:24:431
302582586,10sleep10-21swapper/119:27:581
2886225815,26sleep328859-21apt-get22:48:273
215392586,41sleep00-21swapper/000:04:470
120612588,17sleep10-21swapper/120:09:541
678825715,24sleep00-21swapper/019:53:410
449925717,22sleep30-21swapper/319:48:153
3194225714,20sleep00-21swapper/021:13:460
2241525718,23sleep022451-21ntp_states20:43:410
1785525714,27sleep117872-21/usr/sbin/munin22:13:241
1785525714,27sleep117872-21/usr/sbin/munin22:13:241
664925612,27sleep16678-21sshd19:53:381
2857425614,24sleep028602-21munin-run22:48:110
2416225618,22sleep324282-21sed20:48:453
2254425613,19sleep30-21swapper/322:28:283
3214725515,24sleep332142-21apt-get19:33:293
3214725515,24sleep332142-21apt-get19:33:293
3177825514,20sleep30-21swapper/321:13:413
300225512,23sleep30-21swapper/319:43:223
2920925511,30sleep05666-21snmpd21:04:250
2899025521,17sleep00-21swapper/000:28:380
2726525517,24sleep30-21swapper/320:58:473
2726525517,24sleep30-21swapper/320:58:473
1744225514,24sleep017436-21apt-get23:53:320
1637825514,25sleep316382-21diskstats22:08:333
1551325518,19sleep10-21swapper/120:23:141
1519825516,23sleep115204-21ntp_states22:03:381
1519825516,23sleep115204-21ntp_states22:03:381
125025518,18sleep00-21swapper/019:38:310
423525413,24sleep34231-21apt-get23:13:263
2600025419,20sleep10-21swapper/119:13:321
2448925417,23sleep324552-21switchtime00:13:433
118325421,18sleep10-21swapper/123:03:311
1163925412,24sleep011624-21apt-get21:53:320
549825312,17sleep00-21swapper/023:18:110
3200225316,20sleep30-21swapper/322:58:283
2456525317,16sleep00-21swapper/000:13:440
2067525312,17sleep00-21swapper/020:38:360
1404725314,23sleep314043-21apt-get23:43:263
1310725313,24sleep113113-21ntp_states20:13:411
492525213,14sleep10-21swapper/121:33:131
2979325218,21sleep10-21swapper/121:08:261
2551725214,18sleep00-21swapper/020:53:390
1264825212,24sleep312644-21apt-get20:13:273
3064225114,21sleep330649-21df_inode19:28:323
1430025114,21sleep014335-21fschecks_count20:18:330
32772509,27sleep03326-21irqstats19:43:520
32772509,27sleep03326-21irqstats19:43:520
2227925012,23sleep022305-21munin-run22:28:130
200825012,21sleep02027-21apt-get21:23:230
200825012,21sleep02027-21apt-get21:23:230
2583824913,21sleep125844-21df_inode22:38:311
2230924913,20sleep321821-21apt-get00:08:173
1516024812,22sleep015197-21ntp_states22:03:370
878124715,17sleep00-21swapper/021:43:510
878124715,17sleep00-21swapper/021:43:510
2421424715,16sleep00-21swapper/022:33:300
282512469,14sleep30-21swapper/321:03:313
2575924613,15sleep00-21swapper/000:18:360
1580824612,17sleep10-21swapper/123:48:201
1580824612,17sleep10-21swapper/123:48:201
106892439,19sleep110707-21apt-get23:33:271
106892439,19sleep110707-21apt-get23:33:271
77852398,15sleep10-21swapper/119:58:151
77852398,15sleep10-21swapper/119:58:151
247182398,12sleep10-21swapper/122:33:471
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional