You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-26 - 14:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Tue May 26, 2026 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29974287136,8695sleep026768-21kworker/0:123:33:000
15538224638,2437sleep323530-1kworker/3:2H00:29:183
1217029735,955sleep05587-1kworker/0:0H00:18:580
5353254119,502sleep15400-40ogrotate23:58:441
5353254119,502sleep15400-40ogrotate23:58:441
2933924806,403sleep0363-21snmpd21:46:110
23539247418,433sleep023682-21unixbench_singl19:44:160
23539247418,433sleep023682-21unixbench_singl19:44:160
3692241818,386sleep03761-21fschecks_time22:09:040
1552023647,319sleep05587-1kworker/0:0H00:29:030
278032327135,175sleep027877-21switchtime23:24:160
18067228418,31sleep00-21swapper/022:54:050
899522638,240sleep39052-21/usr/sbin/munin00:09:043
22270225385,34sleep022037-21/usr/sbin/munin21:24:070
15481223014,197sleep015484-21cron21:03:450
8900222696,117sleep08947-21ntpq22:24:100
9390222450,49sleep08760-21/usr/sbin/munin00:09:130
5414222024,180sleep00-21swapper/022:14:130
5414222024,180sleep00-21swapper/022:14:130
2770221323,170sleep02784-21apt-get20:23:450
2975622019,145sleep00-21swapper/020:04:150
10707218722,73sleep00-21swapper/000:14:090
3185721850,7sleep30-21swapper/323:38:473
31409218412,157sleep031507-21switchtime20:09:150
31409218412,157sleep031507-21switchtime20:09:150
26403218318,136sleep00-21swapper/023:19:160
21354217719,100sleep00-21swapper/019:39:030
3186217427,134sleep02784-21apt-get20:23:490
1160221747,23sleep00-21swapper/020:49:160
12189217113,99sleep00-21swapper/019:09:160
15115216810,20sleep015318-21latency_hist00:28:470
18006216712,137sleep118121-21/usr/sbin/munin21:09:151
7639216614,139sleep27681-21ntp_offset00:04:102
2632216525,118sleep12673-21fschecks_count23:49:041
2632216525,118sleep12673-21fschecks_count23:49:041
24921216412,134sleep00-21swapper/021:33:480
18319216134,32sleep018352-21modprobe00:38:440
9288216013,93sleep00-21swapper/020:43:470
20988216013,111sleep00-21swapper/023:03:580
13249215756,21sleep00-21swapper/020:54:160
18704215537,28sleep018780-21switchtime19:29:140
2304215412,108sleep02347-21latency_hist23:48:460
14444215413,126sleep314582-21unixbench_singl00:24:183
1831321536,111sleep00-21swapper/021:09:340
19414214732,35sleep019478-21mailstats21:14:140
19845214513,120sleep319842-21vmstat21:14:223
32499214414,113sleep132578-21find23:39:151
30718214314,51sleep030754-21ntp_kernel_pll_23:34:090
28485214220,20sleep10-21swapper/123:28:451
12161213819,97sleep312159-21apt-get00:18:573
11258213810,90sleep011428-21latency_hist19:08:470
20714213714,84sleep01-21systemd23:03:460
5819213412,70sleep027731-1kworker/u9:220:29:580
3211621348,86sleep032260-21awk21:54:190
1668021336,11sleep00-21swapper/019:24:050
2609421306,94sleep00-21swapper/021:34:370
1657021309,17sleep20-21swapper/222:49:062
2457121289,72sleep00-21swapper/019:48:560
49221268,83sleep00-21swapper/021:58:590
11631212515,72sleep00-21swapper/022:33:590
252621229,41sleep02567-21uname22:04:160
16139212182,18sleep00-21swapper/021:04:090
32634212028,77sleep232653-21cron21:58:452
10596211932,64sleep310731-21grep19:04:153
19189211811,94sleep119139-21latency_hist22:58:461
30042211719,58sleep00-21swapper/021:49:060
22160211622,71sleep322265-21fschecks_time21:24:063
26855211514,84sleep327002-21/usr/sbin/munin19:54:163
26855211514,84sleep327002-21/usr/sbin/munin19:54:163
26807211413,34sleep126814-21df_inode21:39:021
14263211213,78sleep114334-21/usr/sbin/munin00:24:161
5878211119,76sleep25918-21ps22:14:272
5878211119,76sleep25918-21ps22:14:272
14335211116,78sleep214442-21grep00:24:172
16433211074,22sleep00-21swapper/022:49:040
29287210912,81sleep329338-21sshd20:04:013
2385221099,63sleep023932-21date23:13:470
31035210818,76sleep331091-21if_eth020:09:063
31035210818,76sleep331091-21if_eth020:09:063
5386210722,65sleep14879-21apt-get22:13:571
5386210722,65sleep14879-21apt-get22:13:571
2873021075,50sleep00-21swapper/020:00:190
19683210731,47sleep219680-21apt-get19:33:572
17474210618,61sleep217566-21sort21:09:042
695721059,15sleep00-21swapper/022:18:570
695721059,15sleep00-21swapper/022:18:570
28535210423,63sleep228598-21latency_hist23:28:462
27201210412,14sleep00-21swapper/021:39:130
16043210419,61sleep315746-21/usr/sbin/munin21:04:083
4865210213,74sleep14914-21apt-config22:13:451
20918210253,33sleep020961-21ntpq21:19:110
12140210217,64sleep211631-21apt-get00:18:562
27003210017,63sleep227034-21/usr/sbin/munin21:39:072
1504421008,79sleep015112-21df19:19:010
1324621008,79sleep013258-21ntp_states19:13:520
11537210012,73sleep111546-21uname20:49:131
3040829916,56sleep230447-21fschecks_count23:34:032
1978329914,67sleep119876-21fschecks_time19:34:041
1290929922,53sleep113122-21latency_hist19:13:491
29982989,15sleep00-21swapper/023:49:110
29982989,15sleep00-21swapper/023:49:110
2143329813,72sleep121490-21basename19:39:041
952629710,14sleep00-21swapper/020:43:590
773329741,38sleep00-21swapper/020:38:460
418129712,63sleep24247-21fschecks_time23:54:052
418129712,63sleep24247-21fschecks_time23:54:052
3093129714,65sleep231034-21fschecks_time20:09:052
3093129714,65sleep231034-21fschecks_time20:09:052
75222967,48sleep00-21swapper/000:04:080
568429616,66sleep15698-21cpu23:59:011
2177129625,55sleep221782-21sh21:23:452
113312965,46sleep00-21swapper/022:33:340
113312965,46sleep00-21swapper/022:33:340
1703629520,28sleep10-21swapper/100:33:571
196862948,70sleep00-21swapper/019:33:590
1447129421,38sleep10-21swapper/122:43:451
51142938,75sleep0363-21snmpd23:55:410
51142938,75sleep0363-21snmpd23:55:410
1715129221,46sleep217301-21latency_hist21:08:472
2263729118,58sleep122730-21switchtime21:24:171
1480429116,58sleep314905-21latency_hist19:18:473
1017529117,59sleep310284-21fschecks_time22:29:053
1017529117,59sleep310284-21fschecks_time22:29:053
1458329013,59sleep114612-21nfs4_client20:59:101
66282895,12sleep00-21swapper/023:59:440
2615528918,55sleep126217-21wc23:19:131
852728823,50sleep38530-21/usr/sbin/munin22:24:013
3135128716,21sleep30-21swapper/321:53:573
159362866,44sleep00-21swapper/022:47:390
2107628512,57sleep121106-21grep23:04:031
1177228514,44sleep211920-21grep20:49:182
1863828417,50sleep318667-21idleruntime-cro21:13:453
250942836,45sleep00-21swapper/023:15:280
250942836,45sleep00-21swapper/023:15:280
3902825,42sleep00-21swapper/023:39:460
1916728112,55sleep119201-21irqstats21:14:071
165952819,15sleep10-21swapper/119:24:021
11202815,41sleep00-21swapper/020:17:470
961627922,42sleep29703-21fschecks_time20:44:052
136322797,15sleep00-21swapper/022:39:110
491527818,31sleep20-21swapper/222:13:462
255527816,40sleep323758-21sshd20:21:013
2532927826,31sleep10-21swapper/121:34:051
1921327818,28sleep20-21swapper/222:58:472
2749227620,25sleep20-21swapper/219:58:472
2749227620,25sleep20-21swapper/219:58:472
2407327621,27sleep10-21swapper/121:29:131
336627522,24sleep30-21swapper/322:08:473
2569327515,43sleep225763-21/usr/sbin/munin21:34:142
2821727416,36sleep127704-21/usr/sbin/munin19:59:151
2318227313,45sleep123216-21ntp_kernel_err19:44:091
2318227313,45sleep123216-21ntp_kernel_err19:44:091
204627314,43sleep22118-21sensors20:19:142
1939827321,37sleep319419-21idleruntime-cro19:33:453
875927219,27sleep10-21swapper/100:08:541
3113727220,20sleep10-21swapper/121:53:471
2923527218,37sleep329274-21wc23:29:123
1540027013,44sleep2363-21snmpd21:02:422
1860226917,21sleep20-21swapper/219:29:122
1839926921,26sleep20-21swapper/200:38:462
1389626915,31sleep10-21swapper/120:58:451
54902680,7sleep20-21swapper/220:29:152
2003926828,21sleep120115-21awk22:59:151
1609226819,30sleep316237-21grep22:48:473
1596126814,25sleep10-21swapper/121:04:061
1155726816,31sleep30-21swapper/320:49:153
75342679,45sleep1363-21snmpd20:37:061
2619526711,16sleep00-21swapper/019:54:010
2619526711,16sleep00-21swapper/019:54:010
2519826720,31sleep30-21swapper/319:49:323
2413326726,21sleep20-21swapper/219:48:462
2413326726,21sleep20-21swapper/219:48:462
1554926718,22sleep10-21swapper/119:19:141
2865426615,33sleep228685-21/usr/sbin/munin21:44:112
2262826616,36sleep122682-21grep23:09:041
1348626624,22sleep10-21swapper/100:23:471
841026510,10sleep10-21swapper/100:07:361
785726513,38sleep17886-21who22:19:181
785726513,38sleep17886-21who22:19:181
457526515,34sleep34639-21grep23:54:143
457526515,34sleep34639-21grep23:54:143
2591226521,23sleep20-21swapper/219:53:472
2576426518,23sleep30-21swapper/321:34:153
2406426520,24sleep10-21swapper/123:13:561
2406426520,24sleep10-21swapper/123:13:561
1666426516,25sleep20-21swapper/219:24:032
1481126510,40sleep214837-21df22:44:022
144626513,22sleep10-21swapper/121:59:451
1394026516,32sleep214083-21sed22:39:182
1118026523,26sleep211215-21munin-run19:08:452
1042326517,28sleep30-21swapper/300:14:043
766126420,25sleep30-21swapper/320:38:453
611326414,34sleep36152-21modprobe20:33:453
1616226414,21sleep20-21swapper/200:29:382
1382626415,23sleep20-21swapper/219:14:102
104162647,14sleep20-21swapper/200:14:032
3265526317,22sleep10-21swapper/120:14:091
2837126317,28sleep10-21swapper/121:44:061
232626318,25sleep20-21swapper/222:04:102
220012637,11sleep30-21swapper/323:05:343
169342638,35sleep317053-21/usr/sbin/munin22:49:163
1191826314,34sleep211955-21/usr/sbin/munin19:09:102
3174926210,38sleep131352-21/usr/sbin/munin21:54:121
3008326216,27sleep230209-21latency_hist23:33:472
2935526217,23sleep10-21swapper/123:29:141
2461726214,28sleep10-21swapper/119:49:101
2342426220,21sleep30-21swapper/319:44:153
2342426220,21sleep30-21swapper/319:44:153
2182326220,23sleep121951-21ls21:23:481
2094726222,24sleep326135-21kworker/u8:119:38:463
199882628,11sleep20-21swapper/221:16:062
1940926217,16sleep20-21swapper/222:58:552
1832026216,31sleep318360-21ps22:54:123
983926116,18sleep10-21swapper/122:28:471
961426114,32sleep19615-21diskstats20:44:041
923726122,21sleep20-21swapper/220:43:452
2741526116,27sleep327569-21awk21:39:163
1289426118,26sleep10-21swapper/122:38:441
124726116,29sleep11267-21irqstats23:44:071
1181626122,20sleep20-21swapper/222:34:052
1054726113,30sleep110311-21/usr/sbin/munin00:14:061
360226016,26sleep33597-21apt-get22:08:593
3092026019,23sleep10-21swapper/120:09:041
3092026019,23sleep10-21swapper/120:09:041
2995926012,18sleep30-21swapper/321:49:053
2761926012,28sleep327657-21/usr/sbin/munin23:24:113
1512826018,20sleep30-21swapper/319:19:043
70125910,37sleep2363-21snmpd23:43:442
1504025913,27sleep215034-21apt-get19:18:592
1216625912,21sleep30-21swapper/322:34:143
331525815,20sleep10-21swapper/122:08:451
2621825817,20sleep20-21swapper/223:19:142
2261525814,18sleep30-21swapper/323:09:023
412525713,25sleep34255-21/usr/sbin/munin20:25:013
3031925715,25sleep130313-21apt-get23:33:581
2800525715,21sleep30-21swapper/319:59:093
2291025712,23sleep20-21swapper/223:09:092
1822625720,22sleep10-21swapper/119:29:031
1429225719,18sleep30-21swapper/320:59:043
1343125714,28sleep113433-21/usr/sbin/munin19:13:571
1139625718,23sleep111536-21sed22:33:471
1139625718,23sleep111536-21sed22:33:471
581425614,23sleep30-21swapper/322:14:243
581425614,23sleep30-21swapper/322:14:243
438125612,25sleep20-21swapper/220:26:252
2576025614,26sleep325764-21diskstats23:19:033
2099025613,24sleep20-21swapper/223:04:002
1317825619,18sleep30-21swapper/320:54:153
1300825613,28sleep213054-21/usr/sbin/munin20:54:102
67325517,22sleep20-21swapper/221:59:052
279225513,21sleep10-21swapper/120:23:461
180625513,24sleep31816-21munin-run22:03:463
1550425514,20sleep10-21swapper/100:28:561
1473525513,25sleep114754-21apt-get22:43:571
970625418,18sleep30-21swapper/320:44:063
510425413,23sleep15127-21irqstats20:29:061
2781125414,24sleep227817-21diskstats19:59:042
2050525415,24sleep320508-21apt-get21:18:583
1369125421,17sleep30-21swapper/322:39:153
725525313,16sleep30-21swapper/322:19:083
725525313,16sleep30-21swapper/322:19:083
460525312,22sleep10-21swapper/120:28:461
3235425312,18sleep20-21swapper/220:13:582
2879025319,19sleep30-21swapper/321:44:173
845125217,18sleep10-21swapper/120:39:161
40825214,25sleep3477-21mailstats20:14:153
33952527,16sleep20-21swapper/223:49:172
33952527,16sleep20-21swapper/223:49:172
2956125212,22sleep229654-21memory20:04:082
2928525213,23sleep129279-21apt-get20:03:591
2153625215,17sleep30-21swapper/319:39:063
167125217,19sleep20-21swapper/223:44:162
145825219,18sleep30-21swapper/323:44:133
1053325214,22sleep210539-21ntp_states22:29:122
1053325214,22sleep210539-21ntp_states22:29:122
858825111,25sleep38677-21timerandwakeup20:39:183
37632515,10sleep30-21swapper/323:53:133
37632515,10sleep30-21swapper/323:53:133
2360625112,23sleep323601-21apt-get21:28:593
2070225115,21sleep320703-21cron23:03:453
1651325114,21sleep10-21swapper/122:49:051
697925010,24sleep26975-21apt-get22:18:592
697925010,24sleep26975-21apt-get22:18:592
654025014,21sleep36587-21users23:59:193
2416225016,21sleep324168-21df_inode23:14:023
2416225016,21sleep324168-21df_inode23:14:023
2273125015,19sleep20-21swapper/221:24:182
2252425012,24sleep222554-21munin-run19:43:462
915524914,17sleep20-21swapper/200:09:072
70224911,21sleep3730-21munin-run23:43:453
604724913,15sleep20-21swapper/223:59:102
1748024914,17sleep30-21swapper/300:34:113
99232489,14sleep10-21swapper/119:04:011
768224812,15sleep30-21swapper/300:04:113
2655824814,20sleep126602-21ntp_offset19:54:101
2655824814,20sleep126602-21ntp_offset19:54:101
1686124719,15sleep30-21swapper/319:24:093
1150524711,18sleep30-21swapper/319:08:593
89822468,15sleep20-21swapper/222:24:122
648524610,15sleep20-21swapper/220:34:042
2980324616,16sleep20-21swapper/221:48:562
2490924414,17sleep20-21swapper/221:33:462
2455924411,16sleep20-21swapper/223:14:132
2455924411,16sleep20-21swapper/223:14:132
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional