You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-25 - 04:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackbslot7s.osadl.org (updated Thu Jun 25, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
26132267419,639sleep126141-21/usr/sbin/munin00:00:201
26132267419,639sleep126141-21/usr/sbin/munin00:00:201
3082826508,629sleep030869-21ntpq00:15:210
3082826508,629sleep030869-21ntpq00:15:210
3929258813,560sleep23939-21ntpq22:50:232
16258248370,388sleep00-21swapper/020:04:570
17728247115,440sleep217699-21latency_hist23:34:572
116712465430,23sleep311673-21sshd21:30:333
26247245921,418sleep326265-21cron22:19:553
29531238925,348sleep329622-21/usr/sbin/munin20:45:263
604234811,41sleep00-21swapper/019:15:060
165892320157,145sleep016631-21fschecks_count20:05:140
21276228094,159sleep0364-21snmpd20:20:160
20808226189,64sleep00-21swapper/023:44:560
19304226066,181sleep0225-21systemd-journal20:14:550
19304226066,181sleep0225-21systemd-journal20:14:550
27868222225,171sleep025921-1kworker/0:2H22:24:570
5543222112,145sleep05576-21idleruntime-cro21:14:550
255852215131,23sleep00-21swapper/020:34:580
6919220431,20sleep00-21swapper/019:34:570
22125220317,138sleep00-21swapper/022:05:220
21302220214,169sleep021067-21/usr/sbin/munin23:45:160
10837219910,161sleep010864-21fschecks_count21:30:130
719321897,166sleep00-21swapper/023:00:300
3424218331,133sleep03573-21unixbench_multi21:05:260
11921218315,143sleep00-21swapper/019:50:160
11921218315,143sleep00-21swapper/019:50:160
30494218210,152sleep00-21swapper/020:50:120
30494218210,152sleep00-21swapper/020:50:120
6710217842,62sleep06601-21gpgv22:59:580
2936921778,155sleep029383-21apt-config22:29:530
2991621735,156sleep00-21swapper/020:46:120
1738521698,96sleep00-21swapper/021:50:200
31038216718,135sleep031069-21fschecks_count19:10:140
1486216615,137sleep11528-21sed00:25:191
1778216510,15sleep00-21swapper/021:00:260
1085216314,111sleep00-21swapper/000:25:060
146372162124,27sleep014695-21latency_hist23:24:570
146372162124,27sleep014695-21latency_hist23:24:570
26347216112,68sleep026419-21cat00:00:270
26347216112,68sleep026419-21cat00:00:270
15554216110,135sleep015697-21grep23:25:260
1256821618,19sleep00-21swapper/021:35:180
319952160127,17sleep00-21swapper/022:35:270
9659215980,64sleep00-21swapper/021:25:210
1851921558,131sleep00-21swapper/021:55:130
26115215412,124sleep00-21swapper/020:35:180
22996215038,14sleep00-21swapper/020:25:160
22996215038,14sleep00-21swapper/020:25:160
13085214912,121sleep013119-21latency_hist23:19:570
7942214732,16sleep00-21swapper/021:20:160
1652021399,57sleep0364-21snmpd21:47:540
4805213811,109sleep04806-21munin-run22:54:570
27723213610,41sleep00-21swapper/000:05:190
24952213611,44sleep00-21swapper/022:15:080
2856721345,10sleep00-21swapper/020:43:460
14699213417,96sleep00-21swapper/019:59:570
14699213417,96sleep00-21swapper/019:59:570
5259213317,100sleep25257-21apt-get22:55:072
26565213329,76sleep226559-21apt-get22:20:102
1880121328,16sleep00-21swapper/023:35:270
24452212910,106sleep024491-21apt-get22:14:550
2908521289,37sleep00-21swapper/000:10:160
3016321229,41sleep030740-21cyclictest22:30:220
20173212211,85sleep00-21swapper/022:00:140
256121218,36sleep00-21swapper/019:20:440
256121218,36sleep00-21swapper/019:20:440
1513121208,35sleep00-21swapper/021:44:550
821821185,41sleep00-21swapper/019:37:230
27982211821,73sleep228102-21timerwakeupswit00:05:252
344621177,33sleep00-21swapper/000:30:270
23240211310,31sleep00-21swapper/023:50:250
23240211310,31sleep00-21swapper/023:50:250
9388211114,85sleep29448-21cat21:25:152
6926211111,78sleep16568-21apt-get22:59:591
5588211126,35sleep30-21swapper/319:30:083
30132211116,77sleep330165-21awk00:14:553
31071211012,81sleep131147-21cut19:10:151
21066211016,77sleep221083-21apt23:45:062
5458210929,59sleep35509-21munin-plugin-st00:39:553
24377210921,70sleep324466-21sort20:30:163
21091210932,48sleep121086-21apt-get23:45:091
9406210811,87sleep09527-21timerwakeupswit19:40:260
9111210672,20sleep00-21swapper/023:05:230
26006210519,69sleep326040-21irqstats00:00:173
26006210519,69sleep326040-21irqstats00:00:173
4398210412,61sleep10-21swapper/119:26:061
23374210318,72sleep123434-21cat22:10:161
2489121026,75sleep20-21swapper/220:30:272
28480210113,72sleep228483-21/usr/sbin/munin22:25:192
18601210116,67sleep318613-21/usr/sbin/munin21:55:163
25341210027,62sleep030740-21cyclictest23:58:370
19597210023,61sleep00-21swapper/020:15:110
17464210019,11sleep00-21swapper/023:32:080
3228429935,50sleep032359-21latency00:20:170
1182629717,63sleep211838-21/usr/sbin/munin23:15:142
992529621,57sleep10-21swapper/121:25:261
85129646,32sleep2880-21munin-run20:59:552
715729418,11sleep00-21swapper/021:17:340
66829418,12sleep00-21swapper/020:57:350
417529418,30sleep00-21swapper/000:35:030
133829418,11sleep00-21swapper/022:40:560
1149029411,65sleep00-21swapper/023:14:560
1134029414,10sleep00-21swapper/019:47:250
291129318,29sleep00-21swapper/022:46:170
2282129323,46sleep122912-21cut20:25:131
2282129323,46sleep122912-21cut20:25:131
1301029311,69sleep313083-21systemd-run23:19:553
1245429321,51sleep212275-21/usr/sbin/munin21:35:162
2536829116,62sleep125410-21ntp_kernel_pll_22:15:291
2415629125,28sleep30-21swapper/323:55:043
3069029018,51sleep030740-21cyclictest19:09:580
512828915,12sleep00-21swapper/019:27:260
281032890,8sleep30-21swapper/300:05:253
2895928819,21sleep20-21swapper/200:10:142
2012928810,64sleep320198-21/usr/sbin/munin20:15:263
1610128811,60sleep316102-21sshd23:29:333
1383328819,56sleep313865-21proc_pri19:55:233
1383328819,56sleep313865-21proc_pri19:55:233
562728719,34sleep20-21swapper/221:14:572
532228626,40sleep35354-21munin-run19:29:553
2191428610,62sleep221999-21latency22:05:162
1734128624,42sleep317382-21ntpq21:50:193
3181728514,56sleep231963-21cat20:54:582
3181728514,56sleep231963-21cat20:54:582
2482328452,17sleep20-21swapper/223:55:242
1662828420,18sleep10-21swapper/123:30:151
102532847,13sleep30-21swapper/319:45:063
3176028331,31sleep30-21swapper/320:54:553
3176028331,31sleep30-21swapper/320:54:553
427328222,37sleep20-21swapper/219:26:032
1473928228,39sleep114881-21sed19:59:581
1473928228,39sleep114881-21sed19:59:581
742028122,28sleep20-21swapper/221:19:582
1864928023,37sleep318797-21grep23:35:263
1776328021,31sleep30-21swapper/323:34:583
3181127910,56sleep131815-21latency_hist20:54:571
3181127910,56sleep131815-21latency_hist20:54:571
3074727922,28sleep10-21swapper/119:09:591
168727924,30sleep10-21swapper/122:44:571
1805227818,30sleep10-21swapper/120:10:111
1805227818,30sleep10-21swapper/120:10:111
1710527829,25sleep30-21swapper/320:05:253
2768527710,55sleep1364-21snmpd22:23:081
1027827721,36sleep310361-21fschecks_time23:10:143
394027616,42sleep30-21swapper/322:50:243
2456427617,36sleep224711-21grep22:14:572
891527515,43sleep29007-21/usr/sbin/munin23:05:172
576527512,28sleep30-21swapper/321:14:583
46352758,14sleep10-21swapper/100:35:201
3264227515,44sleep3320-21latency_hist22:39:583
2914227526,31sleep10-21swapper/100:10:171
1930727519,28sleep20-21swapper/223:39:572
1888627523,25sleep10-21swapper/121:55:221
1830427521,28sleep20-21swapper/221:54:592
1414127518,27sleep30-21swapper/321:40:183
74027422,34sleep1677-21apt-get19:15:481
272127411,45sleep10-21swapper/121:05:071
1150327423,25sleep10-21swapper/123:14:571
1076427419,30sleep10-21swapper/119:45:241
802227318,37sleep17630-21/usr/sbin/munin21:20:171
6042736,54sleep10-21swapper/120:56:471
782127215,41sleep37942-21timerwakeupswit19:35:293
2011827225,25sleep20-21swapper/223:40:262
396727119,22sleep30-21swapper/300:34:583
3260827122,28sleep232748-21store19:15:002
2614227119,37sleep226179-21ntpq00:00:212
2614227119,37sleep226179-21ntpq00:00:212
2594527114,41sleep126031-21fw_packets20:35:151
1963027121,21sleep20-21swapper/220:15:132
1599527114,42sleep216088-21sed21:45:262
125527122,20sleep10-21swapper/121:00:141
7522709,47sleep3789-21ntpq22:40:183
588427014,37sleep25924-21sort19:30:192
489527023,25sleep10-21swapper/122:54:581
200392708,18sleep10-21swapper/123:40:251
1312127016,30sleep10-21swapper/123:19:581
1308027027,25sleep30-21swapper/319:54:553
1308027027,25sleep30-21swapper/319:54:553
2990226829,24sleep229344-21/usr/sbin/munin19:05:242
267352686,11sleep30-21swapper/320:35:293
2164526818,36sleep321672-21grep22:05:093
211826825,19sleep30-21swapper/319:19:583
1671926812,36sleep116693-21latency_hist21:49:571
1334726819,22sleep20-21swapper/219:55:062
1334726819,22sleep20-21swapper/219:55:062
2782426723,29sleep327866-21true22:24:563
818426621,27sleep28258-21mailstats21:20:222
3110226614,38sleep231219-21cat00:15:272
3110226614,38sleep231219-21cat00:15:272
2544226613,32sleep125507-40mandb23:59:551
2250326614,36sleep222644-21sed20:24:582
1462626624,27sleep3292-21systemd-logind23:24:563
1462626624,27sleep3292-21systemd-logind23:24:563
123002667,38sleep10-21swapper/121:35:111
840626517,20sleep20-21swapper/223:04:562
68126519,29sleep2677-21apt-get19:15:082
486626523,18sleep20-21swapper/200:35:262
398026519,22sleep20-21swapper/219:25:422
398026519,22sleep20-21swapper/219:25:422
3201326515,27sleep10-21swapper/100:20:101
2630826518,31sleep226388-21mailstats20:35:242
2263826515,30sleep222709-21/usr/sbin/munin23:50:112
2263826515,30sleep222709-21/usr/sbin/munin23:50:112
1696926519,32sleep117032-21cut20:05:231
1550426514,21sleep30-21swapper/321:45:153
1372026519,27sleep10-21swapper/123:20:181
1372026519,27sleep10-21swapper/123:20:181
557726418,28sleep15613-21munin-run21:14:561
259742646,49sleep20-21swapper/222:16:312
213312646,49sleep20-21swapper/222:04:362
2063126427,19sleep30-21swapper/322:00:253
1671226420,19sleep20-21swapper/220:05:162
1313426415,25sleep113213-21date19:54:571
1313426415,25sleep113213-21date19:54:571
1113026420,29sleep211179-21awk21:30:202
2997926325,21sleep330100-21sed19:05:253
2398326319,22sleep20-21swapper/220:29:552
2398326319,22sleep20-21swapper/220:29:552
2137726318,23sleep10-21swapper/120:20:171
202026320,23sleep20-21swapper/222:45:142
1859526317,20sleep30-21swapper/320:10:263
1859526317,20sleep30-21swapper/320:10:263
138026318,20sleep30-21swapper/300:25:173
695626217,25sleep10-21swapper/119:34:581
289122627,17sleep20-21swapper/220:45:012
3230626119,26sleep332316-21irqstats20:55:163
3103026114,31sleep331036-21diskstats19:10:133
1225526116,27sleep212278-21ntp_states19:50:222
1225526116,27sleep212278-21ntp_states19:50:222
998726014,28sleep19988-21cron19:44:551
791926015,21sleep30-21swapper/321:20:153
77362609,38sleep27820-21switchtime19:35:282
2865726011,32sleep128704-21systemd-run20:44:561
2243426016,30sleep322452-21awk20:24:553
170626015,29sleep21662-21latency_hist22:44:582
1562326020,17sleep30-21swapper/320:00:283
1534926014,12sleep10-21swapper/123:25:231
152926011,28sleep20-21swapper/200:25:202
1438026015,25sleep20-21swapper/221:40:252
1111726016,23sleep10-21swapper/121:30:191
880925918,25sleep38815-21diskstats19:40:133
872025913,28sleep10-21swapper/123:05:121
568325914,32sleep35720-21awk22:55:373
305725921,20sleep13096-21ntpq00:30:201
210725916,25sleep30-21swapper/322:45:153
200642599,20sleep10-21swapper/122:00:061
1391925912,31sleep113945-21df21:40:121
1019125916,26sleep210185-21apt-get23:10:092
2027725815,22sleep10-21swapper/120:15:281
325525719,19sleep30-21swapper/319:21:033
325525719,19sleep30-21swapper/319:21:033
299225712,28sleep30-21swapper/321:05:173
2894825719,21sleep10-21swapper/120:45:121
2829725712,24sleep30-21swapper/322:25:163
1229725715,25sleep312293-21apt-get21:35:093
428425612,27sleep34280-21apt-get21:10:103
322925620,16sleep20-21swapper/200:30:252
2790125614,30sleep127980-21switchtime00:05:241
2389625615,27sleep123917-21munin-run23:54:541
2389625615,27sleep123917-21munin-run23:54:541
1545125618,20sleep10-21swapper/120:00:261
913125522,18sleep20-21swapper/219:40:202
86132557,11sleep30-21swapper/323:05:023
3248225516,22sleep20-21swapper/200:20:232
3100925514,24sleep10-21swapper/100:15:261
3100925514,24sleep10-21swapper/100:15:261
263725514,24sleep12646-21diskstats19:20:491
263725514,24sleep12646-21diskstats19:20:491
2469625514,24sleep124732-21ntp_states20:30:221
214592556,12sleep30-21swapper/323:45:193
1494925516,17sleep20-21swapper/220:00:112
523825410,25sleep14761-21apt-get22:55:061
3134425415,23sleep231348-21diskstats22:35:122
2290325415,16sleep10-21swapper/122:09:561
2263425413,24sleep322630-21apt-get23:50:083
2263425413,24sleep322630-21apt-get23:50:083
1823325419,16sleep20-21swapper/220:10:162
1823325419,16sleep20-21swapper/220:10:162
170625419,18sleep30-21swapper/321:00:243
1696525413,24sleep116959-21apt-get21:50:091
1509125414,24sleep215140-21irqstats23:25:162
322142535,11sleep10-21swapper/122:35:561
2970725311,25sleep329702-21apt-get22:30:093
1977125316,17sleep20-21swapper/221:59:552
1888525216,18sleep10-21swapper/123:35:321
94492519,16sleep30-21swapper/321:25:163
718225114,21sleep37191-21diskstats23:00:293
324325113,20sleep10-21swapper/122:49:571
3046925113,21sleep330491-21df00:15:113
3046925113,21sleep330491-21df00:15:113
154025119,18sleep20-21swapper/221:00:192
1375825116,18sleep20-21swapper/223:20:192
1375825116,18sleep20-21swapper/223:20:192
692925011,24sleep16982-21users21:15:331
29532508,15sleep20-21swapper/221:05:162
2870625017,17sleep20-21swapper/220:44:572
2795425015,16sleep30-21swapper/320:40:263
1070725012,22sleep210731-21ntp_states19:45:212
3017024914,15sleep10-21swapper/122:30:231
3008624913,20sleep230130-21ntpq22:30:202
228002497,16sleep30-21swapper/320:25:113
228002497,16sleep30-21swapper/320:25:113
2080324810,23sleep30-21swapper/323:44:553
2819924714,19sleep10-21swapper/122:25:141
320102469,20sleep332006-21apt-get00:20:083
2319124310,16sleep30-21swapper/322:10:103
716023710,15sleep10-21swapper/119:35:101
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional