You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-21 - 11:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Mar 21, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
171482193218,1892sleep028956-21kworker/u8:320:33:030
413429816,964sleep1590-1kworker/1:0H23:18:111
221229337,889sleep32258-21ntpq21:28:103
4557289025,853sleep34601-21ntpq23:19:023
25188263320,595sleep125309-21/usr/sbin/munin00:23:191
862249811,473sleep3919-21fschecks_time23:08:083
2391624380,9sleep20-21swapper/219:12:562
1699424065,388sleep317041-21ntpq23:58:103
18547233810,247sleep318588-21ntpq00:03:113
8663227116,198sleep3222-21systemd-journal20:05:243
23186225811,134sleep323344-21sed19:12:493
21026225364,173sleep321075-21latency_hist00:12:483
252192251116,119sleep325239-21kernelversion20:58:103
57192248107,22sleep312-21rcu_sched23:23:023
18964224821,204sleep119048-21latency_hist22:22:481
18964224821,204sleep119048-21latency_hist22:22:481
268352242108,115sleep326857-21modprobe19:22:473
15116224149,178sleep315138-21fschecks_count23:53:053
17418222512,191sleep317545-21latency_hist22:17:493
17418222512,191sleep317545-21latency_hist22:17:493
13271222219,184sleep313418-21latency_hist23:47:493
3796220530,136sleep33938-21cat23:17:493
2758122048,146sleep327664-21wc22:48:173
25217219610,148sleep325366-21date22:42:483
25217219610,148sleep325366-21date22:42:483
30224218913,162sleep330299-21latency_hist19:32:493
1711621898,145sleep316635-21apt-get20:32:583
22394217220,126sleep122482-21fschecks_time22:33:051
22394217220,126sleep122482-21fschecks_time22:33:051
14495216422,120sleep114633-21grep20:23:191
14495216422,120sleep114633-21grep20:23:191
11395216416,81sleep30-21swapper/321:57:583
5448216131,118sleep35530-21cut21:38:163
5448216131,118sleep35530-21cut21:38:163
22195216019,122sleep021768-21/usr/sbin/munin19:08:100
3050521590,29sleep230509-21unixbench_singl21:13:182
2187221597,15sleep10-21swapper/119:08:031
1917721598,135sleep319196-21apt-get22:22:583
1917721598,135sleep319196-21apt-get22:22:583
29862215817,125sleep129938-21sed22:57:481
13460215618,79sleep30-21swapper/322:03:153
13460215618,79sleep30-21swapper/322:03:153
356121559,128sleep03588-21munin-run23:17:460
20818215525,109sleep30-21swapper/320:43:163
20914215129,89sleep30-21swapper/319:03:173
3096121508,127sleep331047-21cat21:17:483
18506214318,102sleep218648-21cat20:37:492
12738214316,110sleep112706-21latency_hist22:02:491
30464213814,111sleep130533-21df19:33:031
29784213872,25sleep3222-21systemd-journal22:57:463
21273213838,35sleep321346-21mailstats22:28:153
359021356,33sleep30-21swapper/319:48:233
3705213319,92sleep23795-21sed23:17:482
145121315,31sleep30-21swapper/321:26:293
12456213010,78sleep30-21swapper/320:18:083
12456213010,78sleep30-21swapper/320:18:083
5565212611,44sleep30-21swapper/319:57:483
8699212412,96sleep08738-21grep21:48:140
8699212412,96sleep08738-21grep21:48:140
3270721247,78sleep3363-21cat19:38:183
19335212431,38sleep321570-21cyclictest20:38:193
2303021235,11sleep30-21swapper/320:52:043
1039521215,10sleep30-21swapper/320:12:463
833221205,11sleep30-21swapper/323:30:163
2937521207,59sleep30-21swapper/321:12:483
2937521207,59sleep30-21swapper/321:12:483
7353211926,81sleep37450-21latency_hist20:02:493
2281621198,64sleep30-21swapper/322:33:163
2281621198,64sleep30-21swapper/322:33:163
1479521197,76sleep30-21swapper/323:52:493
782421185,11sleep30-21swapper/321:45:223
24119211818,83sleep124168-21unin-run00:22:471
1102021185,11sleep30-21swapper/321:56:413
19924211518,77sleep219693-21/usr/sbin/munin00:08:062
2323121138,57sleep30-21swapper/300:18:133
2134921130,11sleep021324-21mailstats22:28:150
27830211119,68sleep127915-21latency_hist21:07:491
21469211139,43sleep121513-21modprobe19:07:471
4088211012,82sleep23585-21apt-get23:17:562
25057211026,74sleep313441-21snmpd19:15:083
21986210826,56sleep022131-21unixbench_singl00:13:170
1003621085,68sleep30-21swapper/323:37:193
1003621085,68sleep30-21swapper/323:37:193
30548210716,72sleep030613-21fschecks_time19:33:060
27765210718,26sleep30-21swapper/321:07:473
23839210710,77sleep00-21swapper/020:53:140
28756210510,74sleep228398-21apt-get19:27:512
2439210516,65sleep12562-21latency_hist19:47:491
28761210329,31sleep30-21swapper/300:37:473
21081210312,76sleep021226-21latency_hist00:12:490
17629210319,18sleep30-21swapper/322:17:573
17629210319,18sleep30-21swapper/322:17:573
1559521035,85sleep30-21swapper/322:10:223
1559521035,85sleep30-21swapper/322:10:223
28883210223,60sleep128398-21apt-get19:27:581
2472421029,52sleep324869-21sed20:57:493
32212210110,76sleep132213-21sshd21:19:361
27584210116,69sleep127592-21awk19:23:131
27584210116,69sleep127592-21awk19:23:131
31739210014,71sleep031832-21grep23:03:040
1529229913,74sleep215309-21kernelversion23:53:082
1236229916,61sleep212451-21fschecks_time20:18:072
1236229916,61sleep212451-21fschecks_time20:18:072
2906829616,64sleep229131-21swap22:53:172
2364529610,70sleep023737-21latency_hist22:37:490
2364529610,70sleep023737-21latency_hist22:37:490
114212965,79sleep30-21swapper/323:39:533
1086529611,71sleep210920-21ps23:38:152
735729512,64sleep07417-21/usr/sbin/munin23:28:070
409329525,29sleep00-21swapper/023:17:580
33892957,16sleep321570-21cyclictest21:33:033
192529514,57sleep11946-21fschecks_count21:28:041
1753129512,67sleep117577-21ntpq20:33:241
896129418,63sleep29075-21timerwakeupswit21:48:202
896129418,63sleep29075-21timerwakeupswit21:48:202
3046629310,67sleep230488-21/usr/sbin/munin22:58:102
2941929318,17sleep00-21swapper/021:12:490
2067129317,63sleep020720-21processes19:03:140
998129212,15sleep00-21swapper/023:36:350
998129212,15sleep00-21swapper/023:36:350
314029217,53sleep12652-21/usr/sbin/munin19:48:171
2877729219,47sleep128924-21grep21:08:171
2877729219,47sleep128924-21grep21:08:171
1563029211,64sleep315337-21/usr/sbin/munin20:28:093
151412929,22sleep00-21swapper/023:53:060
210842910,7sleep10-21swapper/122:28:081
311992908,65sleep30-21swapper/321:18:013
3035728917,40sleep10-21swapper/121:13:181
2126528821,20sleep30-21swapper/300:13:003
152392885,11sleep10-21swapper/123:53:071
3073028732,18sleep321570-21cyclictest19:33:093
1881128718,35sleep00-21swapper/020:38:080
890128617,53sleep28996-21perl23:33:062
890128617,53sleep28996-21perl23:33:062
2440728611,33sleep30-21swapper/300:23:013
1582228615,58sleep215839-21latency_hist22:12:472
1582228615,58sleep215839-21latency_hist22:12:472
1348728621,18sleep00-21swapper/023:48:020
191392848,15sleep20-21swapper/220:38:142
112828413,14sleep312-21rcu_sched19:43:023
1068628427,20sleep310678-21apt-get20:13:033
1068628427,20sleep310678-21apt-get20:13:033
2375028121,39sleep123068-21apt-get19:12:561
2310028119,42sleep223184-21gpgv19:12:482
138112818,55sleep30-21swapper/320:23:003
138112818,55sleep30-21swapper/320:23:003
376628020,22sleep10-21swapper/121:33:131
305628018,46sleep23100-21ntpq19:48:142
1513028012,51sleep015235-21wc22:08:170
1513028012,51sleep015235-21wc22:08:170
61272799,16sleep20-21swapper/219:58:122
2688827929,26sleep10-21swapper/119:22:491
313442785,62sleep30-21swapper/323:02:313
258802789,32sleep30-21swapper/322:43:133
2035527826,29sleep10-21swapper/120:43:061
1583127815,44sleep115894-21/usr/sbin/munin20:28:151
1297727817,47sleep212978-21cpu22:03:022
1297727817,47sleep212978-21cpu22:03:022
2826727718,49sleep30-21swapper/319:26:183
2826727718,49sleep30-21swapper/319:26:183
2673827725,25sleep20-21swapper/222:47:482
133427714,47sleep01387-21interrupts19:43:070
1733927622,35sleep017374-21basename22:17:460
1733927622,35sleep017374-21basename22:17:460
907627518,14sleep30-21swapper/321:48:213
907627518,14sleep30-21swapper/321:48:213
264032757,52sleep30-21swapper/300:28:143
1674827519,33sleep216640-21latency_hist20:32:492
2844927425,27sleep00-21swapper/019:27:490
2844927425,27sleep00-21swapper/019:27:490
1928927426,29sleep00-21swapper/022:23:070
1928927426,29sleep00-21swapper/022:23:070
1428727417,33sleep014258-21latency_hist22:07:490
1428727417,33sleep014258-21latency_hist22:07:490
1352427417,40sleep113538-21idleruntime-cro20:22:461
1352427417,40sleep113538-21idleruntime-cro20:22:461
430927313,44sleep24199-21/usr/sbin/munin19:53:072
359627321,24sleep10-21swapper/123:17:471
3188827316,39sleep231922-21/usr/sbin/munin23:03:062
1461027317,28sleep10-21swapper/122:08:061
1461027317,28sleep10-21swapper/122:08:061
679927212,28sleep00-21swapper/019:58:260
3083927216,47sleep00-21swapper/021:17:130
2141227225,27sleep10-21swapper/100:13:061
3210527123,33sleep30-21swapper/323:03:113
678927014,40sleep26791-21sshd23:25:232
2888127014,47sleep029575-1kworker/0:0H19:27:570
2861227018,25sleep00-21swapper/022:53:070
2595927016,37sleep026008-21cpuspeed00:28:030
226292707,18sleep10-21swapper/120:48:181
107752705,11sleep20-21swapper/220:13:102
107752705,11sleep20-21swapper/220:13:102
960326930,18sleep10-21swapper/121:52:471
960326930,18sleep10-21swapper/121:52:471
940826916,39sleep29467-21latency20:08:092
858126818,22sleep10-21swapper/123:32:491
1099426815,24sleep00-21swapper/023:38:170
817626714,29sleep00-21swapper/020:03:190
56526715,32sleep20-21swapper/219:39:222
2274526714,39sleep20-21swapper/220:48:242
18982676,13sleep00-21swapper/023:11:240
109872675,18sleep00-21swapper/021:56:180
2081026626,26sleep220913-21sed19:03:162
1942926620,29sleep20-21swapper/200:07:462
1737626614,27sleep20-21swapper/223:58:172
1366826617,33sleep10-21swapper/123:48:071
108502668,12sleep10-21swapper/123:38:141
635326516,28sleep06291-21munin-run21:42:490
635326516,28sleep06291-21munin-run21:42:490
4552658,12sleep00-21swapper/021:23:060
3274526515,25sleep132740-21apt-get21:22:591
2434426516,26sleep00-21swapper/022:38:130
2434426516,26sleep00-21swapper/022:38:130
1405726515,21sleep20-21swapper/223:48:162
643426419,22sleep10-21swapper/121:42:501
643426419,22sleep10-21swapper/121:42:501
2886826420,19sleep20-21swapper/200:37:502
2867426410,35sleep028776-21/usr/sbin/munin21:08:160
2641926415,31sleep126460-21proc_pri00:28:161
1872726414,26sleep10-21swapper/120:38:051
1089926414,30sleep010659-21/usr/sbin/munin20:13:120
1089926414,30sleep010659-21/usr/sbin/munin20:13:120
1010926416,30sleep10-21swapper/121:53:081
675226317,25sleep20-21swapper/221:43:032
472126318,18sleep00-21swapper/021:37:480
3223426319,20sleep10-21swapper/119:38:081
1967026321,25sleep10-21swapper/122:23:161
1967026321,25sleep10-21swapper/122:23:161
1633826316,28sleep20-21swapper/222:13:082
1633826316,28sleep20-21swapper/222:13:082
730326218,28sleep17336-21idleruntime-cro20:02:471
3213226220,25sleep00-21swapper/019:38:070
3117526217,27sleep031194-21apt-get21:17:570
2750226216,28sleep10-21swapper/100:33:001
2075226219,17sleep20-21swapper/220:43:152
2026826216,29sleep020262-21apt-get20:43:000
117902627,13sleep00-21swapper/021:58:120
102992627,43sleep00-21swapper/020:11:300
1029726211,36sleep210303-21ntp_states21:53:132
84426118,24sleep20-21swapper/221:23:172
674626118,26sleep06737-21apt-get21:43:010
346526115,29sleep23472-21/usr/sbin/munin21:33:062
2782526114,31sleep127878-21users22:48:201
2368926115,29sleep023223-21munin-node19:12:550
2267626123,18sleep20-21swapper/222:33:102
2267626123,18sleep20-21swapper/222:33:102
1894126111,35sleep018963-21munin-run22:22:470
1894126111,35sleep018963-21munin-run22:22:470
1844226111,20sleep10-21swapper/100:03:091
1205626110,35sleep10-21swapper/123:43:061
733126018,24sleep17402-21/usr/sbin/munin21:43:511
2428926010,12sleep20-21swapper/220:53:202
2294826016,28sleep222974-21fschecks_count00:18:082
220126014,22sleep20-21swapper/221:28:092
1607026014,27sleep016209-21/usr/sbin/munin20:28:180
1441726011,34sleep014494-21grep20:23:180
1441726011,34sleep014494-21grep20:23:180
1203026020,17sleep10-21swapper/120:17:491
1203026020,17sleep10-21swapper/120:17:491
1193726015,26sleep011931-21apt-get23:43:010
661325911,33sleep06626-21vmstat23:23:190
573425912,26sleep20-21swapper/221:38:192
573425912,26sleep20-21swapper/221:38:192
483425918,22sleep10-21swapper/119:53:181
3067125919,17sleep00-21swapper/022:58:160
2749225916,24sleep20-21swapper/219:23:102
2749225916,24sleep20-21swapper/219:23:102
256702598,12sleep20-21swapper/219:18:062
2541325916,26sleep025414-21ntp_states20:58:150
83525812,29sleep20-21swapper/223:08:072
767925810,34sleep17711-21tune2fs20:03:071
626825815,25sleep16342-21sort23:23:161
265992587,39sleep00-21swapper/022:46:370
2646225815,22sleep20-21swapper/200:28:172
2626225818,18sleep10-21swapper/121:02:481
2518325813,30sleep225215-21latency_hist22:42:472
2518325813,30sleep225215-21latency_hist22:42:472
1688725812,23sleep10-21swapper/123:58:081
110125823,19sleep20-21swapper/219:42:572
567625712,28sleep05861-21gpgv19:57:490
2718825719,20sleep20-21swapper/221:03:182
2337025712,23sleep00-21swapper/000:18:180
2313925720,21sleep00-21swapper/020:52:500
2046025712,22sleep10-21swapper/100:08:181
2477425619,22sleep00-21swapper/000:23:100
2466425617,25sleep224381-21/usr/sbin/munin00:23:082
2386225616,23sleep10-21swapper/120:53:151
208625616,17sleep10-21swapper/123:12:501
942625514,20sleep10-21swapper/123:33:171
942625514,20sleep10-21swapper/123:33:171
2828625518,16sleep20-21swapper/222:52:492
2513925522,20sleep11-21systemd22:42:461
2513925522,20sleep11-21systemd22:42:461
1820725514,24sleep218281-21cat22:18:172
1820725514,24sleep218281-21cat22:18:172
1792325517,18sleep00-21swapper/000:02:470
148902554,10sleep20-21swapper/220:25:262
148902554,10sleep20-21swapper/220:25:262
768125418,18sleep20-21swapper/223:28:132
2750125414,22sleep027495-21apt-get00:32:590
266662548,38sleep00-21swapper/019:20:340
2074525414,25sleep220742-21apt-get22:27:582
268025313,23sleep02671-21apt-get19:48:040
2628125316,15sleep20-21swapper/221:02:492
184025313,24sleep01835-21apt-get21:27:590
3055225211,24sleep130558-21ntp_states22:58:131
290682527,33sleep20-21swapper/221:09:272
290682527,33sleep20-21swapper/221:09:272
2823425215,21sleep028262-21munin-run22:52:470
1840825214,22sleep018441-21irqstats00:03:080
1269725213,23sleep012720-21ntp_states20:18:130
1269725213,23sleep012720-21ntp_states20:18:130
2133625112,22sleep221345-21df_inode00:13:032
321502506,32sleep20-21swapper/221:18:482
3053425013,21sleep230541-21df_inode19:33:042
1244025015,19sleep20-21swapper/223:43:152
936724914,20sleep19407-21irqstats20:08:081
805524912,20sleep20-21swapper/220:03:172
3202324912,20sleep10-21swapper/123:03:091
1971424911,20sleep20-21swapper/222:23:172
1971424911,20sleep20-21swapper/222:23:172
1614024913,23sleep116147-21df_inode22:13:031
1614024913,23sleep116147-21df_inode22:13:031
2657724612,21sleep026583-21df_inode21:03:030
1297624510,17sleep10-21swapper/122:03:011
1297624510,17sleep10-21swapper/122:03:011
120412447,16sleep20-21swapper/221:58:182
254252407,15sleep10-21swapper/122:42:581
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional