You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-30 - 14:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Apr 30, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
198332490916,4881sleep13894-21kworker/1:020:33:031
198332490916,4881sleep13894-21kworker/1:020:33:031
805242577,4239sleep324904-21kworker/3:123:28:013
268532107925,1012sleep326905-21unin-run23:00:263
9305273514,703sleep29309-21/usr/sbin/munin20:00:572
9305273514,703sleep29309-21/usr/sbin/munin20:00:572
3184226816,663sleep331882-21ntpq21:10:493
8067242144,330sleep32-21kthreadd21:40:263
25239241612,310sleep325318-21latency_hist19:10:293
25239241612,310sleep325318-21latency_hist19:10:293
4107241313,385sleep14124-21/usr/sbin/munin23:45:351
27543236457,281sleep326866-21kworker/u8:423:05:273
1918523407,321sleep319229-21ntpq20:31:073
1918523407,321sleep319229-21ntpq20:31:073
13522231914,287sleep013592-21/usr/sbin/munin00:25:570
31264231776,92sleep30-21swapper/323:20:253
1180230019,264sleep01268-21/usr/sbin/munin21:15:550
29679227291,86sleep30-21swapper/321:05:283
31174226816,237sleep131224-21unin-run21:10:271
7890225814,229sleep27897-21awk19:56:082
7890225814,229sleep27897-21awk19:56:082
26527225146,187sleep326669-21unixbench_multi19:11:023
26527225146,187sleep326669-21unixbench_multi19:11:023
21199225157,93sleep321220-21kernelversion22:35:473
21645225072,84sleep30-21swapper/320:40:283
15703224533,197sleep315787-21latency20:20:493
29093222036,119sleep329237-21latency_hist23:10:283
881922038,162sleep38834-21awk21:40:543
13676220310,182sleep013441-21snmpd22:00:580
12674220278,79sleep30-21swapper/321:55:573
1005422029,132sleep310089-21awk00:10:253
1486721959,38sleep30-21swapper/300:30:283
1486721959,38sleep30-21swapper/300:30:283
17477219140,114sleep31-21systemd20:25:543
31908218914,159sleep031813-21latency_hist19:30:290
783721787,18sleep00-21swapper/019:56:060
783721787,18sleep00-21swapper/019:56:060
15511217719,138sleep215709-21apt-key00:35:282
15511217719,138sleep215709-21apt-key00:35:282
6729217276,57sleep30-21swapper/323:55:513
149121718,130sleep31502-21vmstat21:15:583
7798216836,95sleep37834-21ntpq19:56:053
7798216836,95sleep37834-21ntpq19:56:053
16296216811,140sleep30-21swapper/322:15:273
14135215548,91sleep314220-21timerandwakeup00:26:123
12602215416,21sleep20-21swapper/221:55:562
1304721499,125sleep313119-21apt-config00:25:273
29688214819,33sleep329708-21modprobe23:15:253
29688214819,33sleep329708-21modprobe23:15:253
26563214711,120sleep326663-21latency_hist20:55:303
12379214768,37sleep30-21swapper/320:10:433
194721449,35sleep30-21swapper/323:35:403
194721449,35sleep30-21swapper/323:35:403
10460214411,116sleep310502-21true20:05:263
10460214411,116sleep310502-21true20:05:263
3237721395,11sleep10-21swapper/121:10:591
1993821395,12sleep30-21swapper/322:29:533
7454213710,115sleep013441-21snmpd23:58:150
5501213528,17sleep30-21swapper/321:30:483
5191213333,62sleep35205-21vmstat19:46:033
10106213218,96sleep10-21swapper/100:10:271
2498213124,86sleep30-21swapper/321:20:513
23430213112,58sleep30-21swapper/320:45:413
156921309,39sleep31613-21ntpq19:35:583
156921309,39sleep31613-21ntpq19:35:583
23327212713,96sleep323346-21apt-get22:45:363
23192212314,95sleep223159-21latency_hist20:45:282
20899212311,98sleep120988-21mailstats20:36:381
9231211913,22sleep00-21swapper/021:40:590
29378211027,46sleep329414-21ntpq19:20:593
16161211022,66sleep016298-21unixbench_singl20:20:580
29444210917,19sleep20-21swapper/219:21:022
28609210912,26sleep128632-21irqstats21:00:501
2514421095,11sleep30-21swapper/322:53:523
12388210816,70sleep012481-21fschecks_time20:10:440
22155210710,81sleep322223-21ls20:40:483
4815210510,58sleep34958-21grep23:45:583
21056210517,63sleep120909-21/usr/sbin/munin22:35:451
10703210427,60sleep30-21swapper/300:15:263
31967210312,75sleep132111-21cat23:25:281
31967210312,75sleep132111-21cat23:25:281
14518210214,19sleep30-21swapper/322:05:463
331621019,51sleep32716-21/usr/sbin/munin19:40:593
31430210021,43sleep30-21swapper/319:26:043
23998210013,71sleep224065-21ls20:45:562
1191421009,56sleep311929-21awk00:15:583
1167621005,11sleep30-21swapper/321:53:513
610929923,38sleep30-21swapper/323:55:273
2049729915,51sleep30-21swapper/322:33:533
188429921,52sleep22021-21unixbench_multi19:36:052
188429921,52sleep22021-21unixbench_multi19:36:052
2841029825,40sleep313441-21snmpd19:16:553
2841029825,40sleep313441-21snmpd19:16:553
213029811,74sleep12191-21cat23:35:461
213029811,74sleep12191-21cat23:35:461
1848329813,65sleep118539-21latency_hist22:25:271
876029716,62sleep28277-21apt-get00:05:362
49162975,36sleep30-21swapper/321:30:043
36302960,10sleep20-21swapper/221:25:292
309882965,35sleep30-21swapper/321:08:023
2812129615,68sleep028125-21latency_hist21:00:280
139182965,30sleep30-21swapper/322:04:043
132892969,73sleep113291-21sshd20:12:031
2910229517,60sleep129110-21fschecks_count19:20:531
961429418,57sleep29622-21modprobe21:45:262
416929417,60sleep14253-21cut21:25:541
181222945,33sleep30-21swapper/322:21:253
1034629215,57sleep010357-21ntp_states21:45:520
1296929118,53sleep112978-21modprobe00:25:251
323432909,65sleep30-21swapper/319:30:533
323432909,65sleep30-21swapper/319:30:533
317092899,15sleep00-21swapper/021:10:460
117442895,23sleep30-21swapper/320:07:043
1079928911,63sleep110722-21latency_hist00:15:281
84872889,66sleep38471-21munin-run00:05:283
788228812,61sleep17889-21ntp_states19:56:071
788228812,61sleep17889-21ntp_states19:56:071
3231128811,60sleep232398-21fschecks_time23:25:462
2747528813,59sleep227500-21sh19:15:412
2747528813,59sleep227500-21sh19:15:412
2467728815,60sleep224691-21latency_hist22:50:282
1396028613,35sleep30-21swapper/320:15:443
1480928510,60sleep214866-21cut00:30:272
1121628514,55sleep111258-21wc20:05:571
1177628416,46sleep210995-21/usr/sbin/munin00:15:572
2755028312,55sleep127689-21awk20:56:011
2736028312,20sleep30-21swapper/320:55:593
165342839,34sleep30-21swapper/322:15:393
165342839,34sleep30-21swapper/322:15:393
389628224,25sleep10-21swapper/123:45:271
3194228130,17sleep331952-21munin-run23:25:263
3194228130,17sleep331952-21munin-run23:25:263
2558328117,19sleep20-21swapper/222:55:412
2360728133,31sleep20-21swapper/219:05:272
10552807,56sleep20-21swapper/221:15:532
63352799,13sleep00-21swapper/021:33:240
2677427924,27sleep20-21swapper/220:55:382
2469727915,48sleep024742-21systemd-run20:50:260
2225527918,46sleep122300-21awk20:40:501
2002027917,47sleep220060-21awk20:35:262
2002027917,47sleep220060-21awk20:35:262
1661427915,42sleep00-21swapper/022:15:450
1661427915,42sleep00-21swapper/022:15:450
1467827919,42sleep214707-21/usr/sbin/munin22:05:492
695827815,46sleep17037-21threads23:55:571
384527811,32sleep30-21swapper/323:45:253
429427616,31sleep00-21swapper/019:45:440
2640927615,45sleep126504-21apt-key20:55:281
2620427611,51sleep026205-21sendmail-mta20:54:550
2479327616,40sleep024807-21vmstat19:06:100
2479327616,40sleep024807-21vmstat19:06:100
2896627525,24sleep20-21swapper/221:00:592
2800527523,28sleep00-21swapper/019:16:000
2800527523,28sleep00-21swapper/019:16:000
2452827517,44sleep224648-21ls19:06:082
2452827517,44sleep224648-21ls19:06:082
2298627510,22sleep323191-21apt-config22:45:283
62822745,30sleep30-21swapper/319:50:563
62822745,30sleep30-21swapper/319:50:563
181042745,59sleep00-21swapper/022:21:100
1580227414,32sleep10-21swapper/120:20:511
1297927422,26sleep20-21swapper/200:25:262
1084327414,30sleep00-21swapper/020:05:470
100012746,45sleep20-21swapper/221:45:462
657127317,43sleep26674-21latency_hist21:35:282
3146227313,40sleep231529-21/usr/sbin/munin21:10:402
3122527320,27sleep20-21swapper/221:10:282
293972738,55sleep00-21swapper/023:11:460
293972738,55sleep00-21swapper/023:11:460
1682527315,28sleep10-21swapper/122:15:501
1682527315,28sleep10-21swapper/122:15:501
1050427310,20sleep00-21swapper/020:05:270
1050427310,20sleep00-21swapper/020:05:270
241822720,7sleep00-21swapper/022:46:130
1513827215,41sleep015155-21munin-plugin-st20:20:260
2826527114,14sleep10-21swapper/123:05:541
541527024,26sleep10-21swapper/121:30:461
2956127016,37sleep029627-21sed19:21:040
2824427013,28sleep00-21swapper/023:05:530
1905227019,29sleep00-21swapper/020:31:040
1905227019,29sleep00-21swapper/020:31:040
153742706,55sleep00-21swapper/022:07:530
12492705,30sleep30-21swapper/323:30:313
348126914,22sleep10-21swapper/121:25:291
2635226915,36sleep126446-21grep19:11:001
2635226915,36sleep126446-21grep19:11:001
2237726922,28sleep322384-21forks19:00:453
1540026910,41sleep015467-21http00:35:260
1540026910,41sleep015467-21http00:35:260
1166126924,28sleep10-21swapper/100:15:561
2493426819,22sleep20-21swapper/222:51:092
2234726818,25sleep00-21swapper/020:40:530
1827926825,24sleep00-21swapper/020:30:270
110892685,28sleep30-21swapper/321:49:303
766926720,22sleep20-21swapper/200:00:272
603426715,31sleep26072-40mandb23:55:262
460226713,30sleep00-21swapper/023:45:550
2842926714,40sleep028435-21df_inode21:00:440
2069726717,33sleep020815-21date22:35:270
3052526620,21sleep00-21swapper/021:05:570
2976726617,35sleep129835-21latency_hist23:15:271
2976726617,35sleep129835-21latency_hist23:15:271
268062668,13sleep00-21swapper/019:11:390
268062668,13sleep00-21swapper/019:11:390
251152664,10sleep00-21swapper/022:53:290
1836826611,39sleep118455-21apt-key20:30:281
3029326518,26sleep20-21swapper/223:15:482
1058226517,19sleep10-21swapper/121:45:571
751826418,24sleep20-21swapper/221:35:562
2644826417,26sleep20-21swapper/219:11:012
2644826417,26sleep20-21swapper/219:11:012
203826410,36sleep02129-21perl23:35:450
203826410,36sleep02129-21perl23:35:450
138852646,13sleep10-21swapper/120:15:421
1179926416,29sleep011820-21munin-run21:55:250
106212647,13sleep00-21swapper/000:14:220
394926319,19sleep20-21swapper/223:45:282
2416326310,27sleep10-21swapper/119:05:591
2416326310,27sleep10-21swapper/119:05:591
2723026217,24sleep00-21swapper/023:02:010
1407626223,21sleep20-21swapper/222:05:292
855326113,27sleep08277-21apt-get00:05:290
6622616,10sleep20-21swapper/219:32:462
6622616,10sleep20-21swapper/219:32:462
2474326111,35sleep124777-21sed20:50:271
2378526114,27sleep10-21swapper/122:46:071
226572617,11sleep00-21swapper/022:43:110
14272615,10sleep00-21swapper/023:32:460
366026015,28sleep03679-21apt-get21:25:360
310926017,28sleep13120-21netstat19:40:521
2561226016,24sleep20-21swapper/220:50:572
2229526023,23sleep022332-21grep19:00:420
1677726012,32sleep216913-21cat20:25:282
137322605,43sleep10-21swapper/122:01:401
3204725916,25sleep032066-21apt-get19:30:380
3204725916,25sleep032066-21apt-get19:30:380
274125914,20sleep20-21swapper/219:40:412
143702597,27sleep20-21swapper/220:15:552
3027025821,18sleep10-21swapper/123:15:471
2534525817,17sleep10-21swapper/122:55:281
2355425818,19sleep10-21swapper/119:05:261
234525813,22sleep00-21swapper/021:20:480
2283025817,25sleep10-21swapper/122:45:261
1405425813,30sleep114124-21sensors00:26:101
1313825813,23sleep00-21swapper/000:25:280
1239025812,27sleep012479-21load21:55:500
859525719,19sleep10-21swapper/121:40:481
492825716,23sleep10-21swapper/119:46:011
221572576,16sleep00-21swapper/022:40:000
2002325714,29sleep020006-21latency_hist22:30:270
1229425715,27sleep212290-21apt-get20:10:382
3040625616,24sleep030453-21ntpq23:15:500
147025619,19sleep10-21swapper/119:35:561
147025619,19sleep10-21swapper/119:35:561
741825512,26sleep17493-21timerandwakeup21:35:551
3245725522,17sleep00-21swapper/023:25:480
3119225516,23sleep131309-21cat19:26:021
190125514,26sleep12010-21cut21:20:281
1621825515,22sleep10-21swapper/122:15:251
651625417,23sleep06641-21cat19:51:010
651625417,23sleep06641-21cat19:51:010
2558225413,24sleep125576-21apt-get22:55:401
702425315,22sleep07076-21irqstats21:35:460
583925312,23sleep25830-21apt-get19:50:432
583925312,23sleep25830-21apt-get19:50:432
44142537,14sleep20-21swapper/219:45:502
2615725314,18sleep00-21swapper/022:55:550
2084025315,24sleep220847-21ntp_states20:36:352
1538525314,22sleep10-21swapper/122:08:021
37592528,30sleep23767-21diskstats21:25:432
3248025210,19sleep10-21swapper/123:25:491
301525217,19sleep00-21swapper/019:40:510
273362526,10sleep10-21swapper/123:03:231
2289025213,25sleep222985-21sed22:45:272
1854025215,21sleep218683-21kworker/u8:122:25:282
174372528,32sleep217474-21ps20:25:532
161625211,18sleep00-21swapper/019:35:590
161625211,18sleep00-21swapper/019:35:590
1287825214,19sleep10-21swapper/121:55:591
1075225213,22sleep210748-21apt-get20:05:412
875825112,23sleep08277-21apt-get00:05:340
5622514,36sleep10-21swapper/119:31:311
5622514,36sleep10-21swapper/119:31:311
3060125113,24sleep230607-21df_inode19:25:472
219225114,22sleep22217-21irqstats23:35:472
219225114,22sleep22217-21irqstats23:35:472
2141725116,19sleep20-21swapper/222:35:542
596125013,20sleep10-21swapper/119:50:501
596125013,20sleep10-21swapper/119:50:501
58782505,34sleep00-21swapper/023:53:260
1879425011,22sleep118800-21df_inode22:25:431
163325011,21sleep11704-21systemd-run23:35:261
2742824912,17sleep10-21swapper/119:15:401
2742824912,17sleep10-21swapper/119:15:401
830524811,18sleep10-21swapper/100:05:261
2120224710,22sleep021294-21users20:36:410
1700224716,17sleep20-21swapper/222:15:552
1700224716,17sleep20-21swapper/222:15:552
1187324716,16sleep20-21swapper/221:55:272
206172468,17sleep20-21swapper/222:35:252
1546924611,19sleep10-21swapper/100:35:271
1546924611,19sleep10-21swapper/100:35:271
927424511,20sleep19286-21df_inode20:00:481
927424511,20sleep19286-21df_inode20:00:481
83922458,16sleep20-21swapper/200:05:272
3195424412,18sleep00-21swapper/023:25:270
3195424412,18sleep00-21swapper/023:25:270
278582448,20sleep227862-21df_inode23:05:422
1183024410,15sleep10-21swapper/121:55:261
160924315,16sleep20-21swapper/223:35:062
114472437,17sleep00-21swapper/000:15:510
60082414,27sleep10-21swapper/123:55:041
269962375,13sleep20-21swapper/223:00:292
162812377,16sleep20-21swapper/222:15:262
381523611,14sleep00-21swapper/023:45:020
314922366,11sleep20-21swapper/223:20:292
314922366,11sleep20-21swapper/223:20:292
158652359,14sleep00-21swapper/022:11:000
102192334,11sleep20-21swapper/200:10:282
293182318,12sleep20-21swapper/223:10:472
293182318,12sleep20-21swapper/223:10:472
182412318,13sleep20-21swapper/222:22:572
35472305,12sleep20-21swapper/223:41:372
204172308,11sleep10-21swapper/122:32:501
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional