You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-14 - 00:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Mon Jul 13, 2026 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
79432250910,2487sleep22982-21kworker/2:210:33:312
29738211189,1094sleep129215-21kworker/u8:309:58:591
30142253712,509sleep030165-21users08:14:230
6839240770,228sleep16900-21grep12:09:161
16422237019,331sleep116461-21fschecks_count07:34:071
8646230052,132sleep10-21swapper/110:34:271
3035227622,235sleep23028-21apt-get08:34:032
26166223317,59sleep126330-21latency_hist09:48:491
9347223065,26sleep10-21swapper/108:54:081
31542222817,192sleep331595-21/usr/sbin/munin10:04:143
13650221452,122sleep10-21swapper/110:49:001
852122119,189sleep08570-21awk10:34:180
24284220617,37sleep124309-21munin-run11:23:471
2700219530,142sleep12722-21modprobe11:58:471
401218714,151sleep132371-21/usr/sbin/munin08:24:171
401218714,151sleep132371-21/usr/sbin/munin08:24:171
18121218710,25sleep118140-21apt-get07:38:591
14594218216,150sleep214527-21latency_hist07:28:512
12628218131,64sleep10-21swapper/109:04:091
2032421788,154sleep120454-21sed07:44:201
837521719,150sleep18496-21sed08:49:201
15594216810,25sleep10-21swapper/110:54:131
2733521656,149sleep1364-21snmpd11:33:121
17661216514,120sleep10-21swapper/109:19:191
2814621599,15sleep00-21swapper/011:34:280
8812215817,118sleep10-21swapper/112:14:211
2797215810,130sleep12925-21latency_hist08:33:491
2037121589,138sleep120453-21grep11:09:141
2037121589,138sleep120453-21grep11:09:141
8744215623,113sleep3225-21systemd-journal07:09:183
283572156112,24sleep10-21swapper/109:54:151
21317215516,15sleep30-21swapper/307:49:063
21317215516,15sleep30-21swapper/307:49:063
10545215412,128sleep110597-21if_eth010:39:231
22705215210,128sleep122735-21munin-plugin-st11:18:471
4795214412,117sleep022175-1kworker/0:0H12:04:030
26492214311,116sleep226568-21cut09:49:072
12367214310,118sleep112518-21date12:28:501
8122214014,75sleep10-21swapper/110:33:501
15263214012,111sleep115339-21latency_hist09:13:501
15263214012,111sleep115339-21latency_hist09:13:501
1497421378,13sleep00-21swapper/007:29:090
10923213310,63sleep10-21swapper/108:59:091
10923213310,63sleep10-21swapper/108:59:091
6898213215,90sleep37050-21sed08:44:213
12185213214,100sleep10-21swapper/110:44:071
12185213214,100sleep10-21swapper/110:44:071
12154213218,93sleep20-21swapper/209:03:502
12154213218,93sleep20-21swapper/209:03:502
21222212913,60sleep10-21swapper/107:49:001
21222212913,60sleep10-21swapper/107:49:001
409421279,103sleep14176-21mailstats10:19:171
409421279,103sleep14176-21mailstats10:19:171
4783212519,85sleep34301-21apt-get12:03:583
3738212518,86sleep03884-21sed11:59:190
3229621246,63sleep10-21swapper/110:05:041
11615212417,23sleep20-21swapper/210:43:482
25926212311,96sleep126064-21latency_hist08:03:501
1956121225,78sleep10-21swapper/109:24:321
5681212113,92sleep0225-21systemd-journal08:41:270
1550121219,56sleep10-21swapper/109:14:021
990321199,63sleep110072-21sed07:13:561
28930211932,66sleep328953-21modprobe11:38:473
2137121175,19sleep10-21swapper/109:32:551
6581211619,80sleep26644-21/usr/sbin/munin07:04:052
6581211619,80sleep26644-21/usr/sbin/munin07:04:052
23953211654,44sleep124081-21timerwakeupswit09:39:211
23953211654,44sleep124081-21timerwakeupswit09:39:211
1860821158,63sleep10-21swapper/111:04:101
2294521147,53sleep10-21swapper/107:53:531
18275211427,32sleep20-21swapper/207:39:122
11053211414,80sleep00-21swapper/012:24:010
24563211219,77sleep324593-21munin-run09:43:473
24563211219,77sleep324593-21munin-run09:43:473
604821115,93sleep10-21swapper/112:08:281
12478211027,56sleep312567-21fschecks_time09:04:073
1531921085,91sleep10-21swapper/112:37:341
8485210710,22sleep20-21swapper/210:33:592
6272210627,58sleep06528-21cp07:03:510
6272210627,58sleep06528-21cp07:03:510
26408210514,74sleep326478-21df09:49:043
347210414,69sleep0377-21/usr/sbin/munin11:49:170
24941210411,41sleep124966-21awk11:24:121
24941210411,41sleep124966-21awk11:24:121
5363210315,71sleep25413-21/usr/sbin/munin10:24:102
28422210314,76sleep028475-21processes09:54:180
6461210212,75sleep16570-21sed10:28:491
18312210210,76sleep218331-21apt-get11:03:582
1748921020,5sleep00-21swapper/007:37:280
6338210130,44sleep06357-21apt-get12:08:590
9469210023,52sleep29529-21latency_hist07:13:492
815621006,47sleep10-21swapper/107:09:001
1031421008,77sleep022175-1kworker/0:0H10:39:020
303162995,59sleep10-21swapper/111:41:191
2377129919,52sleep20-21swapper/209:39:182
2377129919,52sleep20-21swapper/209:39:182
120712999,18sleep30-21swapper/310:43:573
120712999,18sleep30-21swapper/310:43:573
2170829822,59sleep221750-21proc_pri07:49:182
2170829822,59sleep221750-21proc_pri07:49:182
1557029814,66sleep015593-21/usr/sbin/munin10:54:120
2193829714,67sleep222017-21sendmail_mailtr11:14:202
2953429633,46sleep229239-21/usr/sbin/munin08:14:122
490929513,69sleep24961-21cut12:04:132
1618429522,53sleep016322-21grep09:14:220
1325029557,26sleep17922-21cyclictest07:24:061
369429430,34sleep30-21swapper/310:19:073
369429430,34sleep30-21swapper/310:19:073
308482949,66sleep10-21swapper/108:19:041
308482949,66sleep10-21swapper/108:19:041
2985929415,62sleep329749-21/usr/sbin/munin09:59:133
361529317,18sleep10-21swapper/108:34:181
2747529317,55sleep027562-21grep08:08:500
2006329117,55sleep219673-21/usr/sbin/munin07:44:132
18102918,11sleep30-21swapper/311:54:133
428529011,26sleep14314-21munin-run12:03:481
221302905,71sleep10-21swapper/109:34:121
230052899,14sleep00-21swapper/007:54:000
1802428912,62sleep218025-21sshd11:03:302
3197628836,41sleep10-21swapper/111:47:451
1610028810,20sleep116170-21cut07:33:501
2689428420,44sleep327034-21grep08:04:203
1328728414,50sleep313419-21tail12:29:213
2151328218,17sleep00-21swapper/007:49:120
2151328218,17sleep00-21swapper/007:49:120
1386528212,32sleep10-21swapper/112:33:461
171592819,15sleep10-21swapper/110:59:131
748627915,49sleep07637-21cat08:48:500
285122797,14sleep10-21swapper/108:09:191
95882788,12sleep10-21swapper/112:19:051
1994127816,48sleep319996-21latency_hist09:28:493
49732777,13sleep10-21swapper/108:39:101
323482777,14sleep10-21swapper/111:49:021
2181927727,31sleep30-21swapper/309:34:063
202427720,19sleep20-21swapper/210:14:052
1706227721,31sleep217081-21apt-get09:19:002
1218827727,30sleep30-21swapper/309:03:513
4192768,52sleep10-21swapper/110:09:041
935827523,29sleep20-21swapper/208:54:092
89227524,24sleep30-21swapper/310:09:183
328427526,26sleep20-21swapper/211:59:082
24772757,13sleep10-21swapper/110:14:171
290102749,45sleep10-21swapper/111:38:491
2278427418,42sleep322791-21latency_hist11:18:493
2199927421,38sleep222018-21tr09:34:092
1940527418,32sleep20-21swapper/209:24:212
65302738,13sleep10-21swapper/107:03:521
65302738,13sleep10-21swapper/107:03:521
2973727318,37sleep029228-21apt-get09:58:580
2768627314,42sleep027690-21find08:08:580
2279527318,26sleep00-21swapper/011:18:500
2045527310,12sleep00-21swapper/009:29:110
1223127319,12sleep10-21swapper/107:19:171
710227218,11sleep10-21swapper/108:44:471
3047727219,11sleep10-21swapper/108:18:111
2743327226,22sleep227570-21cat11:33:492
2742227219,31sleep227437-21idleruntime-cro08:08:482
2120027214,34sleep00-21swapper/011:13:490
2120027214,34sleep00-21swapper/011:13:490
165227215,42sleep01710-21irqstats11:54:100
1196927214,22sleep20-21swapper/207:19:102
1062727225,24sleep30-21swapper/308:58:503
700827119,26sleep30-21swapper/307:04:383
700827119,26sleep30-21swapper/307:04:383
2025227112,17sleep20-21swapper/209:29:052
1874127119,29sleep00-21swapper/011:04:130
1766627120,27sleep30-21swapper/307:38:493
128752719,15sleep00-21swapper/009:04:160
2739527018,10sleep10-21swapper/108:08:301
2465827020,29sleep00-21swapper/011:24:070
2465827020,29sleep00-21swapper/011:24:070
1839727022,25sleep00-21swapper/009:23:500
152827015,24sleep20-21swapper/211:54:082
866326917,25sleep00-21swapper/012:14:200
2970226918,27sleep20-21swapper/211:39:152
264062698,14sleep10-21swapper/109:49:021
2059526923,27sleep30-21swapper/307:44:223
1527926913,40sleep215286-21diskstats10:54:052
1486726916,36sleep213931-21/usr/sbin/munin09:09:232
1486726916,36sleep213931-21/usr/sbin/munin09:09:232
139492699,42sleep00-21swapper/009:09:010
139492699,42sleep00-21swapper/009:09:010
1138226917,31sleep011407-21munin-run07:18:480
3111826812,37sleep00-21swapper/008:19:110
3111826812,37sleep00-21swapper/008:19:110
2670526826,20sleep30-21swapper/311:29:223
996826712,41sleep310005-21ps12:19:153
959326720,23sleep20-21swapper/212:19:062
167042677,14sleep20-21swapper/207:34:122
1557926715,37sleep215586-21diskstats09:14:072
1528726717,21sleep30-21swapper/310:54:063
1496426723,22sleep20-21swapper/210:53:492
799626615,36sleep28005-21diskstats12:14:062
796726620,27sleep38026-21apt-config10:33:483
767326619,32sleep27811-21latency_hist12:13:502
364226615,30sleep30-21swapper/311:59:173
302152668,12sleep30-21swapper/308:14:543
2461026621,21sleep00-21swapper/007:59:050
1439426622,20sleep00-21swapper/010:49:190
586426520,22sleep20-21swapper/208:43:472
2928626522,25sleep329375-21store09:58:483
2615426517,22sleep00-21swapper/009:48:480
2483326515,28sleep30-21swapper/311:24:103
2483326515,28sleep30-21swapper/311:24:103
1238826510,34sleep00-21swapper/010:44:190
1238826510,34sleep00-21swapper/010:44:190
3210826418,22sleep00-21swapper/011:48:490
24002646,47sleep30-21swapper/308:29:233
1552726417,23sleep00-21swapper/012:38:500
1392526415,33sleep314018-21load10:49:103
127826416,25sleep20-21swapper/208:28:502
127826416,25sleep20-21swapper/208:28:502
1146126415,32sleep211567-21latency_hist07:18:502
3079826317,28sleep230794-21apt-get11:44:032
1577426314,32sleep315831-21/usr/sbin/munin09:14:123
1391826323,23sleep00-21swapper/012:33:500
1102126314,21sleep30-21swapper/308:59:113
271052627,45sleep00-21swapper/011:30:120
2694026218,21sleep00-21swapper/009:49:170
1959826216,29sleep319646-21/usr/sbin/munin11:08:483
817426119,22sleep20-21swapper/208:49:172
741226118,26sleep27441-21/usr/sbin/munin08:48:482
317862616,12sleep30-21swapper/308:19:433
317862616,12sleep30-21swapper/308:19:433
3004626115,26sleep20-21swapper/209:59:202
171462618,12sleep00-21swapper/010:59:120
92762605,10sleep00-21swapper/007:11:580
837526014,29sleep38381-21ntp_states12:14:153
2468126015,27sleep224688-21df_inode07:59:072
158326024,19sleep00-21swapper/008:29:080
1254826017,24sleep012601-21users07:19:200
119326018,25sleep31179-21/usr/sbin/munin11:53:483
1081126014,31sleep010829-21/usr/sbin/munin08:58:590
1081126014,31sleep010829-21/usr/sbin/munin08:58:590
61525917,25sleep0777-21tail08:24:200
61525917,25sleep0777-21tail08:24:200
3132225919,23sleep20-21swapper/208:19:172
3132225919,23sleep20-21swapper/208:19:172
2311025915,27sleep323117-21diskstats07:54:073
1945925917,20sleep30-21swapper/307:43:503
1739625924,19sleep30-21swapper/310:59:193
88525815,28sleep2886-21ntp_states10:09:172
802825816,27sleep08121-21apt-config10:33:490
2920825814,30sleep229285-21apt-config09:58:472
2298725821,21sleep00-21swapper/009:38:470
1169925818,21sleep30-21swapper/307:19:033
3246925710,36sleep20-21swapper/210:07:192
240025711,31sleep30-21swapper/310:14:143
1038725714,23sleep30-21swapper/307:14:153
2388325615,17sleep00-21swapper/009:39:200
2388325615,17sleep00-21swapper/009:39:200
2114725617,23sleep221173-21munin-run11:13:472
2114725617,23sleep221173-21munin-run11:13:472
1754825618,21sleep30-21swapper/309:19:173
1541425625,17sleep20-21swapper/212:38:482
1025525614,24sleep39797-21apt-get10:38:573
814325514,25sleep08166-21ntp_states08:49:150
397725511,28sleep04016-21ntpq10:19:120
397725511,28sleep04016-21ntpq10:19:120
3247525516,22sleep332479-21diskstats08:24:073
3247525516,22sleep332479-21diskstats08:24:073
2587525514,23sleep325901-21munin-run08:03:483
2464225514,25sleep224649-21df_inode11:24:052
2464225514,25sleep224649-21df_inode11:24:052
1822225516,18sleep00-21swapper/007:39:060
1506225518,23sleep20-21swapper/207:29:112
460625413,24sleep34601-21apt-get08:39:003
429025419,19sleep34362-21systemd-run08:38:483
3140525410,26sleep331497-21timerandwakeup11:44:203
1366325410,18sleep30-21swapper/307:24:163
1210625414,24sleep212100-21apt-get10:44:042
1210625414,24sleep212100-21apt-get10:44:042
957825311,15sleep00-21swapper/012:19:040
63032538,36sleep30-21swapper/310:27:263
513325312,24sleep05145-21apt-get10:24:000
328625313,25sleep03312-21munin-run10:18:470
2770925311,26sleep227706-21apt-get08:09:002
2485725313,24sleep224851-21apt-get09:44:022
1513225313,22sleep315160-21memory07:29:123
2395225221,16sleep00-21swapper/011:19:310
1114025220,17sleep20-21swapper/212:24:072
32425114,21sleep2335-21ntp_states11:49:152
1863625111,24sleep018630-21apt-get09:24:020
902125017,18sleep30-21swapper/308:53:503
851925013,22sleep38482-21apt-get10:34:163
323052504,37sleep00-21swapper/010:05:110
310425011,14sleep00-21swapper/008:34:050
2149025012,20sleep30-21swapper/309:33:503
1370225012,14sleep00-21swapper/007:24:170
125792506,32sleep20-21swapper/212:28:592
663324916,13sleep30-21swapper/312:09:103
2942424913,17sleep00-21swapper/011:39:080
1440924912,21sleep314415-21ntp_states09:09:163
1440924912,21sleep314415-21ntp_states09:09:163
933924816,16sleep00-21swapper/008:54:070
230012488,22sleep20-21swapper/207:53:582
1186324814,17sleep30-21swapper/312:24:203
49124710,19sleep00-21swapper/010:09:070
2922324712,19sleep329242-21apt-get11:38:593
2300024711,21sleep223018-21apt-get11:19:002
173124713,20sleep21758-21irqstats08:29:102
368924612,19sleep23693-21diskstats10:19:062
368924612,19sleep23693-21diskstats10:19:062
2778824611,20sleep327687-21/usr/sbin/munin08:09:053
1548824613,20sleep30-21swapper/312:38:493
285232456,14sleep20-21swapper/211:34:342
2611224510,12sleep30-21swapper/309:48:473
2433424514,15sleep30-21swapper/311:23:493
2118724514,18sleep321195-21systemd-run11:13:483
2118724514,18sleep321195-21systemd-run11:13:483
1969724512,17sleep00-21swapper/007:44:030
123922418,17sleep20-21swapper/209:04:022
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional