You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-06 - 18:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Dec 06, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
14291252875,5269sleep0378-21kworker/0:007:58:270
306762440616,4377sleep318503-21kworker/3:010:33:323
27757271711,690sleep127823-21fschecks_time08:43:141
22013266810,642sleep322023-21/usr/sbin/munin10:08:083
28111256415,534sleep228113-21sshd12:09:162
18577253921,499sleep318580-21true09:57:503
9533224945,101sleep19578-21latency_hist11:12:511
22954224751,183sleep123018-21sendmail_mailst11:53:201
32183224213,211sleep232260-21timerandwakeup07:13:222
32724222916,23sleep10-21swapper/108:58:161
24828222936,143sleep14257-21kworker/u8:010:17:511
11086222121,133sleep111211-21latency_hist11:17:511
17738222045,60sleep117741-21cron08:12:491
30046221456,61sleep10-21swapper/112:17:511
27938219713,171sleep027996-21latency_hist10:27:520
30309219352,58sleep10-21swapper/110:33:221
3125421789,79sleep131257-21cron07:12:491
1383921718,130sleep113879-21/usr/sbin/munin07:58:201
22918216831,117sleep10-21swapper/108:28:121
2634216618,129sleep22773-21tail10:48:252
20379216513,133sleep10-21swapper/110:03:051
8403216110,98sleep18483-21grep09:23:241
30766216130,93sleep10-21swapper/108:53:071
57312159104,37sleep10-21swapper/110:58:211
1082521585,142sleep00-21swapper/011:15:080
11427215424,50sleep10-21swapper/111:18:241
11427215424,50sleep10-21swapper/111:18:241
2555215091,18sleep10-21swapper/110:48:241
26833214725,99sleep026865-21munin-run12:07:490
8001214622,105sleep38043-21fschecks_count09:23:143
29651214614,24sleep10-21swapper/108:48:191
346421455,34sleep10-21swapper/107:27:351
27815214543,33sleep10-21swapper/107:02:511
27815214543,33sleep10-21swapper/107:02:511
2490921410,8sleep20-21swapper/210:17:512
534121398,95sleep15352-21latency_hist07:32:521
10323213952,54sleep110324-21diskstats07:48:101
2572021377,13sleep10-21swapper/108:36:561
2572021377,13sleep10-21swapper/108:36:561
23088213714,109sleep323219-21expr11:53:223
21132213714,61sleep10-21swapper/108:23:031
27778213533,75sleep027814-21wc07:02:500
27778213533,75sleep027814-21wc07:02:500
3529213422,90sleep00-21swapper/010:53:070
30768213411,23sleep330835-21df08:53:103
2385221345,12sleep10-21swapper/111:57:521
12607213221,95sleep312681-21cat11:22:503
12607213221,95sleep312681-21cat11:22:503
679213116,98sleep0828-21latency_hist12:27:510
678121309,18sleep10-21swapper/111:03:101
2911321298,109sleep15666-21snmpd10:28:411
2692212914,100sleep12703-21modprobe09:07:491
6910212731,81sleep17059-21date07:37:521
1805821248,99sleep10-21swapper/108:13:061
19103212348,27sleep10-21swapper/111:42:501
295521229,20sleep10-21swapper/109:08:011
295521229,20sleep10-21swapper/109:08:011
11038212221,82sleep10-21swapper/109:33:051
24519212112,94sleep023997-21apt-get08:33:030
24519212112,94sleep023997-21apt-get08:33:030
1728212185,16sleep10-21swapper/110:47:511
817721205,33sleep10-21swapper/107:39:581
26505211921,64sleep10-21swapper/112:03:381
947211815,31sleep31030-21sed07:18:123
947211815,31sleep31030-21sed07:18:123
12183211828,53sleep10-21swapper/107:53:151
40721165,35sleep10-21swapper/112:25:021
20069211621,54sleep10-21swapper/108:18:191
16292211625,77sleep10-21swapper/109:48:191
14764211637,23sleep10-21swapper/111:28:211
14764211637,23sleep10-21swapper/111:28:211
938211549,17sleep10-21swapper/112:28:211
938211549,17sleep10-21swapper/112:28:211
18590211318,22sleep10-21swapper/109:57:521
31666211210,83sleep331720-21fschecks_count07:13:113
29405211211,62sleep10-21swapper/107:07:521
22447211119,40sleep00-21swapper/011:53:030
3148421105,94sleep10-21swapper/112:22:151
29084211013,16sleep30-21swapper/312:13:133
18524211010,16sleep30-21swapper/308:13:203
479421099,83sleep10-21swapper/110:57:481
9222210815,75sleep09292-21latency_hist09:27:510
1415321087,91sleep15666-21snmpd11:24:581
1415321087,91sleep15666-21snmpd11:24:581
17818210712,78sleep317881-21cut08:12:513
29437210612,30sleep329460-21irqstats08:48:133
3980210515,70sleep23495-21apt-get07:28:002
31755210515,23sleep331761-21ntp_states10:38:163
29358210517,71sleep029404-21cut07:07:510
28777210520,62sleep328917-21grep07:03:223
17743210513,77sleep217817-21unin-run08:12:502
14432210513,75sleep214114-21/usr/sbin/munin09:43:162
27340210425,62sleep027368-21grep08:42:490
26556210432,51sleep226649-21grep06:58:132
26556210432,51sleep226649-21grep06:58:132
2624221045,66sleep10-21swapper/110:21:431
1339121029,56sleep113491-21users09:38:221
1339121029,56sleep113491-21users09:38:221
1161210261,20sleep10-21swapper/107:18:181
1161210261,20sleep10-21swapper/107:18:181
30854210116,59sleep330249-21/usr/sbin/munin12:18:213
27925210119,70sleep227926-21/usr/sbin/munin10:27:492
19436210119,28sleep30-21swapper/311:43:093
12530210020,57sleep012674-21grep07:53:220
351529923,58sleep03578-21unin-run07:27:500
233229810,72sleep22419-21fschecks_time07:23:122
1865129815,67sleep218792-21modprobe09:57:532
1865129815,67sleep218792-21modprobe09:57:532
2894929718,56sleep20-21swapper/208:47:512
2487829712,70sleep224959-21latency08:34:112
2487829712,70sleep224959-21latency08:34:112
2470929723,48sleep324787-21cut08:34:073
2470929723,48sleep324787-21cut08:34:073
1637029718,55sleep216514-21grep09:48:202
2049829523,48sleep220564-21fschecks_time10:03:102
2242729410,25sleep10-21swapper/110:08:181
1176929424,44sleep211911-21unixbench_singl09:33:222
475129314,64sleep04806-21if_err_eth009:13:090
323062928,24sleep10-21swapper/110:39:311
1441229253,25sleep114431-21kernelversion09:43:151
125652907,70sleep00-21swapper/009:38:010
125652907,70sleep00-21swapper/009:38:010
328828913,62sleep03226-21latency_hist10:52:510
3185428912,62sleep231938-21switchtime10:38:192
2633028924,12sleep10-21swapper/108:38:131
558828814,60sleep35658-21df07:33:063
422028815,58sleep04282-21if_eth007:28:270
2712028846,27sleep127195-21mailstats10:23:201
2182428717,47sleep021955-21timerwakeupswit08:23:220
526928624,42sleep35329-21sendmail-msp07:32:503
2328528612,59sleep223436-21latency_hist10:12:512
1554028613,61sleep015628-21timerandwakeup08:03:280
2396628416,52sleep224012-21modprobe08:32:492
2215128417,52sleep222215-21modprobe11:52:492
2215128417,52sleep222215-21modprobe11:52:492
156882848,21sleep00-21swapper/009:48:020
868828321,46sleep10-21swapper/107:43:051
206792839,19sleep10-21swapper/111:47:511
121132837,12sleep00-21swapper/009:35:260
188022826,42sleep10-21swapper/109:58:011
188022826,42sleep10-21swapper/109:58:011
2665128124,25sleep30-21swapper/306:58:143
2665128124,25sleep30-21swapper/306:58:143
222352819,17sleep10-21swapper/111:52:521
222352819,17sleep10-21swapper/111:52:521
1492428117,15sleep10-21swapper/108:03:041
131152819,14sleep30-21swapper/311:23:063
131152819,14sleep30-21swapper/311:23:063
999528011,52sleep210046-21latency_hist07:47:512
985928012,15sleep10-21swapper/109:28:171
690028029,30sleep06907-21latency_hist07:37:510
55372805,41sleep10-21swapper/109:14:031
2890528014,50sleep229048-21sed10:28:222
1936027923,31sleep20-21swapper/208:17:512
1285827921,32sleep30-21swapper/309:38:123
1285827921,32sleep30-21swapper/309:38:123
63212786,14sleep10-21swapper/109:17:551
2762827815,12sleep10-21swapper/112:08:181
1488227715,39sleep315031-21unixbench_singl09:43:253
2552827616,41sleep025560-21proc_pri10:18:190
141527618,38sleep20-21swapper/212:28:342
141527618,38sleep20-21swapper/212:28:342
632727520,36sleep25844-21apt-get09:18:002
2951627526,24sleep30-21swapper/310:32:513
2688327512,46sleep226952-21latency_hist12:07:512
1071527522,23sleep30-21swapper/307:48:203
456927414,40sleep34543-21/usr/sbin/munin09:13:043
2404427417,42sleep00-21swapper/008:32:510
2064627439,18sleep00-21swapper/011:47:500
75927322,29sleep00-21swapper/007:18:090
75927322,29sleep00-21swapper/007:18:090
641527314,15sleep00-21swapper/011:02:520
209602737,15sleep00-21swapper/011:48:060
209602737,15sleep00-21swapper/011:48:060
93727220,32sleep0932-21apt-get12:28:200
93727220,32sleep0932-21apt-get12:28:200
761927122,23sleep30-21swapper/309:22:493
2056727117,39sleep320625-21if_eth010:03:113
128302715,11sleep30-21swapper/307:54:343
3179027036,18sleep10-21swapper/107:13:131
3097527019,21sleep00-21swapper/008:53:150
2302027022,28sleep20-21swapper/211:53:212
2241727020,30sleep3311-21in:imuxsock08:27:493
1813627024,22sleep00-21swapper/008:13:090
750626911,29sleep20-21swapper/207:38:142
716726915,21sleep20-21swapper/211:03:192
3206826915,22sleep00-21swapper/010:38:210
1072126914,36sleep210790-21systemd-run09:32:502
740726816,35sleep07479-21latency07:38:120
3047926813,38sleep030561-21mailstats07:08:200
2397926815,34sleep00-21swapper/010:13:180
225626817,28sleep30-21swapper/312:32:513
225626817,28sleep30-21swapper/312:32:513
1748826828,29sleep10-21swapper/111:37:381
639026718,24sleep30-21swapper/311:02:513
2758926711,26sleep00-21swapper/012:08:170
2202626711,36sleep022131-21fschecks_time10:08:090
1755326717,20sleep20-21swapper/211:37:512
142042675,50sleep00-21swapper/011:25:380
142042675,50sleep00-21swapper/011:25:380
336026623,24sleep33409-21ntpq09:08:163
336026623,24sleep33409-21ntpq09:08:163
2984126616,33sleep229855-21fschecks_count10:33:112
214912665,11sleep00-21swapper/010:05:550
1309426616,26sleep20-21swapper/209:38:192
1309426616,26sleep20-21swapper/209:38:192
1024026624,20sleep20-21swapper/207:48:072
605226516,24sleep20-21swapper/207:33:172
407226514,27sleep20-21swapper/210:53:212
1345826514,36sleep313525-21fschecks_time07:58:103
974726418,21sleep30-21swapper/311:13:013
269952646,11sleep20-21swapper/208:38:272
1788626414,30sleep30-21swapper/311:38:123
1078526415,21sleep00-21swapper/007:48:210
603826318,27sleep26212-21sed09:17:522
3172226313,17sleep00-21swapper/007:13:120
3077626311,40sleep230848-21mailstats12:18:202
2943426315,16sleep20-21swapper/212:13:202
2848226315,21sleep30-21swapper/312:12:523
1600426315,25sleep30-21swapper/311:32:513
1600426315,25sleep30-21swapper/311:32:513
863926218,27sleep38645-21ntp_states11:08:163
189526214,24sleep30-21swapper/309:03:193
123042626,10sleep00-21swapper/011:19:110
123042626,10sleep00-21swapper/011:19:110
73812616,43sleep00-21swapper/009:19:440
348126116,20sleep30-21swapper/307:27:493
2091826116,29sleep221040-21latency_hist08:22:522
1897526114,31sleep319043-21basename09:58:113
1897526114,31sleep319043-21basename09:58:113
1787426121,23sleep20-21swapper/209:53:212
1616226115,27sleep00-21swapper/011:32:560
1616226115,27sleep00-21swapper/011:32:560
816126018,24sleep08180-21apt-get11:08:010
2819926021,19sleep30-21swapper/308:43:243
2349126021,22sleep30-21swapper/310:13:013
2270826016,28sleep322700-21apt-get08:28:063
1779526013,29sleep217789-21apt-get11:38:072
1647126018,26sleep016466-21apt-get08:08:040
1319626012,22sleep20-21swapper/207:57:522
3240625915,23sleep30-21swapper/308:58:103
2412125916,25sleep324208-21perl11:58:113
1933425918,20sleep00-21swapper/009:58:200
1933425918,20sleep00-21swapper/009:58:200
413925819,22sleep30-21swapper/310:53:223
3157925817,27sleep031544-21latency_hist12:22:510
3065425817,23sleep00-21swapper/010:33:250
3001925816,19sleep00-21swapper/012:17:500
2172125818,24sleep321790-21users11:48:223
2172125818,24sleep321790-21users11:48:223
2120525816,29sleep221231-21df08:23:082
1816425818,22sleep018209-21awk11:38:170
173122585,15sleep00-21swapper/011:35:230
1492525810,34sleep214993-21df08:03:052
2955525718,15sleep20-21swapper/207:07:532
2602025715,19sleep30-21swapper/310:18:543
164125713,28sleep01692-21if_eth009:03:120
1583625719,20sleep30-21swapper/309:48:093
1134425717,19sleep20-21swapper/211:18:212
1134425717,19sleep20-21swapper/211:18:212
946425612,22sleep00-21swapper/007:43:250
905225617,19sleep30-21swapper/307:43:163
723825625,17sleep30-21swapper/311:03:203
47925611,17sleep30-21swapper/310:43:093
3203725614,26sleep232088-21irqstats12:23:132
2918925613,26sleep029182-21apt-get08:48:060
2605725610,18sleep30-21swapper/308:38:033
2016025615,26sleep320240-21sed08:18:213
983025513,26sleep09858-21nfs4_client09:28:160
789625516,23sleep07923-21munin-run11:07:490
1960925511,25sleep019602-21apt-get08:18:050
946325413,24sleep39458-21apt-get09:28:063
60025419,20sleep20-21swapper/210:43:122
50592548,18sleep20-21swapper/210:58:022
2670325413,26sleep326733-21fschecks_count10:23:103
255132549,25sleep20-21swapper/210:18:172
195625414,21sleep32026-21df10:48:093
1542325420,20sleep00-21swapper/009:47:500
958025312,19sleep20-21swapper/211:12:522
291072536,11sleep00-21swapper/010:28:360
1979525312,24sleep219801-21ntp_states11:43:162
143425313,24sleep21427-21apt-get09:03:062
3076725211,24sleep230759-21apt-get08:53:082
113025212,16sleep20-21swapper/209:02:502
2998525112,23sleep329982-21apt-get07:08:023
2809725111,16sleep00-21swapper/008:43:210
2466025114,14sleep20-21swapper/211:58:222
1745025112,20sleep00-21swapper/009:53:110
1358925112,18sleep20-21swapper/211:23:342
1358925112,18sleep20-21swapper/211:23:342
1204625110,26sleep212081-21uname07:53:122
1191225116,20sleep30-21swapper/309:33:233
755325011,22sleep37589-21ntp_states07:38:153
3231325012,22sleep032316-21apt-get08:58:030
2804925012,23sleep228043-21apt-get07:03:052
917024916,15sleep20-21swapper/209:27:492
2421024917,16sleep00-21swapper/011:58:120
1490024811,21sleep214949-21irqstats11:28:262
1490024811,21sleep214949-21irqstats11:28:262
433024711,15sleep20-21swapper/209:12:512
433024711,15sleep20-21swapper/209:12:512
298024715,18sleep00-21swapper/009:08:050
298024715,18sleep00-21swapper/009:08:050
292352467,16sleep20-21swapper/208:48:082
48824514,16sleep00-21swapper/010:43:100
3227324410,17sleep30-21swapper/312:23:183
151412447,16sleep30-21swapper/308:03:103
258022438,20sleep325842-21irqstats12:03:113
1027924212,14sleep20-21swapper/211:13:202
9012409,15sleep20-21swapper/207:18:112
9012409,15sleep20-21swapper/207:18:112
164592389,13sleep30-21swapper/311:33:203
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional