You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 05:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Thu Dec 04, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
940257115,540sleep3952-21/usr/sbin/munin20:53:223
290824549,430sleep2776-21kworker/u8:019:18:152
2962242937,337sleep23145-21sed00:28:002
18968237951,216sleep218999-21cron21:52:572
18968237951,216sleep218999-21cron21:52:572
4535236735,293sleep24623-21tune2fs22:48:182
21872235499,190sleep221920-21latency_hist20:18:002
30882232915,300sleep330893-21ntpq22:28:283
30882232915,300sleep330893-21ntpq22:28:283
29534232018,279sleep129704-21latency_hist19:03:001
2108231110,289sleep22128-21dpkg20:58:102
28577228916,258sleep328584-21ntpq18:58:273
170092278105,153sleep217083-21fschecks_time23:28:202
170092278105,153sleep217083-21fschecks_time23:28:202
28718227621,239sleep128763-21grep20:38:221
3913226286,25sleep20-21swapper/200:28:452
2743322519,231sleep05666-21snmpd00:01:290
7205224530,200sleep27241-21ntpq21:13:242
21350221144,54sleep221426-21awk21:58:312
2941422100,8sleep20-21swapper/200:08:162
9001220830,121sleep29105-21timerandwakeup21:18:372
31074220550,140sleep231146-21fschecks_time00:13:182
26769220516,175sleep226779-21acpi20:33:102
26378220445,121sleep226468-21fschecks_time23:58:202
1929219511,170sleep22078-21latency_hist20:58:022
30500217617,139sleep030580-21threads20:43:300
16530217624,107sleep20-21swapper/221:43:242
8207217512,149sleep08286-21find19:33:280
22572217532,131sleep222606-21ps20:18:242
13979217317,46sleep20-21swapper/221:33:352
30048217192,23sleep20-21swapper/220:43:202
25834216517,54sleep20-21swapper/220:28:302
32014216421,124sleep331991-21apt-get22:33:173
27722216422,123sleep20-21swapper/222:18:262
13531216347,56sleep213534-21cron23:17:582
14412216184,20sleep20-21swapper/219:53:292
304332160107,13sleep20-21swapper/222:28:152
304332160107,13sleep20-21swapper/222:28:152
596215917,118sleep20-21swapper/219:11:292
27645215837,24sleep20-21swapper/200:03:002
21682215510,34sleep221762-21fschecks_time23:43:182
3647215322,34sleep30-21swapper/300:28:373
18501215160,46sleep218552-21cpuspeed23:33:162
18501215160,46sleep218552-21cpuspeed23:33:162
4159213813,104sleep24176-21awk22:47:572
8050213710,114sleep28115-21awk22:58:272
31274213420,91sleep131390-21latency_hist19:08:011
23144213136,35sleep20-21swapper/223:48:102
2078321299,98sleep220931-21head23:38:292
2078321299,98sleep220931-21head23:38:292
20474212918,84sleep20-21swapper/220:13:032
25373212710,63sleep225553-21latency_hist22:13:002
4996212530,60sleep25006-21munin-run21:07:592
11079212510,102sleep210684-21/usr/sbin/munin19:43:232
23003212433,55sleep222373-21/usr/sbin/munin22:03:312
27412212220,77sleep027427-21fschecks_count22:18:190
15605212114,61sleep20-21swapper/219:58:092
23666211710,65sleep20-21swapper/220:23:132
19158211721,20sleep10-21swapper/123:33:311
19158211721,20sleep10-21swapper/123:33:311
31212211620,75sleep231222-21modprobe19:07:582
2939211616,81sleep12893-21apt-get19:18:391
1396211623,38sleep20-21swapper/200:22:592
12259211613,40sleep229569-21cyclictest19:48:122
24821211520,73sleep124712-21/usr/sbin/munin23:53:181
26557211421,70sleep126645-21wc20:33:011
8289211313,31sleep18417-21unixbench_multi19:33:291
2752211313,84sleep12387-21apt-get19:18:021
32013211222,49sleep20-21swapper/222:33:162
25313211216,77sleep325419-21tune2fs20:28:193
9865211118,76sleep09997-21awk23:03:300
8872211111,87sleep28875-21cron19:37:582
28802211113,65sleep00-21swapper/020:38:250
26092211041,22sleep20-21swapper/222:13:572
6569210951,16sleep20-21swapper/219:28:292
11246210924,33sleep00-21swapper/021:28:000
11246210924,33sleep00-21swapper/021:28:000
9394210710,56sleep20-21swapper/223:03:202
8844210623,67sleep08865-21awk23:02:580
5783210611,82sleep35820-21cat22:52:593
172721038,36sleep20-21swapper/222:39:342
22398210218,66sleep30-21swapper/322:03:163
28214210142,41sleep028310-21uniq18:58:190
25742210117,64sleep325234-21apt-get22:13:093
19449210045,39sleep219511-21cat21:53:182
1704521009,52sleep20-21swapper/220:03:022
943129922,23sleep00-21swapper/019:38:200
2092629914,70sleep320933-21diskstats21:58:203
1694629812,19sleep00-21swapper/020:03:000
1564029715,22sleep315733-21memory23:23:203
1129029713,23sleep20-21swapper/223:08:292
2792829625,51sleep128016-21fschecks_time00:03:201
290929520,56sleep32893-21apt-get19:18:163
241329550,30sleep22462-21apt-key19:17:592
3234229213,64sleep132275-21latency_hist00:18:051
2970929253,19sleep20-21swapper/222:23:302
2002629225,39sleep120170-21sed21:53:291
118229226,44sleep0832-21apt-get22:38:020
3153829121,46sleep031576-21grep20:48:190
3135729115,60sleep331397-21ntpq00:13:233
24052917,68sleep00-21swapper/000:23:300
3230229013,55sleep331871-21/usr/sbin/munin19:08:273
225429011,22sleep32346-21users22:39:433
1520129022,34sleep10-21swapper/119:58:001
415728911,19sleep10-21swapper/121:03:261
71142885,74sleep00-21swapper/019:30:370
2834028821,13sleep20-21swapper/220:38:112
1461528834,41sleep214646-21grep21:38:162
1223728813,18sleep10-21swapper/119:48:111
1782328713,57sleep017825-21users23:28:340
1782328713,57sleep017825-21users23:28:340
1206428729,42sleep20-21swapper/221:28:352
1206428729,42sleep20-21swapper/221:28:352
2365628614,54sleep0225-21systemd-journal22:07:580
2360728613,59sleep123615-21ntpq23:48:251
2192328618,50sleep322067-21latency_hist20:18:013
809528510,56sleep00-21swapper/021:17:580
257392857,14sleep00-21swapper/022:13:060
919328423,45sleep20-21swapper/219:38:142
1327328315,51sleep313325-21/usr/sbin/munin21:33:203
826428216,50sleep18416-21unixbench_multi22:58:301
659628247,24sleep26658-21sed21:13:002
648728211,15sleep20-21swapper/222:53:272
2285928222,42sleep122887-21awk23:47:571
916128119,46sleep19192-21cpu19:38:131
2570028119,31sleep00-21swapper/020:28:280
2471928134,32sleep20-21swapper/223:53:112
2000328120,41sleep120023-21apt-get20:12:581
2346428025,33sleep30-21swapper/320:23:023
1718828020,38sleep317180-21apt-get20:03:153
1514728017,49sleep015200-21cmp19:57:590
1001928012,16sleep20-21swapper/221:23:182
1001928012,16sleep20-21swapper/221:23:182
3237427920,11sleep20-21swapper/220:48:412
2605027920,33sleep30-21swapper/323:58:003
2085027913,50sleep00-21swapper/021:58:160
2948327816,40sleep129573-21grep00:08:201
2052827814,48sleep320590-21/usr/sbin/munin23:38:263
2052827814,48sleep320590-21/usr/sbin/munin23:38:263
1683127814,49sleep116952-21sed21:43:321
1627927819,11sleep20-21swapper/223:23:312
724327712,47sleep37288-21/usr/sbin/munin21:13:253
611227727,26sleep10-21swapper/122:53:171
3126427720,36sleep30-21swapper/320:48:013
17372776,58sleep10-21swapper/120:56:351
358127617,36sleep33737-21grep22:43:303
3247127614,44sleep132477-21ntp_states22:33:311
2932027616,45sleep029330-21ntpq22:23:240
1817827623,22sleep30-21swapper/323:32:573
1817827623,22sleep30-21swapper/323:32:573
524627515,43sleep00-21swapper/021:08:130
485227534,28sleep20-21swapper/219:23:242
2791527515,41sleep327921-21df_inode00:03:173
1971427514,48sleep00-21swapper/020:09:260
1476727537,28sleep20-21swapper/223:18:592
973127424,35sleep30-21swapper/319:38:283
1825927417,34sleep118199-21latency_hist23:32:591
1825927417,34sleep118199-21latency_hist23:32:591
304982735,56sleep00-21swapper/000:10:500
295727312,27sleep00-21swapper/019:18:500
2211127311,48sleep10-21swapper/120:18:111
2013327213,41sleep320222-21apt-config20:13:003
1382027217,22sleep10-21swapper/123:18:131
198127118,24sleep30-21swapper/319:13:323
1698327121,33sleep10-21swapper/120:03:011
1167527118,26sleep10-21swapper/121:28:251
1167527118,26sleep10-21swapper/121:28:251
602527022,30sleep36022-21apt-get22:53:113
522127017,34sleep15223-21/usr/sbin/munin21:08:101
781726920,30sleep30-21swapper/319:33:193
475926916,32sleep00-21swapper/000:32:580
3090226917,23sleep10-21swapper/122:28:301
3090226917,23sleep10-21swapper/122:28:301
813626818,30sleep18155-21ls21:18:001
737426816,21sleep10-21swapper/119:33:001
60126819,22sleep00-21swapper/000:18:290
500926818,28sleep00-21swapper/021:08:010
306432685,10sleep30-21swapper/300:12:383
1907426816,26sleep00-21swapper/023:33:300
1907426816,26sleep00-21swapper/023:33:300
1557526819,21sleep10-21swapper/123:23:181
55726721,26sleep30-21swapper/300:18:283
2406126714,22sleep00-21swapper/020:23:230
2060626722,27sleep30-21swapper/321:57:593
171902677,34sleep00-21swapper/020:03:160
1791326622,28sleep30-21swapper/321:48:183
1791326622,28sleep30-21swapper/321:48:183
1306526615,25sleep10-21swapper/121:33:141
85926520,28sleep00-21swapper/019:13:000
3202626515,22sleep30-21swapper/320:48:303
3058126517,30sleep130709-21unixbench_multi20:43:311
117592659,44sleep30-21swapper/323:10:283
32426415,26sleep10-21swapper/120:53:001
2789926413,38sleep327982-21sed22:18:313
2778826410,24sleep10-21swapper/122:18:291
1352026416,23sleep00-21swapper/019:52:580
564726321,21sleep30-21swapper/321:08:243
2955626325,17sleep10-21swapper/122:23:291
1772026315,27sleep10-21swapper/120:03:291
1075726314,27sleep10-21swapper/123:08:161
811626213,18sleep30-21swapper/322:58:283
3169026217,28sleep031709-21modprobe22:32:570
3169026217,28sleep031709-21modprobe22:32:570
2762626217,28sleep327695-21/usr/sbin/munin20:33:323
23992628,13sleep10-21swapper/120:58:211
1061526210,39sleep310739-21unixbench_multi21:23:313
1061526210,39sleep310739-21unixbench_multi21:23:313
582726117,21sleep00-21swapper/022:53:000
2995926112,22sleep00-21swapper/019:03:240
2065926117,16sleep10-21swapper/123:38:281
2065926117,16sleep10-21swapper/123:38:281
3019226017,17sleep30-21swapper/322:28:003
2143526020,21sleep00-21swapper/020:13:580
1552626013,34sleep1369-21sshd21:38:351
1484526010,21sleep00-21swapper/021:38:220
1213726012,19sleep30-21swapper/321:28:363
1213726012,19sleep30-21swapper/321:28:363
1195326020,19sleep00-21swapper/021:28:330
1195326020,19sleep00-21swapper/021:28:330
1145326016,28sleep111458-21sort21:28:091
2872525915,17sleep20-21swapper/218:58:302
1449325914,18sleep30-21swapper/319:53:303
2781325816,27sleep127816-21sshd20:34:551
213102588,32sleep121348-21grep21:58:301
1868325814,26sleep318746-21if_eth023:33:213
1868325814,26sleep318746-21if_eth023:33:213
1106825814,27sleep111077-21needreboot19:43:221
1043925816,18sleep10-21swapper/123:07:591
729425715,26sleep37329-21munin-run19:32:583
497825710,12sleep30-21swapper/319:23:293
424125718,24sleep34308-21awk21:03:283
297125718,24sleep12977-21df_inode22:43:161
293825714,27sleep12947-21munin-run00:27:591
2492725717,25sleep324971-21systemd-run20:27:583
2338225717,24sleep323416-21/usr/sbin/munin23:48:193
1449725712,23sleep00-21swapper/023:18:280
1094325716,20sleep30-21swapper/319:43:203
875925617,18sleep30-21swapper/321:18:303
2897025617,23sleep329096-21sed20:38:283
2531025621,21sleep00-21swapper/023:53:290
2245225615,23sleep322499-21users23:43:313
171712565,17sleep00-21swapper/021:44:480
491525510,30sleep04938-21ntp_states19:23:260
444925513,25sleep14443-21apt-get22:48:131
1582925513,28sleep015853-21proc_pri23:23:250
1565125513,23sleep315718-21df19:58:263
1426425511,29sleep314275-21ntpq23:18:233
514725417,19sleep10-21swapper/119:23:321
345825418,16sleep00-21swapper/021:03:000
3201725411,17sleep10-21swapper/119:08:211
267372544,10sleep10-21swapper/122:14:131
266225415,18sleep10-21swapper/122:43:001
2366525413,26sleep123660-21apt-get20:23:121
2113225414,23sleep121201-21sensors_temp20:13:551
1273225419,18sleep30-21swapper/323:13:263
1234525414,24sleep112352-21df_inode23:13:161
116525411,27sleep01171-21diskstats19:13:170
1132125412,22sleep00-21swapper/019:43:290
1033825419,21sleep10-21swapper/121:23:251
3149425315,21sleep031573-21systemctl00:13:280
2998525316,20sleep30-21swapper/319:03:393
2239625314,24sleep122391-21apt-get22:03:141
2211025313,24sleep022105-21apt-get20:18:100
2201325312,23sleep022050-21ntp_states23:43:240
1464725312,24sleep314686-21df21:38:173
2431325214,17sleep30-21swapper/322:08:253
2359625214,23sleep023606-21ntpq23:48:240
921825113,17sleep30-21swapper/323:03:173
3254325114,19sleep10-21swapper/100:18:141
2521525115,20sleep00-21swapper/022:12:580
227225114,21sleep30-21swapper/300:23:293
1779125112,22sleep117816-21grep21:48:161
1779125112,22sleep117816-21grep21:48:161
1275525113,19sleep00-21swapper/023:13:280
2899025012,21sleep00-21swapper/000:06:340
289725014,20sleep02894-21apt-get22:43:110
2814625012,16sleep00-21swapper/020:38:010
1351125015,16sleep00-21swapper/021:33:250
847024914,16sleep10-21swapper/121:18:241
1928424913,20sleep019278-21apt-get21:53:120
184524915,21sleep00-21swapper/022:39:360
1827224917,15sleep00-21swapper/021:48:270
1827224917,15sleep00-21swapper/021:48:270
1616024911,22sleep316156-21apt-get21:43:133
1363924916,14sleep30-21swapper/319:53:013
409524817,15sleep10-21swapper/100:28:481
2629424710,20sleep326364-21df23:58:173
1086624714,17sleep00-21swapper/023:08:190
2047624611,20sleep020500-21ntp_states23:38:230
2047624611,20sleep020500-21ntp_states23:38:230
21292458,14sleep30-21swapper/320:58:113
75824415,16sleep00-21swapper/020:53:190
2759724412,19sleep127644-21wc00:02:591
243762448,17sleep10-21swapper/122:08:271
84682427,14sleep00-21swapper/021:18:230
69452388,13sleep00-21swapper/022:53:370
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional