You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-08 - 06:16

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Mon Jun 08, 2026 00:43:48)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
27212250513,466sleep027392-21apt-config00:23:010
2096424086,367sleep1363-21snmpd22:21:571
1137240858,21sleep10-21swapper/119:33:291
16402239644,265sleep116465-21fschecks_time23:48:171
28138239533,252sleep128258-21timerwakeupswit00:23:421
19434237410,350sleep119486-21cat20:33:201
7373235816,326sleep07398-21/usr/sbin/munin21:38:160
1108923086,270sleep1363-21snmpd23:32:431
2335923007,256sleep1363-21snmpd22:28:271
15294227516,201sleep115448-21unixbench_multi20:18:311
16077226210,237sleep116046-21latency_hist23:48:011
24940225644,127sleep125035-21fschecks_time19:08:181
24940225644,127sleep125035-21fschecks_time19:08:181
25778225146,79sleep125839-21latency_hist22:38:001
2848222174,20sleep10-21swapper/121:23:191
2848222174,20sleep10-21swapper/121:23:191
28741220819,61sleep10-21swapper/119:18:231
31727220327,142sleep131834-21latency_hist21:13:011
18498219953,27sleep10-21swapper/123:53:301
16238219551,21sleep10-21swapper/120:23:191
5002219114,164sleep35097-21sed19:48:013
4584218652,51sleep10-21swapper/119:43:311
22606218415,133sleep11-21systemd22:27:591
21467217418,139sleep00-21swapper/000:03:260
28855217218,140sleep228919-21/usr/sbin/munin21:03:132
23297216211,135sleep123441-21tail20:43:301
13107216213,135sleep113180-21fschecks_time20:13:181
11940216127,71sleep10-21swapper/121:53:111
7275215920,61sleep10-21swapper/119:53:281
13226215919,31sleep10-21swapper/123:38:241
4510215812,90sleep14667-21sed23:13:001
8632215129,85sleep18755-21wc21:43:011
19008214340,45sleep10-21swapper/120:33:001
5922214220,99sleep36070-21unixbench_multi19:48:293
2113214011,90sleep12123-21acpi19:38:091
19721213915,52sleep119757-21latency23:58:221
6972213817,20sleep00-21swapper/023:18:280
6972213817,20sleep00-21swapper/023:18:280
6751213610,55sleep10-21swapper/123:18:131
6751213610,55sleep10-21swapper/123:18:131
11583213612,88sleep111644-21if_err_eth023:33:191
31918212928,62sleep131949-21date00:38:001
31918212928,62sleep131949-21date00:38:001
11108212649,25sleep011134-21munin-run23:32:580
31225212464,45sleep10-21swapper/119:28:001
10418212317,93sleep110485-21cpuspeed21:48:141
50521198,67sleep10-21swapper/122:58:271
25706211918,82sleep0225-21systemd-journal22:37:590
26397211735,43sleep126518-21timerwakeupswit20:53:311
28820211618,77sleep328964-21cat00:28:003
32242211530,68sleep332364-21latency_hist22:58:003
25932211511,89sleep30-21swapper/322:38:023
10525211546,53sleep10-21swapper/120:03:291
1023211515,80sleep31059-21wc19:33:273
32043211377,17sleep10-21swapper/119:28:291
2715021129,61sleep127163-21needreboot19:13:231
24892211216,19sleep30-21swapper/322:33:263
2115421117,42sleep121223-21basename00:03:191
1946521119,82sleep20-21swapper/222:17:582
27700211016,48sleep10-21swapper/120:58:241
205021098,42sleep12088-21/usr/sbin/munin23:03:271
13756210952,16sleep10-21swapper/121:58:231
9600210713,78sleep09601-21sshd20:02:350
31633210718,70sleep231774-21grep22:53:312
2962721077,35sleep10-21swapper/122:48:211
2962721077,35sleep10-21swapper/122:48:211
1240210626,63sleep10-21swapper/121:18:181
31176210439,21sleep10-21swapper/122:53:221
27408210342,44sleep10-21swapper/122:43:001
2519321035,59sleep10-21swapper/120:49:521
2693421026,14sleep00-21swapper/022:38:350
9508210114,52sleep19626-21cat21:43:301
8575210119,62sleep38277-21/usr/sbin/munin23:23:323
2297121019,80sleep123005-21ntpq00:08:241
1829821017,78sleep00-21swapper/023:53:250
17231210115,73sleep317233-21sshd23:48:353
25275210043,35sleep10-21swapper/100:16:361
639429910,74sleep16558-21apt-config23:18:011
243302996,11sleep10-21swapper/119:05:361
243302996,11sleep10-21swapper/119:05:361
189829913,70sleep01934-21cat19:38:000
289902968,12sleep10-21swapper/122:47:001
289902968,12sleep10-21swapper/122:47:001
1942629635,36sleep219445-21apt-get23:58:112
83862948,70sleep10-21swapper/123:23:261
574829334,14sleep10-21swapper/119:48:271
3058529315,56sleep230392-21/usr/sbin/munin21:08:222
2812929312,66sleep228078-21latency_hist19:18:002
2627729313,65sleep026346-21switchtime00:18:290
2173429219,58sleep321781-21ntpq22:23:253
1819629112,37sleep10-21swapper/122:13:131
1765029130,28sleep10-21swapper/122:09:421
1128929132,46sleep111193-21sh20:08:011
3245129015,21sleep00-21swapper/021:13:290
210342908,38sleep10-21swapper/120:38:091
426728934,16sleep10-21swapper/121:28:151
2952328926,47sleep10-21swapper/100:28:271
2749528913,19sleep20-21swapper/200:23:152
158162880,6sleep00-21swapper/022:03:300
2893928714,36sleep10-21swapper/121:03:161
2567528726,27sleep125695-21dpkg00:18:091
2390828734,30sleep30-21swapper/319:03:283
2390828734,30sleep30-21swapper/319:03:283
2285828718,48sleep022877-21apt-get22:28:100
778228619,19sleep30-21swapper/321:38:273
493428622,43sleep34910-21apt-get23:13:233
2490928624,38sleep025038-21unixbench_singl20:48:340
398628523,32sleep20-21swapper/221:28:002
398628523,32sleep20-21swapper/221:28:002
2906528531,32sleep20-21swapper/222:47:582
2906528531,32sleep20-21swapper/222:47:582
2264228518,40sleep322730-21tune2fs00:08:183
624728414,15sleep00-21swapper/019:49:430
3196428420,48sleep10-21swapper/121:13:131
1797028415,55sleep218044-21cat20:28:212
1266128411,57sleep012739-21apt-config23:37:580
571728313,57sleep25718-21munin-node21:33:092
960228218,31sleep10-21swapper/120:02:361
574228218,18sleep10-21swapper/121:33:121
26712818,60sleep3363-21snmpd23:05:483
169842810,5sleep20-21swapper/220:23:322
2381428020,27sleep20-21swapper/200:12:572
688627917,40sleep06975-21grep19:53:190
2292527919,46sleep022969-21ntp_kernel_err00:08:230
3245227820,43sleep332454-21/usr/sbin/munin22:58:093
1139827815,43sleep311466-21/usr/sbin/munin23:33:133
871027716,46sleep28734-21ntp_kernel_pll_23:23:352
207602776,10sleep00-21swapper/022:19:220
1451627722,27sleep00-21swapper/023:42:590
2978127617,17sleep00-21swapper/000:28:300
2444127617,41sleep324500-21latency20:48:243
166302767,48sleep30-21swapper/322:08:103
1595827623,14sleep10-21swapper/122:03:321
2958927421,33sleep229653-21/usr/sbin/munin00:28:282
2684927415,46sleep3225-21systemd-journal00:21:343
494927322,32sleep04952-21cron19:47:580
2546227317,32sleep00-21swapper/000:18:000
2259227324,28sleep00-21swapper/022:27:580
2057727315,42sleep020664-21apt-config20:38:000
1868127319,30sleep30-21swapper/322:13:273
1497427320,26sleep20-21swapper/220:18:242
2700327226,26sleep227034-21munin-run00:22:592
2170127213,26sleep20-21swapper/222:23:242
1415927215,40sleep314256-21timerandwakeup21:58:343
125452728,53sleep00-21swapper/020:10:340
1031927212,31sleep20-21swapper/223:28:362
923727115,41sleep29263-21netstat21:43:232
812627113,26sleep20-21swapper/219:58:012
3125127127,25sleep30-21swapper/300:33:343
3125127127,25sleep30-21swapper/300:33:343
2451527130,22sleep20-21swapper/219:07:582
2451527130,22sleep20-21swapper/219:07:582
2232427116,41sleep222352-21latency_hist20:42:582
1460527116,40sleep00-21swapper/020:18:140
1384427123,29sleep20-21swapper/223:39:262
1080727113,22sleep20-21swapper/221:48:232
439527012,40sleep34434-21apt-get23:12:583
2450027018,29sleep30-21swapper/300:13:233
103802709,48sleep210390-21open_files20:03:262
999226913,38sleep010000-21/usr/sbin/munin20:03:160
446526914,41sleep24554-21load21:28:212
266492696,16sleep00-21swapper/019:13:050
1362526915,38sleep213632-21diskstats21:58:202
93002686,50sleep00-21swapper/023:24:380
457126811,34sleep04611-21ntp_kernel_pll_21:28:230
2943226811,36sleep029439-21/usr/sbin/munin22:48:160
2943226811,36sleep029439-21/usr/sbin/munin22:48:160
264226813,42sleep02710-21/usr/sbin/munin21:23:140
264226813,42sleep02710-21/usr/sbin/munin21:23:140
1362226818,21sleep00-21swapper/020:13:290
894826713,37sleep38343-21/usr/sbin/munin19:58:303
722326718,20sleep30-21swapper/323:18:363
722326718,20sleep30-21swapper/323:18:363
423726713,41sleep24308-21switchtime19:43:272
289426713,37sleep22935-21latency_hist23:08:012
2301626714,25sleep30-21swapper/320:43:243
2090026720,26sleep30-21swapper/320:38:033
160326719,20sleep20-21swapper/221:18:272
1257226723,25sleep00-21swapper/021:53:300
1137126723,24sleep20-21swapper/223:33:082
759126622,24sleep20-21swapper/221:38:222
3173526615,31sleep00-21swapper/000:36:370
3173526615,31sleep00-21swapper/000:36:370
1919626622,29sleep219207-21munin-run23:57:592
1741126620,30sleep017433-21modprobe20:27:580
676226521,23sleep30-21swapper/319:53:093
672126515,20sleep20-21swapper/223:18:102
672126515,20sleep20-21swapper/223:18:102
65526515,36sleep2755-21fschecks_time19:33:182
619526519,25sleep26219-21munin-run23:17:582
41582657,17sleep00-21swapper/023:10:010
237626518,21sleep00-21swapper/019:38:210
2173626517,31sleep321858-21timerwakeupswit20:39:003
1874826522,21sleep00-21swapper/022:13:280
108712656,13sleep00-21swapper/021:48:250
3216326420,28sleep032204-21systemctl22:57:570
3090826426,20sleep30-21swapper/321:08:313
2226426316,31sleep322275-21modprobe00:07:583
204126310,38sleep02049-21ntpq23:03:260
1322326315,26sleep30-21swapper/323:38:223
574126219,27sleep05737-21apt-get21:33:110
299752628,43sleep00-21swapper/021:06:100
2617026217,19sleep00-21swapper/019:13:010
2617026217,19sleep00-21swapper/019:13:010
2462826214,24sleep00-21swapper/019:08:000
2462826214,24sleep00-21swapper/019:08:000
2389426216,31sleep223926-21systemd-run20:47:592
1746326217,24sleep30-21swapper/320:28:023
1965026119,23sleep20-21swapper/220:33:262
1237226120,22sleep30-21swapper/321:53:253
94202605,10sleep20-21swapper/220:00:152
287526021,22sleep23025-21sed19:38:312
2522226020,23sleep20-21swapper/219:08:232
2522226020,23sleep20-21swapper/219:08:232
1835026018,27sleep318424-21mailstats23:53:283
1497126018,20sleep30-21swapper/323:43:193
2934025912,31sleep329422-21mailstats21:03:263
2851825915,25sleep00-21swapper/022:43:520
2851825915,25sleep00-21swapper/022:43:520
1759125914,30sleep217622-21munin-run23:52:582
3270825819,17sleep30-21swapper/321:13:323
3013825816,23sleep20-21swapper/222:48:312
3013825816,23sleep20-21swapper/222:48:312
2859125812,30sleep228640-21irqstats19:18:202
2657025817,25sleep326647-21perl19:13:043
2419225818,25sleep20-21swapper/222:33:002
1211325821,22sleep20-21swapper/221:53:202
494825720,19sleep30-21swapper/321:28:303
2991925716,24sleep329915-21apt-get19:23:143
269582577,42sleep00-21swapper/020:57:380
2000025712,18sleep20-21swapper/222:18:192
3116625621,16sleep20-21swapper/200:33:332
3116625621,16sleep20-21swapper/200:33:332
303592567,10sleep30-21swapper/322:49:453
303592567,10sleep30-21swapper/322:49:453
1692125611,28sleep217014-21memory22:08:222
168562568,24sleep00-21swapper/022:08:200
1658525616,18sleep00-21swapper/023:48:200
1533725613,22sleep00-21swapper/023:43:280
2765925513,22sleep327289-21apt-get22:43:023
2765925513,22sleep327289-21apt-get22:43:023
2622925515,25sleep326302-21/usr/sbin/munin20:53:293
232325516,22sleep32324-21sshd21:22:363
1570625516,25sleep315811-21timerandwakeup22:03:303
1173725516,17sleep30-21swapper/321:53:003
629025411,23sleep00-21swapper/023:18:000
584025411,28sleep35949-21perl21:33:183
3128125411,17sleep20-21swapper/219:28:012
2450725415,24sleep20-21swapper/222:33:172
2407925412,23sleep224098-21apt-get00:13:092
2005025417,19sleep30-21swapper/322:18:203
1850325411,18sleep00-21swapper/020:28:310
1318125415,17sleep20-21swapper/220:13:192
1116225416,19sleep30-21swapper/323:33:013
886625313,24sleep38862-21apt-get21:43:113
3214025312,19sleep20-21swapper/219:28:302
3193925312,25sleep231958-21apt-get21:13:102
2963025313,16sleep00-21swapper/019:22:580
2423625313,23sleep224245-21df_inode20:48:152
1966325313,22sleep019720-21irqstats23:58:210
1204625315,23sleep312143-21cat20:08:283
942325217,15sleep00-21swapper/021:43:290
330425216,16sleep30-21swapper/321:23:293
330425216,16sleep30-21swapper/321:23:293
2903725212,24sleep329032-21apt-get00:28:133
2651925213,24sleep226627-21users20:53:322
2102125212,23sleep020526-21apt-get20:38:040
1564225217,18sleep20-21swapper/222:03:292
109425213,24sleep01087-21apt-get21:18:130
3037525111,24sleep030381-21ntp_states19:23:270
286412519,17sleep30-21swapper/319:18:213
2729925111,23sleep227293-21apt-get20:58:132
2231725113,14sleep20-21swapper/200:08:012
148702519,18sleep20-21swapper/223:43:182
1301625113,22sleep313012-21apt-get20:13:123
3192925016,16sleep30-21swapper/319:28:273
213625010,23sleep32133-21apt-get19:38:123
1770025012,22sleep317693-21apt-get20:28:143
1460325011,22sleep314598-21apt-get20:18:133
134325015,20sleep20-21swapper/223:03:002
116782509,29sleep211728-21/usr/sbin/munin20:08:182
38724913,21sleep0417-21awk22:58:220
356124912,21sleep23585-21ntp_states23:08:252
3146624912,21sleep031462-21apt-get19:28:110
3042124914,20sleep00-21swapper/021:08:160
2738624915,17sleep30-21swapper/320:58:183
1076024910,17sleep30-21swapper/320:03:313
2646524812,21sleep226472-21ntp_states22:38:252
1990724811,22sleep319913-21ntp_states23:58:293
568824611,14sleep20-21swapper/219:48:242
248422469,19sleep30-21swapper/319:08:123
248422469,19sleep30-21swapper/319:08:123
2315224615,16sleep30-21swapper/322:28:213
1426624616,14sleep00-21swapper/021:58:350
230652457,18sleep00-21swapper/020:43:250
2766924211,13sleep00-21swapper/020:58:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional