You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-28 - 03:57

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sun Jun 28, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
134072155821,1521sleep213408-21ntp_states20:20:292
155072111517,1082sleep315513-21diskstats22:10:013
1501254744,471sleep0364-21snmpd23:08:200
1501254744,471sleep0364-21snmpd23:08:200
10847254130,489sleep010937-21fschecks_time21:55:040
10847254130,489sleep010937-21fschecks_time21:55:040
805425279,502sleep28092-21ntpq21:45:092
22158249624,456sleep222231-21/usr/sbin/munin22:30:112
22158249624,456sleep222231-21/usr/sbin/munin22:30:112
19758240415,371sleep019819-21sendmail_mailqu00:05:140
21944235025,306sleep322071-21/usr/sbin/munin19:05:153
10305233326,295sleep010340-21ntpq23:35:100
26433231624,205sleep026580-21grep00:25:170
1421323167,255sleep00-21swapper/023:48:470
24437229613,199sleep0225-21systemd-journal22:39:180
18442229320,118sleep018590-21unixbench_singl00:00:180
14287229014,258sleep114336-21unin-run23:49:451
165142274102,82sleep00-21swapper/023:55:090
5226226544,197sleep05383-21tail21:35:220
9885225950,191sleep010015-21grep21:50:160
9885225950,191sleep010015-21grep21:50:160
20473225410,207sleep020485-21needreboot22:25:090
7744225070,95sleep00-21swapper/021:45:030
11743223250,107sleep011790-21ntpq20:15:140
7412222763,89sleep07553-21unixbench_singl20:00:200
195192211115,76sleep00-21swapper/020:40:140
28365220431,138sleep028420-21tune2fs19:25:110
30623220112,75sleep00-21swapper/021:16:000
3014218811,100sleep03047-21latency21:30:050
2967521858,17sleep00-21swapper/022:55:000
18058217815,107sleep018121-21mailstats20:35:170
18058217815,107sleep018121-21mailstats20:35:170
12023217517,140sleep212184-21awk20:15:202
6634217235,43sleep10-21swapper/119:59:561
3006021727,153sleep030100-21latency19:30:110
9660216589,17sleep00-21swapper/023:34:450
1856321599,95sleep018564-21cpu22:20:000
611321588,35sleep00-21swapper/021:39:580
2291821545,97sleep00-21swapper/022:34:370
2291821545,97sleep00-21swapper/022:34:370
16125215419,122sleep316140-21grep23:54:593
2106215318,120sleep11622-21apt-get23:09:551
17516215044,13sleep00-21swapper/020:34:570
17516215044,13sleep00-21swapper/020:34:570
2679121457,14sleep00-21swapper/022:45:110
2539421445,28sleep00-21swapper/000:23:520
2416214317,110sleep32510-21memory23:10:133
2778221428,83sleep027846-21mailstats00:30:140
2325321415,28sleep00-21swapper/020:51:280
28557214017,62sleep228605-21unin-run00:34:442
22216214021,21sleep30-21swapper/300:14:463
13393213956,42sleep00-21swapper/020:20:270
1245221385,30sleep00-21swapper/023:41:030
15877213218,100sleep115892-21latency_hist23:54:451
17124213110,107sleep017190-21ls22:15:030
27824212916,33sleep00-21swapper/019:23:090
18316212926,80sleep318448-21unixbench_singl20:35:203
18316212926,80sleep318448-21unixbench_singl20:35:203
25529212617,36sleep00-21swapper/021:00:090
25529212617,36sleep00-21swapper/021:00:090
2474221255,11sleep00-21swapper/020:55:400
2474221255,11sleep00-21swapper/020:55:400
28327212326,20sleep10-21swapper/119:25:081
25263212333,68sleep00-21swapper/019:15:130
3349212015,84sleep13402-21latency_hist19:49:471
3349212015,84sleep13402-21latency_hist19:49:471
2218212015,20sleep00-21swapper/019:45:100
2218212015,20sleep00-21swapper/019:45:100
15896212010,95sleep015952-21latency_hist20:29:480
1325212021,17sleep00-21swapper/021:25:080
21311211421,67sleep00-21swapper/020:45:230
21311211421,67sleep00-21swapper/020:45:230
22332211311,62sleep022099-21apt-get00:14:470
3428211220,72sleep13580-21unixbench_multi21:30:141
29117211242,16sleep00-21swapper/000:35:060
29117211242,16sleep00-21swapper/000:35:060
8332211012,80sleep28350-21/usr/sbin/munin23:29:552
23886211012,83sleep023931-21awk00:19:440
3333210918,78sleep00-21swapper/023:13:580
2270121096,92sleep33426-1kworker/3:2H00:15:073
22360210917,74sleep122366-21fschecks_count20:50:041
28982210831,62sleep029062-21grep21:10:140
28982210831,62sleep029062-21grep21:10:140
363921079,13sleep10-21swapper/119:50:031
13414210718,75sleep313477-21cut20:20:303
12010210714,78sleep312060-21readlink21:59:443
12010210714,78sleep312060-21readlink21:59:443
2877210619,71sleep22954-21fw_packets21:30:032
17814210513,75sleep317916-21fschecks_time00:00:043
11163210520,67sleep211210-21idleruntime-cro23:39:442
3179421049,26sleep10-21swapper/119:35:171
3179421049,26sleep10-21swapper/119:35:171
26644210422,67sleep326665-21idleruntime-cro21:04:443
26644210422,67sleep326665-21idleruntime-cro21:04:443
7298210322,64sleep37405-21ls20:00:193
7071210212,76sleep27085-21uname23:25:062
13638210028,52sleep00-21swapper/022:04:450
10777210010,74sleep0364-21snmpd20:11:180
2268729918,59sleep222815-21latency_hist19:09:462
693629727,18sleep07019-21perl23:25:040
679329726,45sleep26934-21sed21:40:172
595029722,51sleep16106-21grep23:20:171
282482978,77sleep028257-21munin-run21:09:460
3133629628,50sleep331425-21tune2fs23:00:043
2556429621,58sleep225478-21latency_hist00:24:472
967129515,66sleep19816-21latency_hist23:34:461
324612958,76sleep032526-21cat21:20:160
324612958,76sleep032526-21cat21:20:160
1475329522,53sleep00-21swapper/022:05:180
236322945,11sleep30-21swapper/319:10:243
236322945,11sleep30-21swapper/319:10:243
2705729221,51sleep327059-21/usr/sbin/munin19:20:093
2245029213,62sleep322507-21interrupts20:50:063
1988829219,57sleep219250-21/usr/sbin/munin00:05:162
1953629218,55sleep119579-21awk20:40:151
1552229224,49sleep115599-21cut22:10:031
326329126,50sleep31-21systemd19:49:453
326329126,50sleep31-21systemd19:49:453
313312908,68sleep031325-21apt-get19:35:000
313312908,68sleep031325-21apt-get19:35:000
2179229022,47sleep021805-21/usr/sbin/munin19:05:120
145522907,16sleep00-21swapper/023:49:560
481528925,31sleep20-21swapper/219:54:452
3018528918,56sleep330238-21latency_hist00:39:463
3018528918,56sleep330238-21latency_hist00:39:463
1619828916,58sleep116207-21diskstats23:55:021
73528842,30sleep00-21swapper/019:40:130
73528842,30sleep00-21swapper/019:40:130
52592888,45sleep05261-21apt-get23:19:550
36562887,14sleep00-21swapper/023:14:550
3124428822,14sleep00-21swapper/022:59:570
2487228821,12sleep00-21swapper/022:40:020
1117228812,60sleep211248-21latency_hist20:14:472
824628718,14sleep00-21swapper/020:05:040
824628718,14sleep00-21swapper/020:05:040
166362875,71sleep00-21swapper/022:14:020
377428657,14sleep00-21swapper/019:50:060
2368828621,50sleep323753-21find22:35:133
2551228414,55sleep125562-21latency_hist00:24:461
1300128416,50sleep113008-21/usr/sbin/munin20:20:031
1274228419,45sleep112771-21modprobe23:44:441
721128315,51sleep17253-21ntpq23:25:091
314832839,54sleep10-21swapper/123:00:061
3051428315,50sleep230256-21/usr/sbin/munin21:15:582
221128339,20sleep20-21swapper/223:10:042
2260328126,34sleep122684-21systemd-run19:09:451
185028115,49sleep30-21swapper/321:25:203
80328022,34sleep31004-21cp21:24:473
80328022,34sleep31004-21cp21:24:473
824727814,26sleep10-21swapper/120:05:051
824727814,26sleep10-21swapper/120:05:051
2943527822,24sleep20-21swapper/222:54:462
2647827833,25sleep20-21swapper/219:19:472
284622770,6sleep20-21swapper/219:25:122
2372427725,37sleep123796-21sort19:10:261
2372427725,37sleep123796-21sort19:10:261
2966927611,15sleep10-21swapper/122:54:571
2809427639,18sleep00-21swapper/022:49:560
2141927630,25sleep10-21swapper/122:29:461
1167727619,36sleep211722-21/usr/sbin/munin23:40:042
2764027522,31sleep20-21swapper/222:49:452
2207927528,27sleep122102-21/usr/sbin/munin00:14:441
1100227514,46sleep211022-21irqstats21:55:062
1100227514,46sleep211022-21irqstats21:55:062
716727416,38sleep17229-21/usr/sbin/munin20:00:171
230222747,32sleep00-21swapper/019:09:480
230222747,32sleep00-21swapper/019:09:480
1652727416,23sleep10-21swapper/120:30:211
1652727416,23sleep10-21swapper/120:30:211
2019627329,23sleep30-21swapper/322:25:033
61227225,21sleep10-21swapper/123:05:071
61227225,21sleep10-21swapper/123:05:071
264352726,55sleep20-21swapper/221:02:042
264352726,55sleep20-21swapper/221:02:042
689827113,42sleep36934-21/usr/sbin/munin23:25:033
66392717,48sleep00-21swapper/020:00:000
267127132,19sleep20-21swapper/219:45:202
267127132,19sleep20-21swapper/219:45:202
2223527120,26sleep30-21swapper/320:49:583
177827119,24sleep20-21swapper/221:25:192
501827020,29sleep20-21swapper/223:19:452
42927016,40sleep2474-21fschecks_count23:05:032
42927016,40sleep2474-21fschecks_count23:05:032
428827028,21sleep30-21swapper/323:15:143
3162627021,21sleep30-21swapper/321:19:463
466626915,33sleep14752-21/usr/sbin/munin21:35:111
1450326923,27sleep214630-21cat22:05:162
2785226822,21sleep10-21swapper/100:30:151
2426126827,27sleep224267-21ntp_states20:55:182
2426126827,27sleep224267-21ntp_states20:55:182
2275326818,33sleep222849-21grep20:50:172
1738026811,12sleep10-21swapper/122:15:081
12872686,14sleep10-21swapper/121:25:051
1236026813,44sleep112366-21run-parts20:16:441
1209226820,25sleep10-21swapper/121:59:461
1209226820,25sleep10-21swapper/121:59:461
1066126818,29sleep39883-21/usr/sbin/munin23:35:173
1048226821,33sleep110533-21sort23:35:151
64812678,12sleep30-21swapper/321:40:103
451326722,23sleep30-21swapper/321:34:573
2447126719,25sleep30-21swapper/300:20:073
2212726719,33sleep222215-21apt-config00:14:452
1835426722,25sleep30-21swapper/322:19:463
400726614,35sleep34022-21apt-get21:34:443
2858126619,20sleep10-21swapper/121:10:031
2858126619,20sleep10-21swapper/121:10:031
2586926617,26sleep10-21swapper/121:00:181
2586926617,26sleep10-21swapper/121:00:181
1890026615,37sleep118931-21sed20:39:461
1890026615,37sleep118931-21sed20:39:461
172452665,48sleep30-21swapper/320:34:193
172452665,48sleep30-21swapper/320:34:193
692226518,18sleep20-21swapper/220:00:102
2604726523,25sleep10-21swapper/122:44:441
198202658,12sleep10-21swapper/100:05:151
1976726515,35sleep319874-21timerandwakeup20:40:213
3000026411,23sleep30-21swapper/319:30:103
2018826417,26sleep20-21swapper/222:25:022
1751826423,26sleep217512-21apt-get20:34:592
1751826423,26sleep217512-21apt-get20:34:592
66626313,22sleep20-21swapper/219:40:112
66626313,22sleep20-21swapper/219:40:112
310412636,46sleep20-21swapper/219:34:422
2706026318,21sleep10-21swapper/100:29:461
241632637,11sleep20-21swapper/219:11:322
241632637,11sleep20-21swapper/219:11:322
228322636,16sleep30-21swapper/322:33:313
228322636,16sleep30-21swapper/322:33:313
2202426313,21sleep10-21swapper/122:30:061
2202426313,21sleep10-21swapper/122:30:061
1784526317,21sleep20-21swapper/220:35:112
1784526317,21sleep20-21swapper/220:35:112
164726323,25sleep31669-21munin-run23:09:453
164726323,25sleep31669-21munin-run23:09:453
1505026310,24sleep20-21swapper/223:50:142
967226214,29sleep10-21swapper/121:50:131
967226214,29sleep10-21swapper/121:50:131
965126218,20sleep30-21swapper/321:50:113
965126218,20sleep30-21swapper/321:50:113
872026212,26sleep20-21swapper/220:05:192
872026212,26sleep20-21swapper/220:05:192
637526220,25sleep16392-21cut21:40:081
415726212,25sleep10-21swapper/123:15:121
3195526214,24sleep10-21swapper/121:20:041
3195526214,24sleep10-21swapper/121:20:041
2809726218,27sleep127610-21apt-get22:49:571
2701726216,24sleep10-21swapper/122:45:161
279792617,11sleep20-21swapper/221:07:012
2727426111,19sleep30-21swapper/300:29:573
2687126112,27sleep30-21swapper/322:45:143
2195026114,31sleep121964-21apt20:49:441
2195026114,31sleep121964-21apt20:49:441
157342616,16sleep30-21swapper/323:53:263
3233426018,23sleep20-21swapper/221:20:142
3233426018,23sleep20-21swapper/221:20:142
3173526017,28sleep331769-21/usr/sbin/munin19:35:153
3173526017,28sleep331769-21/usr/sbin/munin19:35:153
1780326015,28sleep217813-21/usr/sbin/munin00:00:032
1748226019,18sleep20-21swapper/223:59:472
1443026016,24sleep10-21swapper/122:05:151
3146025924,21sleep20-21swapper/219:35:102
3146025924,21sleep20-21swapper/219:35:102
2560525922,15sleep10-21swapper/119:15:221
2321425916,25sleep223262-21wc00:15:302
1343025915,27sleep213432-21sshd22:02:372
402225822,17sleep30-21swapper/319:50:133
2851525815,19sleep30-21swapper/319:25:143
2572625816,22sleep30-21swapper/300:24:563
2367425812,21sleep20-21swapper/222:35:122
175825820,23sleep11768-21munin-run19:44:461
175825820,23sleep11768-21munin-run19:44:461
920425716,25sleep29201-21apt-get21:49:582
920425716,25sleep29201-21apt-get21:49:582
3096825719,22sleep10-21swapper/121:16:091
2905225710,24sleep20-21swapper/200:35:042
2905225710,24sleep20-21swapper/200:35:042
2886125716,18sleep30-21swapper/321:10:093
2886125716,18sleep30-21swapper/321:10:093
2536725714,25sleep325360-21apt-get20:59:593
2536725714,25sleep325360-21apt-get20:59:593
1987225720,21sleep220012-21latency_hist22:24:462
792625618,19sleep30-21swapper/321:45:063
657525616,23sleep20-21swapper/223:24:462
211812567,12sleep20-21swapper/220:45:212
211812567,12sleep20-21swapper/220:45:212
2049125615,22sleep20-21swapper/220:44:482
1038725612,23sleep30-21swapper/320:10:203
802625515,24sleep18053-21/usr/sbin/munin21:45:081
522625520,22sleep25310-21grep19:55:052
3124625513,24sleep231240-21apt-get22:59:592
3028425513,18sleep30-21swapper/322:55:163
2975425514,27sleep229783-21munin-run21:14:442
2975425514,27sleep229783-21munin-run21:14:442
2710725513,25sleep127144-21ntp_states19:20:111
2601225510,18sleep20-21swapper/200:25:072
2510725514,24sleep325102-21apt-get19:15:003
2479325520,18sleep20-21swapper/222:40:002
2348325519,16sleep10-21swapper/122:35:071
1578125513,17sleep20-21swapper/220:29:462
510425414,23sleep35101-21apt-get19:54:583
1542825413,24sleep215425-21apt-get22:09:562
1504525416,19sleep30-21swapper/320:25:143
1259025419,18sleep10-21swapper/122:00:061
845625314,23sleep38513-21irqstats20:05:113
845625314,23sleep38513-21irqstats20:05:113
2967225314,23sleep229667-21apt-get22:54:582
2819325316,17sleep30-21swapper/321:09:453
2506025313,24sleep11-21systemd20:59:451
2506025313,24sleep11-21systemd20:59:451
2478925312,25sleep324785-21apt-get22:39:573
2284925315,19sleep10-21swapper/100:15:211
1831825317,16sleep20-21swapper/222:19:452
1350725317,18sleep20-21swapper/223:45:132
872725212,17sleep30-21swapper/323:30:083
524025213,23sleep35257-21apt23:19:543
308322526,35sleep10-21swapper/119:32:021
2714525215,22sleep227151-21ntp_states19:20:122
1970825217,21sleep319709-21sshd22:23:183
1270525220,22sleep30-21swapper/323:44:173
97972516,17sleep10-21swapper/120:10:061
579325113,16sleep20-21swapper/221:39:442
2921225112,18sleep10-21swapper/100:35:081
2921225112,18sleep10-21swapper/100:35:081
1447525117,14sleep10-21swapper/120:25:001
1310125113,22sleep313110-21df_inode23:45:023
281542506,15sleep30-21swapper/322:50:193
1904525013,20sleep10-21swapper/122:20:131
1233125013,19sleep30-21swapper/322:00:003
90024913,20sleep3916-21ntp_states19:40:163
90024913,20sleep3916-21ntp_states19:40:163
2734324816,16sleep10-21swapper/121:05:141
192662489,15sleep30-21swapper/300:04:573
1771824814,19sleep10-21swapper/123:59:581
207012478,24sleep320768-21df20:45:053
1397124710,19sleep314066-21perl22:05:053
118072459,15sleep30-21swapper/320:15:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional