You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-21 - 03:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Tue Apr 21, 2026 00:43:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
62072143818,1405sleep122469-21kworker/u8:222:06:241
62072143818,1405sleep122469-21kworker/u8:222:06:241
32329278312,720sleep032303-21latency_hist23:30:590
2938026346,614sleep029422-21ntpq21:36:230
2726625357,491sleep027302-21ntpq19:46:240
23802227759,61sleep00-21swapper/019:36:140
24171226018,228sleep324216-21ntpq19:36:253
6891225570,125sleep06915-21modprobe20:25:570
6891225570,125sleep06915-21modprobe20:25:570
17767225250,189sleep017824-21latency00:26:190
18663224148,177sleep018769-21fschecks_time22:46:170
16687221838,102sleep00-21swapper/020:56:170
7743221614,186sleep07757-21fschecks_count22:11:160
25950220933,139sleep026040-21timerandwakeup19:41:270
23243220340,82sleep00-21swapper/023:01:110
23243220340,82sleep00-21swapper/023:01:110
22447220312,150sleep022488-21proc_pri19:31:260
1358121988,177sleep013626-21idleruntime20:46:160
26428218831,120sleep026439-21awk23:11:170
24415218718,155sleep124526-21sed19:36:321
20300218635,131sleep020303-21cron00:35:560
31107218123,141sleep231119-21/usr/sbin/munin23:26:172
9412217810,93sleep00-21swapper/020:31:300
16250216119,121sleep20-21swapper/219:11:492
16250216119,121sleep20-21swapper/219:11:492
8998215915,128sleep28946-21latency_hist22:15:582
14954215916,93sleep014956-21sshd20:51:130
12332215937,35sleep00-21swapper/022:26:100
12332215937,35sleep00-21swapper/022:26:100
20193215827,52sleep00-21swapper/021:06:270
12433215317,118sleep312538-21fschecks_time22:26:173
12433215317,118sleep312538-21fschecks_time22:26:173
1078121528,127sleep210870-21apt-config00:05:582
1353821519,126sleep00-21swapper/019:06:100
30778214715,117sleep030824-21latency_hist23:25:580
3717214660,18sleep00-21swapper/023:41:250
3717214660,18sleep00-21swapper/023:41:250
29057214544,74sleep00-21swapper/019:51:290
29057214544,74sleep00-21swapper/019:51:290
6269214026,104sleep013441-21snmpd22:06:280
6269214026,104sleep013441-21snmpd22:06:280
16096214013,112sleep316166-21basename00:21:183
1452214029,96sleep01605-21sed23:36:000
1452214029,96sleep01605-21sed23:36:000
18887213847,18sleep00-21swapper/019:21:120
18887213847,18sleep00-21swapper/019:21:120
1728321387,118sleep017433-21latency_hist00:25:590
11773213821,21sleep10-21swapper/100:06:341
31489213718,34sleep00-21swapper/021:41:530
30976213545,77sleep00-21swapper/019:57:020
30976213545,77sleep00-21swapper/019:57:020
22780213313,105sleep00-21swapper/021:16:120
16368213313,107sleep016381-21vmstat22:36:320
16368213313,107sleep016381-21vmstat22:36:320
10967213313,100sleep011163-21latency_hist00:06:000
2558621299,78sleep025629-21true21:25:570
2094921299,35sleep00-21swapper/022:51:300
15709212913,97sleep015777-21latency_hist00:20:580
15709212913,97sleep015777-21latency_hist00:20:580
471821285,108sleep04753-21ntpq22:01:230
4036212618,57sleep00-21swapper/020:16:120
4036212618,57sleep00-21swapper/020:16:120
28389212651,60sleep00-21swapper/023:16:250
20382212618,81sleep220525-21sed21:06:292
921621238,70sleep09226-21find22:16:090
25062212332,16sleep00-21swapper/023:06:200
30637212217,91sleep130723-21cut21:41:171
5853212118,82sleep35917-21cat20:21:203
5853212118,82sleep35917-21cat20:21:203
2165212125,80sleep22227-21wc23:36:262
2165212125,80sleep22227-21wc23:36:262
7587212024,70sleep27640-40ogrotate23:55:572
2761021200,7sleep10-21swapper/123:15:571
6943211920,79sleep27025-21latency_hist20:25:592
6943211920,79sleep27025-21latency_hist20:25:592
1084021195,65sleep00-21swapper/020:36:270
21689211625,65sleep221683-21apt-get22:56:122
2449211515,80sleep01-21systemd21:55:580
22093211519,85sleep00-21swapper/022:56:230
1623211324,64sleep31765-21sed20:06:303
52621086,90sleep00-21swapper/020:04:530
5186210825,62sleep24757-21/usr/sbin/munin23:46:242
2539210823,66sleep22604-21fschecks_time20:11:172
20210210812,80sleep320354-21latency_hist19:26:003
20210210812,80sleep320354-21latency_hist19:26:003
20896210718,72sleep00-21swapper/019:26:250
20896210718,72sleep00-21swapper/019:26:250
6933210630,27sleep07020-21ls23:51:260
32070210645,43sleep00-21swapper/021:46:080
28669210615,70sleep328709-21irqstats19:51:193
28669210615,70sleep328709-21irqstats19:51:193
22069210615,76sleep322094-21grep19:31:173
168721057,82sleep00-21swapper/023:36:120
168721057,82sleep00-21swapper/023:36:120
21565210428,61sleep021604-21ntpq21:11:240
348921036,86sleep00-21swapper/020:13:010
17186210343,17sleep00-21swapper/022:41:160
17186210343,17sleep00-21swapper/022:41:160
13934210314,73sleep113968-21ntp_kernel_err19:06:231
29098210113,72sleep329193-21fschecks_time21:36:183
18863210124,30sleep20-21swapper/219:21:102
18863210124,30sleep20-21swapper/219:21:102
24694210019,63sleep124301-21/usr/sbin/munin21:21:221
16987210014,25sleep20-21swapper/222:41:062
16987210014,25sleep20-21swapper/222:41:062
1398721005,28sleep00-21swapper/000:14:440
1398721005,28sleep00-21swapper/000:14:440
13301210041,40sleep213350-21latency_hist19:06:002
13301210041,40sleep213350-21latency_hist19:06:002
594729918,70sleep00-21swapper/023:49:450
832029715,70sleep00-21swapper/020:29:240
68192976,12sleep30-21swapper/322:06:553
68192976,12sleep30-21swapper/322:06:553
2597429729,19sleep026065-21cut21:26:180
1917829714,66sleep319295-21cat22:46:283
1760629742,19sleep017620-21grep19:16:200
1760629742,19sleep017620-21grep19:16:200
1457729715,63sleep213870-21/usr/sbin/munin22:31:332
155432969,41sleep015644-21processes19:11:070
155432969,41sleep015644-21processes19:11:070
1377729612,14sleep20-21swapper/200:12:122
1377729612,14sleep20-21swapper/200:12:122
1310829622,51sleep213128-21modprobe20:45:582
2020229517,62sleep320290-21cut22:51:173
3141729424,46sleep231541-21latency_hist20:01:002
3141729424,46sleep231541-21latency_hist20:01:002
3160229316,54sleep130995-21/usr/sbin/munin23:26:281
3016529316,61sleep130228-21idleruntime19:56:181
3016529316,61sleep130228-21idleruntime19:56:181
2797029311,37sleep00-21swapper/021:31:270
1248529318,53sleep112602-21ls20:41:291
1197829322,53sleep011979-21cpu19:01:140
1197829322,53sleep011979-21cpu19:01:140
90229219,17sleep00-21swapper/020:06:100
151092910,6sleep20-21swapper/219:10:592
119082915,11sleep00-21swapper/022:24:120
609229017,52sleep16238-21sed23:50:591
314428912,62sleep13188-21ntpq21:56:231
99028722,41sleep21022-21fschecks_count20:06:172
1713728725,43sleep217218-21cat20:56:282
83532860,6sleep10-21swapper/123:56:231
2119728613,50sleep321170-21/usr/sbin/munin21:11:153
1242028613,56sleep212432-21/usr/sbin/munin22:26:162
1242028613,56sleep212432-21/usr/sbin/munin22:26:162
79152858,15sleep00-21swapper/023:56:110
267732850,8sleep30-21swapper/323:11:233
252128522,49sleep12550-21sed21:55:591
112572857,37sleep00-21swapper/000:06:120
1007828515,55sleep110161-21wc00:01:291
956628413,55sleep19632-21users20:31:311
2451628413,57sleep224575-21if_eth021:21:192
2026128415,54sleep120381-21sed21:06:281
1256628423,31sleep20-21swapper/200:10:592
399728315,57sleep00-21swapper/022:00:300
1389728355,12sleep00-21swapper/022:31:150
3082628214,52sleep130976-21latency_hist23:25:591
1914228224,27sleep20-21swapper/200:31:182
432328118,47sleep34295-21/usr/sbin/munin22:01:143
2344128112,52sleep223661-21latency_hist19:36:002
1155928117,47sleep311229-21/usr/sbin/munin00:06:283
17642809,42sleep30-21swapper/323:36:153
17642809,42sleep30-21swapper/323:36:153
478127910,50sleep30-21swapper/323:46:123
1323627827,31sleep113298-21systemd-run19:05:591
1323627827,31sleep113298-21systemd-run19:05:591
2296427619,41sleep222756-21/usr/sbin/munin21:16:182
1747527614,27sleep30-21swapper/322:41:233
1747527614,27sleep30-21swapper/322:41:233
37802750,7sleep10-21swapper/123:41:251
37802750,7sleep10-21swapper/123:41:251
1941027522,24sleep10-21swapper/121:06:001
1672627523,28sleep10-21swapper/122:40:571
1672627523,28sleep10-21swapper/122:40:571
1101227512,29sleep20-21swapper/220:36:292
2969927315,40sleep129720-21ls19:55:581
2969927315,40sleep129720-21ls19:55:581
2581727310,48sleep20-21swapper/219:41:252
2432427324,32sleep324320-21apt-get21:21:123
1401227318,41sleep214094-21grep20:46:262
918127217,42sleep29228-21/usr/sbin/munin00:00:572
745927213,23sleep20-21swapper/220:26:232
70222720,7sleep10-21swapper/123:51:261
440727219,28sleep30-21swapper/320:16:233
440727219,28sleep30-21swapper/320:16:233
185632727,47sleep00-21swapper/021:01:260
899927116,34sleep30-21swapper/320:31:213
412527121,32sleep24205-21fschecks_time20:16:182
412527121,32sleep24205-21fschecks_time20:16:182
2052027121,29sleep30-21swapper/319:26:163
2052027121,29sleep30-21swapper/319:26:163
1640627111,45sleep216414-21ntpq00:21:252
1662027020,28sleep30-21swapper/320:56:163
1011227021,32sleep210261-21sed20:36:002
84382690,6sleep10-21swapper/122:11:281
369426917,28sleep30-21swapper/323:41:243
369426917,28sleep30-21swapper/323:41:243
3056826925,30sleep330570-21sshd23:23:553
2743326917,27sleep20-21swapper/219:46:302
2301226923,20sleep10-21swapper/123:01:001
1780626924,25sleep20-21swapper/219:16:262
1780626924,25sleep20-21swapper/219:16:262
1539926922,28sleep114835-21apt-get19:11:031
1539926922,28sleep114835-21apt-get19:11:031
2876426819,30sleep10-21swapper/121:36:001
200742686,11sleep30-21swapper/300:33:053
1129326821,33sleep311382-21mailstats22:21:293
1078026818,31sleep30-21swapper/320:36:263
331126719,29sleep20-21swapper/221:56:282
2682626716,35sleep226889-21/usr/sbin/munin23:11:272
2531626719,27sleep125335-21apt-get19:41:081
2127926718,31sleep221282-21/usr/sbin/munin21:11:182
1934426718,32sleep319382-21munin-run21:05:573
1779926712,38sleep117805-21ntp_states19:16:251
1779926712,38sleep117805-21ntp_states19:16:251
1747126724,22sleep10-21swapper/100:26:081
2487926620,19sleep10-21swapper/123:06:171
2071726619,29sleep220794-21/usr/sbin/munin22:51:282
1786626618,34sleep217902-21ntpq00:26:212
823026519,25sleep30-21swapper/322:11:273
1555726516,24sleep20-21swapper/222:36:182
1555726516,24sleep20-21swapper/222:36:182
2971126414,19sleep20-21swapper/223:21:202
2374626418,25sleep30-21swapper/323:01:263
2374626418,25sleep30-21swapper/323:01:263
2062026414,34sleep120648-21/usr/sbin/munin19:26:181
2062026414,34sleep120648-21/usr/sbin/munin19:26:181
118492648,13sleep00-21swapper/020:41:110
440626319,20sleep20-21swapper/222:01:172
3219226322,23sleep10-21swapper/120:01:311
3185826316,25sleep30-21swapper/321:45:593
3039726315,32sleep330442-21grep19:56:243
3039726315,32sleep330442-21grep19:56:243
2961126319,22sleep20-21swapper/221:36:292
2606826313,33sleep126126-21/usr/sbin/munin21:26:191
1965326322,21sleep10-21swapper/100:31:291
1001226318,26sleep110140-21/usr/sbin/munin22:16:301
1399326217,19sleep10-21swapper/122:31:201
95226122,23sleep21033-21latency_hist21:50:592
2954326111,27sleep10-21swapper/121:36:281
276726111,28sleep30-21swapper/321:56:133
2651326119,22sleep30-21swapper/321:26:293
2610126119,20sleep30-21swapper/323:10:593
2333826124,15sleep10-21swapper/119:35:591
2333126116,32sleep123379-21fschecks_count23:01:161
2333126116,32sleep123379-21fschecks_count23:01:161
1654926118,23sleep10-21swapper/100:21:281
1349326122,23sleep30-21swapper/320:46:153
1087426116,31sleep110942-21grep22:21:181
572326013,29sleep15917-21latency_hist22:06:001
34226016,26sleep3486-21/usr/sbin/munin21:46:323
2314226017,25sleep10-21swapper/121:16:211
2012826012,21sleep30-21swapper/321:06:263
1911026016,24sleep20-21swapper/222:46:272
331825914,18sleep20-21swapper/223:41:152
331825914,18sleep20-21swapper/223:41:152
2828025913,17sleep30-21swapper/321:31:303
1835625920,18sleep20-21swapper/221:01:202
1087125920,24sleep30-21swapper/300:05:593
90325813,27sleep1898-21apt-get20:06:111
3223625821,20sleep10-21swapper/121:46:181
2788725818,24sleep20-21swapper/221:31:252
169125815,20sleep10-21swapper/123:36:131
169125815,20sleep10-21swapper/123:36:131
1593325819,26sleep30-21swapper/322:36:273
1593325819,26sleep30-21swapper/322:36:273
129932586,10sleep30-21swapper/319:02:543
129932586,10sleep30-21swapper/319:02:543
608125720,23sleep26101-21dpkg22:06:092
608125720,23sleep26101-21dpkg22:06:092
430825716,19sleep10-21swapper/120:16:211
430825716,19sleep10-21swapper/120:16:211
2972325722,19sleep20-21swapper/219:55:592
2972325722,19sleep20-21swapper/219:55:592
2792425715,26sleep127921-21apt-get23:16:101
2285525717,24sleep322864-21/usr/sbin/munin21:16:153
1856025711,26sleep118557-21apt-get22:46:101
1649825719,17sleep30-21swapper/319:11:543
1649825719,17sleep30-21swapper/319:11:543
1566625718,21sleep10-21swapper/120:51:301
1233525713,30sleep312409-21mailstats20:41:273
2923425618,17sleep20-21swapper/223:21:002
282125616,23sleep32427-21/usr/sbin/munin20:11:223
272225614,28sleep12777-21cat20:11:201
2011125615,24sleep120106-21apt-get22:51:111
1629425621,21sleep30-21swapper/320:56:003
96732559,18sleep30-21swapper/322:16:243
322252558,28sleep20-21swapper/220:01:322
3102425516,20sleep30-21swapper/323:26:143
2495825519,18sleep20-21swapper/223:06:182
1880525512,30sleep318911-21latency_hist00:30:593
1723425514,23sleep117284-21irqstats22:41:171
1723425514,23sleep117284-21irqstats22:41:171
1448625514,21sleep30-21swapper/300:16:173
1448625514,21sleep30-21swapper/300:16:173
136925515,21sleep20-21swapper/223:35:582
136925515,21sleep20-21swapper/223:35:582
764825416,23sleep37669-21awk20:26:283
2595525413,25sleep225962-21diskstats21:26:162
2587725412,21sleep325948-21sensors_temp19:41:263
2094325413,15sleep20-21swapper/219:26:272
2094325413,15sleep20-21swapper/219:26:272
1784625417,19sleep30-21swapper/319:16:273
1784625417,19sleep30-21swapper/319:16:273
153312546,12sleep20-21swapper/200:16:442
153312546,12sleep20-21swapper/200:16:442
1030325412,24sleep110297-21apt-get20:36:121
56625315,24sleep1612-21grep23:31:251
56625315,24sleep1612-21grep23:31:251
255125313,16sleep20-21swapper/221:56:002
1492525311,25sleep115003-21mailstats00:16:271
1492525311,25sleep115003-21mailstats00:16:271
1280125313,23sleep112795-21apt-get00:11:101
1280125313,23sleep112795-21apt-get00:11:101
1233325312,25sleep112328-21apt-get22:26:111
1233325312,25sleep112328-21apt-get22:26:111
1078325314,22sleep210780-21apt-get22:21:112
931325213,14sleep20-21swapper/222:16:152
673825214,22sleep36780-21ntp_states23:51:213
559125214,22sleep15657-21df20:21:141
559125214,22sleep15657-21df20:21:141
2858125217,17sleep30-21swapper/323:16:293
2376725210,22sleep20-21swapper/219:36:082
1394125218,15sleep10-21swapper/120:46:251
68425114,16sleep30-21swapper/323:31:273
68425114,16sleep30-21swapper/323:31:273
1889025112,22sleep118882-21apt-get19:21:131
1889025112,22sleep118882-21apt-get19:21:131
164525113,22sleep31651-21ntp_states21:51:243
1356225111,23sleep313557-21apt-get19:06:133
1294525112,23sleep313012-21fw_packets00:11:163
1294525112,23sleep313012-21fw_packets00:11:163
2819125011,22sleep328201-21munin-run19:50:593
2160625013,16sleep10-21swapper/121:11:251
1652825011,23sleep116523-21apt-get20:56:101
3128024917,16sleep20-21swapper/221:41:302
1580924912,22sleep215859-21users20:51:312
2689424810,22sleep126895-21sshd19:46:131
856224711,18sleep30-21swapper/323:56:303
2324724711,14sleep20-21swapper/223:01:132
2324724711,14sleep20-21swapper/223:01:132
2215824714,17sleep30-21swapper/322:56:263
847924614,16sleep10-21swapper/120:30:591
3208524616,17sleep30-21swapper/320:01:293
1490524210,18sleep313441-21snmpd22:32:513
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional