You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-31 - 08:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Mon Aug 31, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
71222250914,2469sleep125654-1kworker/1:0H23:58:511
31565244518,410sleep131533-21latency_hist23:33:521
139824108,384sleep0367-21snmpd20:14:210
11719233074,191sleep011733-21fschecks_count20:49:090
11719233074,191sleep011733-21fschecks_count20:49:090
19518232735,126sleep019602-21apt-config21:13:520
16059227043,213sleep016100-21dpkg21:03:510
880122528,221sleep00-21swapper/020:39:120
17741224250,114sleep017785-21ls19:24:110
8775223919,205sleep38849-21sort22:19:233
8290223972,91sleep00-21swapper/020:38:520
13487222438,15sleep00-21swapper/022:34:260
24058221814,192sleep324082-21cut19:44:123
11405221812,192sleep011420-21awk00:09:230
5089220846,78sleep00-21swapper/022:09:070
6082220515,177sleep06096-21grep23:54:130
31714218222,145sleep331772-21cut20:09:113
1141621809,111sleep011522-21fschecks_time22:29:130
48221738,15sleep00-21swapper/020:13:530
1440621737,149sleep00-21swapper/019:14:010
14980217049,65sleep015132-21latency_hist00:23:530
14980217049,65sleep015132-21latency_hist00:23:530
2927221688,142sleep029404-21latency_hist21:43:520
2927221688,142sleep029404-21latency_hist21:43:520
2723216815,137sleep32765-21/usr/sbin/munin20:19:173
23754216617,134sleep023758-21find23:09:000
7378216247,34sleep00-21swapper/023:59:020
23877216247,103sleep023912-21fschecks_count19:44:090
31382216110,137sleep031431-21cat20:08:520
18297215914,130sleep218344-21sshd00:34:042
18297215914,130sleep218344-21sshd00:34:042
12040215518,113sleep212189-21unixbench_singl22:29:272
21273215212,127sleep121346-21find19:34:211
22004215070,21sleep00-21swapper/019:38:520
8335214415,110sleep18235-21latency_hist20:38:531
5870214457,32sleep05383-21/usr/sbin/munin20:29:190
20750213832,53sleep00-21swapper/022:59:070
26362213439,24sleep00-21swapper/021:34:030
26362213439,24sleep00-21swapper/021:34:030
27398212613,97sleep327433-21irqstats23:19:133
3269211835,20sleep00-21swapper/023:44:200
15202211723,21sleep00-21swapper/000:24:050
15202211723,21sleep00-21swapper/000:24:050
454021159,88sleep04246-21/usr/sbin/munin23:49:130
11755211539,49sleep011855-21latency_hist19:08:520
11755211539,49sleep011855-21latency_hist19:08:520
7054211444,57sleep07091-21ntp_states22:14:200
20670211432,44sleep30-21swapper/322:59:023
28806211316,77sleep00-21swapper/023:24:100
28474211244,56sleep028540-21ls19:59:070
19802211216,75sleep319940-21grep19:29:223
19802211216,75sleep319940-21grep19:29:223
11392211236,16sleep00-21swapper/020:48:530
7625211110,14sleep30-21swapper/323:59:133
25571211131,63sleep00-21swapper/023:14:050
17883211111,76sleep00-21swapper/021:08:510
17883211111,76sleep00-21swapper/021:08:510
31102211033,17sleep00-21swapper/023:29:230
2075210915,19sleep00-21swapper/023:39:240
11075210933,16sleep00-21swapper/022:28:530
24631210815,74sleep324788-21latency_hist21:28:533
3246210710,15sleep00-21swapper/022:03:500
31779210734,15sleep00-21swapper/023:34:030
26146210631,15sleep00-21swapper/019:49:220
25061210637,12sleep00-21swapper/021:29:140
1418210614,75sleep01470-21modprobe21:58:500
12341210638,14sleep00-21swapper/019:09:290
12341210638,14sleep00-21swapper/019:09:290
10081210618,74sleep210101-21cpu20:44:072
4001210514,69sleep14037-21proc_pri22:04:201
21706210511,28sleep00-21swapper/021:19:010
9421210420,67sleep00-21swapper/000:04:190
31059210443,13sleep00-21swapper/021:49:050
31059210443,13sleep00-21swapper/021:49:050
612210319,64sleep2631-21cron23:38:492
22820210316,70sleep00-21swapper/019:39:210
16829210335,13sleep00-21swapper/000:29:100
16829210335,13sleep00-21swapper/000:29:100
19125210231,13sleep00-21swapper/019:29:070
19125210231,13sleep00-21swapper/019:29:070
8224210134,13sleep00-21swapper/022:19:080
31624210115,34sleep20-21swapper/220:09:062
1322221006,83sleep00-21swapper/000:17:230
10680210032,46sleep110743-21latency19:04:131
10680210032,46sleep110743-21latency19:04:131
2921429916,70sleep229294-21readlink23:24:202
2336829933,13sleep00-21swapper/021:24:100
172582989,73sleep017287-21munin-run22:48:500
1858529714,68sleep318607-21/usr/sbin/munin21:09:193
272829622,53sleep12879-21grep21:59:271
232092965,12sleep00-21swapper/023:05:130
1515529610,67sleep115282-21/usr/sbin/munin19:15:151
1252029513,30sleep212526-21ntp_states00:14:182
3235829221,56sleep232382-21latency_hist21:53:512
3235829221,56sleep232382-21latency_hist21:53:512
1169929213,22sleep211710-21df_inode20:49:072
1169929213,22sleep211710-21df_inode20:49:072
666429117,54sleep36681-21ls20:33:503
3024129123,47sleep230388-21unixbench_singl21:44:222
3024129123,47sleep230388-21unixbench_singl21:44:222
2976929010,61sleep00-21swapper/021:44:120
2976929010,61sleep00-21swapper/021:44:120
35122899,18sleep00-21swapper/022:04:020
260482898,24sleep30-21swapper/323:14:323
2134928925,46sleep221493-21grep19:34:222
557328813,60sleep15632-21basename20:29:121
2375728813,58sleep223264-21/usr/sbin/munin21:24:242
182952889,18sleep00-21swapper/000:34:020
182952889,18sleep00-21swapper/000:34:020
464128712,60sleep24673-21/usr/sbin/munin23:49:152
1987828712,18sleep00-21swapper/021:14:000
3023228614,55sleep330250-21apt23:29:013
2042028619,33sleep20-21swapper/221:14:272
577928523,39sleep15936-21unixbench_multi22:09:241
495528522,42sleep15097-21grep23:49:221
44382859,17sleep00-21swapper/020:24:220
184482856,41sleep00-21swapper/022:49:230
1652828540,34sleep01-21systemd19:19:220
131602858,19sleep00-21swapper/020:54:020
131602858,19sleep00-21swapper/020:54:020
2555228413,53sleep125608-21/usr/sbin/munin19:49:111
374728311,15sleep20-21swapper/222:04:132
1909828314,18sleep00-21swapper/022:54:070
23112818,16sleep00-21swapper/020:19:050
1491128116,20sleep10-21swapper/100:23:501
1491128116,20sleep10-21swapper/100:23:501
894228023,37sleep19083-21head22:19:251
867828043,20sleep20-21swapper/220:39:102
2775928017,19sleep20-21swapper/221:38:532
2118028026,35sleep121206-21apt-get21:18:501
194428016,50sleep21449-21apt-get21:59:002
1167128031,36sleep311752-21apt-config19:08:513
1167128031,36sleep311752-21apt-config19:08:513
1670927914,45sleep316712-21fschecks_time21:04:143
1670927914,45sleep316712-21fschecks_time21:04:143
260232789,16sleep20-21swapper/223:14:312
2428427811,53sleep224363-21systemctl23:09:192
2665827725,30sleep30-21swapper/321:34:143
2665827725,30sleep30-21swapper/321:34:143
165552778,52sleep10-21swapper/121:04:041
165552778,52sleep10-21swapper/121:04:041
1112327712,42sleep211251-21timerwakeupswit00:09:212
2978427621,38sleep329806-21apt20:03:503
2684327629,22sleep10-21swapper/123:18:501
2128827624,22sleep30-21swapper/321:18:523
790027511,49sleep37977-21systemd-run22:18:513
397927527,26sleep13990-21modprobe23:48:491
2574227512,27sleep30-21swapper/321:29:303
1610127523,35sleep116240-21date21:03:521
1262927517,41sleep312678-21grep22:33:523
2842027433,20sleep00-21swapper/023:23:500
2035527415,44sleep320393-21cat22:58:503
749427321,23sleep10-21swapper/123:59:111
320527315,25sleep30-21swapper/323:44:193
2526827326,20sleep10-21swapper/123:13:541
199327324,32sleep1231-21systemd-journal20:18:501
910827220,38sleep19174-21idleruntime00:04:111
49222729,15sleep20-21swapper/222:08:532
1965827223,28sleep219738-21switchtime22:54:212
1961227217,39sleep119402-21latency_hist21:13:531
716227113,43sleep37215-21interrupts20:34:103
2689327110,46sleep00-21swapper/019:54:060
1365627110,15sleep20-21swapper/200:19:062
1365627110,15sleep20-21swapper/200:19:062
798127024,27sleep10-21swapper/122:18:531
3001427019,24sleep10-21swapper/123:28:511
2754027019,28sleep20-21swapper/223:19:162
2724827018,37sleep127339-21fschecks_time23:19:101
253127014,23sleep30-21swapper/321:59:253
2400127014,21sleep20-21swapper/219:44:112
2113627017,39sleep321199-21ps19:34:193
63226924,18sleep30-21swapper/323:38:503
398026916,24sleep20-21swapper/220:24:122
3145826919,26sleep30-21swapper/321:49:163
3145826919,26sleep30-21swapper/321:49:163
2379326913,13sleep10-21swapper/119:44:061
1602926918,35sleep316044-21apt-get21:03:503
1415426913,14sleep30-21swapper/300:19:193
1415426913,14sleep30-21swapper/300:19:193
292926812,37sleep13061-21grep20:19:231
2614926824,27sleep20-21swapper/221:33:532
2094826815,22sleep10-21swapper/122:59:131
144482687,45sleep00-21swapper/022:39:020
1220926819,29sleep111695-21apt-get19:09:011
1220926819,29sleep111695-21apt-get19:09:011
1087126822,28sleep210893-21/usr/sbin/munin19:04:192
1087126822,28sleep210893-21/usr/sbin/munin19:04:192
870926720,30sleep20-21swapper/222:19:222
306326713,25sleep20-21swapper/220:19:242
2269326727,22sleep20-21swapper/223:04:182
2231626715,23sleep10-21swapper/123:04:091
2113726719,27sleep20-21swapper/222:59:182
1541826719,23sleep20-21swapper/220:59:212
1513126718,22sleep20-21swapper/222:39:202
1499626719,20sleep30-21swapper/319:15:133
871926615,28sleep20-21swapper/200:03:522
1701726614,21sleep20-21swapper/221:04:212
1701726614,21sleep20-21swapper/221:04:212
147826616,37sleep11492-21apt-key21:58:511
1454226618,22sleep20-21swapper/220:58:522
1454226618,22sleep20-21swapper/220:58:522
1424926618,30sleep214290-21munin-node19:13:592
1335426619,25sleep20-21swapper/222:34:242
118892666,15sleep10-21swapper/100:13:521
704026517,21sleep10-21swapper/120:34:081
435626514,25sleep30-21swapper/320:24:213
394726521,23sleep10-21swapper/120:24:111
2235126510,41sleep222441-21grep19:39:102
1732926518,23sleep20-21swapper/219:23:522
173026520,28sleep10-21swapper/123:39:211
2436426418,26sleep30-21swapper/323:09:203
232572647,18sleep10-21swapper/121:23:591
1938126419,27sleep20-21swapper/221:13:502
1177926415,35sleep111785-21ntp_states22:29:221
69226315,31sleep2763-21/usr/sbin/munin21:54:202
3007326215,29sleep130066-21apt-get20:04:041
1360526222,19sleep10-21swapper/122:34:271
1242726223,23sleep30-21swapper/300:14:153
1142126218,26sleep111418-21vmstat00:09:241
1025526213,35sleep210323-21systemctl22:24:232
3229126124,20sleep10-21swapper/123:34:201
3076826116,29sleep130770-21cron21:48:491
3076826116,29sleep130770-21cron21:48:491
2850126113,20sleep30-21swapper/321:39:233
2850126113,20sleep30-21swapper/321:39:233
228292616,12sleep10-21swapper/121:21:401
1963126126,18sleep10-21swapper/119:29:201
1963126126,18sleep10-21swapper/119:29:201
1263726118,25sleep312663-21netstat19:09:363
1263726118,25sleep312663-21netstat19:09:363
1097026116,27sleep310993-21/usr/sbin/munin00:09:183
815526013,28sleep28157-21sshd20:38:052
1797726015,26sleep118001-21ntp_states22:49:161
169632606,11sleep20-21swapper/222:45:052
1021726018,28sleep310237-21munin-run00:08:503
2221725918,22sleep328391-1kworker/3:2H21:19:283
1939125914,31sleep219419-21kernelversion19:29:132
1939125914,31sleep219419-21kernelversion19:29:132
1817525923,19sleep218258-21grep19:24:212
1684225918,18sleep20-21swapper/200:29:122
1684225918,18sleep20-21swapper/200:29:122
1601825915,18sleep30-21swapper/319:19:053
1341425917,26sleep313485-21/usr/sbin/munin22:34:253
1318225915,27sleep113179-21apt-get20:54:031
1318225915,27sleep113179-21apt-get20:54:031
549625816,23sleep30-21swapper/322:09:183
2732325811,31sleep327346-21ntp_states19:54:173
45412576,15sleep30-21swapper/322:05:353
2957625711,31sleep329587-21df_inode21:44:073
2957625711,31sleep329587-21df_inode21:44:073
2262825712,17sleep30-21swapper/323:04:153
2043625720,18sleep10-21swapper/122:58:521
1890525715,22sleep10-21swapper/100:34:191
1890525715,22sleep10-21swapper/100:34:191
180725724,17sleep20-21swapper/223:39:222
1044925714,19sleep30-21swapper/320:44:163
477925618,16sleep30-21swapper/323:49:203
2710025621,17sleep20-21swapper/219:54:122
2481925614,24sleep124831-21apt-get21:29:031
1805725618,17sleep10-21swapper/100:33:511
1805725618,17sleep10-21swapper/100:33:511
1791225614,22sleep30-21swapper/322:49:143
122525617,17sleep30-21swapper/323:39:073
977325513,19sleep10-21swapper/122:24:091
399325512,26sleep24030-21munin-run23:48:502
283362555,12sleep20-21swapper/221:39:182
283362555,12sleep20-21swapper/221:39:182
2822025513,28sleep128176-21/usr/sbin/munin19:58:521
2638925517,21sleep20-21swapper/219:51:062
1888325513,17sleep20-21swapper/222:53:532
1651125516,16sleep30-21swapper/300:28:523
1651125516,16sleep30-21swapper/300:28:523
120062557,13sleep10-21swapper/120:49:141
120062557,13sleep10-21swapper/120:49:141
76425415,18sleep30-21swapper/321:54:213
3143425414,23sleep131353-21latency_hist20:08:531
3054925413,17sleep30-21swapper/320:04:193
2954725412,25sleep229544-21apt-get20:01:072
2534025415,24sleep325333-21apt-get19:49:053
226962546,38sleep20-21swapper/221:20:002
207225415,18sleep30-21swapper/320:18:523
582225312,20sleep20-21swapper/223:54:072
2796125311,25sleep127955-21apt-get21:39:041
2796125311,25sleep127955-21apt-get21:39:041
1876725316,24sleep318831-21ps00:34:173
1876725316,24sleep318831-21ps00:34:173
1221225316,20sleep30-21swapper/320:49:203
1221225316,20sleep30-21swapper/320:49:203
91762527,16sleep20-21swapper/200:04:122
621925218,19sleep30-21swapper/323:54:163
2846825213,22sleep128465-21apt-get19:59:041
1646525210,15sleep10-21swapper/119:19:191
1625525213,22sleep116268-21irqstats22:44:111
660625110,16sleep20-21swapper/222:14:012
2739025112,27sleep127392-21sshd21:35:331
2739025112,27sleep127392-21sshd21:35:331
1764625112,23sleep117652-21df_inode19:24:081
1757025110,18sleep30-21swapper/319:24:053
1444925112,23sleep114442-21apt-get22:39:031
1222725111,18sleep20-21swapper/219:09:022
1222725111,18sleep20-21swapper/219:09:022
1106425115,21sleep311073-21munin-run22:28:523
71825011,23sleep2715-21apt-get20:14:032
3025725013,22sleep230253-21apt-get23:29:042
2233025018,16sleep10-21swapper/119:39:091
1519625013,21sleep115193-21apt-get00:24:021
1519625013,21sleep115193-21apt-get00:24:021
1454925010,16sleep30-21swapper/322:39:093
1016225012,22sleep110166-21diskstats20:44:101
563424913,20sleep25677-21irqstats20:29:132
2231024812,21sleep322321-21df_inode19:39:073
1385924810,18sleep30-21swapper/320:54:213
1385924810,18sleep30-21swapper/320:54:213
860224713,20sleep38611-21df_inode20:39:073
3007424715,16sleep20-21swapper/220:04:062
268042479,14sleep20-21swapper/221:34:172
1570824712,17sleep20-21swapper/200:24:192
1570824712,17sleep20-21swapper/200:24:192
2916224612,17sleep30-21swapper/323:24:173
1610624613,16sleep30-21swapper/322:44:093
3133424412,18sleep131428-21memory21:49:141
3133424412,18sleep131428-21memory21:49:141
1675824414,16sleep30-21swapper/300:29:093
1675824414,16sleep30-21swapper/300:29:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional