You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-06 - 03:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackbslot7s.osadl.org (updated Mon Apr 06, 2026 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
210932142368,14209sleep218326-21kworker/2:222:42:182
232562526437,5189sleep218326-21kworker/2:222:51:182
11632235175,3499sleep011596-21kworker/0:222:13:180
11632235175,3499sleep011596-21kworker/0:222:13:180
15997234815,313sleep315876-21latency_hist20:46:533
15352224861,168sleep215442-21cut00:07:122
15352224861,168sleep215442-21cut00:07:122
18076220029,95sleep218223-21grep22:32:222
3019921989,164sleep230278-21cat19:47:122
22707217114,141sleep322713-21df_inode21:07:083
1187621618,104sleep213441-21snmpd22:16:192
1187621618,104sleep213441-21snmpd22:16:192
12764215817,125sleep012779-21kernelversion22:17:300
21102215618,23sleep00-21swapper/019:17:190
6710215445,96sleep26797-21wc23:41:522
6710215445,96sleep26797-21wc23:41:522
1862215212,18sleep20-21swapper/220:01:502
721721509,58sleep20-21swapper/223:42:132
32036214816,77sleep20-21swapper/223:17:142
22379214613,120sleep221479-21/usr/sbin/munin00:27:232
21965214513,111sleep121978-21/usr/sbin/munin00:27:181
26517213953,24sleep226576-21grep19:36:512
19400213713,105sleep119006-21/usr/sbin/munin19:12:151
19400213713,105sleep119006-21/usr/sbin/munin19:12:151
1189213613,103sleep21225-21proc_pri23:22:182
15424213526,85sleep20-21swapper/222:26:532
23444213412,18sleep00-21swapper/019:26:520
2208921347,13sleep10-21swapper/122:47:061
27320213174,13sleep20-21swapper/223:02:202
246521310,8sleep30-21swapper/321:46:523
246521310,8sleep30-21swapper/321:46:523
25343212923,88sleep10-21swapper/119:32:091
19290212921,36sleep20-21swapper/220:56:532
19290212921,36sleep20-21swapper/220:56:532
22407212722,89sleep022434-21awk22:47:130
2862521268,98sleep228764-21latency_hist21:26:532
29652212549,60sleep229827-21date23:11:532
29652212549,60sleep229827-21date23:11:532
13048212512,87sleep00-21swapper/020:37:050
30775212312,58sleep230791-21netstat21:32:142
30775212312,58sleep230791-21netstat21:32:142
30047212230,72sleep329538-21apt-get23:12:063
18584212217,86sleep118717-21wc20:52:201
18584212217,86sleep118717-21wc20:52:201
20677212118,83sleep220831-21grep00:22:222
4957212043,64sleep21-21systemd20:11:512
25447212020,83sleep025502-21tr21:16:500
23003212049,52sleep20-21swapper/221:07:152
12440211920,79sleep311933-21apt-get22:17:013
12005211844,61sleep212080-21/usr/sbin/munin20:32:232
808721177,17sleep00-21swapper/023:45:180
17357211611,26sleep20-21swapper/220:48:052
2707821149,90sleep227165-21wc21:21:532
926621138,91sleep213441-21snmpd23:47:222
16402211218,80sleep1222-21systemd-journal00:10:181
16402211218,80sleep1222-21systemd-journal00:10:181
24643211168,21sleep20-21swapper/221:12:202
14308211021,75sleep1222-21systemd-journal20:41:501
1010221107,88sleep29767-21apt-get23:51:542
1010221107,88sleep29767-21apt-get23:51:542
2401921097,87sleep20-21swapper/222:52:152
15779210940,20sleep215798-21apt-get22:27:012
28862210836,16sleep20-21swapper/221:27:052
4148210726,56sleep14167-21apt-get21:52:001
24845210714,77sleep124869-21idleruntime-cro22:56:501
20005210724,62sleep119495-21/usr/sbin/munin20:57:211
5714210637,17sleep20-21swapper/220:12:202
10149210513,73sleep210190-21systemd-run22:11:512
14187210435,15sleep20-21swapper/200:03:002
11663210424,58sleep311565-21latency_hist23:56:553
11663210424,58sleep311565-21latency_hist23:56:553
5930210223,53sleep16070-21sed20:12:231
262829922,47sleep32623-21apt-get21:47:053
262829922,47sleep32623-21apt-get21:47:053
2577629919,19sleep20-21swapper/222:57:202
73672989,62sleep10-21swapper/123:42:161
1286429814,24sleep20-21swapper/222:17:322
680629710,72sleep218471-21munin-node20:17:002
2992629733,37sleep00-21swapper/019:47:020
1678729712,68sleep20-21swapper/200:12:012
3204829518,58sleep032139-21switchtime19:52:190
197882957,72sleep219842-21latency20:57:142
38229315,27sleep20-21swapper/219:56:532
3172129315,65sleep031751-21grep23:17:080
93529210,69sleep3985-21/usr/sbin/munin23:22:113
61242928,69sleep20-21swapper/221:57:182
370029214,63sleep13701-21cpu20:07:061
295732925,12sleep20-21swapper/219:45:592
1707729223,38sleep30-21swapper/322:31:503
1387529220,60sleep013918-21fschecks_time00:02:540
151722917,12sleep10-21swapper/120:42:191
1459429110,18sleep20-21swapper/220:42:042
1362129112,64sleep213709-21sed20:37:192
273772908,70sleep226809-21/usr/sbin/munin19:37:192
2354628912,62sleep323630-21awk00:32:213
73952888,67sleep27415-21fschecks_count22:02:102
5132885,11sleep20-21swapper/221:38:202
5132885,11sleep20-21swapper/221:38:202
500728817,26sleep30-21swapper/320:11:523
21522888,18sleep20-21swapper/220:02:062
1317628821,30sleep10-21swapper/122:17:401
2671628714,53sleep326691-21/usr/sbin/munin23:02:053
2515928716,35sleep00-21swapper/022:57:020
222922875,50sleep20-21swapper/221:06:182
1920028719,51sleep319201-21cron20:56:503
1920028719,51sleep319201-21cron20:56:503
3001928610,21sleep229538-21apt-get23:12:012
3105928417,41sleep131188-21timerwakeupswit21:32:211
3105928417,41sleep131188-21timerwakeupswit21:32:211
189472845,43sleep20-21swapper/222:37:062
117342847,15sleep20-21swapper/223:56:582
972828310,17sleep20-21swapper/220:26:532
3223828319,32sleep30-21swapper/323:17:193
224322838,61sleep222475-21cat19:22:132
1851328314,35sleep00-21swapper/020:52:190
1851328314,35sleep00-21swapper/020:52:190
1438828312,55sleep014423-21ntp_kernel_err22:22:150
694728112,56sleep36960-21cpu23:42:063
317428112,53sleep20-21swapper/223:27:232
289628110,14sleep20-21swapper/221:47:122
289628110,14sleep20-21swapper/221:47:122
2711528118,46sleep127158-21/usr/sbin/munin23:02:151
1911528113,47sleep019149-21cut19:12:100
1911528113,47sleep019149-21cut19:12:100
1728028110,56sleep317317-21ps00:12:183
1302328111,53sleep313040-21dpkg00:00:103
729228022,34sleep30-21swapper/320:17:173
730927918,29sleep00-21swapper/022:02:040
231692798,57sleep223257-21perl00:32:112
989427821,30sleep10-21swapper/123:51:531
711527817,21sleep10-21swapper/122:01:531
1945227816,45sleep319540-21grep22:37:193
428527717,41sleep04317-21proc_pri23:32:180
2731727714,49sleep027358-21grep21:22:070
2313627720,29sleep10-21swapper/100:32:101
226912777,59sleep00-21swapper/000:30:560
2147127719,38sleep321049-21/usr/sbin/munin21:02:153
84127622,23sleep30-21swapper/321:41:523
84127622,23sleep30-21swapper/321:41:523
38502767,18sleep00-21swapper/021:51:220
38502767,18sleep00-21swapper/021:51:220
250062760,7sleep00-21swapper/022:56:520
1966427612,49sleep20-21swapper/200:21:502
1557127623,36sleep115614-21fschecks_count19:02:101
112822769,48sleep20-21swapper/220:31:542
2342227517,37sleep323438-21apt19:26:513
1376427523,36sleep313700-21apt-get00:02:453
63942746,11sleep30-21swapper/323:38:263
63942746,11sleep30-21swapper/323:38:263
237122748,49sleep20-21swapper/219:27:062
1962727422,26sleep119757-21grep22:37:211
72732738,13sleep10-21swapper/120:17:151
543627315,40sleep35475-21munin-run21:56:503
51252736,56sleep00-21swapper/021:52:510
46652739,52sleep20-21swapper/221:52:172
233927320,30sleep02349-21modprobe21:46:500
233927320,30sleep02349-21modprobe21:46:500
2179327334,27sleep20-21swapper/219:20:482
1755927310,52sleep20-21swapper/219:07:202
503827242,19sleep213441-21snmpd23:36:152
3002427213,40sleep330132-21cut19:47:093
242722726,15sleep00-21swapper/000:36:210
1215927212,18sleep00-21swapper/020:32:250
86227122,23sleep00-21swapper/019:57:120
3121027118,31sleep131237-21munin-run19:51:521
2659727110,45sleep026701-21latency_hist19:36:530
22682715,18sleep20-21swapper/221:45:552
22682715,18sleep20-21swapper/221:45:552
2102027110,13sleep10-21swapper/122:42:171
1679827123,28sleep316815-21munin-run19:06:523
1652227113,40sleep016564-21systemd-run00:11:500
1652227113,40sleep016564-21systemd-run00:11:500
263927015,25sleep10-21swapper/123:27:141
923826925,24sleep00-21swapper/022:07:170
3155826920,32sleep10-21swapper/119:52:081
2818926918,27sleep00-21swapper/019:41:530
1072326912,42sleep310730-21/usr/sbin/munin22:12:083
1072326912,42sleep310730-21/usr/sbin/munin22:12:083
1021526918,31sleep310250-21apt-key22:11:523
10012698,14sleep10-21swapper/123:22:131
911626824,24sleep00-21swapper/020:22:230
854426817,26sleep10-21swapper/120:22:121
346326811,30sleep03578-21sed20:06:530
2585126813,25sleep00-21swapper/021:17:100
2416826818,21sleep30-21swapper/319:27:193
2239826821,25sleep00-21swapper/021:06:530
2097726824,27sleep30-21swapper/319:17:143
1089326820,23sleep00-21swapper/023:53:050
1089326820,23sleep00-21swapper/023:53:050
864826712,39sleep38354-21/usr/sbin/munin20:22:143
564626714,27sleep30-21swapper/320:12:173
2269726714,24sleep10-21swapper/119:22:201
2037726733,16sleep20-21swapper/219:16:522
2037726733,16sleep20-21swapper/219:16:522
809226622,28sleep08108-21cron20:21:510
551626623,27sleep05528-21kernelversion20:12:140
2579126622,26sleep325862-21/usr/sbin/munin19:32:203
1631626614,18sleep10-21swapper/122:27:491
1394226621,25sleep10-21swapper/100:02:551
868526516,26sleep30-21swapper/323:47:103
2718126516,26sleep00-21swapper/023:02:180
2569826517,31sleep10-21swapper/122:57:191
2422026521,19sleep10-21swapper/119:27:211
1832926514,29sleep10-21swapper/119:11:511
204062646,13sleep00-21swapper/000:22:160
1174426420,20sleep30-21swapper/320:32:153
1170526416,30sleep11-21systemd23:56:571
1170526416,30sleep11-21systemd23:56:571
314272639,12sleep00-21swapper/021:33:420
314272639,12sleep00-21swapper/021:33:420
2898926317,26sleep329098-21wc19:42:213
3139926216,23sleep10-21swapper/123:16:521
2281126224,22sleep30-21swapper/300:31:523
1988826216,24sleep00-21swapper/020:57:160
1371026222,19sleep30-21swapper/320:37:203
1011726219,26sleep110148-21basename22:11:501
577826113,24sleep00-21swapper/023:37:140
577826113,24sleep00-21swapper/023:37:140
2875026115,29sleep128784-21/usr/sbin/munin19:42:151
2180026117,29sleep321877-21latency_hist22:46:513
1971826115,26sleep00-21swapper/000:21:520
625226018,24sleep30-21swapper/321:57:233
49492606,10sleep30-21swapper/323:35:073
292026016,19sleep10-21swapper/120:02:221
2838126016,28sleep028375-21apt-get19:42:060
2095626017,26sleep00-21swapper/022:42:160
1473226014,19sleep30-21swapper/322:22:223
547425916,27sleep15483-21diskstats23:37:071
547425916,27sleep15483-21diskstats23:37:071
4122596,43sleep10-21swapper/123:21:331
3040825914,29sleep330405-21apt-get21:32:033
3040825914,29sleep330405-21apt-get21:32:033
1846525919,19sleep00-21swapper/019:11:540
1744525912,31sleep117450-21diskstats22:32:081
102942599,24sleep30-21swapper/320:27:153
421925816,30sleep14254-21ntp_kernel_pll_23:32:151
2886325814,27sleep328931-21df21:27:073
285202586,18sleep30-21swapper/321:26:103
2550325816,26sleep11-21systemd21:16:511
2153025816,27sleep121535-21sshd21:02:171
1635625816,17sleep00-21swapper/020:47:020
1509325814,24sleep00-21swapper/020:42:180
990025719,22sleep10-21swapper/120:27:011
3005025715,26sleep030038-21apt-get23:12:070
2501225719,17sleep00-21swapper/019:31:520
1674525715,21sleep20-21swapper/219:06:512
1642425715,21sleep00-21swapper/022:27:520
1191525720,22sleep111950-21munin-run22:16:511
1191525720,22sleep111950-21munin-run22:16:511
1186825715,26sleep011860-21apt-get23:57:050
48042566,41sleep00-21swapper/020:09:520
2814025615,17sleep30-21swapper/319:41:523
2622825614,26sleep126229-21ntp_states21:17:191
199452568,12sleep30-21swapper/319:12:253
199452568,12sleep30-21swapper/319:12:253
1736425615,26sleep017359-21apt-get22:32:030
1033625614,25sleep010378-21ntp_states20:27:160
99425518,20sleep30-21swapper/319:57:153
849525515,24sleep08490-21apt-get23:47:030
358825515,18sleep00-21swapper/023:31:530
3016925518,25sleep030142-21latency_hist21:31:520
270425515,19sleep30-21swapper/320:02:203
2701525514,20sleep10-21swapper/119:37:101
2231125514,22sleep322405-21memory22:47:123
2068925513,25sleep120695-21df_inode19:17:081
1442625514,24sleep114471-21ntp_states22:22:161
3207025414,19sleep00-21swapper/021:37:090
3207025414,19sleep00-21swapper/021:37:090
30025413,21sleep1425-21users21:37:221
30025413,21sleep1425-21users21:37:221
2749425418,21sleep10-21swapper/121:22:121
2600225414,25sleep325742-21/usr/sbin/munin21:17:133
1689225413,19sleep00-21swapper/000:12:070
300302537,15sleep10-21swapper/121:30:401
2874825314,24sleep028824-21mailstats23:07:190
2874825314,24sleep028824-21mailstats23:07:190
2417925313,22sleep324176-21apt-get21:12:043
1708825320,17sleep10-21swapper/100:12:121
1680425316,16sleep30-21swapper/320:47:303
1548525312,25sleep015480-21apt-get19:02:050
59225212,22sleep1585-21apt-get19:57:051
393025215,20sleep10-21swapper/121:51:511
393025215,20sleep10-21swapper/121:51:511
2925325210,29sleep1367-21sshd23:08:211
2925325210,29sleep1367-21sshd23:08:211
2367025215,22sleep323677-21forks22:52:083
2337425217,15sleep10-21swapper/121:07:221
2263425216,18sleep00-21swapper/021:07:050
1973125216,21sleep119881-21latency_hist00:21:531
1949425216,18sleep30-21swapper/320:57:033
177332529,22sleep30-21swapper/322:32:153
1715125215,21sleep317155-21diskstats19:07:083
1920525112,21sleep319310-21users00:17:233
1686025113,22sleep116867-21ntp_states20:47:321
1186925116,18sleep10-21swapper/123:57:061
108025117,17sleep10-21swapper/121:42:061
108025117,17sleep10-21swapper/121:42:061
190232508,16sleep00-21swapper/022:37:080
1147925012,13sleep10-21swapper/120:32:091
1995224911,22sleep319949-21apt-get00:22:033
1526224914,20sleep315260-21apt-get00:07:053
1526224914,20sleep315260-21apt-get00:07:053
283224815,16sleep30-21swapper/323:27:203
2525224810,21sleep325344-21perl22:57:093
3168024713,20sleep30-21swapper/319:52:103
2835124713,16sleep30-21swapper/323:07:103
2835124713,16sleep30-21swapper/323:07:103
130524715,13sleep00-21swapper/023:22:200
757124612,19sleep10-21swapper/122:02:131
575724511,17sleep00-21swapper/021:57:070
2025524510,20sleep120322-21memory00:22:131
741624411,15sleep30-21swapper/322:02:113
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional