You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-04 - 13:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackbslot7s.osadl.org (updated Tue Aug 04, 2026 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
84692862723,8568sleep12352-21kworker/1:200:18:301
84692862723,8568sleep12352-21kworker/1:200:18:301
256792196018,1930sleep38024-21kworker/u8:123:32:463
256792196018,1930sleep38024-21kworker/u8:123:32:463
8404287714,849sleep38410-21df_inode22:37:473
30556254618,512sleep230597-21/usr/sbin/munin23:47:502
21164243323,398sleep121200-21ntpq23:17:581
27195238922,351sleep327218-21/usr/sbin/munin21:52:583
7049230330,68sleep17149-21perl19:08:081
20422229715,128sleep10-21swapper/119:48:011
140422284120,148sleep114114-21mailstats19:27:591
27386227080,106sleep127447-21fschecks_time23:37:561
169362235112,62sleep117066-21latency_hist21:22:321
16563223066,55sleep10-21swapper/123:02:561
21194222422,181sleep121341-21unixbench_multi21:33:021
2773222221,125sleep12903-21unixbench_multi22:17:591
12568220112,22sleep10-21swapper/100:32:551
30473219940,74sleep130553-21fschecks_time23:47:491
19292219819,82sleep119331-21fschecks_count23:12:491
19292219819,82sleep119331-21fschecks_count23:12:491
1324521849,163sleep113280-21latency22:52:511
1817218016,149sleep10-21swapper/120:32:591
23173217318,46sleep10-21swapper/119:57:541
24938217218,54sleep10-21swapper/121:47:311
22177217011,144sleep122267-21cut21:37:501
30054216812,61sleep10-21swapper/122:02:531
11340216631,97sleep111449-21ls21:03:021
7490216213,132sleep26983-21apt-get00:17:382
7490216213,132sleep26983-21apt-get00:17:382
2806321560,8sleep028061-21systemd-run21:57:300
9143215315,78sleep10-21swapper/120:57:451
10860214911,55sleep10-21swapper/119:17:571
8008214318,106sleep18040-21awk22:37:291
26496214016,112sleep126588-21latency_hist21:52:321
16310214022,96sleep016450-21unixbench_multi21:18:030
18349213910,116sleep118439-21grep19:43:091
18349213910,116sleep118439-21grep19:43:091
6662213518,99sleep00-21swapper/020:48:030
1498213512,103sleep11516-21open_inodes23:57:591
15159213112,101sleep10-21swapper/119:32:471
9587212615,95sleep19599-21modprobe22:42:301
29542212530,70sleep10-21swapper/120:17:571
16791212422,81sleep216938-21grep23:03:012
11928212361,19sleep10-21swapper/100:32:311
11928212361,19sleep10-21swapper/100:32:311
6216212220,56sleep10-21swapper/120:47:521
5363212224,75sleep25067-21/usr/sbin/munin19:02:532
4922212112,95sleep14956-21ps20:43:001
4922212112,95sleep14956-21ps20:43:001
1094212123,76sleep31219-21latency_hist20:32:323
1792021209,86sleep10-21swapper/123:07:531
26612211825,59sleep10-21swapper/123:33:201
26612211825,59sleep10-21swapper/123:33:201
28286211621,72sleep028426-21unixbench_singl20:13:030
17857211316,81sleep318010-21awk21:23:033
1498211324,57sleep10-21swapper/122:14:031
12948211328,60sleep212967-21apt-get22:52:402
2589211221,52sleep10-21swapper/100:02:431
2589211221,52sleep10-21swapper/100:02:431
25280211228,21sleep10-21swapper/123:32:311
24660211211,45sleep124704-21cat20:02:511
13555211221,34sleep10-21swapper/121:12:311
12817211218,53sleep10-21swapper/119:23:021
4808211110,16sleep10-21swapper/100:08:001
32672211121,50sleep10-21swapper/122:12:301
18605211111,42sleep118797-21apt-key21:27:331
13642211124,59sleep213807-21latency_hist21:12:322
5689210920,34sleep10-21swapper/100:12:441
764721087,37sleep10-21swapper/120:52:491
21401210819,52sleep10-21swapper/119:52:491
21401210819,52sleep10-21swapper/119:52:491
1538621088,16sleep10-21swapper/121:17:321
14641210819,73sleep014722-21grep22:57:510
887521076,13sleep10-21swapper/119:12:441
18933210722,25sleep10-21swapper/121:27:411
17991210714,33sleep10-21swapper/119:42:321
11049210618,34sleep10-21swapper/122:46:211
9151210413,25sleep30-21swapper/300:22:483
9151210413,25sleep30-21swapper/300:22:483
5483210420,62sleep05067-21/usr/sbin/munin19:02:550
421121046,14sleep10-21swapper/120:42:311
3637210420,63sleep03778-21unixbench_singl20:38:020
1777921045,13sleep20-21swapper/221:23:022
9406210325,57sleep39427-21sed20:57:513
519321038,35sleep10-21swapper/122:27:441
26100210319,54sleep226192-21fschecks_time20:07:482
2442521036,12sleep20-21swapper/220:02:352
19943210319,65sleep220076-21grep23:13:032
19943210319,65sleep220076-21grep23:13:032
1037021039,53sleep10-21swapper/100:27:301
1037021039,53sleep10-21swapper/100:27:301
311122997,53sleep10-21swapper/120:22:571
2662029940,19sleep10-21swapper/120:07:591
145512997,33sleep10-21swapper/122:57:431
1180929911,72sleep011956-21latency_hist19:22:320
223172986,33sleep10-21swapper/123:22:421
414329718,65sleep24116-21/usr/sbin/munin00:07:452
543629618,47sleep35446-21sshd00:12:303
2859329410,71sleep323116-1kworker/3:2H20:14:363
1474729417,58sleep014888-21unixbench_multi21:13:110
3190629313,59sleep10-21swapper/123:52:461
3190629313,59sleep10-21swapper/123:52:461
2404729311,68sleep224096-21/usr/sbin/munin21:43:002
1636429328,41sleep016394-21munin-run19:37:300
54829121,51sleep10-21swapper/120:28:021
1014029125,41sleep210166-21munin-run19:17:302
1210529014,33sleep10-21swapper/122:48:031
2851428922,12sleep10-21swapper/120:13:411
2396928813,61sleep223976-21diskstats23:27:482
702428724,43sleep07058-21cat22:32:520
122422870,8sleep00-21swapper/021:07:400
87052868,16sleep30-21swapper/319:12:383
2902028612,33sleep10-21swapper/123:42:531
2856828619,19sleep00-21swapper/023:42:320
2795628619,11sleep10-21swapper/121:57:041
259028620,42sleep02607-21modprobe20:37:300
1544028623,41sleep215331-21latency_hist21:17:332
860328518,29sleep00-21swapper/022:37:520
1486728514,56sleep22165-1kworker/2:0H19:32:322
268592849,12sleep00-21swapper/023:36:300
268592849,12sleep00-21swapper/023:36:300
712428316,35sleep00-21swapper/000:17:320
3170828313,19sleep00-21swapper/023:52:330
2962928330,39sleep129762-21sed22:02:331
1518528312,54sleep215191-21diskstats19:32:482
56572829,61sleep05688-21proc_pri22:27:590
315028214,50sleep23233-21switchtime00:03:002
315028214,50sleep23233-21switchtime00:03:002
3081228111,23sleep30-21swapper/320:22:513
44482809,14sleep10-21swapper/122:23:041
44482809,14sleep10-21swapper/122:23:041
3089528014,47sleep330963-21/usr/sbin/munin23:48:003
2563728015,46sleep30-21swapper/321:47:583
238922807,52sleep00-21swapper/023:27:460
696427924,39sleep27012-21http00:17:302
640127928,30sleep16447-21munin-run19:07:321
2091427912,52sleep320962-21grep21:32:563
2879527819,21sleep00-21swapper/021:58:000
267227813,53sleep22744-21latency_hist20:37:322
752727711,52sleep37595-21df00:17:473
752727711,52sleep37595-21df00:17:473
70627736,24sleep02720-1kworker/0:1H23:57:290
70627736,24sleep02720-1kworker/0:1H23:57:290
51692779,53sleep35186-21/usr/sbin/munin22:27:423
2907827717,37sleep229112-21latency23:42:542
3112427619,39sleep21-21systemd22:07:302
2360027619,25sleep00-21swapper/023:27:300
2236027620,27sleep00-21swapper/021:37:530
1713027633,24sleep00-21swapper/023:04:290
1364227621,31sleep313684-21fschecks_count19:27:483
1274527621,33sleep20-21swapper/200:33:002
118432765,59sleep00-21swapper/000:32:010
118432765,59sleep00-21swapper/000:32:010
822327512,48sleep08259-21latency_hist19:12:310
1582827518,23sleep30-21swapper/319:33:053
922627421,24sleep00-21swapper/020:57:480
701527422,24sleep30-21swapper/300:17:313
362327424,26sleep00-21swapper/022:22:420
362327424,26sleep00-21swapper/022:22:420
181922745,60sleep00-21swapper/021:24:250
503527313,44sleep35121-21sed20:43:023
503527313,44sleep35121-21sed20:43:023
2035127321,22sleep00-21swapper/019:48:000
1260127312,44sleep312684-21/usr/sbin/munin19:23:003
235427211,49sleep02366-21kernelversion22:17:500
1508727212,16sleep20-21swapper/222:58:012
581327114,42sleep25958-21latency_hist20:47:332
2927627117,37sleep029340-21ps23:43:000
85482706,52sleep00-21swapper/000:19:300
85482706,52sleep00-21swapper/000:19:300
172272708,20sleep30-21swapper/323:05:413
390126926,23sleep03962-21latency_hist00:07:310
390126926,23sleep03962-21latency_hist00:07:310
2265326921,22sleep20-21swapper/219:57:312
2265326921,22sleep20-21swapper/219:57:312
2059126918,34sleep320739-21cat23:17:323
1718326913,39sleep217259-21/usr/sbin/munin19:38:012
1251426916,22sleep20-21swapper/219:22:592
2986926818,29sleep329864-21apt-get22:02:473
2605226815,37sleep326078-21df20:07:453
2127226821,24sleep221333-21grep23:18:032
209326821,21sleep20-21swapper/222:17:442
1895026814,28sleep20-21swapper/221:27:422
125126818,32sleep01400-21unixbench_singl22:13:020
926426716,28sleep20-21swapper/200:22:512
926426716,28sleep20-21swapper/200:22:512
598226717,32sleep38440-21munin-node20:47:423
3262826715,32sleep20-21swapper/220:27:552
2340726713,41sleep0278-21dbus-daemon19:58:010
1112426713,39sleep211203-21systemctl00:28:002
1112426713,39sleep211203-21systemctl00:28:002
544826610,39sleep00-21swapper/000:12:310
2858226622,24sleep20-21swapper/221:57:542
1677026630,23sleep016792-21fschecks_count19:37:510
1481426614,19sleep00-21swapper/019:32:300
1176726615,22sleep30-21swapper/319:22:313
117072664,13sleep20-21swapper/219:22:222
688626516,26sleep20-21swapper/219:07:412
323522657,21sleep30-21swapper/322:08:243
1163826516,31sleep311688-21/usr/sbin/munin22:47:523
950926414,36sleep09628-21expr19:13:040
3221026415,20sleep00-21swapper/020:27:410
130626412,26sleep30-21swapper/320:32:403
1033826417,29sleep30-21swapper/322:43:003
55992637,11sleep00-21swapper/020:45:570
55992637,11sleep00-21swapper/020:45:570
517826320,17sleep30-21swapper/319:02:503
1825226310,28sleep30-21swapper/319:43:023
1825226310,28sleep30-21swapper/319:43:023
87002627,12sleep20-21swapper/222:37:542
2290326219,21sleep20-21swapper/219:57:472
1187626217,22sleep30-21swapper/300:32:293
1187626217,22sleep30-21swapper/300:32:293
118242628,10sleep20-21swapper/221:05:502
3160426112,21sleep20-21swapper/220:23:062
267626115,29sleep32683-21/usr/sbin/munin00:02:493
267626115,29sleep32683-21/usr/sbin/munin00:02:493
2370126118,24sleep20-21swapper/223:27:322
893426022,22sleep00-21swapper/020:57:320
260822606,14sleep00-21swapper/020:07:460
2598926012,30sleep226067-21/usr/sbin/munin23:33:092
2598926012,30sleep226067-21/usr/sbin/munin23:33:092
2373126017,25sleep00-21swapper/021:42:540
2336526018,21sleep323403-21ps19:58:003
194152608,12sleep30-21swapper/323:12:513
1282226014,29sleep312941-21/usr/sbin/munin00:33:013
904925912,24sleep00-21swapper/000:22:410
904925912,24sleep00-21swapper/000:22:410
2539725912,24sleep20-21swapper/221:47:522
2391325918,19sleep30-21swapper/321:42:573
1874725914,27sleep218826-21/usr/sbin/munin19:43:182
1874725914,27sleep218826-21/usr/sbin/munin19:43:182
1515825916,23sleep30-21swapper/322:58:023
1432325918,20sleep30-21swapper/321:13:013
324902586,11sleep20-21swapper/222:10:092
1975125814,17sleep00-21swapper/023:13:010
1975125814,17sleep00-21swapper/023:13:010
1344225811,30sleep312949-21/usr/sbin/munin22:52:573
1333425815,28sleep013478-21latency_hist19:27:330
1318625812,27sleep013244-21irqstats22:52:500
79425719,18sleep20-21swapper/223:57:312
79425719,18sleep20-21swapper/223:57:312
3244025714,19sleep30-21swapper/323:53:003
3244025714,19sleep30-21swapper/323:53:003
2987125714,18sleep00-21swapper/022:02:480
255672576,11sleep00-21swapper/020:05:320
2434525714,27sleep324351-21ntp_states23:27:573
1802625716,19sleep30-21swapper/323:07:553
166582576,11sleep30-21swapper/321:19:353
131025718,21sleep00-21swapper/023:57:540
751525611,21sleep30-21swapper/319:08:183
527125616,23sleep25279-21diskstats22:27:492
2797025615,23sleep228007-21ntp_states20:12:572
235325619,19sleep20-21swapper/200:02:322
2078125614,27sleep020795-21kernelversion21:32:530
1793825618,19sleep30-21swapper/319:42:303
1570225612,16sleep20-21swapper/221:17:492
1252625620,20sleep30-21swapper/321:07:533
675025514,23sleep36746-21apt-get22:32:433
3204925514,19sleep00-21swapper/020:27:320
2906925513,24sleep329205-21uptime21:58:033
2134525511,18sleep20-21swapper/221:33:032
1764825514,17sleep20-21swapper/223:07:422
1189225515,21sleep20-21swapper/222:48:002
756025410,16sleep20-21swapper/222:33:022
754025414,17sleep20-21swapper/220:52:422
422125414,17sleep20-21swapper/220:42:322
277172545,15sleep20-21swapper/223:38:062
2287725415,20sleep30-21swapper/323:23:013
2026325413,17sleep30-21swapper/319:47:593
1944125415,24sleep319464-21ntp_states21:28:323
1896725413,17sleep00-21swapper/023:12:320
1069425410,16sleep00-21swapper/022:43:050
996225314,23sleep29968-21diskstats22:42:512
469525314,23sleep24434-21/usr/sbin/munin20:42:532
3150125311,25sleep031543-21fschecks_count22:07:490
2948925317,16sleep30-21swapper/323:43:033
288525311,23sleep30-21swapper/320:37:413
1893125314,22sleep018460-21apt-get21:27:400
1891825313,24sleep019042-21sed19:43:200
1891825313,24sleep019042-21sed19:43:200
727225213,23sleep27299-21munin-run20:52:292
50325213,22sleep3498-21apt-get22:12:443
501125214,22sleep34923-21latency_hist22:27:343
2055925215,19sleep20-21swapper/219:48:022
147025211,17sleep30-21swapper/323:57:573
1069625212,25sleep010690-21apt-get21:02:450
2922425112,22sleep229228-21diskstats20:17:502
291172519,25sleep30-21swapper/320:17:423
2647825113,22sleep026495-21munin-run21:52:310
1077025113,22sleep310776-21df_inode21:02:483
770625015,17sleep30-21swapper/320:52:513
2914625011,16sleep00-21swapper/020:17:480
283212505,36sleep30-21swapper/323:40:003
259725013,16sleep30-21swapper/322:17:573
2491225011,22sleep324918-21ntp_states20:02:573
2087625013,23sleep320880-21diskstats23:17:523
1238025024,13sleep00-21swapper/022:48:340
1216725010,23sleep012162-21apt-get00:32:420
2230924911,22sleep322358-21irqstats21:37:523
2149524916,15sleep00-21swapper/019:52:530
2149524916,15sleep00-21swapper/019:52:530
490224813,13sleep30-21swapper/322:27:303
92142479,14sleep20-21swapper/220:57:462
404724714,16sleep20-21swapper/222:22:572
404724714,16sleep20-21swapper/222:22:572
2209324611,15sleep20-21swapper/221:37:472
93024510,16sleep20-21swapper/222:12:552
2076324411,16sleep00-21swapper/023:17:410
2266824212,15sleep30-21swapper/319:57:323
2266824212,15sleep30-21swapper/319:57:323
267342397,16sleep20-21swapper/221:52:452
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional