You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-17 - 04:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Wed Dec 17, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
224532101714,988sleep222459-21diskstats19:45:092
224532101714,988sleep222459-21diskstats19:45:092
1947427189,691sleep319523-21/usr/sbin/munin19:35:113
1079526029,577sleep110975-21perl20:42:151
807325690,61sleep18079-21unixbench_multi22:05:251
3265243616,294sleep03392-21unixbench_multi20:25:220
3265243616,294sleep03392-21unixbench_multi20:25:220
22066243333,328sleep122098-21modprobe21:09:451
2328238724,339sleep02416-21latency_hist20:24:490
1027235513,325sleep31020-21apt-get20:20:033
1373222967,252sleep113804-21find19:16:061
210922281101,20sleep00-21swapper/019:40:130
19775227231,224sleep119983-21cat22:44:481
16798227182,162sleep016881-21timerandwakeup19:25:240
16798227182,162sleep016881-21timerandwakeup19:25:240
19352226371,179sleep119394-21fschecks_count19:35:091
2370522619,194sleep023817-21date19:49:490
2370522619,194sleep023817-21date19:49:490
20515224915,217sleep020487-21latency_hist00:29:480
23621224044,131sleep123657-21latency_hist00:39:461
7544223825,191sleep07567-21munin-plugin-st23:49:450
22006223253,100sleep122009-21cron00:34:451
22006223253,100sleep122009-21cron00:34:451
19016223039,114sleep119032-21latency_hist20:59:481
27294222429,160sleep127363-21cat20:00:101
26123222451,114sleep126160-21awk21:20:111
32536221912,192sleep032612-21awk21:40:170
11466221849,55sleep10-21swapper/100:00:161
26138221415,160sleep126239-21wc19:55:241
26138221415,160sleep126239-21wc19:55:241
12531221214,181sleep212709-21ts2lt20:42:242
4208220715,174sleep34293-21perl21:55:073
32292220614,134sleep032323-21ntpq20:15:140
20355220494,21sleep10-21swapper/122:45:451
27518220313,176sleep027587-21tr20:00:170
24378220229,26sleep00-21swapper/019:50:170
9284219514,167sleep09290-21cpu19:05:010
20461219413,163sleep120484-21idleruntime-cro00:29:451
16202219414,165sleep216227-21df20:50:032
30620218871,96sleep130634-21grep21:35:101
11070217814,148sleep011175-21fschecks_time00:00:070
1711521745,156sleep130726-1kworker/1:0H20:50:591
28063217113,145sleep328108-21cat23:10:063
372321695,148sleep00-21swapper/023:35:150
17902216415,137sleep117944-21cat19:30:071
8264216012,41sleep10-21swapper/123:50:111
12270215737,40sleep112404-21latency_hist19:14:491
20835215557,82sleep120859-21cpu19:40:061
30046215220,28sleep20-21swapper/220:09:502
15071215211,43sleep10-21swapper/122:29:501
1500321529,129sleep015019-21netstat19:20:100
4959215012,122sleep35015-21interrupts23:40:073
4604214712,93sleep1369-21snmpd20:30:211
24466214717,112sleep024481-21kworker/u8:422:59:450
5681214632,102sleep15691-21cpu22:00:011
3132021448,82sleep1369-21snmpd20:11:021
3132021448,82sleep1369-21snmpd20:11:021
14976214421,109sleep015069-21latency_hist22:29:490
25734214325,20sleep00-21swapper/021:20:020
1124521439,120sleep110293-21/usr/sbin/munin22:15:321
1124521439,120sleep110293-21/usr/sbin/munin22:15:321
28577214216,111sleep028605-21cpu20:05:060
27877214211,118sleep327951-21find21:25:183
2496921410,26sleep224973-21unixbench_multi21:15:192
2496921410,26sleep224973-21unixbench_multi21:15:192
32020214033,67sleep132078-21cut21:40:061
26421213811,114sleep226312-21/usr/sbin/munin23:05:042
6741213719,10sleep00-21swapper/020:37:400
25828213771,20sleep00-21swapper/019:55:150
25828213771,20sleep00-21swapper/019:55:150
14727213713,97sleep014793-21fschecks_time20:45:050
30740213512,111sleep030767-21awk20:10:250
30740213512,111sleep030767-21awk20:10:250
15996213311,105sleep10-21swapper/119:24:481
6787213216,51sleep26865-21sendmail_mailtr23:45:152
30682213213,104sleep130745-21modprobe23:19:451
3910213116,35sleep10-21swapper/121:54:481
22842212816,34sleep222899-21irqstats21:10:272
2148121280,7sleep00-21swapper/022:49:470
22299212630,67sleep022293-21apt-get00:35:010
17131212613,54sleep10-21swapper/119:25:541
17131212613,54sleep10-21swapper/119:25:541
11806212622,71sleep00-21swapper/019:10:390
21672212532,80sleep121700-21grep22:50:011
1076721249,100sleep110863-21latency_hist23:59:481
5645212323,35sleep10-21swapper/120:34:581
1557212335,76sleep11628-21/usr/sbin/munin21:45:151
1892121215,80sleep00-21swapper/019:34:060
12892212012,85sleep012941-21fschecks_time00:05:160
12892212012,85sleep012941-21fschecks_time00:05:160
22574211912,91sleep322093-21apt-get21:09:553
24771211718,32sleep10-21swapper/119:50:221
12909211724,72sleep013101-21cut20:42:260
20166211614,39sleep00-21swapper/019:35:240
18189211518,82sleep317692-21/usr/sbin/munin19:30:173
31464211422,77sleep031499-21proc_pri23:20:130
3106621147,36sleep10-21swapper/123:20:021
12393211421,33sleep10-21swapper/122:20:331
12393211421,33sleep10-21swapper/122:20:331
8882211319,67sleep39064-21latencyplot20:42:053
30444211118,70sleep230474-21fschecks_count21:35:062
31739211011,83sleep331880-21date21:39:483
6000210922,50sleep1228-21systemd-journal23:44:451
2276210911,85sleep12313-21modprobe20:24:461
13111210957,11sleep00-21swapper/019:15:410
14387210811,81sleep214393-21sh19:19:472
24546210617,72sleep224635-21ls22:59:472
23858210614,76sleep023887-21awk21:14:450
19713210612,76sleep219719-21/usr/sbin/munin21:00:132
2940210515,19sleep10-21swapper/123:34:481
522621046,66sleep012408-21munin-node20:32:590
12942210331,54sleep10-21swapper/100:05:171
12942210331,54sleep10-21swapper/100:05:171
26161210211,75sleep226185-21ntp_states21:20:122
2329210220,59sleep02478-21tail23:30:190
27741210111,70sleep10-21swapper/121:25:161
9062210014,67sleep09154-21/usr/sbin/munin22:10:220
28493210020,41sleep10-21swapper/123:10:171
24971210011,74sleep225034-21if_err_eth023:00:082
23210210026,14sleep10-21swapper/122:54:581
22446210021,15sleep10-21swapper/119:45:081
22446210021,15sleep10-21swapper/119:45:081
11668210022,58sleep011718-21latency_hist22:19:470
11668210022,58sleep011718-21latency_hist22:19:470
3232529923,15sleep10-21swapper/120:15:151
1660329917,18sleep10-21swapper/100:16:081
407829811,73sleep24080-21sshd21:54:532
1766629821,16sleep10-21swapper/100:19:591
1766629821,16sleep10-21swapper/100:19:591
1100129716,68sleep311076-21grep19:10:093
1933929619,16sleep10-21swapper/121:00:041
2881629519,14sleep10-21swapper/121:30:011
2881629519,14sleep10-21swapper/121:30:011
1574129421,17sleep10-21swapper/122:30:431
36402935,12sleep20-21swapper/223:35:132
2908729312,46sleep20-21swapper/221:30:102
2908729312,46sleep20-21swapper/221:30:102
917029211,66sleep29211-21ntpq22:10:242
521629210,66sleep25254-21/usr/sbin/munin23:40:142
1700829214,58sleep217086-21users19:25:262
1700829214,58sleep217086-21users19:25:262
1101929217,59sleep311145-21unixbench_multi22:15:303
1101929217,59sleep311145-21unixbench_multi22:15:303
3032529117,60sleep330345-21apt-get20:09:593
3032529117,60sleep330345-21apt-get20:09:593
2273229019,37sleep10-21swapper/121:10:101
1648229022,44sleep016497-21apt22:34:450
87762897,68sleep18768-21apt-get22:10:031
172528913,58sleep31772-21fschecks_count23:30:053
934228814,59sleep39415-21cat22:10:293
75712889,43sleep17594-21munin-run23:49:461
2996528855,20sleep129962-21munin-run20:09:491
1583828812,60sleep315569-21apt-get00:14:483
1583828812,60sleep315569-21apt-get00:14:483
387028717,54sleep10-21swapper/120:29:481
387028717,54sleep10-21swapper/120:29:481
605328519,43sleep06178-21cat23:44:480
3275328425,46sleep10-21swapper/123:25:071
2890428431,24sleep30-21swapper/321:30:073
2890428431,24sleep30-21swapper/321:30:073
2707728428,35sleep327211-21grep23:05:183
1792428421,12sleep10-21swapper/122:38:261
1535028418,11sleep10-21swapper/100:12:041
1535028418,11sleep10-21swapper/100:12:041
510928318,11sleep10-21swapper/121:56:261
263352838,35sleep10-21swapper/123:04:591
1802328320,43sleep318050-21modprobe22:39:453
689828232,32sleep36922-21munin-run20:39:463
430328218,11sleep10-21swapper/123:38:031
2576428218,11sleep10-21swapper/123:01:041
2011828129,30sleep320116-21apt-get22:44:583
2094928014,51sleep221041-21fschecks_time19:40:112
770527912,49sleep20-21swapper/222:05:192
1557127918,11sleep10-21swapper/120:45:591
1096127921,33sleep20-21swapper/219:10:082
908727821,33sleep30-21swapper/323:54:453
254822786,60sleep125491-21munin-run21:19:471
254822786,60sleep125491-21munin-run21:19:471
2084427618,37sleep320892-21fschecks_count00:30:073
2084427618,37sleep320892-21fschecks_count00:30:073
2058227611,49sleep220098-21/usr/sbin/munin22:45:502
1829627619,43sleep318370-21find20:55:173
129202757,19sleep00-21swapper/022:21:370
129202757,19sleep00-21swapper/022:21:370
192292747,13sleep20-21swapper/219:34:532
1158627422,21sleep20-21swapper/222:19:452
1158627422,21sleep20-21swapper/222:19:452
954527321,26sleep00-21swapper/023:55:060
2898227314,44sleep028793-21/usr/sbin/munin21:30:080
2898227314,44sleep028793-21/usr/sbin/munin21:30:080
2366927326,27sleep30-21swapper/319:49:473
2366927326,27sleep30-21swapper/319:49:473
2250827312,28sleep20-21swapper/200:35:072
1883227336,19sleep00-21swapper/022:40:150
1775427318,36sleep317812-21fschecks_count00:20:053
1775427318,36sleep317812-21fschecks_count00:20:053
699227218,27sleep00-21swapper/022:04:490
2865927216,41sleep328662-21/usr/sbin/munin20:05:093
283927219,39sleep22865-21cut20:25:122
283927219,39sleep22865-21cut20:25:122
2516127223,29sleep325181-21munin-plugin-st19:54:453
2228227215,28sleep00-21swapper/021:09:480
978527117,25sleep20-21swapper/219:05:152
458927121,28sleep00-21swapper/021:55:160
2838727127,28sleep30-21swapper/320:04:493
1841027120,37sleep018547-21sed00:20:190
1841027120,37sleep018547-21sed00:20:190
1223527113,44sleep212296-21apt-config00:04:462
1405227017,30sleep20-21swapper/200:09:462
1405227017,30sleep20-21swapper/200:09:462
1338327022,31sleep313480-21/usr/sbin/munin00:05:273
1338327022,31sleep313480-21/usr/sbin/munin00:05:273
946526919,30sleep30-21swapper/323:55:053
70026912,36sleep2721-21awk20:19:452
3254626918,21sleep00-21swapper/023:25:020
312602698,42sleep20-21swapper/223:20:072
2203926922,19sleep20-21swapper/222:50:102
1468426917,29sleep20-21swapper/220:45:032
365526819,28sleep30-21swapper/323:35:143
3258226815,23sleep30-21swapper/320:15:213
1811226810,15sleep20-21swapper/200:20:132
1811226810,15sleep20-21swapper/200:20:132
1310226815,35sleep30-21swapper/319:15:363
2670426719,31sleep226767-21modprobe19:59:462
2670426719,31sleep226767-21modprobe19:59:462
2402526723,21sleep00-21swapper/021:14:490
2402526723,21sleep00-21swapper/021:14:490
1843026720,25sleep20-21swapper/222:40:052
2999726615,24sleep00-21swapper/023:15:160
253272660,9sleep30-21swapper/323:00:163
2527126619,28sleep20-21swapper/219:54:482
2287226621,26sleep30-21swapper/300:35:163
2083826613,35sleep220843-21diskstats00:30:062
2083826613,35sleep220843-21diskstats00:30:062
1555326623,27sleep00-21swapper/000:14:450
1555326623,27sleep00-21swapper/000:14:450
672326516,34sleep36745-21ntp_states23:45:123
1902226520,18sleep20-21swapper/219:34:492
149426516,29sleep00-21swapper/021:45:140
831126412,15sleep00-21swapper/023:50:140
750626426,20sleep00-21swapper/022:05:130
3032426414,37sleep229840-21apt-get20:09:582
3032426414,37sleep229840-21apt-get20:09:582
2834126416,34sleep228359-21open_inodes23:10:132
2833126415,26sleep20-21swapper/220:04:482
241242648,17sleep30-21swapper/321:14:563
241242648,17sleep30-21swapper/321:14:563
1747726422,18sleep30-21swapper/320:54:493
1614626416,21sleep00-21swapper/000:15:460
1595926414,28sleep30-21swapper/320:49:483
1229726415,32sleep312408-21apt-key00:04:473
2554626320,21sleep30-21swapper/319:55:103
2554626320,21sleep30-21swapper/319:55:103
1935226316,32sleep319423-21cut21:00:063
4652627,17sleep20-21swapper/221:41:202
3131326212,36sleep031314-21sshd21:35:530
3097626219,25sleep30-21swapper/323:19:563
1199926213,35sleep312025-21fschecks_count22:20:113
1199926213,35sleep312025-21fschecks_count22:20:113
227726117,26sleep32305-21munin-run21:49:453
221426123,18sleep20-21swapper/223:30:172
2175626111,13sleep00-21swapper/022:50:040
1986626112,32sleep219981-21timerwakeupswit00:25:172
1734726120,22sleep30-21swapper/300:19:453
999026013,31sleep210096-21ls23:55:172
3233526013,31sleep332380-21/usr/sbin/munin21:40:123
293426013,26sleep00-21swapper/021:50:110
274682606,14sleep00-21swapper/023:08:090
2365726010,33sleep30-21swapper/322:55:123
155492609,15sleep30-21swapper/319:20:213
844125915,23sleep20-21swapper/223:50:162
2104225916,27sleep30-21swapper/319:40:123
1676825914,25sleep00-21swapper/020:50:180
1605425913,24sleep20-21swapper/200:14:552
2811125817,24sleep028149-21irqstats23:10:070
1714125816,29sleep217160-21netstat22:35:122
1054025815,24sleep110568-21munin-run19:09:471
702925717,21sleep20-21swapper/220:39:492
567825715,24sleep35674-21apt-get21:59:593
2506025714,27sleep025152-21/usr/sbin/munin23:00:100
1558825718,24sleep20-21swapper/219:20:532
540825616,23sleep30-21swapper/320:34:483
2214625612,28sleep322279-21sed21:09:473
2102425616,18sleep20-21swapper/221:05:062
1510925617,24sleep214716-21apt-get22:29:512
2630825516,17sleep30-21swapper/321:20:173
1996025515,19sleep00-21swapper/021:00:190
76425415,25sleep3888-21sed23:25:173
2780325411,26sleep227876-21mailstats21:25:172
1677425415,24sleep016767-21apt-get22:35:020
1653425416,15sleep30-21swapper/322:34:493
985925321,15sleep30-21swapper/319:05:163
857525318,22sleep38640-21latency_hist22:09:493
569225314,23sleep25753-21df22:00:022
2190725318,17sleep30-21swapper/322:50:073
1822125314,18sleep20-21swapper/220:55:162
1485025319,17sleep30-21swapper/322:29:483
3262225210,27sleep232519-21/usr/sbin/munin23:25:042
302525114,21sleep33031-21ntp_states20:25:173
302525114,21sleep33031-21ntp_states20:25:173
281125113,22sleep22840-21irqstats21:50:092
1933225112,22sleep319418-21perl00:25:063
1663825112,23sleep316644-21ntp_states19:25:193
1663825112,23sleep316644-21ntp_states19:25:193
1801925011,15sleep00-21swapper/020:55:100
1452025012,22sleep214571-21irqstats00:10:072
1452025012,22sleep214571-21irqstats00:10:072
396924912,21sleep24072-21modprobe20:29:492
2795524916,17sleep00-21swapper/021:25:190
1243324913,20sleep212512-21systemctl22:20:342
1243324913,20sleep212512-21systemctl22:20:342
2941924811,22sleep329416-21apt-get23:14:593
256924811,20sleep32564-21apt-get21:50:023
1550324814,19sleep20-21swapper/222:30:362
3021824710,19sleep230321-21users23:15:182
1834624710,21sleep30-21swapper/322:40:023
144624710,19sleep21482-21ntp_states20:20:142
2170824210,18sleep00-21swapper/000:30:530
2170824210,18sleep00-21swapper/000:30:530
110624111,15sleep20-21swapper/221:45:042
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional