You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-04 - 12:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri Jul 04, 2025 00:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
250882681813,6789sleep323169-21kworker/3:122:14:443
7190238286,3805sleep11909-21kworker/u8:022:59:451
28225286710,840sleep028274-21/usr/sbin/munin22:24:470
28225286710,840sleep028274-21/usr/sbin/munin22:24:470
1218925286,510sleep112231-21ntpq23:14:511
280872495141,303sleep128136-21fschecks_time20:39:521
280872495141,303sleep128136-21fschecks_time20:39:521
2911424908,469sleep123137-21kworker/u8:018:59:571
302132369179,48sleep10-21swapper/119:04:501
1837923626,343sleep118418-21ntpq23:34:501
20261235010,325sleep220271-21ntpq23:40:042
5252234310,187sleep15391-21latency_hist19:29:291
21056228322,236sleep021099-40ogrotate23:44:260
13308228266,196sleep113396-21fschecks_time19:54:491
2953223220,191sleep13109-21unixbench_singl22:44:591
30104223158,104sleep130138-21cut22:29:561
30104223158,104sleep130138-21cut22:29:561
953222915,198sleep01006-21/usr/sbin/munin19:14:500
953222915,198sleep01006-21/usr/sbin/munin19:14:500
32323222360,144sleep132360-21munin-run20:54:261
32323222360,144sleep132360-21munin-run20:54:261
46322139,16sleep10-21swapper/122:39:301
2068220811,26sleep10-21swapper/119:19:281
2068220811,26sleep10-21swapper/119:19:281
1902220530,137sleep11905-21cron22:44:261
21223218610,79sleep121312-21latency_hist20:19:291
5939218415,153sleep36066-21timerwakeupswit22:54:583
26879218014,151sleep026957-21grep20:35:020
20493217310,145sleep120508-21fschecks_count21:59:481
28929216612,138sleep028968-21ntp_kernel_pll_18:59:520
1672821639,136sleep116746-21kernelversion23:29:491
25020216211,91sleep10-21swapper/120:29:551
25020216211,91sleep10-21swapper/120:29:551
9533215413,85sleep10-21swapper/121:24:431
2636821549,21sleep10-21swapper/123:59:501
2636821549,21sleep10-21swapper/123:59:501
23903215328,62sleep123939-21ntpq22:09:491
1758021537,14sleep10-21swapper/121:49:501
3649215226,58sleep13743-21sed19:24:281
73021518,128sleep1806-21switchtime20:55:021
1710621510,21sleep126360-21cyclictest20:05:101
1710621510,21sleep126360-21cyclictest20:05:101
6712215010,125sleep16869-21latency_hist22:59:281
17015214456,74sleep117050-21awk21:49:261
2409721395,11sleep20-21swapper/223:50:052
2159421399,30sleep121613-21apt-get23:44:391
15321213913,112sleep015379-21switchtime20:00:010
27270213815,77sleep10-21swapper/120:35:121
3040721335,93sleep10-21swapper/100:11:041
3040721335,93sleep10-21swapper/100:11:041
263721335,94sleep10-21swapper/121:00:111
263721335,94sleep10-21swapper/121:00:111
3894213112,105sleep33961-21df19:24:473
2418521295,91sleep10-21swapper/123:51:141
19464212811,75sleep10-21swapper/120:14:281
1309721289,76sleep113159-21processes21:34:561
19383212618,22sleep30-21swapper/320:14:263
895921239,99sleep18976-21kernelversion23:04:501
7139212319,84sleep37205-21/usr/sbin/munin19:34:483
24702212334,45sleep30-21swapper/323:54:393
14445212317,84sleep214514-21latency21:39:502
14445212317,84sleep214514-21latency21:39:502
30733211813,91sleep030819-21sort19:05:010
20285211857,49sleep120342-21processes23:40:071
16021211811,92sleep216105-21cat21:44:512
725621169,92sleep17333-21latency19:34:501
27758211644,12sleep10-21swapper/122:22:371
29439211445,10sleep10-21swapper/122:29:001
29439211445,10sleep10-21swapper/122:29:001
20064211344,12sleep10-21swapper/121:58:471
9498211255,46sleep1227-21systemd-journal19:40:441
4512211247,54sleep10-21swapper/121:08:431
4512211247,54sleep10-21swapper/121:08:431
401211114,81sleep0462-21sed22:39:280
30696211144,10sleep10-21swapper/120:48:241
30696211144,10sleep10-21swapper/120:48:241
6474211048,10sleep10-21swapper/119:31:391
27557211047,51sleep10-21swapper/118:55:451
27557211047,51sleep10-21swapper/118:55:451
9212210957,40sleep19284-21/usr/sbin/munin21:24:261
3776210921,18sleep10-21swapper/122:49:431
32596210921,68sleep032662-21fschecks_time00:19:480
1812210914,78sleep21831-21fschecks_count20:59:482
1812210914,78sleep21831-21fschecks_count20:59:482
1122121095,12sleep10-21swapper/119:47:301
9292210815,78sleep39395-21latency_hist21:24:283
733621075,11sleep10-21swapper/121:15:141
733621075,11sleep10-21swapper/121:15:141
28944210645,50sleep10-21swapper/100:07:091
28944210645,50sleep10-21swapper/100:07:091
22528210644,51sleep10-21swapper/120:21:591
32716210516,15sleep30-21swapper/320:54:483
24576210513,75sleep224586-21munin-run20:29:272
10195210515,74sleep30-21swapper/319:44:503
14262210410,77sleep314322-21/usr/sbin/munin21:39:473
14262210410,77sleep314322-21/usr/sbin/munin21:39:473
12472210445,15sleep10-21swapper/119:50:011
30937210310,18sleep10-21swapper/100:14:401
15839210316,71sleep315913-21fschecks_time21:44:483
10324210315,72sleep210348-21fschecks_count23:09:432
753621028,67sleep30-21swapper/323:00:073
1156321028,79sleep111632-21mailstats21:29:551
6043210180,11sleep10-21swapper/121:13:281
6043210180,11sleep10-21swapper/121:13:281
5757210114,74sleep05769-21vmstat21:10:000
5757210114,74sleep05769-21vmstat21:10:000
16547210117,59sleep216581-21fschecks_count23:29:462
14915210112,77sleep014960-21/usr/sbin/munin23:24:430
4012210011,67sleep30-21swapper/322:49:493
1379229912,74sleep013885-21cat19:55:000
3185529810,70sleep10-21swapper/119:09:501
3185529810,70sleep10-21swapper/119:09:501
3115229810,75sleep231211-21if_eth000:14:492
727529715,63sleep27339-21fw_packets23:00:022
3115429716,67sleep331166-21fschecks_count20:49:473
3115429716,67sleep331166-21fschecks_count20:49:473
141732978,17sleep10-21swapper/121:39:411
141732978,17sleep10-21swapper/121:39:411
1225629715,66sleep312339-21find19:49:593
2441629674,11sleep10-21swapper/120:27:531
1521829645,14sleep10-21swapper/119:59:581
1239029619,45sleep20-21swapper/221:34:292
8072950,11sleep226369-21cobalt_printf20:55:022
2329729515,64sleep023410-21latency_hist22:09:280
1507729510,67sleep10-21swapper/123:24:481
270029415,59sleep22762-21sendmail_mailqu22:44:562
1855729413,62sleep318624-21/usr/sbin/munin23:34:563
1685829413,66sleep016897-21ntpq23:29:520
1271729412,67sleep0370-21snmpd21:34:470
1024029413,58sleep30-21swapper/323:09:403
2863329316,27sleep30-21swapper/318:59:473
2813629318,59sleep028155-21sed00:04:490
2813629318,59sleep028155-21sed00:04:490
2380329314,63sleep323834-21latency22:09:473
1148629371,11sleep10-21swapper/123:14:141
2485829219,48sleep025011-21latency_hist22:14:290
200029215,61sleep01701-21/usr/sbin/munin20:59:540
200029215,61sleep01701-21/usr/sbin/munin20:59:540
1272729210,70sleep0370-21snmpd19:51:570
228482917,16sleep10-21swapper/122:05:011
1499929114,58sleep315002-21/usr/sbin/munin23:24:463
4232898,68sleep1464-21ntpq00:19:531
348528925,45sleep0227-21systemd-journal22:49:260
2696228913,62sleep226993-21nfs_client18:54:532
2696228913,62sleep226993-21nfs_client18:54:532
2877228715,60sleep228840-21switchtime22:25:012
2877228715,60sleep228840-21switchtime22:25:012
2579728763,13sleep10-21swapper/123:57:031
74228521,43sleep0838-21fschecks_time22:39:490
701328513,59sleep27100-21timerandwakeup21:14:582
701328513,59sleep27100-21timerandwakeup21:14:582
1553428513,53sleep215663-21awk20:00:032
1083328516,55sleep210858-21cat21:29:282
263442846,13sleep10-21swapper/122:19:261
1934428417,20sleep00-21swapper/021:54:580
2525528111,58sleep025315-21idleruntime22:14:490
2510028133,30sleep10-21swapper/118:49:481
249782817,12sleep00-21swapper/018:49:390
1655028120,41sleep016054-21apt-get20:04:380
1655028120,41sleep016054-21apt-get20:04:380
1154028112,21sleep00-21swapper/021:29:540
2080428015,45sleep320848-21awk21:59:543
547327712,47sleep25600-21/usr/sbin/munin21:09:582
547327712,47sleep25600-21/usr/sbin/munin21:09:582
2541527626,27sleep00-21swapper/020:30:060
2541527626,27sleep00-21swapper/020:30:060
145622766,11sleep10-21swapper/123:24:001
2797827514,44sleep228052-21latency_hist22:24:282
85427420,37sleep2887-21grep19:14:482
85427420,37sleep2887-21grep19:14:482
2762627417,41sleep327685-21cat00:04:283
2762627417,41sleep327685-21cat00:04:283
1370527411,27sleep30-21swapper/323:19:503
1000327421,28sleep30-21swapper/323:09:283
35242737,55sleep00-21swapper/019:23:510
2935427318,36sleep229210-21latency_hist20:44:292
2935427318,36sleep229210-21latency_hist20:44:292
1812527315,33sleep218122-21apt-get20:09:402
103522738,14sleep30-21swapper/321:25:013
68692728,17sleep30-21swapper/321:14:553
68692728,17sleep30-21swapper/321:14:553
583727224,31sleep25879-21ntpq19:29:542
286227219,24sleep20-21swapper/219:19:572
2540527225,27sleep20-21swapper/218:49:532
2016427223,19sleep20-21swapper/221:59:282
196092720,8sleep30-21swapper/321:55:003
1740427222,32sleep217510-21fschecks_time21:49:472
80862717,52sleep00-21swapper/023:00:490
2921227117,27sleep20-21swapper/218:59:582
2821427114,41sleep028307-21cut20:39:550
2821427114,41sleep028307-21cut20:39:550
2166627115,30sleep00-21swapper/023:45:150
1941527116,39sleep019458-21unin-run20:14:270
1236527119,29sleep20-21swapper/223:14:562
2789927015,39sleep027933-21latency_hist22:24:260
2330327018,37sleep323354-21fschecks_time23:49:503
1756527022,33sleep00-21swapper/021:49:490
60222697,40sleep30-21swapper/319:30:003
495526933,15sleep30-21swapper/321:09:463
495526933,15sleep30-21swapper/321:09:463
397026820,27sleep30-21swapper/321:05:023
397026820,27sleep30-21swapper/321:05:023
320132686,12sleep00-21swapper/000:16:490
2769326818,31sleep227741-21latency_hist20:39:292
2492126817,33sleep324933-21fschecks_count20:29:533
2492126817,33sleep324933-21fschecks_count20:29:533
2153926815,40sleep221540-21sshd22:02:442
1657826812,22sleep30-21swapper/320:04:413
1657826812,22sleep30-21swapper/320:04:413
258326721,25sleep32668-21latency19:19:503
2458726720,19sleep30-21swapper/320:29:283
1954126718,25sleep00-21swapper/023:39:290
55526616,21sleep00-21swapper/020:54:570
292872665,15sleep30-21swapper/322:27:043
292872665,15sleep30-21swapper/322:27:043
2774426622,22sleep30-21swapper/320:39:303
2366126617,28sleep00-21swapper/023:49:590
1446826624,26sleep214494-21munin-run19:59:272
1388426619,28sleep30-21swapper/321:39:263
51952659,34sleep00-21swapper/019:29:280
49872658,45sleep0370-21snmpd22:53:500
315072655,11sleep30-21swapper/300:14:583
3013926513,23sleep20-21swapper/222:29:582
3013926513,23sleep20-21swapper/222:29:582
2214726514,23sleep00-21swapper/020:20:000
2161726519,29sleep321613-21apt-get23:44:413
2000226520,29sleep320007-21/usr/sbin/munin20:14:463
1849426515,23sleep20-21swapper/223:34:552
1617026515,30sleep216194-21munin-run23:29:262
1245126518,27sleep30-21swapper/323:14:573
1187426515,24sleep20-21swapper/219:49:492
659226413,23sleep00-21swapper/021:14:480
659226413,23sleep00-21swapper/021:14:480
343626418,24sleep00-21swapper/021:04:510
343626418,24sleep00-21swapper/021:04:510
3195826415,24sleep20-21swapper/222:34:542
3082126419,25sleep00-21swapper/020:49:280
3082126419,25sleep00-21swapper/020:49:280
2272126418,27sleep30-21swapper/320:24:273
2225826426,21sleep20-21swapper/222:04:502
772826312,13sleep20-21swapper/219:34:592
512426319,20sleep20-21swapper/222:54:292
320902635,16sleep00-21swapper/020:51:260
320902635,16sleep00-21swapper/020:51:260
311112635,49sleep20-21swapper/219:06:002
1612226314,22sleep00-21swapper/021:44:530
1061926318,29sleep010668-21ntpq23:09:490
2656526213,32sleep026567-21/usr/sbin/munin18:54:390
2656526213,32sleep026567-21/usr/sbin/munin18:54:390
2612826218,26sleep30-21swapper/318:52:443
2563926214,31sleep225739-21users20:30:082
2563926214,31sleep225739-21users20:30:082
1686326225,21sleep20-21swapper/220:05:042
1686326225,21sleep20-21swapper/220:05:042
135462626,15sleep00-21swapper/023:19:470
1256726221,20sleep00-21swapper/023:14:580
841226116,20sleep00-21swapper/019:39:300
3131226116,23sleep00-21swapper/022:34:280
3131226116,23sleep00-21swapper/022:34:280
3090326117,22sleep20-21swapper/220:49:302
3090326117,22sleep20-21swapper/220:49:302
2951926115,27sleep329512-21apt-get20:44:473
2951926115,27sleep329512-21apt-get20:44:473
2506126115,29sleep225078-21/usr/sbin/munin22:14:392
2019926124,16sleep00-21swapper/020:14:590
1010826118,28sleep210102-21apt-get19:44:452
861726019,19sleep00-21swapper/019:39:460
328726018,19sleep20-21swapper/221:04:412
328726018,19sleep20-21swapper/221:04:412
319152600,7sleep20-21swapper/219:09:502
319152600,7sleep20-21swapper/219:09:502
89592594,15sleep20-21swapper/221:21:122
389325917,26sleep23883-21apt-get19:24:462
242192598,40sleep00-21swapper/020:25:210
2186625922,18sleep20-21swapper/220:19:542
1988225915,21sleep20-21swapper/220:14:372
1138825922,23sleep01-21systemd19:49:270
934225818,24sleep39434-21users19:40:013
3259325819,24sleep332594-21diskstats00:19:473
2997425820,22sleep30-21swapper/322:29:383
2997425820,22sleep30-21swapper/322:29:383
19632586,42sleep30-21swapper/319:18:413
19632586,42sleep30-21swapper/319:18:413
1514025812,24sleep30-21swapper/319:59:563
909725720,19sleep00-21swapper/023:04:530
529825713,24sleep05317-21apt-get22:54:390
3012325713,26sleep330119-21apt-get19:04:443
2674925720,21sleep20-21swapper/223:59:592
2674925720,21sleep20-21swapper/223:59:592
2180525716,21sleep221848-21fschecks_time23:45:292
2038025715,16sleep00-21swapper/021:59:390
1417725720,18sleep00-21swapper/021:39:440
1417725720,18sleep00-21swapper/021:39:440
869725616,24sleep28690-21apt-get23:04:422
3228625614,25sleep332413-21/usr/sbin/munin19:10:013
3228625614,25sleep332413-21/usr/sbin/munin19:10:013
3215925618,18sleep30-21swapper/322:34:593
2786225613,26sleep327858-21apt-get00:04:403
2786225613,26sleep327858-21apt-get00:04:403
2526225618,17sleep00-21swapper/023:55:000
172925612,28sleep31796-21df20:59:453
172925612,28sleep31796-21df20:59:453
1345225614,21sleep30-21swapper/319:54:513
90582555,37sleep00-21swapper/021:22:270
3066825514,15sleep30-21swapper/300:14:253
3066825514,15sleep30-21swapper/300:14:253
126125517,16sleep30-21swapper/322:40:003
1116225512,27sleep311165-21/usr/sbin/munin21:29:453
107925514,24sleep21102-21ntp_states22:39:552
2941125414,24sleep029404-21apt-get00:09:420
2941125414,24sleep029404-21apt-get00:09:420
2850425418,18sleep30-21swapper/320:40:023
2850425418,18sleep30-21swapper/320:40:023
2823625415,21sleep20-21swapper/200:04:522
2823625415,21sleep20-21swapper/200:04:522
1979325415,22sleep319785-21apt-get23:39:403
552525324,15sleep00-21swapper/019:29:480
3250625311,25sleep232500-21apt-get00:19:412
3213425314,20sleep00-21swapper/019:09:580
3213425314,20sleep00-21swapper/019:09:580
3119225316,18sleep20-21swapper/220:49:502
3119225316,18sleep20-21swapper/220:49:502
183852534,10sleep30-21swapper/321:52:113
1262725312,25sleep212623-21apt-get21:34:412
2708325212,25sleep327094-21ntp_states22:19:533
2680825212,24sleep326861-21irqstats18:54:503
2680825212,24sleep326861-21irqstats18:54:503
229525214,23sleep32306-21diskstats22:44:453
986725111,24sleep29997-21cat19:44:282
1342125115,19sleep20-21swapper/223:19:452
175192505,33sleep30-21swapper/323:31:533
2541524817,16sleep20-21swapper/223:55:022
2481724817,17sleep20-21swapper/222:14:272
222752479,17sleep30-21swapper/322:04:513
532124612,17sleep20-21swapper/222:54:412
2636224616,15sleep00-21swapper/023:59:490
2636224616,15sleep00-21swapper/023:59:490
100422408,14sleep00-21swapper/021:24:580
2940823911,16sleep20-21swapper/200:09:402
2940823911,16sleep20-21swapper/200:09:402
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional