You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 23:04

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Fri May 02, 2025 12:43:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
25716265014,618sleep125842-21/usr/sbin/munin07:03:411
23085262735,577sleep123086-21ntp_states08:38:381
17950261715,585sleep117942-21apt-get08:23:281
23786246323,421sleep123914-21/usr/sbin/munin06:58:441
2630423257,304sleep226307-21cron07:08:102
8415231812,292sleep28451-21idleruntime-cro11:18:092
13135230145,195sleep213220-21latency_hist11:33:112
434229079,23sleep20-21swapper/209:08:442
1522922849,261sleep015273-21ntpq08:13:380
1522922849,261sleep015273-21ntpq08:13:380
281762243100,40sleep228194-21latency_hist10:38:242
281762243100,40sleep228194-21latency_hist10:38:242
23277223320,200sleep323326-21unin-run10:23:103
28628221918,183sleep128659-21idleruntime-cro12:23:091
15525221621,104sleep215607-21timerandwakeup11:38:402
26865220918,166sleep20-21swapper/207:08:262
21408220912,96sleep221478-21cat11:58:312
5377220513,177sleep2370-21snmpd11:08:122
18077220323,88sleep218227-21unixbench_singl10:03:512
18077220323,88sleep218227-21unixbench_singl10:03:512
5996220110,167sleep26034-21grep09:28:342
31040219212,163sleep231079-21idleruntime-cro10:48:092
31040219212,163sleep231079-21idleruntime-cro10:48:092
7163218616,24sleep07157-21apt-get11:13:270
1060721838,15sleep20-21swapper/207:58:382
911221758,155sleep29158-21ntpq11:18:372
31632217431,130sleep231635-21date10:48:542
27850216758,96sleep227910-21find12:18:382
20595216127,117sleep220656-21fschecks_time10:13:372
6922215837,103sleep27069-21sed11:13:122
3793215832,113sleep23887-21latency_hist11:03:112
16883215720,33sleep10-21swapper/108:18:401
14553215637,95sleep20-21swapper/209:53:462
14553215637,95sleep20-21swapper/209:53:462
14553215637,95sleep20-21swapper/209:53:462
364721505,11sleep00-21swapper/007:37:520
16218215018,115sleep116251-21/usr/sbin/munin09:59:021
4715214927,31sleep10-21swapper/111:03:431
27875214931,22sleep20-21swapper/208:53:412
13706214733,95sleep213724-21awk09:53:092
26209214412,23sleep20-21swapper/208:48:412
10992214413,117sleep011064-21tune2fs09:43:420
10992214413,117sleep011064-21tune2fs09:43:420
1801721427,46sleep20-21swapper/211:48:212
13346214216,112sleep313354-21acpi11:33:203
30427213914,110sleep230481-21processes07:18:402
1647221378,115sleep216480-21cpu11:43:292
628213416,22sleep20-21swapper/207:28:252
24067213436,79sleep215146-21kworker/2:010:23:502
24067213436,79sleep215146-21kworker/2:010:23:502
13884213241,46sleep213965-21sort08:08:432
9886213116,100sleep39932-21sh07:58:103
30252213035,19sleep214549-21kworker/2:012:28:112
20171212718,89sleep3227-21systemd-journal10:13:093
20171212718,89sleep3227-21systemd-journal10:13:093
17083212790,19sleep20-21swapper/210:03:112
26728212519,16sleep20-21swapper/212:13:452
22467212574,15sleep222472-21ntp_states10:18:412
18418212310,92sleep217924-21/usr/sbin/munin08:23:412
10254212228,71sleep310274-21cpu11:23:273
2339212016,87sleep00-21swapper/009:18:100
26575211818,80sleep126727-21sed12:13:441
23251211716,83sleep20-21swapper/206:58:332
20105211619,80sleep020195-21switchtime08:28:440
16162211514,83sleep216234-21latency_hist08:18:122
16162211514,83sleep216234-21latency_hist08:18:122
15664211416,80sleep015268-21apt-get09:58:200
15664211416,80sleep015268-21apt-get09:58:200
15664211416,80sleep015268-21apt-get09:58:200
1428821147,94sleep213347-21/usr/sbin/munin11:33:442
16544211217,72sleep016692-21grep09:59:090
11480211211,85sleep311562-21find09:43:543
11480211211,85sleep311562-21find09:43:543
934621108,86sleep29493-21unixbench_singl07:53:452
20849210919,74sleep120889-21modprobe11:58:101
4444210815,76sleep34524-21mailstats07:38:403
26098210614,78sleep326122-21cut12:13:343
22771210633,46sleep322801-21munin-run06:58:103
906821057,84sleep20-21swapper/209:38:112
906821057,84sleep20-21swapper/209:38:112
17553210510,79sleep317610-21irqstats10:03:403
17553210510,79sleep317610-21irqstats10:03:403
13179210511,71sleep00-21swapper/009:48:510
12989210525,57sleep113015-21munin-plugin-st08:08:101
2198210415,49sleep22248-21unin-run10:58:102
2144521048,81sleep20-21swapper/208:33:352
16741210313,76sleep116773-21latency11:43:361
1139021039,71sleep20-21swapper/209:43:532
1139021039,71sleep20-21swapper/209:43:532
368521028,23sleep23718-21munin-run07:38:102
1978210221,59sleep02015-21apt-key07:33:120
19552210227,27sleep20-21swapper/210:08:442
19552210227,27sleep20-21swapper/210:08:442
1172521018,75sleep222886-21cyclictest08:03:262
6409210014,24sleep16512-21users11:08:391
2192029729,33sleep20-21swapper/206:53:372
2012829712,15sleep20-21swapper/211:53:382
2812429522,37sleep10-21swapper/110:38:231
2812429522,37sleep10-21swapper/110:38:231
2717829515,62sleep026496-21/usr/sbin/munin10:33:450
2717829515,62sleep026496-21/usr/sbin/munin10:33:450
1062429514,67sleep010670-21/usr/sbin/munin11:23:360
95912949,18sleep29602-21grep09:38:402
95912949,18sleep29602-21grep09:38:402
488529417,60sleep05031-21unixbench_singl09:23:440
315922947,14sleep20-21swapper/207:23:332
241742949,13sleep20-21swapper/208:43:282
2317929420,11sleep20-21swapper/212:03:402
2317929420,11sleep20-21swapper/212:03:402
207762947,14sleep20-21swapper/208:33:102
126729462,21sleep21316-21basename09:13:322
707329314,44sleep10-21swapper/107:48:231
683929314,15sleep20-21swapper/207:48:122
226132938,13sleep20-21swapper/208:38:232
293292907,13sleep20-21swapper/207:13:482
2582629021,24sleep20-21swapper/210:28:592
2582629021,24sleep20-21swapper/210:28:592
1559029022,49sleep315736-21grep08:13:453
1559029022,49sleep315736-21grep08:13:453
778128820,37sleep37848-21df09:33:323
778128820,37sleep37848-21df09:33:323
304862888,62sleep20-21swapper/209:03:312
290592889,61sleep20-21swapper/210:38:522
290592889,61sleep20-21swapper/210:38:522
3151128715,28sleep30-21swapper/307:23:303
299582877,40sleep20-21swapper/209:01:082
243762870,8sleep00-21swapper/012:08:290
1306628620,41sleep313173-21grep09:48:503
317028512,11sleep20-21swapper/209:18:452
184792859,56sleep00-21swapper/011:48:360
47082849,11sleep20-21swapper/209:23:422
724028325,42sleep17272-21cut09:33:091
595228313,55sleep35453-21apt-get09:28:193
2811928324,39sleep128142-21munin-plugin-st07:13:101
2324328316,47sleep323318-21wc12:03:413
2324328316,47sleep323318-21wc12:03:413
191728319,51sleep31973-21apt-key07:33:113
50942828,11sleep20-21swapper/207:41:382
2958828218,42sleep329695-21timerwakeupswit12:23:443
293232829,56sleep329360-21ps08:58:393
50372810,6sleep10-21swapper/109:23:441
2861328123,28sleep00-21swapper/008:58:120
2591828118,42sleep025981-21fschecks_time12:13:310
253402815,10sleep20-21swapper/212:10:382
2404228120,45sleep10-21swapper/112:08:111
2404228120,45sleep10-21swapper/112:08:111
1960928128,28sleep10-21swapper/108:28:331
833928021,39sleep38354-21idleruntime-cro07:53:093
2115328022,31sleep00-21swapper/008:33:280
175172807,11sleep20-21swapper/208:21:232
1210228016,45sleep111806-21/usr/sbin/munin11:28:341
1135227916,45sleep111388-21grep09:43:521
1135227916,45sleep111388-21grep09:43:521
81662785,12sleep20-21swapper/207:50:542
2706327815,48sleep327075-21netstat10:33:403
2706327815,48sleep327075-21netstat10:33:403
2525227817,37sleep025267-21fschecks_count10:28:450
2525227817,37sleep025267-21fschecks_count10:28:450
2369127812,51sleep123717-21fschecks_count10:23:331
2369127812,51sleep123717-21fschecks_count10:23:331
1439827811,52sleep10-21swapper/109:53:411
1439827811,52sleep10-21swapper/109:53:411
1439827811,52sleep10-21swapper/109:53:411
1308527824,35sleep013102-21modprobe11:33:090
102162785,11sleep20-21swapper/211:23:142
784127726,28sleep10-21swapper/111:13:441
716427621,25sleep00-21swapper/007:48:300
238927617,30sleep10-21swapper/109:18:111
1761227617,33sleep00-21swapper/010:03:410
1761227617,33sleep00-21swapper/010:03:410
76927514,47sleep1866-21cat10:53:131
521327521,33sleep15241-21munin-run07:43:101
1529127511,29sleep00-21swapper/011:38:340
1017927511,46sleep010234-21sort07:58:280
478127416,30sleep30-21swapper/309:23:433
35512746,11sleep00-21swapper/009:18:510
2492927418,24sleep30-21swapper/307:03:193
2476327413,45sleep124826-21/usr/sbin/munin12:08:401
1414427421,31sleep00-21swapper/011:33:420
1229927413,47sleep312363-21ps11:28:403
2282327321,29sleep022842-21munin-run06:58:110
1561027219,31sleep315766-21grep11:38:413
2085227124,28sleep30-21swapper/310:13:423
785027016,39sleep07857-21sh09:33:330
785027016,39sleep07857-21sh09:33:330
687027019,36sleep36872-21cron11:13:093
3165827014,38sleep031722-21/usr/sbin/munin07:23:350
3060227022,27sleep10-21swapper/109:03:341
2841127011,41sleep10-21swapper/107:13:261
1785227022,25sleep10-21swapper/111:48:121
3214826912,12sleep10-21swapper/110:49:131
2796626912,36sleep328071-21timerwakeupswit08:53:423
1780026925,29sleep017850-21latency_hist11:48:110
1182126924,30sleep311916-21fschecks_time08:03:313
628726814,39sleep06408-21timerwakeupswit11:08:380
595426818,26sleep00-21swapper/009:28:200
131726813,40sleep31353-21uname09:13:333
301602676,12sleep10-21swapper/112:27:581
292892678,40sleep00-21swapper/012:23:370
1967926713,41sleep019733-21fschecks_count11:53:280
590426620,22sleep10-21swapper/107:43:391
3164226612,28sleep031538-21apt-get10:48:560
225226624,24sleep00-21swapper/010:58:120
2122526622,26sleep30-21swapper/311:58:283
1824926616,32sleep30-21swapper/311:48:313
319026515,21sleep30-21swapper/310:59:003
2081626519,31sleep320845-21latency_hist08:33:113
1544926514,18sleep10-21swapper/111:38:391
111492650,7sleep10-21swapper/111:23:441
993526419,24sleep00-21swapper/007:58:110
2413426418,21sleep30-21swapper/310:23:513
2413426418,21sleep30-21swapper/310:23:513
1367926420,19sleep00-21swapper/008:08:370
257426319,18sleep10-21swapper/110:58:301
2090226315,34sleep321002-21latency_hist11:58:123
206426315,25sleep12194-21gpgv07:33:131
1912826319,21sleep00-21swapper/010:08:340
1912826319,21sleep00-21swapper/010:08:340
1684026316,32sleep316873-21/usr/sbin/munin11:43:383
1336426314,34sleep113388-21df_inode08:08:301
315382627,43sleep00-21swapper/009:06:270
2727926216,31sleep027313-21/usr/sbin/munin07:08:490
256726215,32sleep02573-21diskstats10:58:290
2515226216,23sleep30-21swapper/310:28:313
2515226216,23sleep30-21swapper/310:28:313
2089926212,36sleep020929-21ntp_states10:13:430
997626116,21sleep10-21swapper/107:58:121
443626114,28sleep30-21swapper/311:03:363
2791326122,23sleep30-21swapper/312:18:393
2705926120,21sleep10-21swapper/108:53:121
2424826113,32sleep024254-21df_inode08:43:300
208426117,21sleep30-21swapper/310:56:423
1385826117,20sleep10-21swapper/111:33:391
530626014,31sleep05236-21latency_hist07:43:130
2995526021,21sleep10-21swapper/110:43:361
2995526021,21sleep10-21swapper/110:43:361
2993526014,28sleep129954-21apt-get07:18:231
2991126017,28sleep029954-21fschecks_time10:43:350
2991126017,28sleep029954-21fschecks_time10:43:350
2201626017,26sleep021920-21apt-get10:18:270
919325918,18sleep10-21swapper/109:38:141
919325918,18sleep10-21swapper/109:38:141
3175225913,21sleep30-21swapper/309:08:123
308212595,12sleep30-21swapper/310:45:193
308212595,12sleep30-21swapper/310:45:193
3041425915,26sleep030420-21ntp_states07:18:380
1646425914,20sleep30-21swapper/309:59:083
135782596,10sleep10-21swapper/109:51:311
1064925917,28sleep310672-21/usr/sbin/munin07:58:473
2702725816,27sleep30-21swapper/307:08:443
2201725810,34sleep122018-21apt-get10:18:281
1648925822,18sleep00-21swapper/008:18:300
158712585,10sleep10-21swapper/108:15:041
158712585,10sleep10-21swapper/108:15:041
1465425813,18sleep30-21swapper/308:13:123
940725714,27sleep39414-21forks09:38:333
940725714,27sleep39414-21forks09:38:333
2337025720,17sleep00-21swapper/006:58:350
2277325716,26sleep022784-21forks12:03:300
2277325716,26sleep022784-21forks12:03:300
2163725719,22sleep321043-21/usr/sbin/munin08:33:403
2106825714,25sleep10-21swapper/108:33:251
1062025718,19sleep10-21swapper/109:43:111
1062025718,19sleep10-21swapper/109:43:111
65192565,11sleep00-21swapper/007:45:030
2764425613,27sleep027722-21latency12:18:320
2574525615,24sleep325742-21apt-get08:48:253
2074725611,32sleep120443-21/usr/sbin/munin10:13:401
879625515,26sleep08838-21fschecks_count11:18:310
45725514,22sleep30-21swapper/310:53:113
3196525511,17sleep10-21swapper/109:08:221
263325512,25sleep32626-21apt-get09:18:303
2323025514,20sleep00-21swapper/008:38:410
1667225515,20sleep30-21swapper/308:18:333
114252555,18sleep10-21swapper/108:03:051
312125412,26sleep13278-21awk07:33:581
295372546,11sleep30-21swapper/307:16:313
2769825415,24sleep10-21swapper/108:53:361
2616425411,18sleep00-21swapper/008:48:390
2356125414,23sleep023563-21apt-get10:23:240
2356125414,23sleep023563-21apt-get10:23:240
1947225416,23sleep119551-21cat10:08:431
1947225416,23sleep119551-21cat10:08:431
1531925413,19sleep30-21swapper/309:58:113
1531925413,19sleep30-21swapper/309:58:113
1531925413,19sleep30-21swapper/309:58:113
1192525417,17sleep00-21swapper/008:03:320
1078025413,18sleep00-21swapper/009:43:140
897825321,18sleep30-21swapper/311:18:343
550725312,24sleep35500-21apt-get07:43:283
396025318,21sleep14113-21sed09:23:121
3199025313,24sleep031984-21apt-get09:08:250
3191425313,20sleep10-21swapper/107:23:401
2522925313,16sleep10-21swapper/110:28:341
2522925313,16sleep10-21swapper/110:28:341
2444425314,17sleep10-21swapper/108:43:351
2995825211,25sleep329954-21apt-get07:18:253
2949125211,18sleep30-21swapper/310:43:093
2949125211,18sleep30-21swapper/310:43:093
2502425211,22sleep325077-21users08:43:453
214982529,17sleep10-21swapper/106:53:241
1821925213,19sleep00-21swapper/008:23:350
1779925212,24sleep117805-21ntp_states10:03:461
1779925212,24sleep117805-21ntp_states10:03:461
871225111,23sleep18782-21df11:18:281
403325112,23sleep04028-21apt-get11:03:250
3206625112,23sleep332073-21diskstats09:08:283
2977925116,20sleep029908-21cut07:18:140
1013125112,21sleep010229-21users09:38:500
723025015,20sleep37240-21df_inode11:13:303
2926025017,19sleep00-21swapper/008:58:360
238425011,22sleep31894-21apt-get07:33:233
215552509,15sleep10-21swapper/111:58:341
2150825011,25sleep021114-21/usr/sbin/munin11:58:330
1816025011,21sleep318217-21irqstats08:23:343
2954324912,22sleep129617-21latency_hist10:43:111
2954324912,22sleep129617-21latency_hist10:43:111
2646424914,18sleep00-21swapper/007:08:120
2261424811,22sleep322609-21apt-get08:38:243
2868624710,14sleep00-21swapper/010:38:420
2868624710,14sleep00-21swapper/010:38:420
2317624712,20sleep323183-21df_inode06:58:303
289402469,15sleep10-21swapper/108:58:291
2489624615,16sleep30-21swapper/312:08:423
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional