You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 13:11

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #b, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Note that this system runs a non-RT kernel.
Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackbslot7s.osadl.org (updated Sat Nov 22, 2025 00:43:52)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1399821990122,19861sleep027084-21kworker/u8:320:14:250
212492721513,7188sleep315736-21kworker/3:222:19:033
4984263418,6321sleep028377-21kworker/0:123:10:030
54362111213,1035sleep0195-21jbd2/mmcblk2p2-21:29:070
29935264293,525sleep029959-21idleruntime-cro22:48:390
30237251616,482sleep330246-21unin-run19:23:413
30237251616,482sleep330246-21unin-run19:23:413
30502248984,384sleep030518-21modprobe00:33:390
168922475129,214sleep017052-21latency_hist20:23:420
168922475129,214sleep017052-21latency_hist20:23:420
19139240811,384sleep219178-21ntpq20:29:092
5689236016,209sleep05796-21fschecks_time23:13:580
55742343128,194sleep05712-21latency_hist19:48:410
55742343128,194sleep05712-21latency_hist19:48:410
190692296110,35sleep019212-21unixbench_singl23:54:510
701222849,215sleep011-21ksoftirqd/021:34:030
701222849,215sleep011-21ksoftirqd/021:34:030
22955227567,195sleep022956-21cpu00:08:560
22955227567,195sleep022956-21cpu00:08:560
11045226759,193sleep011153-21latency_hist21:48:410
2549225122,205sleep22616-21uniq23:04:032
10615224453,26sleep00-21swapper/023:29:040
40182234180,38sleep00-21swapper/019:43:400
25803223020,189sleep025828-21unixbench_multi20:49:180
235422285,10sleep00-21swapper/019:37:470
3253022190,18sleep032526-21cron21:13:390
14647221614,120sleep00-21swapper/021:58:540
30342221318,180sleep030414-21fschecks_time22:49:000
17842220819,171sleep017934-21fschecks_time22:08:590
8403220610,179sleep025264-21kworker/u8:323:23:390
22006220617,175sleep122084-21find00:04:131
25536220117,31sleep00-21swapper/019:08:480
25110219914,80sleep00-21swapper/019:08:400
25110219914,80sleep00-21swapper/019:08:400
29402219416,156sleep029430-21modprobe21:03:390
641921907,172sleep05787-21/usr/sbin/munin19:49:100
11510217910,111sleep011502-21apt-get21:48:570
13669217711,150sleep313783-21latency_hist20:13:433
2879217415,40sleep00-21swapper/023:04:090
25338217214,143sleep025486-21latency_hist22:33:420
8831216617,125sleep00-21swapper/023:23:590
2192521669,104sleep022005-21mailstats00:04:120
9579216510,125sleep00-21swapper/019:59:110
5987216514,138sleep16034-21awk23:14:031
8660216223,124sleep1225-21systemd-journal19:58:391
16198216214,62sleep00-21swapper/022:03:550
30248214715,22sleep00-21swapper/019:23:420
30248214715,22sleep00-21swapper/019:23:420
13354214615,118sleep013425-21apt-key20:13:400
14327214015,20sleep30-21swapper/320:14:323
31157213818,105sleep131277-21sed19:24:151
21138213710,70sleep01-21systemd00:03:390
21138213710,70sleep01-21systemd00:03:390
2645921369,114sleep026484-21awk00:19:010
937721357,115sleep00-21swapper/021:42:590
937721357,115sleep00-21swapper/021:42:590
5653213221,92sleep25759-21cat21:29:132
2319821318,71sleep023226-21latency_hist20:43:410
2319821318,71sleep023226-21latency_hist20:43:410
28265213015,100sleep328284-21fschecks_count20:59:023
2454321278,64sleep0910-21sendmail-mta00:13:530
2325721278,107sleep1362-21snmpd19:00:171
15907212720,89sleep115939-21munin-plugin-st22:03:391
55021268,108sleep0362-21snmpd19:29:560
349021268,53sleep00-21swapper/021:23:540
1563821267,103sleep015781-21sed23:44:100
21604212510,104sleep01-21systemd20:38:400
21604212510,104sleep01-21systemd20:38:400
20881212510,15sleep00-21swapper/022:18:520
29984212319,17sleep30-21swapper/321:04:113
29984212319,17sleep30-21swapper/321:04:113
24120212318,86sleep124209-21load19:04:011
24120212318,86sleep124209-21load19:04:011
27806212221,82sleep327938-21tail22:39:143
13934212011,51sleep014006-21switchtime23:39:090
11798212023,36sleep10-21swapper/123:33:531
24768211911,53sleep024810-21latency_hist20:48:410
24768211911,53sleep024810-21latency_hist20:48:410
5589211612,90sleep15652-21swap21:29:121
18644211617,81sleep118697-21sort20:28:441
30519211315,81sleep130580-21unin-run00:33:401
124621137,48sleep00-21swapper/021:14:130
124621137,48sleep00-21swapper/021:14:130
30399211220,69sleep00-21swapper/021:04:200
30399211220,69sleep00-21swapper/021:04:200
1704211214,82sleep11822-21date21:18:421
1704211214,82sleep11822-21date21:18:421
3971211115,72sleep30-21swapper/319:43:393
3093621116,45sleep00-21swapper/019:24:100
2666621118,43sleep0362-21snmpd20:53:580
6168211016,81sleep26256-21switchtime23:14:082
1252210919,74sleep31257-21/usr/sbin/munin19:34:033
32523210713,75sleep332527-21cron21:13:393
28048210711,59sleep00-21swapper/019:14:130
18286210617,67sleep118284-21apt-get23:53:511
13594210323,17sleep00-21swapper/021:54:100
32678210220,66sleep232707-21users22:54:142
27899210217,70sleep327938-21date20:58:413
3200210125,56sleep13214-21modprobe21:23:391
23996210119,41sleep20-21swapper/222:28:552
27455210049,31sleep031496-21kworker/u8:400:23:420
980329918,65sleep39835-21tune2fs21:43:593
1591229915,70sleep315998-21mailstats20:19:093
1591229915,70sleep315998-21mailstats20:19:093
1089729760,19sleep00-21swapper/020:04:030
1089729760,19sleep00-21swapper/020:04:030
3082829517,65sleep330924-21cat22:49:113
1686229512,19sleep00-21swapper/023:49:020
3241929414,56sleep00-21swapper/022:54:120
2633629420,23sleep30-21swapper/320:53:413
1947329411,22sleep00-21swapper/020:29:190
707029324,54sleep1311-21in:imuxsock19:53:391
2895529317,20sleep30-21swapper/300:28:413
256532927,30sleep00-21swapper/022:33:580
2043029215,17sleep30-21swapper/323:59:103
2043029215,17sleep30-21swapper/323:59:103
2114329121,44sleep121160-21cut22:19:011
203202915,74sleep00-21swapper/022:15:090
194429124,14sleep00-21swapper/021:18:570
1782029111,16sleep00-21swapper/020:24:270
2294329056,19sleep00-21swapper/022:24:090
2739128918,54sleep127445-21sort19:13:581
222028939,29sleep00-21swapper/023:03:410
82242886,27sleep00-21swapper/019:54:150
1793828815,55sleep117996-21/usr/sbin/munin22:09:001
321192876,13sleep30-21swapper/319:28:583
118692878,20sleep20-21swapper/223:33:582
280992865,28sleep00-21swapper/022:40:030
2039728611,25sleep30-21swapper/320:33:553
2039728611,25sleep30-21swapper/320:33:553
127732845,28sleep00-21swapper/023:34:450
732728316,49sleep37472-21unixbench_singl21:34:103
732728316,49sleep37472-21unixbench_singl21:34:103
2359928313,54sleep323546-21latency_hist19:03:423
937628126,24sleep30-21swapper/323:24:113
941428017,46sleep19436-21/usr/sbin/munin19:59:081
3253528020,40sleep132677-21sed22:54:131
2549628012,47sleep225561-21sendmail_mailst20:49:122
1539028023,39sleep115454-21find23:44:071
527727923,39sleep15298-21latency_hist23:13:391
2152627916,40sleep221590-21fschecks_time00:04:022
2018027925,32sleep30-21swapper/320:33:423
1552527920,18sleep30-21swapper/323:44:093
1277127922,36sleep212773-21cron21:53:392
1514627815,30sleep20-21swapper/221:59:082
914227710,53sleep29152-21ntpq23:24:052
2198527719,38sleep122003-21fschecks_count20:39:011
2198527719,38sleep122003-21fschecks_count20:39:011
1896727717,36sleep30-21swapper/323:54:503
1728727712,49sleep30-21swapper/320:23:573
1668227723,37sleep116696-21apt-get20:23:391
1668227723,37sleep116696-21apt-get20:23:391
1657827719,29sleep20-21swapper/222:04:052
368127614,43sleep2369-21sshd19:39:592
3189727626,29sleep20-21swapper/221:09:142
807727524,34sleep38220-21unixbench_multi19:54:133
394827514,24sleep20-21swapper/221:24:082
2328227512,49sleep223352-21cut20:43:432
2328227512,49sleep223352-21cut20:43:432
2525327425,26sleep30-21swapper/319:08:433
2322727417,41sleep223279-21latency00:09:062
2322727417,41sleep223279-21latency00:09:062
2245027418,33sleep30-21swapper/320:39:153
2245027418,33sleep30-21swapper/320:39:153
2055127417,41sleep220596-21if_eth020:34:022
2055127417,41sleep220596-21if_eth020:34:022
92052737,52sleep10-21swapper/123:24:091
407827324,26sleep10-21swapper/121:24:111
3046127314,40sleep230469-21/usr/sbin/munin19:23:522
1150227329,28sleep311519-21sh23:33:393
2997927222,31sleep230015-21sed22:48:412
2456127215,41sleep324616-21df00:13:563
2261127226,24sleep20-21swapper/218:59:012
1117827215,20sleep20-21swapper/220:04:102
1117827215,20sleep20-21swapper/220:04:102
107262728,52sleep20-21swapper/221:45:282
877827120,30sleep10-21swapper/121:39:091
877827120,30sleep10-21swapper/121:39:091
799927111,46sleep28076-21grep19:54:122
3179627121,22sleep30-21swapper/319:28:423
2009427116,35sleep120113-21latency_hist20:33:401
1180027124,26sleep10-21swapper/120:08:411
1180027124,26sleep10-21swapper/120:08:411
47427021,30sleep30-21swapper/321:14:003
47427021,30sleep30-21swapper/321:14:003
3260227024,24sleep10-21swapper/121:13:411
2629927014,22sleep10-21swapper/100:18:581
1342627025,23sleep10-21swapper/120:13:411
2403126929,23sleep224100-21switchtime20:44:112
2403126929,23sleep224100-21switchtime20:44:112
1909826919,27sleep20-21swapper/222:13:412
1460726913,41sleep20-21swapper/223:43:232
301962686,13sleep20-21swapper/219:23:402
301962686,13sleep20-21swapper/219:23:402
212042680,7sleep20-21swapper/200:03:402
212042680,7sleep20-21swapper/200:03:402
99026715,37sleep1995-21/usr/sbin/munin22:58:581
2740426724,22sleep30-21swapper/300:23:403
2557726717,25sleep20-21swapper/222:33:562
1788726721,27sleep317962-21cmp23:53:413
1769926714,40sleep217733-21ps20:24:252
1435726721,30sleep114382-21munin-run21:58:391
1061726719,28sleep10-21swapper/121:44:141
2980226615,31sleep20-21swapper/221:04:052
2980226615,31sleep20-21swapper/221:04:052
283326614,31sleep30-21swapper/323:04:083
2635526612,20sleep20-21swapper/219:09:142
2422526622,29sleep324269-21ntpq19:04:033
2422526622,29sleep324269-21ntpq19:04:033
1353826621,21sleep20-21swapper/220:13:422
1051326623,23sleep20-21swapper/223:29:022
82592658,14sleep10-21swapper/123:21:461
3121326518,29sleep131221-21/usr/sbin/munin21:08:531
3121326518,29sleep131221-21/usr/sbin/munin21:08:531
1960326519,22sleep10-21swapper/122:14:011
1803626519,22sleep10-21swapper/123:53:431
2683526424,21sleep19959-1kworker/1:2H22:38:391
2578226413,31sleep30-21swapper/322:34:013
2515626418,20sleep10-21swapper/100:14:101
2373026413,21sleep30-21swapper/320:44:043
2373026413,21sleep30-21swapper/320:44:043
2005926417,28sleep20-21swapper/222:14:102
1579426413,21sleep20-21swapper/220:19:032
1579426413,21sleep20-21swapper/220:19:032
2864726318,23sleep30-21swapper/319:18:403
2831426322,22sleep10-21swapper/100:24:111
2879026210,37sleep228791-21sshd00:27:152
1277626215,30sleep31-21systemd21:53:403
1237726219,24sleep30-21swapper/323:34:123
715026113,24sleep30-21swapper/319:53:413
51652615,44sleep30-21swapper/323:12:123
2658626116,27sleep126580-21apt-get20:53:551
1783426117,28sleep317840-21/usr/sbin/munin22:08:583
1434526119,25sleep10-21swapper/123:40:031
541126018,27sleep35431-21netstat21:29:063
2365026018,18sleep30-21swapper/300:09:163
2365026018,18sleep30-21swapper/300:09:163
316525916,19sleep10-21swapper/119:39:051
2436525914,26sleep324396-21/usr/sbin/munin22:29:043
2275025918,17sleep30-21swapper/300:08:433
220925914,26sleep32218-21munin-run23:03:403
2096725913,32sleep220990-21fschecks_count22:18:582
1816425915,24sleep217746-21apt-get23:53:442
1355225917,28sleep313613-21if_err_eth023:39:003
666125813,26sleep26714-21users19:49:122
488725816,23sleep30-21swapper/319:44:123
488725816,23sleep30-21swapper/319:44:123
474925818,20sleep10-21swapper/119:44:101
474925818,20sleep10-21swapper/119:44:101
272332585,12sleep20-21swapper/219:13:422
254425816,25sleep12548-21diskstats23:04:021
894825716,24sleep38944-21apt-get19:58:543
726825715,17sleep30-21swapper/323:19:023
3156225716,18sleep10-21swapper/122:53:411
3036325715,25sleep330360-21apt-get00:32:163
2384825728,15sleep20-21swapper/219:03:532
2384825728,15sleep20-21swapper/219:03:532
165325720,17sleep10-21swapper/119:34:131
984825617,21sleep20-21swapper/219:59:132
586425615,26sleep35957-21memory23:14:013
301852565,42sleep10-21swapper/119:23:311
301852565,42sleep10-21swapper/119:23:311
1985025615,25sleep119848-21apt-get23:58:521
1985025615,25sleep119848-21apt-get23:58:521
1619625614,26sleep316191-21apt-get22:03:533
1320925612,15sleep30-21swapper/321:54:013
1188725617,18sleep30-21swapper/320:08:433
1188725617,18sleep30-21swapper/320:08:433
481225522,17sleep20-21swapper/219:44:112
481225522,17sleep20-21swapper/219:44:112
3243725512,23sleep132482-21/usr/sbin/munin19:29:041
3188025514,17sleep30-21swapper/322:53:593
2938725516,17sleep30-21swapper/322:44:143
2931725513,24sleep229400-21perl00:29:002
290602556,11sleep20-21swapper/220:59:172
2642225521,17sleep10-21swapper/120:53:431
2618725515,17sleep10-21swapper/119:09:091
816625415,17sleep20-21swapper/221:38:542
816625415,17sleep20-21swapper/221:38:542
688825413,23sleep26981-21memory21:34:012
688825413,23sleep26981-21memory21:34:012
65725420,15sleep20-21swapper/221:14:032
65725420,15sleep20-21swapper/221:14:032
440525416,21sleep20-21swapper/223:09:042
19792546,36sleep20-21swapper/223:01:132
1877225413,26sleep218808-21ntp_states23:54:452
141425415,23sleep21438-21irqstats19:34:062
1376625419,18sleep20-21swapper/221:54:122
2463325313,24sleep224725-21perl00:13:592
2247825316,19sleep00-21swapper/018:58:590
1514925313,24sleep214890-21/usr/sbin/munin23:44:002
1389525312,24sleep213872-21apt-get20:14:072
661525212,23sleep16610-21apt-get21:33:521
661525212,23sleep16610-21apt-get21:33:521
2700625213,21sleep227042-21ntp_states20:54:082
2336125213,23sleep123410-21ntp_states00:09:091
2336125213,23sleep123410-21ntp_states00:09:091
2289725215,24sleep222902-21ntp_states22:24:072
194225212,24sleep31937-21apt-get21:18:563
1631725218,16sleep10-21swapper/122:04:001
115002529,24sleep10-21swapper/121:48:541
2867625112,23sleep128671-21apt-get22:43:561
2734625112,24sleep127395-21irqstats22:39:041
2399525112,23sleep123990-21apt-get22:28:541
235212516,33sleep10-21swapper/122:26:241
216552515,12sleep10-21swapper/120:38:411
216552515,12sleep10-21swapper/120:38:411
292372505,10sleep10-21swapper/121:01:301
238625011,23sleep22397-21ntp_states21:19:072
2385025013,23sleep123421-21/usr/sbin/munin20:44:061
2385025013,23sleep123421-21/usr/sbin/munin20:44:061
1951625015,18sleep30-21swapper/322:13:593
1518925016,18sleep10-21swapper/120:18:421
92124914,18sleep20-21swapper/219:33:412
864224916,17sleep30-21swapper/321:39:073
864224916,17sleep30-21swapper/321:39:073
2843024912,14sleep20-21swapper/222:43:412
782824810,17sleep212-21rcu_sched23:19:132
361124812,18sleep30-21swapper/321:24:013
304642489,14sleep20-21swapper/222:49:022
2634024813,21sleep226362-21irqstats00:18:592
2290324814,21sleep30-21swapper/322:24:083
1595824820,19sleep30-21swapper/322:03:413
1158224510,16sleep10-21swapper/123:33:411
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional