You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-31 - 07:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Jul 31, 2026 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
149412237162,24sleep30-21swapper/319:09:173
130332234163,58sleep10-21swapper/119:05:041
148452228153,25sleep20-21swapper/219:08:032
146762202163,26sleep00-21swapper/019:05:510
2537621530,5sleep31510099cyclictest23:10:193
2541721350,5sleep31510099cyclictest00:25:163
15095999693,2cyclictest0-21swapper/221:37:072
299932610,4sleep230-21rcuc/220:55:132
15095995450,3cyclictest0-21swapper/219:27:222
138112540,5sleep113806-21strings21:30:211
106772490,2sleep20-21swapper/223:50:202
1509599412,26cyclictest0-21swapper/220:33:122
15095994112,16cyclictest29157-21missed_timers22:05:202
1509599406,18cyclictest0-21swapper/223:35:212
1509599381,25cyclictest29458-21/usr/sbin/munin00:35:212
1509599371,8cyclictest32145-21sshd22:12:132
1509599371,8cyclictest32145-21sshd22:12:132
1509599371,23cyclictest579-21systemd-journal22:50:012
1509599360,24cyclictest0-21swapper/223:30:122
15095993511,21cyclictest8186-21latency_hist21:20:012
1509599344,21cyclictest87350irq/126-enp1s0-21:10:192
1509599342,27cyclictest23944-21ntp_states00:20:242
15095993414,17cyclictest24196-21crond23:10:002
1509599340,2cyclictest87650irq/129-enp1s0-23:25:212
1509599332,4cyclictest21231-21perl00:15:162
1509599322,3cyclictest87350irq/126-enp1s0-21:45:282
1509599321,4cyclictest0-21swapper/200:10:192
1509599321,26cyclictest18010-21diskstats19:15:032
1509599321,25cyclictest0-21swapper/220:09:332
1509599321,21cyclictest5254-21/usr/sbin/munin23:40:132
1509599321,16cyclictest5179-21fschecks_time22:25:152
1508499321,24cyclictest0-21swapper/022:28:530
1508499321,23cyclictest0-21swapper/023:17:370
1509599313,5cyclictest2280950sshd20:38:132
1509599312,25cyclictest8519-21ntp_states23:45:222
1509599312,25cyclictest7181-21diskstats22:30:142
1509599311,6cyclictest0-21swapper/222:38:502
1509599311,4cyclictest0-21swapper/221:25:132
1509599311,16cyclictest28596-21munin-run23:20:002
15095993111,17cyclictest32428-21ntpq19:45:222
15095993111,17cyclictest11556-21grep20:10:252
1509599310,2cyclictest0-21swapper/223:55:162
1509099311,22cyclictest0-21swapper/121:05:481
1509099311,13cyclictest29281-21ntp_states22:05:221
1509099311,13cyclictest16523-21cat22:50:181
1509099311,12cyclictest0-21swapper/122:11:301
1509099311,12cyclictest0-21swapper/122:11:301
1509099311,12cyclictest0-21swapper/121:59:541
1509099310,24cyclictest0-21swapper/119:31:121
1508499312,21cyclictest17967-21kworker/u8:200:36:480
1509599307,20cyclictest16912-21gltestperf00:05:182
1509599306,21cyclictest25305-21unixbench_multi21:55:272
1509599302,4cyclictest0-21swapper/222:00:292
1509599301,5cyclictest8595-21cat21:20:122
1509599301,5cyclictest0-21swapper/200:25:252
1509599301,4cyclictest28370-21/usr/sbin/munin20:50:162
1509599301,3cyclictest909-21sssd_nss00:00:212
1509599301,2cyclictest0-21swapper/200:30:262
1509599301,2cyclictest0-21swapper/200:30:262
1509099301,6cyclictest0-21swapper/122:24:381
1509099301,11cyclictest0-21swapper/123:15:471
1509099300,23cyclictest0-21swapper/121:16:071
1509599299,17cyclictest827-21runrttasks19:50:192
1509599299,17cyclictest827-21runrttasks19:50:192
1509599296,11cyclictest3493-21ntpq22:20:192
1509599293,21cyclictest13459-21smart_sda20:15:242
1509599292,23cyclictest3749-21diskstats19:55:152
1509599291,7cyclictest0-21swapper/221:50:252
1509599291,4cyclictest0-21swapper/221:40:222
1509599291,4cyclictest0-21swapper/220:40:212
1509599291,3cyclictest87650irq/129-enp1s0-19:40:162
1509599291,3cyclictest25491-21/usr/sbin/munin23:10:202
15095992910,16cyclictest16779-21cstates20:25:122
1509599290,4cyclictest0-21swapper/223:00:272
1509099291,12cyclictest0-21swapper/123:46:431
1509099290,9cyclictest0-21swapper/120:59:361
1508499292,19cyclictest0-21swapper/023:35:260
1509599287,18cyclictest1-21systemd22:40:132
1509599286,19cyclictest15049-21gltestperf20:20:152
1509599281,5cyclictest6661-21ntp_states20:00:222
1509599281,4cyclictest21504-21/usr/sbin/munin19:20:232
1509599281,4cyclictest0-21swapper/219:35:222
1509599281,3cyclictest25450-21hddtemp_smartct19:30:192
1509599281,3cyclictest18932-21/usr/sbin/munin22:55:232
1509599281,2cyclictest25383-21/usr/sbin/munin20:45:192
15095992812,13cyclictest31980-21/usr/sbin/munin21:00:212
15095992812,13cyclictest31980-21/usr/sbin/munin21:00:212
1509599280,3cyclictest87650irq/129-enp1s0-22:50:272
1509099283,11cyclictest0-21swapper/121:28:551
1509099281,9cyclictest0-21swapper/120:20:161
1509099281,8cyclictest0-21swapper/121:13:011
1509099281,13cyclictest16645-21fschecks_count00:05:161
1509099281,12cyclictest0-21swapper/120:04:101
1509099281,11cyclictest0-21swapper/122:59:061
1509099281,11cyclictest0-21swapper/121:49:591
1509099281,11cyclictest0-21swapper/100:35:381
1509099281,11cyclictest0-21swapper/100:16:301
1508499281,22cyclictest0-21swapper/023:53:530
1508499281,21cyclictest0-21swapper/022:08:120
1510099275,7cyclictest401ktimersoftd/320:55:273
1509599274,20cyclictest28943-21tr23:20:112
1509599274,20cyclictest28943-21tr23:20:112
1509599271,3cyclictest1827-21/usr/sbin/munin21:05:152
1509599271,3cyclictest15282-21/usr/sbin/munin19:10:162
1509599271,3cyclictest13714-21sed21:30:202
1509599271,2cyclictest501-21/usr/sbin/munin22:15:132
1509099279,4cyclictest16267-21sed19:10:181
1509099271,12cyclictest0-21swapper/123:41:071
1509099271,11cyclictest0-21swapper/121:24:071
1509099271,11cyclictest0-21swapper/120:42:221
1509099271,10cyclictest0-21swapper/123:51:591
1509099271,10cyclictest0-21swapper/121:42:151
1509099271,10cyclictest0-21swapper/100:32:541
1509099271,10cyclictest0-21swapper/100:32:541
1508499273,22cyclictest0-21swapper/000:17:540
1510099263,15cyclictest22527-21memory21:50:193
1509099261,9cyclictest0-21swapper/100:14:461
1509099261,16cyclictest1656-21grep19:50:131
1509099261,16cyclictest1656-21grep19:50:131
1509099261,13cyclictest0-21swapper/119:48:341
1509099261,10cyclictest0-21swapper/121:36:111
1509099261,10cyclictest0-21swapper/119:28:361
1509099261,10cyclictest0-21swapper/119:19:011
1509099260,6cyclictest0-21swapper/123:28:351
1508499269,15cyclictest16049-21munin-run00:05:010
1508499269,14cyclictest26416-21idleruntime-cro23:15:010
1508499262,23cyclictest87350irq/126-enp1s0-23:32:490
1508499262,11cyclictest4423-21awk22:25:010
169832250,4sleep339-21rcuc/321:40:013
1510099259,3cyclictest87350irq/126-enp1s0-20:55:013
1510099255,10cyclictest0-21swapper/322:37:363
1510099251,5cyclictest0-21swapper/323:39:063
1509099256,9cyclictest0-21swapper/120:16:561
1509099256,7cyclictest29213-21fschecks_count23:20:151
1509099256,7cyclictest29213-21fschecks_count23:20:151
1509099251,9cyclictest0-21swapper/122:33:581
1509099251,9cyclictest0-21swapper/100:24:541
1509099251,12cyclictest0-21swapper/120:27:561
1509099251,10cyclictest0-21swapper/123:05:531
1509099250,14cyclictest0-21swapper/123:35:031
1509099250,12cyclictest0-21swapper/122:15:201
1508499252,11cyclictest31980-21/usr/sbin/munin21:00:250
1508499252,11cyclictest31980-21/usr/sbin/munin21:00:250
1508499251,6cyclictest0-21swapper/023:49:090
1508499251,22cyclictest0-21swapper/019:35:180
1508499251,21cyclictest3797-21diskmemload21:50:150
1508499251,18cyclictest0-21swapper/000:11:050
1508499251,16cyclictest0-21swapper/021:14:550
1510099243,4cyclictest87350irq/126-enp1s0-21:25:013
1510099241,17cyclictest0-21swapper/300:33:173
1510099241,17cyclictest0-21swapper/300:33:173
1510099241,16cyclictest0-21swapper/321:05:263
1510099240,15cyclictest0-21swapper/323:27:163
1509099241,9cyclictest0-21swapper/122:43:541
1509099241,9cyclictest0-21swapper/120:45:541
1509099241,3cyclictest1182-21df_inode23:30:141
1509099241,12cyclictest0-21swapper/121:03:321
1509099241,12cyclictest0-21swapper/121:03:321
1509099241,11cyclictest0-21swapper/122:48:101
1509099241,11cyclictest0-21swapper/121:54:541
1509099241,11cyclictest0-21swapper/120:53:461
1509099241,10cyclictest0-21swapper/120:38:381
1508499241,21cyclictest0-21swapper/021:30:130
1508499241,20cyclictest3797-21diskmemload22:33:370
1508499241,15cyclictest0-21swapper/023:22:490
1508499241,15cyclictest0-21swapper/023:22:490
1508499241,15cyclictest0-21swapper/022:13:160
1508499241,15cyclictest0-21swapper/022:13:160
1508499241,11cyclictest10615-21/usr/sbin/munin21:25:140
1508499240,16cyclictest0-21swapper/022:02:030
15100992321,1cyclictest0-21swapper/322:55:213
1509099237,7cyclictest0-21swapper/119:42:121
1509099236,3cyclictest0-21swapper/119:58:521
1509099232,18cyclictest9942-21memory22:35:181
1509099231,9cyclictest0-21swapper/123:58:141
1509099231,9cyclictest0-21swapper/120:14:521
1509099231,14cyclictest14865-21interrupts00:00:181
1509099231,10cyclictest0-21swapper/122:01:301
1508499237,4cyclictest12050irq/123-ahci[0019:15:020
1508499237,2cyclictest87350irq/126-enp1s0-23:55:160
1508499231,9cyclictest1611-21tune2fs19:50:130
1508499231,9cyclictest1611-21tune2fs19:50:130
1508499231,7cyclictest0-21swapper/022:40:010
1508499231,10cyclictest1827-21/usr/sbin/munin21:05:130
1508499231,10cyclictest16443-21irqstats22:50:180
1508499230,21cyclictest0-21swapper/023:05:180
1510099229,3cyclictest12050irq/123-ahci[0021:35:013
1510099227,8cyclictest0-21swapper/322:34:133
1510099226,8cyclictest87350irq/126-enp1s0-23:05:163
1510099222,7cyclictest24486-21hddtemp_smartct21:55:163
1510099221,4cyclictest0-21swapper/300:17:553
1510099221,15cyclictest0-21swapper/322:25:223
1510099220,15cyclictest0-21swapper/322:00:143
1509099224,3cyclictest0-21swapper/100:27:501
1509099221,9cyclictest0-21swapper/123:01:181
1509099221,7cyclictest0-21swapper/119:36:501
1509099221,15cyclictest0-21swapper/122:28:061
1508499227,13cyclictest0-21swapper/019:30:000
1508499225,4cyclictest11-21rcuc/023:25:130
1508499221,6cyclictest0-21swapper/000:20:360
1508499221,14cyclictest0-21swapper/000:06:320
1508499221,13cyclictest0-21swapper/022:57:320
1508499221,12cyclictest0-21swapper/023:40:560
1508499221,12cyclictest0-21swapper/022:47:200
1508499221,11cyclictest0-21swapper/000:29:500
1508499221,10cyclictest0-21swapper/021:48:340
1508499220,6cyclictest0-21swapper/021:22:050
1510099219,3cyclictest15442entropy19:50:143
1510099219,3cyclictest15442entropy19:50:133
1510099218,8cyclictest0-21swapper/320:33:133
1510099215,8cyclictest87350irq/126-enp1s0-23:50:183
1510099215,8cyclictest87350irq/126-enp1s0-22:45:273
1510099215,3cyclictest10735-21kworker/u8:223:00:163
1510099214,4cyclictest18435-21ps21:40:233
1510099212,11cyclictest16270-21/usr/sbin/munin00:05:193
1510099211,7cyclictest18581-21perf20:30:023
1510099211,7cyclictest0-21swapper/321:15:203
1510099211,17cyclictest2097-21wc23:30:243
1510099211,15cyclictest0-21swapper/322:42:083
1510099211,11cyclictest29804-21df_inode00:35:143
1510099210,15cyclictest0-21swapper/300:20:213
1509099217,3cyclictest0-21swapper/120:30:081
1509099211,9cyclictest7799-21/usr/sbin/munin20:05:161
1508499219,3cyclictest29481-21sshd20:54:130
1508499216,4cyclictest66162netstat20:00:190
1508499212,6cyclictest7-21ksoftirqd/020:48:150
1508499211,9cyclictest3402-21/usr/sbin/munin19:55:140
1508499211,9cyclictest0-21swapper/021:39:120
1508499211,17cyclictest31302-21unixbench_multi20:55:270
1508499211,13cyclictest0-21swapper/021:16:280
1508499211,12cyclictest0-21swapper/021:57:020
1508499211,10cyclictest28091-21kernelversion00:30:180
1508499211,10cyclictest28091-21kernelversion00:30:180
1508499211,10cyclictest0-21swapper/023:04:470
1508499211,10cyclictest0-21swapper/022:18:130
1510099209,3cyclictest839-21lldpd19:15:233
1510099204,9cyclictest0-21swapper/321:00:273
1510099204,9cyclictest0-21swapper/321:00:273
1510099201,9cyclictest11757-21tr21:25:233
1510099201,8cyclictest12036-21cstates23:55:133
1510099201,7cyclictest6173-21memory23:40:203
1510099201,7cyclictest19025-21fschecks_time00:10:203
1510099201,7cyclictest0-21swapper/321:47:083
1510099201,6cyclictest15426-21netstat20:20:203
1510099201,6cyclictest14072-21egrep00:00:123
1510099201,6cyclictest0-21swapper/322:05:173
1510099201,6cyclictest0-21swapper/319:25:173
15100992013,4cyclictest31083-21/usr/sbin/munin22:10:163
15100992013,4cyclictest31083-21/usr/sbin/munin22:10:163
15100992013,2cyclictest41-21ksoftirqd/320:45:223
1510099201,11cyclictest7448-21/usr/sbin/munin23:45:173
1509099206,3cyclictest0-21swapper/119:20:451
15090992012,3cyclictest0-21swapper/123:11:011
1508499202,16cyclictest87350irq/126-enp1s0-19:40:110
1508499201,7cyclictest9995-21/usr/sbin/munin20:10:180
1508499201,13cyclictest7-21ksoftirqd/021:40:130
1508499201,10cyclictest0-21swapper/022:41:120
173892190,2sleep30-21swapper/322:52:273
1510099196,3cyclictest39-21rcuc/320:00:153
1510099193,5cyclictest41-21ksoftirqd/322:15:123
1510099193,4cyclictest808-21snmpd19:55:193
1510099191,7cyclictest0-21swapper/321:12:563
1510099191,6cyclictest31358-21/usr/sbin/munin19:45:193
1510099191,6cyclictest29159-21/usr/sbin/munin19:40:123
15100991915,3cyclictest0-21swapper/323:15:123
15100991915,2cyclictest39-21rcuc/319:15:013
15100991912,4cyclictest27102-21grep19:35:133
1508499195,3cyclictest87350irq/126-enp1s0-20:05:210
1508499191,7cyclictest808-21snmpd19:15:070
1508499191,6cyclictest563-40dnf19:46:120
1508499191,15cyclictest20992-21/usr/sbin/munin20:35:170
1510099188,2cyclictest23185-21/usr/sbin/munin20:40:263
1510099185,3cyclictest87650irq/129-enp1s0-19:30:133
1510099182,4cyclictest30488-21chrt23:21:143
1510099182,4cyclictest30488-21chrt23:21:143
1510099181,5cyclictest20992-21/usr/sbin/munin20:35:243
1510099181,4cyclictest7799-21/usr/sbin/munin20:05:193
1508499185,3cyclictest87550irq/128-enp1s0-20:40:110
1508499182,9cyclictest11-21rcuc/019:30:180
1508499181,6cyclictest16597-21/usr/sbin/munin20:25:120
1508499181,3cyclictest0-21swapper/020:20:180
1510099173,4cyclictest41-21ksoftirqd/322:20:203
15100991712,3cyclictest21074-21idleruntime19:20:163
15100991712,3cyclictest11275-21switchtime20:10:243
15084991711,3cyclictest12197-21/usr/sbin/munin20:15:150
1508499170,5cyclictest0-21swapper/019:20:110
1510099161,4cyclictest0-21swapper/320:15:123
1508499163,2cyclictest18861-21cpuspeed_turbos20:30:120
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional