You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-11 - 18:01

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Thu Jun 11, 2026 12:44:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
190909121180,3sleep10-21swapper/107:45:161
189972521160,3sleep00-21swapper/007:20:000
200743221050,3sleep10-21swapper/112:25:001
19513302940,3sleep10-21swapper/109:45:011
189541827640,13sleep30-21swapper/307:08:063
189422727042,20sleep10-21swapper/107:05:131
189449026746,14sleep00-21swapper/007:05:170
189499626544,13sleep20-21swapper/207:05:262
20046972550,1chrt2004698-21/usr/sbin/munin12:15:163
20017332460,2sleep33056-21PK-Backend12:05:223
20017332460,2sleep33056-21PK-Backend12:05:223
19015422460,4sleep33056-21PK-Backend07:25:003
20125622400,1sleep12012561-21awk12:40:001
139099400,3rtkit-daemon0-21swapper/012:35:060
139099390,3rtkit-daemon0-21swapper/011:52:360
19885492360,4sleep23056-21PK-Backend11:30:122
189570799359,14cyclictest1095-21runrttasks09:12:022
1895707993214,16cyclictest1964925-21timerandwakeup10:20:222
1895707993113,16cyclictest1920744-21cron08:20:002
19072642300,6sleep10-21swapper/107:40:141
1895707993014,14cyclictest1924214-21latency_hist08:30:012
1895709992910,17cyclictest2008258-21cut12:25:183
1895709992910,17cyclictest1972954-21expr10:45:123
189570799292,7cyclictest82332-21update-manager12:21:312
189570799292,24cyclictest0-21swapper/209:25:122
189570799292,10cyclictest2009345-21apt12:30:112
1895707992910,17cyclictest2008245-21grep12:25:172
189570799290,7cyclictest0-21swapper/210:50:112
1895707992811,15cyclictest1934868-21cron08:59:592
189570799279,16cyclictest2003311-21sed12:10:222
189570799278,17cyclictest2000850-21cstates12:05:132
189570799278,17cyclictest2000850-21cstates12:05:132
189570799275,6cyclictest436-20systemd-journal09:34:552
189570999260,4cyclictest1-21systemd08:35:003
189570799267,17cyclictest1974184-21sed10:50:012
189570799267,17cyclictest0-21swapper/209:59:262
189570799267,17cyclictest0-21swapper/209:36:082
1895707992619,6cyclictest0-21swapper/211:05:142
189570799261,6cyclictest1915778-21/usr/sbin/munin08:05:172
1895702992615,6cyclictest141rcu_preempt11:59:480
139099260,4rtkit-daemon1389-21rtkit-daemon11:03:060
139099260,4rtkit-daemon1389-21rtkit-daemon11:03:050
139099260,4rtkit-daemon1389-21rtkit-daemon10:08:361
139099260,4rtkit-daemon1389-21rtkit-daemon09:58:161
139099260,4rtkit-daemon1389-21rtkit-daemon09:00:340
139099260,4rtkit-daemon1389-21rtkit-daemon08:49:041
139099260,4rtkit-daemon1389-21rtkit-daemon07:38:461
19182152250,6sleep00-21swapper/008:10:160
189570999258,16cyclictest1975086-21grep10:50:173
189570999258,15cyclictest2010808-21latency_hist12:35:013
189570999258,15cyclictest1999405-21grep12:00:183
1895709992511,11cyclictest3056-21PK-Backend11:31:293
189570999250,19cyclictest3056-21PK-Backend08:45:173
189570799256,3cyclictest20123172chrt12:37:172
189570799253,4cyclictest0-21swapper/207:37:482
189570799253,19cyclictest1951001-21apt-get09:45:002
189570799252,8cyclictest1903078-21dump-pmu-power07:30:002
189570799251,4cyclictest0-21swapper/208:30:122
1895707992510,13cyclictest1982941-21apt-key11:15:002
189570799250,18cyclictest0-21swapper/207:51:022
1895704992513,8cyclictest2267-21packagekitd09:45:041
139099250,4rtkit-daemon1389-21rtkit-daemon11:28:171
139099250,4rtkit-daemon1389-21rtkit-daemon08:38:521
139099250,4rtkit-daemon1389-21rtkit-daemon08:25:011
19952202240,3sleep23056-21PK-Backend11:50:012
19286222240,3sleep00-21swapper/008:40:160
189570999247,15cyclictest1896023-21cpuspeed_turbos07:10:123
189570999243,19cyclictest1909632-21timerwakeupswit07:45:223
189570999241,5cyclictest436-20systemd-journal12:21:313
1895709992413,10cyclictest1944052-21/usr/sbin/munin09:25:153
189570999240,8cyclictest2011226-21cpuspeed_turbos12:35:123
189570999240,22cyclictest3056-21PK-Backend07:51:023
189570999240,22cyclictest1916384-21memory08:05:163
189570799247,15cyclictest1958817-21cpuspeed_turbos10:05:112
189570799247,15cyclictest0-21swapper/209:08:062
189570799247,15cyclictest0-21swapper/209:08:052
189570799247,15cyclictest0-21swapper/208:05:002
189570799247,15cyclictest0-21swapper/208:04:592
189570799246,3cyclictest19636142chrt10:19:182
189570799246,3cyclictest19293502chrt08:41:022
189570799245,5cyclictest883-21dbus-daemon09:05:012
189570799245,17cyclictest2156-21gnome-shell08:21:092
189570799240,8cyclictest436-20systemd-journal08:54:552
189570799240,7cyclictest1976494-21hddtemp_smartct10:55:152
189570799240,7cyclictest1961933-21perf10:15:002
189570499241,21cyclictest1317-21systemd08:43:251
139099240,4rtkit-daemon1389-21rtkit-daemon10:38:391
139099240,4rtkit-daemon1389-21rtkit-daemon10:19:031
139099240,4rtkit-daemon1389-21rtkit-daemon09:28:190
189570999239,12cyclictest1914901-21processes08:00:213
189570999239,12cyclictest1914901-21processes08:00:213
189570999237,14cyclictest1902165-21latency07:25:163
189570999231,17cyclictest1951005-21perf09:45:003
189570999230,21cyclictest3056-21PK-Backend11:05:123
189570999230,18cyclictest3056-21PK-Backend09:03:153
189570799236,15cyclictest0-21swapper/211:16:162
189570799235,4cyclictest1906840-21sshd07:40:072
189570799234,17cyclictest0-21swapper/210:39:232
189570799233,3cyclictest0-21swapper/212:16:272
189570799233,3cyclictest0-21swapper/211:39:302
189570799233,3cyclictest0-21swapper/208:10:552
189570799232,6cyclictest1986163-21chrt11:21:022
189570799232,3cyclictest0-21swapper/207:13:542
189570799231,6cyclictest1967553-21grep10:30:112
189570799231,5cyclictest1943812-21perf09:25:002
189570799230,8cyclictest2000390-21munin-plugin-st12:04:592
189570799230,4cyclictest436-20systemd-journal10:05:002
189570799230,4cyclictest436-20systemd-journal10:04:592
189570799230,2cyclictest1901335-21latency_hist07:25:002
1895704992321,1cyclictest0-21swapper/107:31:521
189570299232,9cyclictest13-21ksoftirqd/010:15:200
139099230,3rtkit-daemon1389-21rtkit-daemon11:22:581
139099230,3rtkit-daemon1389-21rtkit-daemon10:00:511
139099230,3rtkit-daemon1389-21rtkit-daemon10:00:501
189570999228,13cyclictest3056-21PK-Backend09:12:113
189570999224,4cyclictest1912825-21cut07:55:163
1895709992219,2cyclictest3056-21PK-Backend09:39:383
189570999221,4cyclictest1943811-21munin-run09:25:013
189570999221,17cyclictest1924212-21turbostat.cron08:30:013
189570999220,8cyclictest1969650-21grep10:35:173
189570999220,7cyclictest1978681-21grep11:00:213
189570999220,7cyclictest1978681-21grep11:00:213
189570999220,7cyclictest1929306-21chrt08:40:273
189570999220,20cyclictest3056-21PK-Backend08:57:013
189570999220,20cyclictest3056-21PK-Backend07:17:583
189570799229,11cyclictest1095-21runrttasks11:02:542
189570799229,11cyclictest1095-21runrttasks11:02:542
189570799226,14cyclictest0-21swapper/207:55:302
189570799223,3cyclictest0-21swapper/211:56:582
189570799223,3cyclictest0-21swapper/209:52:482
189570799223,3cyclictest0-21swapper/209:48:312
189570799223,3cyclictest0-21swapper/208:39:552
189570799223,3cyclictest0-21swapper/207:31:372
189570799223,2cyclictest0-21swapper/210:41:402
189570799221,6cyclictest1987454-21/usr/sbin/munin11:25:162
189570799220,8cyclictest1992710-21ntpq11:40:182
189570799220,7cyclictest1996923-21munin-run11:55:002
189570799220,7cyclictest1930687-21ntpq08:45:172
189570799220,6cyclictest1898468-21expr07:15:132
1895704992220,1cyclictest0-21swapper/111:43:111
1895704992220,1cyclictest0-21swapper/111:06:341
1895704992220,1cyclictest0-21swapper/110:53:221
1895704992220,1cyclictest0-21swapper/107:13:061
1895704992213,7cyclictest1999283-21df12:00:131
1895704992212,8cyclictest436-20systemd-journal08:55:001
189570499220,20cyclictest1983655-21df_inode11:15:121
1895702992220,1cyclictest0-21swapper/011:36:360
1895702992215,2cyclictest141rcu_preempt09:10:020
1895702992214,6cyclictest1942734-21fw_packets09:20:140
139099220,3rtkit-daemon1389-21rtkit-daemon10:22:070
189570999218,12cyclictest3056-21PK-Backend10:21:163
189570999217,12cyclictest0-21swapper/311:55:133
1895709992119,1cyclictest3056-21PK-Backend11:28:423
1895709992119,1cyclictest0-21swapper/307:40:213
189570999211,7cyclictest1903350-21apt07:30:113
189570999211,5cyclictest1947288-21sh09:35:003
1895709992113,6cyclictest1956721-21cron09:59:593
189570999210,6cyclictest1951414-21apt-config09:45:023
189570999210,1cyclictest3056-21PK-Backend11:35:203
189570999210,1cyclictest3056-21PK-Backend10:30:093
189570999210,19cyclictest3056-21PK-Backend12:10:193
189570999210,19cyclictest3056-21PK-Backend11:10:113
189570999210,19cyclictest3056-21PK-Backend10:02:233
189570999210,19cyclictest3056-21PK-Backend10:02:233
189570999210,19cyclictest3056-21PK-Backend08:15:023
189570799214,3cyclictest0-21swapper/209:18:082
189570799213,2cyclictest0-21swapper/207:46:042
189570799210,4cyclictest1965929-21expr10:25:122
1895704992119,1cyclictest0-21swapper/111:37:501
1895704992119,1cyclictest0-21swapper/110:44:301
1895704992119,1cyclictest0-21swapper/110:13:551
1895704992119,1cyclictest0-21swapper/109:18:391
1895704992119,1cyclictest0-21swapper/109:02:571
1895704992119,1cyclictest0-21swapper/108:32:221
1895704992119,1cyclictest0-21swapper/108:01:261
1895704992119,1cyclictest0-21swapper/108:01:261
1895704992119,1cyclictest0-21swapper/107:58:491
1895704992119,1cyclictest0-21swapper/107:27:191
189570499210,20cyclictest0-21swapper/109:20:141
189570499210,20cyclictest0-21swapper/109:11:231
189570299219,4cyclictest13-21ksoftirqd/012:23:110
1895702992119,1cyclictest0-21swapper/008:00:190
1895702992119,1cyclictest0-21swapper/008:00:180
1895702992119,1cyclictest0-21swapper/007:54:440
1895702992119,1cyclictest0-21swapper/007:49:030
1895702992119,1cyclictest0-21swapper/007:10:300
1895702992114,2cyclictest141rcu_preempt12:10:100
1895702992114,2cyclictest141rcu_preempt10:47:450
1895702992110,2cyclictest141rcu_preempt10:50:030
189570299210,5cyclictest0-21swapper/012:01:090
189570299210,20cyclictest0-21swapper/012:26:250
189570299210,20cyclictest0-21swapper/010:34:300
189570299210,20cyclictest0-21swapper/010:27:200
189570299210,20cyclictest0-21swapper/009:40:440
139099210,14rtkit-daemon0-21swapper/111:46:251
189570999207,11cyclictest1940323-21/usr/sbin/munin09:15:223
1895709992018,1cyclictest3056-21PK-Backend11:21:183
1895709992018,1cyclictest3056-21PK-Backend10:10:153
1895709992018,1cyclictest3056-21PK-Backend08:21:523
1895709992017,2cyclictest3056-21PK-Backend10:59:553
189570999201,5cyclictest1926962-21cut08:35:173
189570999201,3cyclictest1918397-21python308:10:193
189570999200,4cyclictest1966537-21/usr/sbin/munin10:25:193
189570999200,19cyclictest3056-21PK-Backend11:40:153
189570999200,19cyclictest3056-21PK-Backend09:50:483
189570999200,19cyclictest0-21swapper/307:35:203
1895704992018,1cyclictest0-21swapper/111:32:301
1895704992018,1cyclictest0-21swapper/110:30:051
1895704992014,4cyclictest2003842-21sh12:15:011
189570499200,19cyclictest0-21swapper/111:10:531
189570499200,19cyclictest0-21swapper/110:23:011
189570499200,19cyclictest0-21swapper/109:50:571
189570499200,19cyclictest0-21swapper/109:27:521
189570499200,19cyclictest0-21swapper/107:21:461
189570299209,3cyclictest13-21ksoftirqd/010:37:120
1895702992018,1cyclictest0-21swapper/012:15:290
1895702992018,1cyclictest0-21swapper/011:47:240
1895702992018,1cyclictest0-21swapper/010:01:180
1895702992018,1cyclictest0-21swapper/010:01:180
1895702992018,1cyclictest0-21swapper/009:47:540
1895702992018,1cyclictest0-21swapper/008:55:140
1895702992016,3cyclictest0-21swapper/007:40:110
1895702992015,4cyclictest13-21ksoftirqd/011:31:290
1895702992015,4cyclictest13-21ksoftirqd/010:06:260
1895702992015,4cyclictest13-21ksoftirqd/008:16:230
189570299201,11cyclictest13-21ksoftirqd/007:30:110
1895702992010,3cyclictest13-21ksoftirqd/008:28:010
1895702992010,2cyclictest141rcu_preempt08:21:310
189570299200,19cyclictest0-21swapper/011:41:290
189570299200,19cyclictest0-21swapper/011:28:240
189570299200,19cyclictest0-21swapper/011:19:250
189570999196,12cyclictest3056-21PK-Backend11:19:423
189570999192,3cyclictest3056-21PK-Backend10:07:573
189570999192,2cyclictest0-21swapper/311:50:023
189570999190,6cyclictest1971853-21grep10:40:233
189570999190,4cyclictest1995185-21munin-plugin-st11:50:003
189570999190,4cyclictest19636102sleep310:19:163
189570999190,1cyclictest3056-21PK-Backend08:52:563
189570499193,15cyclictest1095-21runrttasks07:18:001
1895704991917,1cyclictest3056-21PK-Backend10:46:471
1895704991917,1cyclictest0-21swapper/111:03:171
1895704991917,1cyclictest0-21swapper/111:03:161
1895704991917,1cyclictest0-21swapper/110:55:401
1895704991917,1cyclictest0-21swapper/110:28:491
1895704991917,1cyclictest0-21swapper/109:33:191
1895704991917,1cyclictest0-21swapper/108:20:531
1895704991917,1cyclictest0-21swapper/108:05:041
1895704991913,5cyclictest1949014-21apt-get09:40:021
1895704991912,5cyclictest1995754-21expr11:50:111
1895704991911,6cyclictest1937596-21ntpq09:05:191
1895704991911,6cyclictest1937596-21ntpq09:05:181
189570499190,1cyclictest0-21swapper/112:09:271
189570499190,1cyclictest0-21swapper/112:09:271
189570499190,1cyclictest0-21swapper/108:58:181
189570299199,3cyclictest13-21ksoftirqd/007:26:520
189570299197,4cyclictest13-21ksoftirqd/008:09:210
189570299191,7cyclictest13-21ksoftirqd/007:25:000
1895702991917,1cyclictest0-21swapper/009:08:390
1895702991917,1cyclictest0-21swapper/009:08:390
1895702991913,5cyclictest1912871-21cat07:55:170
1895702991912,5cyclictest1932548-21python308:50:210
1895702991911,6cyclictest1976961-21python310:55:200
189570299190,1cyclictest0-21swapper/010:14:370
189570299190,1cyclictest0-21swapper/009:52:260
189570299190,18cyclictest0-21swapper/011:11:410
139099190,15rtkit-daemon1895696-21cyclictest11:07:410
139099190,13rtkit-daemon0-21swapper/112:19:121
1895704991813,3cyclictest1998651-21dump-pmu-power11:59:591
1895704991812,5cyclictest26-21ksoftirqd/108:16:231
189570299189,4cyclictest13-21ksoftirqd/009:30:480
189570299188,3cyclictest13-21ksoftirqd/010:42:210
189570299187,3cyclictest141rcu_preempt09:15:250
189570299183,14cyclictest1095-21runrttasks08:37:110
1895702991813,4cyclictest13-21ksoftirqd/009:40:010
189570299180,4cyclictest13-21ksoftirqd/008:45:210
189570999176,10cyclictest19380422chrt09:06:173
189570999176,10cyclictest19380422chrt09:06:173
1895704991712,3cyclictest1910397-21cpu07:50:141
189570299179,3cyclictest13-21ksoftirqd/007:37:480
189570299178,3cyclictest13-21ksoftirqd/008:31:550
189570299170,9cyclictest1986497-21idleruntime-cro11:25:000
189570299170,9cyclictest13-21ksoftirqd/009:55:210
1895704991611,4cyclictest2010185-21cut12:30:191
1895704991610,5cyclictest1917746-21ls08:10:121
1895702991610,5cyclictest2000652-21awk12:05:100
1895702991610,5cyclictest2000652-21awk12:05:100
189570299160,5cyclictest3056-21PK-Backend12:30:140
20084662150,4sleep10-21swapper/112:25:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional