You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-28 - 10:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackcslot7s.osadl.org (updated Tue Apr 28, 2026 00:44:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19748332980,3sleep10-21swapper/122:55:191
19645612950,3sleep23056-21PK-Backend22:25:202
139099910,77rtkit-daemon0-21swapper/319:05:313
189353528039,13sleep10-21swapper/119:05:301
189362927741,13sleep20-21swapper/219:06:272
189320426240,13sleep00-21swapper/019:05:250
189406299381,25cyclictest1934657-21munin-run21:05:013
19937472300,3chrt0-21swapper/123:50:161
139099300,21rtkit-daemon1939886-21munin-run21:20:002
19571872290,10sleep013-21ksoftirqd/022:05:210
189406299293,24cyclictest1989444-21munin-run23:40:003
189406299285,20cyclictest1949324-21grep21:45:133
189406299280,6cyclictest1991439-21apt23:45:133
1894062992610,14cyclictest1896881-21hddtemp_smartct19:15:173
139099260,19rtkit-daemon3056-21PK-Backend23:37:112
189406299259,14cyclictest1958724-21awk22:10:183
189406299256,17cyclictest1904648-21sed19:35:213
189406299251,22cyclictest883-21dbus-daemon19:55:003
1894062992512,11cyclictest2006375-21wc00:25:213
1894062992510,13cyclictest1920043-21awk20:20:183
1894062992510,13cyclictest1920043-21awk20:20:183
19278092240,3sleep30-21swapper/320:45:003
189406299248,14cyclictest1976425-21sed23:00:183
189406299247,15cyclictest1974082-21find22:55:113
189406299246,16cyclictest1969410-21latency22:40:183
189406299241,20cyclictest0-21swapper/321:10:123
189406299241,20cyclictest0-21swapper/319:10:123
1894062992410,12cyclictest2005392-21cat00:25:003
189406299240,22cyclictest1999247-21python300:05:213
189406299240,22cyclictest1911351-21/usr/sbin/munin19:55:193
1894060992416,6cyclictest883-21dbus-daemon00:29:372
189406099240,5cyclictest978-21snmpd23:50:482
189406099240,22cyclictest2003414-21gpgv00:20:012
189405099241,20cyclictest883-21dbus-daemon23:09:370
189406299238,13cyclictest2008074-21ntpq00:30:193
189406299238,13cyclictest2008074-21ntpq00:30:193
189406299238,13cyclictest1990494-21grep23:40:193
189406299238,13cyclictest1947812-21sed21:40:173
189406299236,15cyclictest1953771-21sed21:55:233
189406299234,17cyclictest436-20systemd-journal20:56:433
189406299234,17cyclictest1942444-21expr21:25:163
189406299233,18cyclictest1993321-21grep23:50:113
189406299233,18cyclictest1905766-21expr19:40:143
189406299231,9cyclictest1984642-21tr23:25:133
189406299231,3cyclictest1983476-21/usr/sbin/munin23:20:183
1894062992310,11cyclictest1954949-21if_enp2s022:00:163
189406299230,4cyclictest1978277-21ntp_kernel_pll_23:05:183
189406299230,21cyclictest2010131-21taskset00:35:223
189406299230,21cyclictest1962692-21awk22:20:183
189406299230,20cyclictest1965433-21apt22:30:123
1894050992320,2cyclictest3056-21PK-Backend19:45:480
1894050992313,7cyclictest1915541-21turbostat20:10:010
1894050992312,8cyclictest1903138-21apt-get19:35:000
19619642220,4sleep23056-21PK-Backend22:20:112
189406299226,14cyclictest2002023-21grep00:15:143
189406299226,14cyclictest1964750-21cat22:25:213
189406299226,14cyclictest1945624-21expr21:35:123
189406299225,15cyclictest1917658-21cpuspeed_turbos20:15:143
189406299224,16cyclictest1907502-21df19:45:143
189406299223,17cyclictest1987098-21seq23:30:193
189406299223,17cyclictest1987098-21seq23:30:183
1894062992211,9cyclictest1994874-21/usr/sbin/munin23:55:153
189406299221,19cyclictest1944569-21timerandwakeup21:30:233
1894062992211,9cyclictest1938474-21cpuspeed21:15:133
189406299221,16cyclictest1972365-21apt22:50:123
1894062992210,10cyclictest2000284-21tr00:10:133
1894062992210,10cyclictest1997014-21df00:00:143
189406299220,2cyclictest2156-21gnome-shell20:06:093
1894060992220,1cyclictest0-21swapper/220:34:522
1894060992220,1cyclictest0-21swapper/220:34:522
1894060992219,2cyclictest3056-21PK-Backend22:18:212
1894060992219,1cyclictest0-21swapper/221:34:202
1894060992215,5cyclictest978-21snmpd19:55:592
1894050992220,1cyclictest3056-21PK-Backend21:34:040
1894050992220,1cyclictest0-21swapper/022:12:430
1894050992220,1cyclictest0-21swapper/020:54:150
1894050992220,1cyclictest0-21swapper/020:32:460
1894050992220,1cyclictest0-21swapper/020:32:450
1894050992220,1cyclictest0-21swapper/000:31:340
1894050992220,1cyclictest0-21swapper/000:31:340
1894050992220,1cyclictest0-21swapper/000:24:240
1894050992219,2cyclictest3056-21PK-Backend19:40:210
189405099220,21cyclictest3056-21PK-Backend20:12:240
189405099220,20cyclictest3056-21PK-Backend23:00:350
139099220,15rtkit-daemon34-21ksoftirqd/221:26:532
139099220,14rtkit-daemon0-21swapper/200:01:292
18988272210,4sleep10-21swapper/119:20:171
189406299216,13cyclictest1982082-21timerwakeupswit23:15:233
189406299216,13cyclictest1940977-21gunzip21:20:193
189406299216,13cyclictest1928141-21tr20:45:133
189406299216,13cyclictest1920733-21latency_hist20:25:003
189406299215,14cyclictest1916811-21timerwakeupswit20:10:233
189406299215,14cyclictest1899174-21cat19:20:223
189406299215,13cyclictest1902471-21proc_pri19:30:193
189406299213,16cyclictest1970619-21find22:45:133
1894062992119,1cyclictest3056-21PK-Backend20:39:553
1894062992119,1cyclictest0-21swapper/322:39:253
1894062992119,1cyclictest0-21swapper/322:15:263
1894062992119,1cyclictest0-21swapper/321:51:063
1894062992119,1cyclictest0-21swapper/320:52:183
189406299211,18cyclictest1956216-21/usr/sbin/munin22:05:163
189406299210,20cyclictest0-21swapper/323:14:543
1894060992119,1cyclictest0-21swapper/223:58:452
1894060992119,1cyclictest0-21swapper/223:33:302
1894060992119,1cyclictest0-21swapper/223:33:302
1894060992119,1cyclictest0-21swapper/223:26:342
1894060992119,1cyclictest0-21swapper/223:23:302
1894060992119,1cyclictest0-21swapper/223:03:362
1894060992119,1cyclictest0-21swapper/222:01:312
1894060992119,1cyclictest0-21swapper/219:35:162
1894060992119,1cyclictest0-21swapper/219:31:382
1894060992119,1cyclictest0-21swapper/219:11:582
1894060992118,2cyclictest3056-21PK-Backend21:04:532
1894060992118,2cyclictest3056-21PK-Backend19:43:342
189406099210,19cyclictest3056-21PK-Backend20:46:272
189406099210,19cyclictest3056-21PK-Backend20:01:322
189406099210,19cyclictest3056-21PK-Backend00:35:152
189405399212,17cyclictest1969273-21diskstats22:40:141
1894053992119,1cyclictest0-21swapper/123:41:241
1894053992113,6cyclictest1931196-21cron20:55:011
189405399210,20cyclictest0-21swapper/123:02:101
189405399210,20cyclictest0-21swapper/122:29:261
189405399210,20cyclictest0-21swapper/121:29:021
189405399210,19cyclictest3056-21PK-Backend20:21:011
189405399210,19cyclictest3056-21PK-Backend20:21:011
1894050992119,1cyclictest3056-21PK-Backend22:02:290
1894050992119,1cyclictest0-21swapper/023:13:470
1894050992119,1cyclictest0-21swapper/022:54:470
1894050992119,1cyclictest0-21swapper/022:21:490
1894050992119,1cyclictest0-21swapper/021:29:440
1894050992119,1cyclictest0-21swapper/020:41:310
1894050992119,1cyclictest0-21swapper/020:28:470
1894050992119,1cyclictest0-21swapper/019:55:070
1894050992119,1cyclictest0-21swapper/019:26:280
1894050992118,2cyclictest3056-21PK-Backend23:41:320
1894050992113,5cyclictest1947012-21latency_hist21:40:010
189405099210,20cyclictest0-21swapper/023:26:040
189405099210,20cyclictest0-21swapper/022:40:140
189405099210,20cyclictest0-21swapper/022:28:450
189405099210,20cyclictest0-21swapper/022:19:250
139099210,15rtkit-daemon3056-21PK-Backend22:53:422
139099210,14rtkit-daemon0-21swapper/222:58:052
139099210,13rtkit-daemon0-21swapper/221:11:482
189406299205,13cyclictest1900430-21grep19:25:163
189406299201,17cyclictest883-21dbus-daemon20:05:003
189406299201,16cyclictest1935668-21ntp_states21:05:193
189406299200,19cyclictest3056-21PK-Backend20:34:203
189406299200,19cyclictest3056-21PK-Backend20:34:203
1894060992018,1cyclictest3056-21PK-Backend20:05:382
1894060992018,1cyclictest3056-21PK-Backend19:19:212
1894060992018,1cyclictest0-21swapper/223:06:132
1894060992018,1cyclictest0-21swapper/222:12:122
1894060992018,1cyclictest0-21swapper/221:56:582
1894060992018,1cyclictest0-21swapper/221:52:252
1894060992018,1cyclictest0-21swapper/221:47:182
1894060992018,1cyclictest0-21swapper/220:20:122
1894060992018,1cyclictest0-21swapper/220:20:112
1894060992018,1cyclictest0-21swapper/219:54:242
1894060992018,1cyclictest0-21swapper/219:47:262
1894060992018,1cyclictest0-21swapper/200:10:072
1894060992013,5cyclictest1992186-21ntpq23:45:212
1894060992012,6cyclictest2004430-21memory00:20:182
189406099200,19cyclictest3056-21PK-Backend21:40:502
189406099200,19cyclictest0-21swapper/223:41:142
189406099200,19cyclictest0-21swapper/221:36:302
189406099200,18cyclictest3056-21PK-Backend20:52:502
189406099200,18cyclictest0-21swapper/222:47:202
1894053992018,1cyclictest0-21swapper/122:23:161
1894053992018,1cyclictest0-21swapper/122:17:561
1894053992018,1cyclictest0-21swapper/121:58:561
1894053992018,1cyclictest0-21swapper/119:43:501
1894053992017,2cyclictest3056-21PK-Backend20:12:091
1894053992012,6cyclictest1995758-21python323:55:191
189405399200,1cyclictest3056-21PK-Backend21:54:241
189405399200,1cyclictest3056-21PK-Backend20:19:261
189405399200,1cyclictest3056-21PK-Backend19:32:561
189405399200,1cyclictest0-21swapper/121:06:511
189405399200,19cyclictest3056-21PK-Backend23:15:351
189405399200,19cyclictest3056-21PK-Backend23:06:221
189405399200,19cyclictest3056-21PK-Backend22:31:481
189405399200,19cyclictest0-21swapper/123:22:381
189405399200,19cyclictest0-21swapper/122:39:321
189405399200,19cyclictest0-21swapper/121:40:381
189405399200,19cyclictest0-21swapper/121:10:231
189405399200,18cyclictest0-21swapper/122:45:061
189405399200,18cyclictest0-21swapper/121:21:461
1894050992018,1cyclictest3056-21PK-Backend21:52:150
1894050992018,1cyclictest3056-21PK-Backend21:42:280
1894050992018,1cyclictest0-21swapper/022:38:380
1894050992018,1cyclictest0-21swapper/019:24:380
1894050992017,2cyclictest3056-21PK-Backend00:07:000
1894050992015,4cyclictest13-21ksoftirqd/020:50:000
1894050992014,5cyclictest1910325-21cron19:55:000
1894050992011,7cyclictest13-21ksoftirqd/023:17:320
189405099200,19cyclictest3056-21PK-Backend00:36:410
189405099200,19cyclictest0-21swapper/023:56:200
189405099200,19cyclictest0-21swapper/021:02:470
189405099200,19cyclictest0-21swapper/000:10:040
19553712190,4sleep10-21swapper/122:00:211
1894060991917,1cyclictest0-21swapper/222:33:482
1894060991917,1cyclictest0-21swapper/220:11:192
1894060991913,4cyclictest436-20systemd-journal19:21:442
189406099190,1cyclictest3056-21PK-Backend23:10:252
189406099190,1cyclictest0-21swapper/222:35:242
189406099190,18cyclictest0-21swapper/220:59:252
189406099190,18cyclictest0-21swapper/220:42:022
1894053991917,1cyclictest3056-21PK-Backend19:52:361
1894053991917,1cyclictest0-21swapper/123:48:061
1894053991917,1cyclictest0-21swapper/123:25:431
1894053991917,1cyclictest0-21swapper/122:12:351
1894053991917,1cyclictest0-21swapper/120:49:341
1894053991917,1cyclictest0-21swapper/119:16:261
1894053991911,6cyclictest1979298-21awk23:10:111
189405399190,1cyclictest3056-21PK-Backend23:31:371
189405399190,1cyclictest3056-21PK-Backend23:31:371
189405399190,1cyclictest0-21swapper/120:26:231
189405399190,1cyclictest0-21swapper/100:02:121
189405399190,18cyclictest3056-21PK-Backend21:47:061
189405399190,18cyclictest0-21swapper/122:51:081
189405399190,18cyclictest0-21swapper/120:40:421
189405099192,13cyclictest1987974-21apt23:35:120
1894050991917,1cyclictest3056-21PK-Backend22:47:240
1894050991917,1cyclictest0-21swapper/023:53:280
1894050991917,1cyclictest0-21swapper/021:22:410
1894050991917,1cyclictest0-21swapper/020:02:170
1894050991914,4cyclictest13-21ksoftirqd/022:35:000
1894050991912,5cyclictest1897973-21idleruntime-cro19:20:000
189405099190,1cyclictest3056-21PK-Backend23:20:220
189405099190,1cyclictest3056-21PK-Backend19:10:120
189405099190,1cyclictest0-21swapper/020:16:260
189405099190,1cyclictest0-21swapper/019:39:190
189405099190,18cyclictest3056-21PK-Backend20:21:500
189405099190,18cyclictest3056-21PK-Backend20:21:500
139099190,3rtkit-daemon3056-21PK-Backend20:25:122
139099190,13rtkit-daemon0-21swapper/223:16:442
19389312180,3sleep1251rcuc/121:15:181
1894060991816,1cyclictest0-21swapper/220:19:242
1894060991812,4cyclictest1941575-21cron21:24:592
189406099180,1cyclictest0-21swapper/221:08:142
189406099180,1cyclictest0-21swapper/220:38:202
1894053991816,1cyclictest3056-21PK-Backend19:47:421
1894053991816,1cyclictest3056-21PK-Backend19:36:341
1894053991816,1cyclictest0-21swapper/121:02:461
1894053991816,1cyclictest0-21swapper/100:33:561
1894053991816,1cyclictest0-21swapper/100:33:561
1894053991816,1cyclictest0-21swapper/100:27:341
1894053991816,1cyclictest0-21swapper/100:24:161
1894053991813,3cyclictest2156-21gnome-shell00:15:091
1894053991813,3cyclictest1943639-21cpu21:30:121
189405399181,16cyclictest2009213-21tr00:35:131
1894053991811,5cyclictest1988495-21cat23:35:171
189405099183,12cyclictest1938365-21/usr/sbin/munin21:15:230
1894050991816,1cyclictest3056-21PK-Backend23:45:240
1894050991811,5cyclictest1925401-21cat20:35:210
189405099180,1cyclictest3056-21PK-Backend21:08:140
189405099180,1cyclictest3056-21PK-Backend00:26:270
189405099180,1cyclictest3056-21PK-Backend00:15:110
189405099180,1cyclictest0-21swapper/021:47:550
189405099180,17cyclictest3056-21PK-Backend21:10:580
18999482170,3sleep10-21swapper/119:25:121
1894060991715,1cyclictest0-21swapper/222:06:512
1894060991713,3cyclictest1999182-21processes00:05:212
1894060991712,4cyclictest2007617-21expr00:30:132
1894060991712,4cyclictest2007617-21expr00:30:132
1894053991711,5cyclictest1895151-21sed19:10:211
1894053991710,5cyclictest1-21systemd20:35:001
1894053991710,5cyclictest1-21systemd20:35:001
1894050991715,1cyclictest0-21swapper/020:57:330
189405399168,6cyclictest1924866-21fw_packets20:35:141
1894053991614,1cyclictest0-21swapper/120:10:001
1894053991614,1cyclictest0-21swapper/100:14:221
1894053991614,1cyclictest0-21swapper/100:06:451
1894053991611,3cyclictest1912718-21entropy20:00:161
1894050991614,1cyclictest0-21swapper/023:33:480
1894050991614,1cyclictest0-21swapper/023:33:480
1894050991613,2cyclictest3056-21PK-Backend21:58:440
1894060991513,1cyclictest3056-21PK-Backend22:40:102
1894060991510,3cyclictest19010742sleep219:25:532
189405399158,5cyclictest1946748-21sshd21:39:431
1894053991513,1cyclictest0-21swapper/119:55:401
1894053991511,3cyclictest1931433-21/usr/sbin/munin20:55:171
1894053991511,3cyclictest1095-21runrttasks22:06:321
189405099150,6cyclictest3056-21PK-Backend22:55:200
189405099143,7cyclictest2176-21JS10:27:480
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional