You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-06 - 06:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackeslot8s.osadl.org (updated Sun Jul 06, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
118553799753748,4cyclictest1530107-21kworker/3:2+events23:58:323
118553799752747,4cyclictest1135055-21kworker/3:0+events20:39:293
118553799750745,4cyclictest1324055-21kworker/3:1+events22:11:233
118553799747738,6cyclictest1135055-21kworker/3:0+events19:23:033
118553799746742,3cyclictest1324055-21kworker/3:1+events22:21:163
118553799746742,3cyclictest1286908-21turbostat20:34:543
118553799746741,4cyclictest1135055-21kworker/3:0+events20:44:033
118553799745740,4cyclictest1324055-21kworker/3:1+events21:46:173
118553799744740,3cyclictest1324055-21kworker/3:1+events23:52:523
118553799744740,3cyclictest1324055-21kworker/3:1+events23:27:223
118553799744739,4cyclictest1135055-21kworker/3:0+events20:15:503
118553799743738,4cyclictest1324055-21kworker/3:1+events23:33:043
118553799743737,5cyclictest1324055-21kworker/3:1+events22:01:483
118553599743738,4cyclictest1305755-21kworker/2:0+events21:10:232
118553599743738,4cyclictest1305755-21kworker/2:0+events21:10:232
118553299743738,4cyclictest1467853-21kworker/1:0+events00:24:341
118553799742738,3cyclictest1135055-21kworker/3:0+events19:10:323
118553799742735,6cyclictest1324055-21kworker/3:1+events21:57:413
118553799741737,3cyclictest1324055-21kworker/3:1+events22:28:023
118553799741737,3cyclictest1324055-21kworker/3:1+events22:06:163
118553799741737,3cyclictest1135055-21kworker/3:0+events19:38:133
118553799741735,5cyclictest1324055-21kworker/3:1+events23:17:483
118553799741735,5cyclictest1135055-21kworker/3:0+events20:12:093
118553799740736,3cyclictest1324055-21kworker/3:1+events22:36:113
118553799740736,3cyclictest1135055-21kworker/3:0+events20:23:153
118553799740736,3cyclictest1135055-21kworker/3:0+events20:04:043
118553799740735,4cyclictest1324055-21kworker/3:1+events22:56:393
118553799739735,3cyclictest1324055-21kworker/3:1+events21:26:253
118553799739734,4cyclictest1324055-21kworker/3:1+events23:43:473
118553799739734,4cyclictest1324055-21kworker/3:1+events23:03:343
118553799739734,4cyclictest1324055-21kworker/3:1+events21:37:533
118553799739732,4cyclictest1324055-21kworker/3:1+events23:13:383
118553799738734,3cyclictest1135055-21kworker/3:0+events20:28:433
118553799738733,4cyclictest1324055-21kworker/3:1+events23:06:593
118553799738733,4cyclictest1324055-21kworker/3:1+events21:24:113
118553299738730,3cyclictest1467853-21kworker/1:0+events00:25:201
118553799737733,3cyclictest1324055-21kworker/3:1+events22:45:433
118553799737733,3cyclictest1135055-21kworker/3:0+events19:56:493
118553799737732,4cyclictest1135055-21kworker/3:0+events20:49:293
118553799737731,5cyclictest1135055-21kworker/3:0+events20:07:523
118553299737732,4cyclictest1467853-21kworker/1:0+events00:05:231
118553799735730,3cyclictest1135055-21kworker/3:0+events20:58:213
118553299734725,4cyclictest1467853-21kworker/1:0+events00:10:221
118553799733728,4cyclictest1135055-21kworker/3:0+events19:47:523
118553799731727,3cyclictest1324055-21kworker/3:1+events22:18:173
118553799731726,4cyclictest1135055-21kworker/3:0+events20:54:323
118553799731725,5cyclictest1324055-21kworker/3:1+events23:46:593
118553799730727,2cyclictest1324055-21kworker/3:1+events21:40:523
118553799730726,3cyclictest1324055-21kworker/3:1+events21:54:223
118553799728724,3cyclictest1135055-21kworker/3:0+events19:53:063
118553299727724,2cyclictest1467853-21kworker/1:0+events00:18:221
118553299725722,2cyclictest3258197-21kworker/1:2+events19:31:501
118553799720709,9cyclictest1530107-21kworker/3:2+events00:00:233
118553299720717,2cyclictest1467853-21kworker/1:0+events00:30:211
118553799718713,4cyclictest1449287-21turbostat22:44:553
118553299716712,3cyclictest3258197-21kworker/1:2+events19:27:201
118553299716711,4cyclictest3258197-21kworker/1:2+events21:32:301
118553599715710,4cyclictest1318136-21turbostat20:59:552
118553299712708,3cyclictest1467853-21kworker/1:0+events00:35:521
118553799706702,3cyclictest1324055-21kworker/3:1+events22:32:443
118553799706697,7cyclictest1135055-21kworker/3:0+events19:40:213
118553599706701,4cyclictest1343057-21kworker/2:2+events21:18:382
118553599704700,3cyclictest1305755-21kworker/2:0+events21:03:172
118553599704700,3cyclictest1305755-21kworker/2:0+events21:03:162
118553599700695,4cyclictest1305755-21kworker/2:0+events21:05:222
118553299694691,2cyclictest1548641-21turbostat00:04:561
118553799690685,4cyclictest1212642-21turbostat19:34:563
11855309969010,674cyclictest1274242-21kworker/0:0+events22:40:220
118553799689685,3cyclictest1135055-21kworker/3:0+events19:17:083
118553099687682,4cyclictest1443057-21turbostat22:39:540
118553799681679,1cyclictest1324055-21kworker/3:1+events22:51:033
118553799673669,3cyclictest1324055-21kworker/3:1+events23:36:513
118553299671665,5cyclictest1200228-21turbostat19:24:561
118553799657654,2cyclictest1343388-21turbostat21:19:553
118553799646641,4cyclictest1324055-21kworker/3:1+events23:20:453
118553599623619,3cyclictest1591974-21turbostat00:39:542
118553799621617,3cyclictest1361980-21turbostat21:34:563
118553299603601,1cyclictest1355799-21turbostat21:29:561
1185530992623,2cyclictest1573459-21systemctl00:20:010
118553599240,3cyclictest1225006-21idleruntime-cro19:40:002
1185535992319,2cyclictest1467825-21latency_hist22:54:592
1185535992218,2cyclictest0-21swapper/220:00:012
1185535992216,4cyclictest1299251-21cron20:40:002
1185530992219,2cyclictest1374623-21sh21:40:010
1185530992118,2cyclictest0-21swapper/000:39:590
1185530992117,3cyclictest506-21in:imuxsock20:50:000
1185535992017,2cyclictest1286906-21idleruntime-cro20:30:002
1185535992013,5cyclictest1498798-21cron23:20:002
1185530992017,2cyclictest493-21dbus-daemon21:00:010
1185530992016,3cyclictest1330794-21cron21:05:010
1185530992016,3cyclictest1330794-21cron21:05:000
1185530992016,2cyclictest0-21swapper/019:55:000
1185530992012,6cyclictest0-21swapper/022:15:000
118553599197,6cyclictest1474021-21munin-run23:00:002
1185535991916,2cyclictest1455455-21cron22:45:002
1185535991916,2cyclictest1370151-21kthreadcore21:35:182
1185530991916,2cyclictest1548705-21latency_hist00:00:000
1185530991916,2cyclictest1530130-21munin-plugin-st23:45:000
1185530991915,3cyclictest509-21rs:main0
1185535991814,3cyclictest1449296-21munin-run22:39:592
1185535991814,2cyclictest0-21swapper/223:25:012
1185535991813,3cyclictest1218794-21turbostat.cron19:35:012
1185532991815,2cyclictest0-21swapper/122:20:001
1185530991816,1cyclictest1355791-21cron21:25:000
1185535991714,2cyclictest0-21swapper/221:55:012
1185535991714,2cyclictest0-21swapper/219:25:002
1185535991711,4cyclictest493-21dbus-daemon19:15:002
1185532991714,2cyclictest502-21systemd-logind19:20:011
1185532991714,2cyclictest1387766-21irqcore21:50:161
1185530991714,2cyclictest1555374-21fschecks_count00:05:140
1185530991713,3cyclictest1275046-21gltestperf20:20:140
1185535991613,2cyclictest1461637-21cron22:49:592
1185532991613,2cyclictest2873-21cron22:29:591
1185532991613,2cyclictest1243940-21cpuspeed_turbos19:55:141
1185532991612,3cyclictest1287240-21cpuspeed_turbos20:30:131
1185532991612,3cyclictest1243557-21cron19:55:001
1185530991614,1cyclictest1212642-21turbostat19:30:000
1185530991613,2cyclictest1336733-21unixbench_multi21:05:250
1185530991613,2cyclictest1256292-21cpuspeed_turbos20:05:130
1185530991612,3cyclictest171ktimers/019:45:130
118553599159,4cyclictest1554902-21sh00:05:012
118553599152,7cyclictest1262141-21munin-plugin-st20:10:002
1185535991512,2cyclictest1548652-21sendmail00:00:002
1185535991512,2cyclictest1380833-21cron21:45:002
1185535991512,2cyclictest0-21swapper/219:45:012
1185535991511,2cyclictest0-21swapper/221:30:002
1185535991511,2cyclictest0-21swapper/220:19:592
1185535991511,2cyclictest0-21swapper/200:10:002
118553299159,4cyclictest3614154-21systemd-journal22:00:001
118553299157,6cyclictest523-21gdbus23:05:001
1185532991512,2cyclictest1543061-21gltestperf23:55:151
1185532991512,2cyclictest1524557-21gltestperf23:40:191
1185532991512,2cyclictest1514330-21kthreadcore23:30:191
1185532991512,2cyclictest1350119-21fschecks_time21:20:141
1185532991512,2cyclictest1349566-21cron21:20:001
1185532991512,2cyclictest1230723-21unixbench_multi19:40:261
1185532991512,2cyclictest0-21swapper/122:15:001
1185530991512,2cyclictest1274242-21kworker/0:0+kacpid22:25:160
1185530991512,2cyclictest1224036-21needreboot19:35:210
1185530991512,2cyclictest1199810-21unixbench_singl19:15:240
1185530991512,2cyclictest0-21swapper/020:35:000
1185530991510,4cyclictest3614154-21systemd-journal23:50:000
11794802152,7sleep10-21swapper/119:05:141
1185537991411,2cyclictest1579980-21cpuspeed_turbos00:25:133
1185537991410,3cyclictest1206905-21fschecks_count19:25:153
118553599149,3cyclictest1311939-21sh20:50:002
118553599148,4cyclictest493-21dbus-daemon22:05:012
118553599148,4cyclictest1541772-21python323:50:242
118553599141,3cyclictest0-21swapper/223:29:552
1185535991411,2cyclictest1578653-21needreboot00:20:202
1185535991411,2cyclictest1567785-21fschecks_time00:15:162
1185535991411,2cyclictest1529408-21systemctl23:40:262
1185535991410,2cyclictest0-21swapper/219:30:002
1185535991410,2cyclictest0-21swapper/200:25:162
118553299148,5cyclictest523-21gdbus20:15:001
1185532991412,1cyclictest1340269-21kthreadcore21:10:201
1185532991412,1cyclictest1340269-21kthreadcore21:10:191
1185532991412,1cyclictest0-21swapper/121:00:011
1185532991411,2cyclictest1455915-21fschecks_count22:45:181
1185532991411,2cyclictest1443399-21cpuspeed_turbos22:35:141
1185532991411,2cyclictest1299892-21cpuspeed_turbos20:40:161
1185532991411,2cyclictest1256687-21sed20:05:161
1185532991411,2cyclictest0-21swapper/123:20:001
1185532991411,2cyclictest0-21swapper/122:35:001
1185532991410,3cyclictest1467384-21unixbench_multi22:50:241
1185532991410,3cyclictest1191191-21unixbench_multi19:10:221
1185532991410,3cyclictest0-21swapper/120:30:001
118553099149,4cyclictest1567254-21cron00:15:000
1185530991412,2cyclictest1498602-21kworker/0:2+kacpid00:29:530
1185530991412,1cyclictest1292753-21kworker/0:1+kacpid23:07:010
1185530991412,1cyclictest1292753-21kworker/0:1+kacpid22:52:330
1185530991412,1cyclictest1274242-21kworker/0:0+kacpid22:34:180
1185530991412,1cyclictest1191744-21latency_hist19:15:000
1185530991411,2cyclictest491-21avahi-daemon20:40:020
1185530991411,2cyclictest1381431-21gltestperf21:45:140
1185530991411,2cyclictest1280917-21/usr/sbin/munin20:25:230
1185530991411,2cyclictest1274242-21kworker/0:0+kacpid21:32:440
1185530991411,2cyclictest1212884-21kworker/0:1+kacpid20:03:340
1185530991411,2cyclictest0-21swapper/023:10:170
1185530991411,2cyclictest0-21swapper/021:30:010
1185530991411,2cyclictest0-21swapper/020:55:000
1185530991411,1cyclictest0-21swapper/021:55:010
1185530991410,3cyclictest1498602-21kworker/0:2+kacpi_notify00:23:250
1185530991410,3cyclictest1492347-21kworker/0:0+kacpi_notify23:15:210
118553799139,2cyclictest1331312-21/usr/sbin/munin21:05:153
118553799139,2cyclictest0-21swapper/300:05:153
118553799138,3cyclictest0-21swapper/321:05:013
118553799138,3cyclictest0-21swapper/321:05:013
1185537991310,2cyclictest1591961-21cron00:34:593
118553799130,12cyclictest1337453-21/usr/sbin/munin21:10:173
118553799130,12cyclictest1337453-21/usr/sbin/munin21:10:163
118553599139,2cyclictest3614154-21systemd-journal21:30:022
118553599139,2cyclictest0-21swapper/223:10:012
118553599138,4cyclictest3614154-21systemd-journal20:43:202
118553599135,6cyclictest1250248-21uniq20:00:172
1185535991311,1cyclictest0-21swapper/219:50:162
1185535991310,2cyclictest1412557-21irqcore22:10:182
1185535991310,2cyclictest1194491-21fschecks_time19:15:152
1185535991310,2cyclictest0-21swapper/223:15:012
1185535991310,2cyclictest0-21swapper/222:05:142
1185535991310,2cyclictest0-21swapper/220:50:242
1185535991310,2cyclictest0-21swapper/220:30:152
118553299139,3cyclictest1399793-21cpuspeed_turbos22:00:161
118553299139,2cyclictest0-21swapper/120:40:001
1185532991311,1cyclictest1450033-21irqcore22:40:181
1185532991310,2cyclictest1518029-21cpuspeed_turbos23:35:141
1185532991310,2cyclictest1480734-21fschecks_time23:05:141
1185532991310,2cyclictest1312284-21cpuspeed_turbos20:50:141
1185532991310,2cyclictest1231754-21gltestperf19:45:151
118553299130,12cyclictest0-21swapper/120:15:181
118553099139,3cyclictest1492347-21kworker/0:0+kacpi_notify23:30:170
118553099139,3cyclictest1455104-21kworker/0:2+kacpi_notify22:55:250
118553099139,3cyclictest1399011-21unixbench_singl21:55:250
118553099137,5cyclictest523-21gdbus23:55:010
118553099136,1cyclictest295798rtkit-daemon22:20:060
118553099133,2cyclictest295798rtkit-daemon19:40:200
1185530991311,1cyclictest493040-21kworker/0:3+kacpid19:20:130
1185530991311,1cyclictest1498602-21kworker/0:2+kacpid23:35:440
1185530991311,1cyclictest1498602-21kworker/0:2+kacpid00:30:180
1185530991311,1cyclictest1292753-21kworker/0:1+kacpid22:45:200
1185530991310,2cyclictest1346955-21kthreadcore21:15:180
1185530991310,2cyclictest1268720-21cpuspeed_turbos20:15:140
1185530991310,2cyclictest1212884-21kworker/0:1+kacpid19:56:430
118553799129,2cyclictest0-21swapper/300:39:593
118553799128,3cyclictest0-21swapper/300:20:223
118553799126,5cyclictest493-21dbus-daemon00:20:013
118553799126,4cyclictest1567104-21sshd00:13:093
118553599129,2cyclictest0-21swapper/223:35:002
118553599129,2cyclictest0-21swapper/200:30:272
118553599128,2cyclictest0-21swapper/223:00:182
118553599128,2cyclictest0-21swapper/222:30:162
118553599126,5cyclictest1429862-21ntpq22:20:212
118553299129,2cyclictest1530495-21cpuspeed_turbos23:45:141
118553299129,2cyclictest1487007-21gltestperf23:10:161
118553299129,2cyclictest1250554-21irqcore20:00:191
118553299129,2cyclictest0-21swapper/122:20:151
118553299128,3cyclictest1510709-21grep23:25:251
118553299128,2cyclictest0-21swapper/119:40:001
118553299127,4cyclictest2997-21gdbus22:07:021
118553299125,6cyclictest1306205-21uniq20:45:141
118553299120,3cyclictest0-21swapper/121:05:051
118553299120,1cyclictest0-21swapper/121:35:201
118553099127,4cyclictest3614154-21systemd-journal21:41:200
118553099126,4cyclictest0-21swapper/020:10:270
118553099125,6cyclictest1212997-21cut19:30:170
1185530991210,1cyclictest1274242-21kworker/0:0+kacpid22:05:200
1185530991210,1cyclictest1274242-21kworker/0:0+kacpid22:00:200
118553599118,2cyclictest1381416-21grep21:45:152
118553599117,2cyclictest0-21swapper/219:45:152
118553599116,4cyclictest3614154-21systemd-journal22:16:592
118553599116,4cyclictest1518543-21cut23:35:172
118553599116,4cyclictest1279838-21awk20:20:212
118553599116,3cyclictest1531360-21irqrtprio23:45:182
118553599116,3cyclictest0-21swapper/221:25:012
118553299119,1cyclictest1503985-21missed_timers23:20:201
118553299119,1cyclictest0-21swapper/120:20:121
118553299118,2cyclictest1375019-21cpuspeed_turbos21:40:131
118553299118,2cyclictest0-21swapper/121:45:131
118553299116,4cyclictest1538357-21cut23:50:191
118553299110,10cyclictest0-21swapper/122:55:571
118553099118,2cyclictest1292753-21kworker/0:1+kacpid22:20:000
118553099118,1cyclictest0-21swapper/021:10:230
118553099118,1cyclictest0-21swapper/021:10:220
118553099117,3cyclictest3614154-21systemd-journal23:25:000
118553099115,5cyclictest523-21gdbus00:05:010
118553099111,1cyclictest62850irq/126-eno1:rx-023:03:360
11850532118,2sleep20-21swapper/219:05:592
11850232113,1sleep30-21swapper/319:05:363
118553599107,2cyclictest0-21swapper/220:10:162
118553599106,2cyclictest1566876-21grep00:10:262
118553599104,4cyclictest1393742-21/usr/sbin/munin21:55:152
118553599101,7cyclictest3614154-21systemd-journal22:30:002
118553299109,1cyclictest0-21swapper/121:00:331
118553299109,1cyclictest0-21swapper/121:00:321
118553099101,8cyclictest1505232-21/usr/sbin/munin23:25:170
29579880,7rtkit-daemon2956-21rtkit-daemon19:05:190
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional