You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 05:59

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot8s.osadl.org (updated Fri May 09, 2025 00:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
132034899754750,3cyclictest1428499-21kworker/3:1+events21:18:273
132034899753749,3cyclictest1510358-21kworker/3:0+events21:55:413
132034899752748,3cyclictest1334459-21kworker/3:1+events19:22:463
132034299752748,3cyclictest1571791-21kworker/1:1+events22:30:211
132034899749745,3cyclictest1428499-21kworker/3:1+events20:57:523
132034899749745,3cyclictest1428499-21kworker/3:1+events20:57:523
132034899747743,3cyclictest1510358-21kworker/3:0+events23:46:033
132034899747742,4cyclictest1372259-21kworker/3:1+events19:54:053
132034899746742,3cyclictest1697040-21kworker/3:2+events00:12:213
132034899745741,3cyclictest1510358-21kworker/3:0+events23:19:063
132034899745741,3cyclictest1397388-21kworker/3:2+events20:16:163
132034899745741,3cyclictest1359762-21kworker/3:2+events19:43:123
132034899745741,3cyclictest1334459-21kworker/3:1+events19:37:063
132034299745741,3cyclictest1571791-21kworker/1:1+events22:56:481
132034899744740,3cyclictest1510358-21kworker/3:0+events23:14:553
132034899744740,3cyclictest1428499-21kworker/3:1+events21:02:413
132034299744740,3cyclictest1615891-21kworker/1:0+events00:05:211
132034299744736,4cyclictest1571791-21kworker/1:1+events22:49:201
132034899743739,3cyclictest1697040-21kworker/3:2+events00:21:033
132034899743739,3cyclictest1671738-21kworker/3:2+events23:51:143
132034899743739,3cyclictest1510358-21kworker/3:0+events22:12:523
132034899743739,3cyclictest1428499-21kworker/3:1+events20:38:033
132034899743739,3cyclictest1372259-21kworker/3:1+events19:57:533
132034899743739,3cyclictest1334459-21kworker/3:1+events19:30:523
132034899743738,4cyclictest1678220-21kworker/3:1+events23:58:233
132034899743738,4cyclictest1466250-21kworker/3:0+events21:09:063
132034899742738,3cyclictest1510358-21kworker/3:0+events23:44:273
132034899742738,3cyclictest1510358-21kworker/3:0+events23:26:223
132034899742738,3cyclictest1510358-21kworker/3:0+events22:09:433
132034899742737,4cyclictest1697040-21kworker/3:2+events00:15:303
132034299742738,3cyclictest1615891-21kworker/1:0+events00:00:541
132034899741737,3cyclictest1428499-21kworker/3:1+events20:42:183
132034899740736,3cyclictest3413334-21kworker/3:2+events19:13:033
132034899740735,4cyclictest1428499-21kworker/3:1+events21:43:253
132034899740735,4cyclictest1428499-21kworker/3:1+events20:45:213
132034299740736,3cyclictest1571791-21kworker/1:1+events23:05:431
132034299740736,3cyclictest1571791-21kworker/1:1+events22:52:441
132034299740736,3cyclictest1428889-21kworker/1:2+events22:25:211
132034899739735,3cyclictest1697040-21kworker/3:2+events00:28:203
132034899739735,3cyclictest1510358-21kworker/3:0+events23:36:483
132034899739735,3cyclictest1428499-21kworker/3:1+events21:47:533
132034299739735,3cyclictest1571791-21kworker/1:1+events22:36:151
132034299739734,4cyclictest1609431-21turbostat23:04:551
132034899738734,3cyclictest1510358-21kworker/3:0+events23:22:423
132034899738734,3cyclictest1428499-21kworker/3:1+events20:54:103
132034899737733,3cyclictest3413334-21kworker/3:2+events19:17:073
132034899736733,2cyclictest1397388-21kworker/3:2+events20:27:193
132034899736732,3cyclictest1510358-21kworker/3:0+events21:51:373
132034899736732,3cyclictest1397388-21kworker/3:2+events20:12:423
132034299736731,4cyclictest1571791-21kworker/1:1+events22:40:431
132034899734730,3cyclictest1510358-21kworker/3:0+events23:34:133
132034899733729,3cyclictest1428499-21kworker/3:1+events21:12:013
132034899733729,3cyclictest1409894-21turbostat20:24:553
132034899733729,3cyclictest1372259-21kworker/3:1+events20:09:293
132034899732728,3cyclictest1510358-21kworker/3:0+events23:04:413
132034899731726,4cyclictest1397388-21kworker/3:2+events20:34:513
132034299730726,3cyclictest1428889-21kworker/1:2+events22:20:431
132034899729726,2cyclictest1428499-21kworker/3:1+events21:22:533
132033899726721,4cyclictest1485206-21turbostat21:24:560
132034899725721,3cyclictest1372259-21kworker/3:1+events20:01:293
132033899721718,2cyclictest1491142-21kworker/0:2+events21:26:040
132033899715711,3cyclictest1491142-21kworker/0:2+events21:34:290
132034899704701,2cyclictest1690814-21turbostat00:09:553
132034899699695,3cyclictest1334459-21kworker/3:1+events19:28:423
132034899696690,5cyclictest1510358-21kworker/3:0+events22:17:203
132034299694687,6cyclictest1553485-21turbostat22:19:551
132034899691688,2cyclictest1697040-21kworker/3:2+events00:30:233
132034899685683,1cyclictest1510358-21kworker/3:0+events22:03:133
132033899682676,4cyclictest1491142-21kworker/0:2+events21:36:490
132034299679677,1cyclictest1678321-21turbostat23:59:551
132034899678674,3cyclictest1615651-21turbostat23:09:553
132034899664660,3cyclictest1359762-21kworker/3:2+events19:45:213
132034899643638,4cyclictest1697040-21kworker/3:2+events00:35:213
132034899623620,2cyclictest1603197-21turbostat22:59:553
132034899602598,3cyclictest1503854-21turbostat21:39:563
17079302240,1sleep20-21swapper/200:15:212
1320346992320,2cyclictest0-21swapper/200:00:002
1320346992319,3cyclictest1690831-21systemd-run00:05:012
1320346992219,2cyclictest1603198-21munin-plugin-st22:55:002
1320346992017,2cyclictest1672015-21latency_hist23:50:012
1320346992017,2cyclictest1535653-21irqcore22:00:182
1320346991915,2cyclictest0-21swapper/220:10:002
1320338991916,2cyclictest1620905-21needreboot23:05:200
15224802180,8sleep10-21swapper/121:50:011
1320346991815,2cyclictest502-21systemd-logind20:40:012
1320346991814,2cyclictest0-21swapper/221:20:002
132034299183,2cyclictest191rcu_preempt23:15:221
1320342991815,2cyclictest1485947-21sed21:20:181
1320338991814,2cyclictest0-21swapper/023:20:000
1320348991714,2cyclictest1589962-21ntp_kernel_pll_22:40:213
1320348991711,4cyclictest1565699-21sshd22:22:183
1320346991714,2cyclictest523-21gdbus00:10:002
1320346991714,2cyclictest1428983-21latency_hist20:35:012
1320346991714,2cyclictest0-21swapper/222:10:002
1320342991714,2cyclictest1523127-21gltestperf21:50:181
1320342991714,2cyclictest1460469-21cpuspeed_turbos21:00:141
1320342991714,2cyclictest1357037-21kthreadcore19:35:181
132034299170,16cyclictest0-21swapper/120:55:001
132033899174,11cyclictest3614154-21systemd-journal22:39:590
1320338991714,2cyclictest502-21systemd-logind23:31:580
1320338991714,2cyclictest502-21systemd-logind20:11:590
1320338991713,3cyclictest491-21avahi-daemon22:55:000
1320338991712,3cyclictest21750irq/159-ahci[0000:00:17.0]23:15:000
1320346991613,2cyclictest1510355-21apt-get21:40:122
132034299165,2cyclictest191rcu_preempt20:40:141
1320342991614,1cyclictest509-21rs:main1
1320342991613,2cyclictest1646984-21turbostat.cron23:30:011
1320342991613,2cyclictest1384425-21unixbench_singl19:55:241
1320342991610,4cyclictest1365980-21sshd19:42:121
132034299160,4cyclictest17251001kthreadcore00:30:191
132033899168,6cyclictest3614154-21systemd-journal19:25:000
132033899163,2cyclictest1313878-21kworker/0:1+kacpid19:10:140
1320338991613,2cyclictest386-21systemd-udevd22:13:580
1320338991613,2cyclictest386-21systemd-udevd00:13:580
1320338991612,3cyclictest1559699-21idleruntime-cro22:20:000
1320338991612,3cyclictest1546329-21needreboot22:05:210
1320338991612,3cyclictest1479351-21cpuspeed_turbos21:15:120
1320338991612,3cyclictest1378231-21unixbench_singl19:50:220
1320338991610,4cyclictest1334827-21sh19:20:010
13143012162,8sleep00-21swapper/019:05:120
132034699159,4cyclictest1609407-21systemctl23:00:002
1320346991512,2cyclictest1710668-21tr00:20:162
1320346991512,2cyclictest1503467-21unixbench_singl21:30:252
1320346991512,2cyclictest1422136-21unixbench_singl20:25:242
1320346991512,2cyclictest0-21swapper/220:25:012
1320346991511,2cyclictest0-21swapper/219:25:002
1320346991510,4cyclictest3614154-21systemd-journal21:56:172
132034299157,6cyclictest523-21gdbus19:35:011
1320342991513,1cyclictest0-21swapper/121:45:011
1320342991512,2cyclictest1503854-21turbostat.cron21:35:001
1320342991512,2cyclictest0-21swapper/100:39:591
1320342991510,3cyclictest1659992-21sort23:40:161
132034299150,14cyclictest1325741-21cut19:10:211
132033899153,3cyclictest1594770-21tr22:45:190
1320338991512,2cyclictest502-21systemd-logind20:43:590
1320338991512,2cyclictest386-21systemd-udevd21:41:590
1320338991512,2cyclictest1525767-21kthreadcore21:50:220
1320338991512,2cyclictest0-21swapper/019:30:010
132034899149,4cyclictest1-21systemd22:55:003
1320348991411,2cyclictest0-21swapper/321:30:003
132034699148,5cyclictest1579956-21sshd22:35:172
132034699148,4cyclictest1372756-21sshd19:50:122
132034699146,2cyclictest1403700-21munin-run20:15:002
1320346991411,2cyclictest1584999-21cpuspeed_turbos22:40:142
1320346991411,2cyclictest1547784-21fschecks_count22:10:142
1320346991411,2cyclictest1408899-21needreboot20:15:182
1320346991411,2cyclictest1390640-21unixbench_singl20:00:222
1320346991411,2cyclictest0-21swapper/222:15:172
1320346991410,3cyclictest1660056-21gltestperf23:40:152
1320346991410,2cyclictest13788842sleep219:55:122
132034299144,2cyclictest191rcu_preempt00:25:131
132034299143,2cyclictest191rcu_preempt23:50:191
132034299143,2cyclictest191rcu_preempt21:20:001
1320342991412,1cyclictest1478977-21latency_hist21:15:011
1320342991411,2cyclictest506-21in:imuxsock23:15:011
1320342991411,2cyclictest1710678-21cut00:20:161
1320342991411,2cyclictest1423780-21tr20:30:161
1320342991411,2cyclictest1391646-21gltestperf20:05:161
1320342991411,2cyclictest1388116-21kthreadcore20:00:161
1320342991411,2cyclictest1374141-21kthreadcore19:50:161
1320342991410,3cyclictest3614154-21systemd-journal20:28:111
1320342991410,3cyclictest311ktimers/120:55:141
1320342991410,3cyclictest311ktimers/120:55:131
132033899149,4cyclictest1568551-21sshd22:25:180
132033899148,4cyclictest493-21dbus-daemon00:00:000
132033899146,7cyclictest1672521-21cut23:50:120
132033899141,4cyclictest1429260-21/usr/sbin/munin20:35:220
1320338991411,2cyclictest1659914-21fschecks_count23:40:150
1320338991411,2cyclictest1354967-21irqrtprio19:35:170
1320338991410,3cyclictest1728437-21cpuspeed_turbos00:35:130
1320338991410,3cyclictest1390073-21needreboot20:00:190
1320338991410,2cyclictest1529417-21irqcore21:55:130
1320338991410,2cyclictest0-21swapper/020:10:000
132034899139,3cyclictest491ktimers/322:30:123
1320348991311,1cyclictest0-21swapper/322:39:593
1320348991310,2cyclictest1591384-21gltestperf22:45:153
132034699139,3cyclictest1727248-21grep00:30:232
132034699137,4cyclictest3614154-21systemd-journal23:13:182
132034699136,5cyclictest1360285-21sshd19:40:122
1320346991311,1cyclictest0-21swapper/223:20:202
1320346991310,2cyclictest499-21polkitd00:39:592
1320346991310,2cyclictest1341668-21cpuspeed_turbos19:25:152
1320346991310,2cyclictest0-21swapper/222:30:162
1320346991310,2cyclictest0-21swapper/200:30:002
132034299139,3cyclictest493-21dbus-daemon00:20:001
132034299139,3cyclictest1434213-21wc20:35:171
132034299138,3cyclictest1653839-21gltestperf23:35:161
132034299133,2cyclictest295798rtkit-daemon23:46:031
132034299133,2cyclictest191rcu_preempt20:25:011
132034299132,3cyclictest2956-21rtkit-daemon19:45:101
1320342991310,2cyclictest1341495-21apt-key19:25:011
1320342991310,2cyclictest0-21swapper/120:15:151
132033899139,3cyclictest1720825-21needreboot00:25:200
132033899139,3cyclictest1703137-21kworker/0:0+kacpi_notify00:30:190
132033899139,3cyclictest1641311-21gltestperf23:25:180
132033899138,4cyclictest1371958-21tail19:45:250
132033899138,4cyclictest0-21swapper/020:15:110
132033899138,3cyclictest1535532-21gltestperf22:00:170
132033899137,4cyclictest3614154-21systemd-journal00:10:010
132033899136,5cyclictest1590834-21munin-plugin-st22:45:000
132033899131,3cyclictest1384831-21munin-run20:00:010
1320338991312,1cyclictest1621769-21kworker/0:3+kacpid23:47:520
1320338991312,1cyclictest1403453-21kworker/0:3+kacpid21:14:070
1320338991311,1cyclictest1703137-21kworker/0:0+kacpid00:17:340
1320338991311,1cyclictest1621769-21kworker/0:3+kacpid23:20:040
1320338991311,1cyclictest1515984-21kworker/0:1+kacpid21:45:030
1320338991311,1cyclictest1491142-21kworker/0:2+kacpid23:00:090
1320338991311,1cyclictest1491142-21kworker/0:2+kacpid22:56:180
1320338991311,1cyclictest1403453-21kworker/0:3+kacpid21:06:300
1320338991311,1cyclictest1403453-21kworker/0:3+kacpid20:21:400
1320338991311,1cyclictest1340943-21kworker/0:0+kacpid19:30:340
1320338991311,1cyclictest1326326-21kworker/0:2+kacpid19:43:210
1320338991310,2cyclictest1690371-21unixbench_multi00:00:290
1320338991310,2cyclictest1578013-21unixbench_singl22:30:240
1320338991310,2cyclictest1560441-21grep22:20:150
1320338991310,2cyclictest1448872-21irqrtprio20:50:140
1320338991310,2cyclictest1423027-21fschecks_count20:30:140
132034899129,2cyclictest1685059-21fschecks_time00:00:183
132034899121,2cyclictest191rcu_preempt21:30:143
132034699129,2cyclictest509-21rs:main2
132034699129,2cyclictest502-21systemd-logind20:45:002
132034699129,2cyclictest1647359-21cpuspeed_turbos23:30:152
132034699129,2cyclictest0-21swapper/223:15:122
132034699129,2cyclictest0-21swapper/222:24:592
132034699127,4cyclictest3614154-21systemd-journal21:12:122
132034699125,5cyclictest3614154-21systemd-journal19:15:012
132034299129,2cyclictest311ktimers/120:10:021
132034299129,2cyclictest1468010-21kthreadcore21:05:181
132034299129,2cyclictest0-21swapper/122:05:241
132034299129,1cyclictest0-21swapper/122:10:161
132034299128,3cyclictest311ktimers/123:20:171
132034299128,3cyclictest311ktimers/119:20:091
132034299120,3cyclictest0-21swapper/121:25:301
132033899128,3cyclictest1713836-21kthreadcore00:20:190
1320338991211,1cyclictest1621769-21kworker/0:3+kacpid23:36:340
1320338991211,1cyclictest1447414-21kworker/0:1+kacpid20:57:430
1320338991211,1cyclictest1447414-21kworker/0:1+kacpid20:57:420
1320338991211,1cyclictest1403453-21kworker/0:3+kacpid21:00:090
1320338991211,1cyclictest1403453-21kworker/0:3+kacpid20:45:450
1320338991211,1cyclictest1403453-21kworker/0:3+kacpid20:26:550
132034899118,2cyclictest0-21swapper/322:25:253
132034699119,1cyclictest0-21swapper/221:25:012
132034699118,2cyclictest1618923-21kthreadcore23:05:192
132034699118,2cyclictest1491885-21cpuspeed_turbos21:25:192
132034699118,2cyclictest0-21swapper/221:04:592
132034699118,2cyclictest0-21swapper/220:45:252
132034699117,3cyclictest1522485-21cron21:50:012
132034699117,3cyclictest0-21swapper/223:25:222
132034699117,2cyclictest0-21swapper/219:35:242
132034699116,4cyclictest1528687-21cron21:55:012
132034699110,10cyclictest0-21swapper/220:55:252
132034699110,10cyclictest0-21swapper/220:55:242
132034299117,3cyclictest1532263-21tr21:55:151
132034299110,11cyclictest0-21swapper/123:30:321
132034299110,11cyclictest0-21swapper/120:46:421
13198762117,2sleep30-21swapper/319:06:063
13198742117,2sleep10-21swapper/119:06:051
132034699109,1cyclictest0-21swapper/220:50:222
132034699107,2cyclictest1653975-21irqcore23:35:172
132034699107,2cyclictest1347889-21cpuspeed_turbos19:30:142
132034699107,2cyclictest0-21swapper/219:15:012
132034699105,4cyclictest1698854-21tr00:10:192
132034699105,4cyclictest1614393-21tr23:00:182
132034699105,4cyclictest1371486-21cut19:45:202
132034699104,2cyclictest1469560-21kthreadcore21:05:192
132034699102,6cyclictest1596603-21unixbench_singl22:45:252
132034299106,3cyclictest1506648-21kthreadcore21:35:181
132034299100,2cyclictest191rcu_preempt00:10:191
132034299100,10cyclictest0-21swapper/122:00:131
13198752106,2sleep20-21swapper/219:06:052
13203469996,2cyclictest0-21swapper/221:35:202
13203469985,2cyclictest1672195-21apt-key23:50:012
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional