You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-06 - 00:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails
  Congatec
Click here to display the system's profile data.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rackeslot8s.osadl.org (updated Sat Jul 05, 2025 12:43:50)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
144822199752748,3cyclictest1692190-21kworker/3:2+events10:26:373
144822199752748,3cyclictest1692190-21kworker/3:2+events10:26:363
144822199750746,3cyclictest1704993-21kworker/3:1+events10:40:453
144822199749744,4cyclictest1679497-21kworker/3:1+events10:20:213
144822199748744,3cyclictest1518157-21kworker/3:1+events08:57:243
144822199748744,3cyclictest1518157-21kworker/3:1+events08:35:323
144822199747743,3cyclictest1717460-21kworker/3:2+events10:48:463
144822199747741,5cyclictest1828794-21kworker/3:0+events12:27:423
144822199747741,5cyclictest1828794-21kworker/3:0+events12:27:423
144822199746742,3cyclictest1642418-21kworker/3:2+events10:14:313
144822199746742,3cyclictest1410088-21kworker/3:3+events07:18:133
144822199746741,4cyclictest1410088-21kworker/3:3+events07:25:543
144822199745741,3cyclictest1766845-21kworker/3:1+events12:12:303
144822199744740,3cyclictest1518157-21kworker/3:1+events08:41:393
144822199744739,4cyclictest1642418-21kworker/3:2+events10:03:043
144822199743739,3cyclictest1828794-21kworker/3:0+events12:23:433
144822199743739,3cyclictest1766845-21kworker/3:1+events12:18:433
144822199743739,3cyclictest1766845-21kworker/3:1+events12:18:423
144822199743739,3cyclictest1717460-21kworker/3:2+events11:03:033
144822199743739,3cyclictest1717460-21kworker/3:2+events10:52:403
144822199743739,3cyclictest1518157-21kworker/3:1+events08:21:173
144822199743738,4cyclictest1518157-21kworker/3:1+events08:29:433
144822199743738,4cyclictest1410088-21kworker/3:3+events07:24:063
144822199741737,3cyclictest1766845-21kworker/3:1+events11:59:313
144822199741737,3cyclictest1518157-21kworker/3:1+events09:21:373
144822199741737,3cyclictest1481053-21kworker/3:1+events07:38:373
144822199741736,4cyclictest1692190-21kworker/3:2+events10:31:493
144822199740736,3cyclictest1766845-21kworker/3:1+events12:07:333
144822199740736,3cyclictest1679497-21kworker/3:1+events10:17:503
144822199739735,3cyclictest1518157-21kworker/3:1+events08:16:593
144822199739735,3cyclictest1518157-21kworker/3:1+events08:14:003
144822199739732,5cyclictest1493685-21kworker/3:2+events08:01:043
144822199738732,5cyclictest1766845-21kworker/3:1+events12:00:203
144822199737733,3cyclictest1766845-21kworker/3:1+events11:53:043
144822199737733,3cyclictest1717460-21kworker/3:2+events11:16:283
144822199737733,3cyclictest1642418-21kworker/3:2+events10:08:433
144822199737733,3cyclictest1518157-21kworker/3:1+events08:47:443
144822199737733,3cyclictest1481053-21kworker/3:1+events07:52:213
144822199737732,4cyclictest1717460-21kworker/3:2+events11:13:463
144822199736732,3cyclictest1766845-21kworker/3:1+events11:29:293
144822199736732,3cyclictest1704993-21kworker/3:1+events10:39:133
144821599736731,4cyclictest2511645-21kworker/1:1+events11:32:591
144822199734723,6cyclictest1410088-21kworker/3:3+events07:10:443
144821899734730,3cyclictest1530872-21kworker/2:1+events09:56:592
144822199733729,3cyclictest1518157-21kworker/3:1+events08:09:293
144822199729725,3cyclictest1410088-21kworker/3:3+events07:33:373
144822199728724,3cyclictest1518157-21kworker/3:1+events09:04:543
144822199727724,2cyclictest1518157-21kworker/3:1+events08:51:503
144822199727723,3cyclictest1518157-21kworker/3:1+events09:18:463
144822199723718,4cyclictest1828794-21kworker/3:0+events12:35:523
144821899718709,7cyclictest1530872-21kworker/2:1+events09:50:222
144821599714710,3cyclictest2511645-21kworker/1:1+events11:37:201
144821899709701,7cyclictest1530872-21kworker/2:1+events11:45:212
144822199702694,7cyclictest1828794-21kworker/3:0+events12:30:203
144821899701696,4cyclictest1487563-21turbostat07:44:552
144822199699694,4cyclictest1518157-21kworker/3:1+events08:34:053
144821899698693,4cyclictest1779428-21turbostat11:39:552
144821899697693,3cyclictest1530872-21kworker/2:1+events11:44:092
144822199693691,1cyclictest1717460-21kworker/3:2+events10:57:313
144822199691687,3cyclictest1481053-21kworker/3:1+events07:40:203
144822199686682,3cyclictest1493731-21turbostat07:49:543
144821899686681,4cyclictest1530872-21kworker/2:1+events09:36:572
144822199684678,5cyclictest1717460-21kworker/3:2+events11:05:223
144821899681677,3cyclictest1530872-21kworker/2:1+events09:48:162
144821899681675,5cyclictest2413769-21kworker/2:2+events07:45:232
144822199678674,3cyclictest1518157-21kworker/3:1+events09:05:223
144821899675671,3cyclictest1530872-21kworker/2:1+events09:40:422
144822199667663,3cyclictest1518157-21kworker/3:1+events09:11:493
144821899667664,2cyclictest1530872-21kworker/2:1+events09:30:222
144821899665660,4cyclictest1617710-21turbostat09:29:562
144822199661656,3cyclictest1760866-21turbostat11:24:563
144822199659655,3cyclictest1518157-21kworker/3:1+events09:25:223
144822199655651,3cyclictest1655041-21turbostat09:59:553
144822199650647,2cyclictest1493685-21kworker/3:2+events07:55:223
144822199646643,2cyclictest1791822-21turbostat11:49:553
144821599643637,5cyclictest1767060-21turbostat11:29:561
1448218993330,2cyclictest1679811-21turbostat10:15:002
1448218992522,2cyclictest1462589-21sendmail07:20:002
295798230,8rtkit-daemon1698665-21runrttasks10:26:062
295798230,8rtkit-daemon1698665-21runrttasks10:26:052
1448218992219,2cyclictest1549369-21sh08:29:592
1448213992118,2cyclictest1785635-21grep11:40:000
1448213992118,2cyclictest1742383-21latency_hist11:05:010
1448213992117,3cyclictest506-21in:imuxsock08:05:000
18107152200,1sleep10-21swapper/112:00:131
1448215992017,2cyclictest1798747-21irqcore11:50:181
1448218991914,3cyclictest1562602-21gltestperf08:40:182
1448215991916,2cyclictest491-21avahi-daemon12:25:001
1448213991916,2cyclictest1661240-21cron10:00:000
1448221991815,2cyclictest0-21swapper/311:35:003
1448218991815,2cyclictest523-21gdbus07:35:002
1448218991815,2cyclictest0-21swapper/211:00:002
1448218991815,2cyclictest0-21swapper/207:15:012
1448215991814,3cyclictest1543531-21cpuspeed_turbos08:25:151
1448213991815,2cyclictest312-21jbd2/sda2-810:25:000
1448213991815,2cyclictest1847947-21cron12:30:000
1448213991815,2cyclictest1847947-21cron12:30:000
1448213991815,2cyclictest1474983-21turbostat07:30:000
1448213991813,4cyclictest1586742-21cron09:00:000
14479592188,2sleep30-21swapper/307:08:423
1448221991714,2cyclictest1790889-21needreboot11:40:203
1448218991715,1cyclictest1835290-21sendmail12:20:002
1448218991715,1cyclictest1835290-21sendmail12:20:002
1448218991714,2cyclictest493-21dbus-daemon08:55:012
1448218991714,2cyclictest0-21swapper/207:40:002
1448218991712,3cyclictest1549974-21gltestperf08:30:162
1448215991713,3cyclictest1617699-21cron09:25:001
1448215991712,3cyclictest0-21swapper/111:20:001
1448213991714,2cyclictest1524689-21cut08:10:010
1448213991714,2cyclictest1493716-21cron07:45:000
1448213991714,2cyclictest0-21swapper/012:40:000
144821899167,6cyclictest3614154-21systemd-journal10:20:002
1448218991613,2cyclictest1667049-21unixbench_singl10:00:262
1448218991612,3cyclictest1524258-21unixbench_singl08:05:252
1448215991613,2cyclictest502-21systemd-logind12:30:001
1448215991613,2cyclictest502-21systemd-logind12:29:591
1448215991613,2cyclictest0-21swapper/109:50:001
1448215991611,3cyclictest1761514-21gltestperf11:20:151
144821399167,1cyclictest1530855-21turbostat08:15:000
1448213991613,2cyclictest1810457-21turbostat12:00:000
1448213991613,2cyclictest0-21swapper/009:20:010
1448213991612,3cyclictest1724104-21cpuspeed_turbos10:50:140
1448213991610,5cyclictest1555539-21idleruntime-cro08:35:000
144822199159,4cyclictest3614154-21systemd-journal09:49:593
1448221991512,2cyclictest1650106-21tr09:50:163
1448218991513,1cyclictest1824837-21kthreadcore12:10:172
1448218991511,3cyclictest1448250-21ls07:10:012
1448218991511,2cyclictest0-21swapper/211:30:182
1448218991511,2cyclictest0-21swapper/211:20:002
1448218991510,4cyclictest1586777-21sendmail09:00:012
144821599158,5cyclictest523-21gdbus12:15:001
144821599158,5cyclictest523-21gdbus07:24:591
1448215991513,1cyclictest0-21swapper/112:20:011
1448215991513,1cyclictest0-21swapper/112:20:011
1448215991512,2cyclictest1754284-21unixbench_singl11:10:271
1448215991512,2cyclictest0-21swapper/112:10:001
144821399154,2cyclictest295798rtkit-daemon10:05:140
1448213991513,1cyclictest0-21swapper/011:00:000
1448213991512,2cyclictest509-21rs:main0
15706682140,2sleep0211rcuc/008:45:180
1448221991411,2cyclictest1629638-21unixbench_singl09:30:263
144821899148,4cyclictest493-21dbus-daemon07:25:252
144821899148,4cyclictest1536243-21systemctl08:15:232
1448218991412,1cyclictest0-21swapper/210:40:182
1448218991411,2cyclictest1854523-21cpuspeed_turbos12:35:152
1448218991411,2cyclictest1705691-21fschecks_time10:35:152
1448218991411,2cyclictest1601531-21kthreadcore09:10:182
1448218991410,3cyclictest401ktimers/211:25:132
1448218991410,2cyclictest1592567-21grep09:00:232
1448218991410,2cyclictest1530404-21tail08:10:252
1448218991410,2cyclictest1468348-21tail07:20:272
1448218991410,2cyclictest0-21swapper/210:50:182
144821899140,5cyclictest1816669-21latency_hist12:05:002
144821599149,4cyclictest1748483-21cron11:10:011
144821599148,5cyclictest3614154-21systemd-journal07:25:211
144821599146,7cyclictest1711749-21cut10:40:141
1448215991411,2cyclictest1649275-21cpuspeed_turbos09:50:131
1448215991410,3cyclictest6950irq/9-acpi08:20:141
1448215991410,3cyclictest1729933-21cron10:55:001
1448215991410,3cyclictest1562548-21gltestperf08:40:181
1448215991410,3cyclictest1555520-21cron08:35:001
1448215991410,2cyclictest1668021-21tail10:05:151
144821599140,1cyclictest0-21swapper/110:00:131
144821399149,3cyclictest3614154-21systemd-journal09:09:220
1448213991412,1cyclictest1605042-21kworker/0:1+kacpid09:47:130
1448213991411,2cyclictest491-21avahi-daemon11:10:010
1448213991411,2cyclictest1810819-21cpuspeed_turbos12:00:140
1448213991411,2cyclictest1785528-21kworker/0:0+kacpid12:15:550
1448213991411,2cyclictest1785528-21kworker/0:0+kacpid12:15:550
1448213991411,2cyclictest1779428-21turbostat11:34:590
1448213991411,2cyclictest171ktimers/010:46:580
1448213991411,2cyclictest171ktimers/009:28:580
1448213991411,2cyclictest1512288-21cron08:00:000
1448213991411,2cyclictest1445818-21kworker/0:1+kacpid07:10:190
1448213991410,3cyclictest1456724-21cpuspeed_turbos07:15:150
1448213991410,2cyclictest1767662-21tail11:25:160
144822199139,3cyclictest0-21swapper/309:40:163
144821899139,3cyclictest401ktimers/210:05:132
144821899139,2cyclictest0-21swapper/212:05:192
144821899137,5cyclictest493-21dbus-daemon08:00:002
144821899137,4cyclictest3614154-21systemd-journal12:21:152
144821899137,4cyclictest1599255-21sed09:10:002
144821899131,9cyclictest1810433-21cron11:59:592
1448218991310,2cyclictest1736493-21cpuspeed_turbos11:00:142
1448218991310,2cyclictest0-21swapper/208:45:152
1448218991310,2cyclictest0-21swapper/208:00:142
144821899130,4cyclictest15012571irqrtprio07:50:192
144821899130,1cyclictest18490601irqcore12:30:202
144821899130,1cyclictest1745569-21kthreadcore11:05:222
144821599139,3cyclictest1718272-21irqcore10:45:161
144821599138,3cyclictest1531121-21apt-get08:15:121
1448215991310,2cyclictest1854702-21gltestperf12:35:171
1448215991310,2cyclictest1737873-21kthreadcore11:00:201
1448215991310,2cyclictest1605709-21cpuspeed_turbos09:15:141
1448215991310,2cyclictest1593354-21cpuspeed_turbos09:05:161
1448215991310,2cyclictest0-21swapper/107:36:531
144821399137,4cyclictest1828191-21ntpq12:10:200
144821399132,9cyclictest1711647-21apt-get10:40:140
144821399131,2cyclictest1562257-21apt-get08:40:160
1448213991312,1cyclictest1605042-21kworker/0:1+kacpid09:50:120
1448213991311,1cyclictest1574219-21kworker/0:0+kacpid08:50:280
1448213991310,2cyclictest1798735-21sed11:50:180
1448213991310,2cyclictest1786229-21gltestperf11:40:160
1448213991310,2cyclictest171ktimers/012:23:580
1448213991310,2cyclictest171ktimers/011:11:580
1448213991310,2cyclictest171ktimers/007:33:590
1448213991310,2cyclictest1494317-21gltestperf07:45:160
1448213991310,2cyclictest1453983-21kworker/0:2+kacpid07:20:060
144821399130,12cyclictest0-21swapper/010:15:000
144822199127,4cyclictest1636219-21cat09:40:003
144821899129,2cyclictest1842219-21fschecks_count12:25:142
144821899129,2cyclictest1842219-21fschecks_count12:25:132
144821899129,2cyclictest1686625-21cpuspeed_turbos10:20:142
144821899129,1cyclictest0-21swapper/208:35:202
144821899120,3cyclictest1605994-21sed09:15:162
144821599129,2cyclictest1853673-21unixbench_multi12:30:281
144821599129,2cyclictest1705898-21irqcore10:35:171
144821599129,2cyclictest1655394-21cpuspeed_turbos09:55:141
144821599129,2cyclictest1500670-21sed07:50:181
144821599129,2cyclictest0-21swapper/107:40:171
144821599128,3cyclictest506-21in:imuxsock08:50:011
144821599128,3cyclictest491-21avahi-daemon07:15:011
144821599128,3cyclictest1624214-21cpuspeed_turbos09:30:151
144821599127,4cyclictest523-21gdbus09:25:251
144821599127,4cyclictest1507190-21sed07:55:181
144821599126,4cyclictest1788581-21tr11:40:191
144821599126,4cyclictest1592912-21sshd09:04:211
1448215991210,1cyclictest0-21swapper/110:55:171
144821599120,1cyclictest1461648-21strings07:15:211
144821599120,11cyclictest1481118-21seq07:34:401
144821599120,11cyclictest0-21swapper/111:45:231
144821599120,11cyclictest0-21swapper/110:33:411
144821599120,11cyclictest0-21swapper/108:12:221
144821599120,11cyclictest0-21swapper/108:00:211
144821399129,2cyclictest1630643-21gltestperf09:35:150
144821399129,2cyclictest1599537-21cpuspeed_turbos09:10:140
144821399129,2cyclictest1592980-21turbostat09:05:010
144821399129,2cyclictest1561974-21idleruntime-cro08:40:010
144821399129,2cyclictest1537441-21cpuspeed_turbos08:20:150
144821399129,2cyclictest0-21swapper/011:15:270
144821399129,2cyclictest0-21swapper/008:19:590
144821399129,2cyclictest0-21swapper/007:35:150
144821399127,4cyclictest1704310-21ntpq10:30:210
144821399126,4cyclictest3614154-21systemd-journal10:05:000
144821399122,8cyclictest1816943-21apt-get12:05:140
144821399121,2cyclictest1624149-21apt-get09:30:140
144822199110,2cyclictest1780011-21grep11:35:173
144821899118,2cyclictest0-21swapper/210:30:162
144821899118,1cyclictest0-21swapper/211:20:162
144821899117,2cyclictest1798666-21idleruntime11:50:182
144821899117,2cyclictest17546072sleep211:14:222
144821899116,4cyclictest1719094-21sed10:45:162
144821599118,2cyclictest0-21swapper/110:10:211
144821599118,2cyclictest0-21swapper/107:45:231
144821599117,3cyclictest1686731-21fschecks_count10:20:161
144821599116,4cyclictest1695521-21tr10:25:201
144821599116,4cyclictest1695521-21tr10:25:201
144821599115,4cyclictest1576347-21tr08:50:191
144821599110,1cyclictest0-21swapper/109:10:211
144821599110,1cyclictest0-21swapper/108:38:021
144821599110,11cyclictest0-21swapper/109:44:241
144821599110,11cyclictest0-21swapper/109:35:201
144821599110,10cyclictest0-21swapper/111:55:021
144821599110,10cyclictest0-21swapper/110:15:491
144821599110,10cyclictest0-21swapper/108:07:561
144821399119,1cyclictest1605042-21kworker/0:1+kacpid10:15:070
144821399118,2cyclictest1605042-21kworker/0:1+kacpi_notify10:35:140
144821399118,2cyclictest0-21swapper/009:40:190
144821399116,4cyclictest1698558-21grep10:25:260
144821399116,4cyclictest1698558-21grep10:25:260
1448213991110,1cyclictest1730224-21kworker/0:2+kacpid11:20:500
144821399110,9cyclictest1546226-21kthreadcore08:25:190
144821399110,1cyclictest0-21swapper/011:45:420
14479872113,4sleep10-21swapper/107:09:051
144821899104,4cyclictest1539079-21cut08:20:192
144821899101,7cyclictest1611798-21/usr/sbin/munin09:20:132
144821599106,3cyclictest1585028-21tr08:55:171
144821399107,2cyclictest0-21swapper/012:30:170
144821399100,9cyclictest1612235-21idleruntime09:20:160
14482139990,2cyclictest1448070-21latency_hist07:10:010
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional