You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 23:07

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Wed Dec 03, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1376003994438,4cyclictest1567614-21latency_hist10:15:011
1376007993928,9cyclictest2698871-21systemd-journal08:05:242
1376010993631,3cyclictest1651829-21apt-get11:35:073
1376010993528,5cyclictest1437707-21latency_hist08:10:013
1375999993531,3cyclictest1442854-21kworker/u8:0+flush-259:009:20:000
1375999993530,3cyclictest1651625-21kworker/u8:2+flush-259:012:00:210
1375999993530,3cyclictest1608868-21kworker/u8:2+flush-259:011:05:000
1376010993429,3cyclictest1515953-21/usr/sbin/munin09:25:143
1375999993228,3cyclictest1693019-21kworker/u8:0+flush-259:012:29:590
1376007993120,9cyclictest1682746-21latency_hist12:05:012
1376003993125,4cyclictest1677796-21/usr/sbin/munin12:00:141
1375999993127,3cyclictest1562468-21kworker/u8:0+flush-259:011:25:210
1376003993025,3cyclictest1311503-21kworker/u8:2+flush-259:007:25:171
1376003993024,4cyclictest1386222-21latency_hist07:20:001
1375999992824,3cyclictest1468615-21kworker/u8:1+events_unbound09:10:020
1375999992822,4cyclictest492-21dbus-daemon10:35:000
1376003992722,3cyclictest1578197-21/usr/sbin/munin10:25:151
1375999992723,3cyclictest1562468-21kworker/u8:0+flush-259:011:40:150
1375999992721,4cyclictest1462489-21ntpq08:30:210
1375999992720,5cyclictest1526023-21latency_hist09:35:000
1376010992622,3cyclictest1375918-21kworker/u8:3+flush-259:007:40:173
1376010992520,4cyclictest1662195-21/usr/sbin/munin11:45:173
1376007992519,4cyclictest2698871-21systemd-journal11:25:242
1376010992419,4cyclictest1526258-21/usr/sbin/munin09:35:223
1375999992422,1cyclictest1422249-21grep07:55:000
1375999992421,2cyclictest1468615-21kworker/u8:1+flush-259:009:20:210
1375999992420,3cyclictest1562468-21kworker/u8:0+events_unbound10:25:240
1376010992218,3cyclictest1432773-21kworker/u8:4+flush-259:008:25:243
1376010992217,3cyclictest0-21swapper/308:25:003
1376003992217,4cyclictest1661952-21latency_hist11:45:001
1376003992216,5cyclictest1712783-21ntpq12:30:221
1376003992214,7cyclictest1577963-21turbostat.cron10:25:001
1375999992219,3cyclictest1504523-21kworker/u8:2+events_unbound09:35:190
1375999992218,3cyclictest1432773-21kworker/u8:4+events_unbound08:40:210
1375999992215,5cyclictest2698871-21systemd-journal08:20:000
1375999992215,5cyclictest1448230-21/usr/sbin/munin08:20:160
1376010992119,1cyclictest1624659-21kworker/u8:3+events_unbound11:20:243
1376010992118,2cyclictest1651625-21kworker/u8:2+events_unbound12:20:143
1376010992117,3cyclictest1588284-21kworker/u8:1+flush-259:010:45:003
1376010992115,5cyclictest1547023-21latency_hist09:55:013
1376010992115,5cyclictest1547023-21latency_hist09:55:003
1376010992115,4cyclictest1698167-21awk12:20:003
1376010992115,4cyclictest1698167-21awk12:20:003
1375999992118,2cyclictest1562468-21kworker/u8:0+flush-259:011:45:280
1375999992118,2cyclictest1468615-21kworker/u8:1+flush-259:009:05:170
1375999992117,3cyclictest1386483-21apt-get07:20:120
1376010992017,2cyclictest1693019-21kworker/u8:0+flush-259:012:25:193
1376010992016,3cyclictest1624659-21kworker/u8:3+flush-259:011:15:193
1376010992015,3cyclictest0-21swapper/310:15:003
1376007992016,3cyclictest1311503-21kworker/u8:2+flush-259:007:20:172
1376007992014,4cyclictest1677562-21latency_hist12:00:002
1375999992014,4cyclictest2698871-21systemd-journal08:15:000
1376010991915,3cyclictest1421854-21kworker/u8:1+flush-259:007:55:003
1376010991915,3cyclictest1396811-21apt-get07:30:123
1376010991915,2cyclictest1468615-21kworker/u8:1+flush-259:009:20:163
1376010991914,4cyclictest1613105-21ntpq10:55:223
1376010991912,5cyclictest1432547-21latency_hist08:05:013
1376007991917,1cyclictest0-21swapper/211:00:012
1376007991914,4cyclictest1562468-21kworker/u8:0+flush-259:010:35:172
1376007991913,4cyclictest1646869-21diskstats11:30:142
137600799190,2cyclictest1412985-21sed07:45:172
137600399195,12cyclictest1651959-21cpuspeed_turbos11:35:131
1376003991916,2cyclictest3000063-21rs:main1
1376010991814,3cyclictest1635466-21kworker/u8:1+events_unbound11:55:243
137600799187,9cyclictest1717710-21needreboot12:35:212
1376007991815,2cyclictest1484028-21idleruntime-cro08:55:002
1376007991814,3cyclictest1411925-21cron07:45:002
1376007991813,4cyclictest1662222-21apt-get11:45:122
1376007991812,4cyclictest1463411-21latency_hist08:35:022
1376007991810,7cyclictest1474077-21cpuspeed_turbos08:45:142
1376007991810,7cyclictest1406369-21unixbench_multi07:35:282
137600399189,7cyclictest0-21swapper/107:45:001
137600399181,5cyclictest1587282-21cut10:30:191
1376003991815,2cyclictest1468615-21kworker/u8:1+flush-259:010:00:181
1376003991814,3cyclictest1588284-21kworker/u8:1+flush-259:010:40:001
1376003991812,4cyclictest2698871-21systemd-journal07:13:251
1376003991812,4cyclictest1453553-21diskstats08:25:141
137600399180,6cyclictest1417071-21awk07:50:001
137600399180,5cyclictest1630910-21grep11:15:181
137600399180,16cyclictest1515287-21latency_hist09:25:001
1375999991815,2cyclictest1562468-21kworker/u8:0+events_unbound10:45:210
1375999991815,2cyclictest1478911-21kworker/u8:2+events_unbound08:51:250
1375999991814,3cyclictest1651625-21kworker/u8:2+flush-259:012:15:260
1375999991814,3cyclictest1651625-21kworker/u8:2+flush-259:012:15:250
1375999991814,3cyclictest1635466-21kworker/u8:1+events_unbound11:55:190
1375999991814,3cyclictest1588284-21kworker/u8:1+events_unbound10:40:250
1375999991814,2cyclictest60650irq/133-enp0s29f1:rx-007:10:200
1375999991814,2cyclictest0-21swapper/008:05:000
1375999991813,2cyclictest1624659-21kworker/u8:3+events_unbound11:15:180
1376010991713,3cyclictest1624900-21apt-get11:10:123
1376010991712,4cyclictest3000063-21rs:main3
1376010991711,4cyclictest2698871-21systemd-journal08:59:593
137600799179,7cyclictest0-21swapper/210:25:002
137600799175,10cyclictest1692105-21ldconfig12:10:222
1376007991714,1cyclictest0-21swapper/207:30:002
1376007991713,3cyclictest1635854-21cat11:20:022
137600799171,2cyclictest1400768-21ntpq07:30:212
137600399176,10cyclictest0-21swapper/109:45:001
137600399175,2cyclictest1683089-21cpuspeed_turbos12:05:171
137600399175,2cyclictest1552134-21sh10:00:001
137600399172,3cyclictest2698871-21systemd-journal11:35:001
137600399171,5cyclictest1718743-21systemd-run12:40:011
137600399171,5cyclictest1488717-21grep08:55:261
137600399171,3cyclictest2698871-21systemd-journal11:54:591
137600399171,3cyclictest2698871-21systemd-journal11:00:011
137600399171,3cyclictest1640883-21cron11:25:011
137600399171,3cyclictest1619524-21idleruntime-cro11:05:001
137600399171,3cyclictest1427364-21cron08:00:001
137600399171,3cyclictest1422240-21idleruntime-cro07:55:001
1376003991713,3cyclictest1542125-21apt-get09:50:121
1376003991713,3cyclictest1542125-21apt-get09:50:111
1376003991713,3cyclictest1489455-21apt-get09:00:121
1376003991713,3cyclictest1432773-21kworker/u8:4+flush-259:008:35:011
137600399171,2cyclictest1708409-21awk12:30:001
137600399170,5cyclictest1499454-21cron09:09:591
137600399170,2cyclictest1473753-21latency_hist08:45:011
1375999991714,2cyclictest60650irq/133-enp0s29f1:rx-012:10:260
1375999991714,2cyclictest1562434-21idleruntime-cro10:10:000
1375999991713,3cyclictest1562468-21kworker/u8:0+events_unbound11:34:250
1375999991713,3cyclictest1562468-21kworker/u8:0+events_unbound11:00:000
1375999991711,4cyclictest1713597-21latency_hist12:35:010
1376010991612,3cyclictest1567844-21/usr/sbin/munin10:15:203
1376010991612,3cyclictest1311503-21kworker/u8:2+flush-259:007:35:193
137600799163,11cyclictest1572354-21awk10:15:252
137600799161,3cyclictest1447265-21/usr/sbin/munin08:15:232
137600799161,2cyclictest2698871-21systemd-journal12:30:002
137600799161,2cyclictest1561526-21ntpq10:05:212
137600799161,2cyclictest1547016-21apt09:55:012
137600799161,2cyclictest1547016-21apt09:55:002
137600799161,2cyclictest1489191-21ls08:59:592
137600799161,2cyclictest1421469-21ps07:50:222
137600799160,3cyclictest1504133-21ntpq09:10:232
137600799160,2cyclictest1602932-21ps10:45:232
137600799160,1cyclictest1458326-21cat08:30:012
137600399168,7cyclictest0-21swapper/107:35:271
137600399166,1cyclictest0-21swapper/107:20:181
137600399165,10cyclictest0-21swapper/110:50:171
137600399163,2cyclictest0-21swapper/112:25:001
137600399162,3cyclictest1641322-21df11:25:131
137600399161,3cyclictest1474420-21grep08:45:171
137600399161,3cyclictest1-21systemd11:50:011
1376003991613,1cyclictest1693674-21irqcore12:15:171
1376003991613,1cyclictest1693674-21irqcore12:15:161
137600399161,2cyclictest1468587-21turbostat.cron08:40:001
1376003991612,2cyclictest1437690-21cron08:10:001
137600399160,5cyclictest1539393-21kthreadcore09:45:181
137600399160,5cyclictest1519884-21needreboot09:25:191
137600399160,4cyclictest1624619-21cron11:10:011
137600399160,4cyclictest1562429-21cron10:10:001
137600399160,4cyclictest1431565-21grep08:00:231
137600399160,3cyclictest1629794-21apt11:15:001
137600399160,3cyclictest1526024-21apt09:35:001
137600399160,3cyclictest1500299-21tr09:10:151
137600399160,3cyclictest1447973-21cron08:20:001
137599999169,5cyclictest1515896-21cat09:25:030
1375999991614,2cyclictest60650irq/133-enp0s29f1:rx-008:58:270
1375999991614,1cyclictest60650irq/133-enp0s29f1:rx-012:37:120
1375999991613,2cyclictest60650irq/133-enp0s29f1:rx-007:55:080
1375999991613,2cyclictest1635466-21kworker/u8:1+flush-259:011:50:180
1375999991613,2cyclictest1635466-21kworker/u8:1+flush-259:011:35:210
1375999991613,2cyclictest1562468-21kworker/u8:0+events_unbound10:20:130
1375999991613,2cyclictest1504523-21kworker/u8:2+events_unbound10:04:240
1375999991613,2cyclictest1375918-21kworker/u8:3+flush-259:007:15:210
1375999991612,3cyclictest1442854-21kworker/u8:0+flush-259:008:35:210
1375999991612,3cyclictest1311503-21kworker/u8:2+events_unbound07:35:170
1376010991512,2cyclictest0-21swapper/308:45:003
1376010991511,3cyclictest1651625-21kworker/u8:2+flush-259:012:00:133
1376010991511,3cyclictest1468615-21kworker/u8:1+events_unbound09:15:243
1376010991511,3cyclictest1391386-21latency_hist07:25:003
1376010991510,4cyclictest1391622-21/usr/sbin/munin07:25:233
137600799154,9cyclictest1685886-21kthreadcore12:05:222
137600799154,10cyclictest0-21swapper/208:15:002
137600799153,10cyclictest1580918-21cut10:25:182
137600799151,2cyclictest1661948-21cron11:44:592
137600799151,2cyclictest1551234-21ntpq09:55:232
137600799151,2cyclictest1535408-21ntpq09:40:222
137600799151,2cyclictest1526023-21latency_hist09:35:002
137600799151,2cyclictest1498627-21ntpq09:05:212
137600799151,2cyclictest1453132-21munin-plugin-st08:25:002
137600799151,1cyclictest1712710-21ntpq12:30:202
137600799151,1cyclictest1519974-21ntpq09:25:222
137600799150,5cyclictest1699422-21/usr/sbin/munin12:20:182
137600799150,5cyclictest1653079-21grep11:35:182
137600799150,5cyclictest1630641-21df11:15:152
137600799150,5cyclictest1469258-21/usr/sbin/munin08:40:152
137600799150,5cyclictest1463956-21fw_forwarded_lo08:35:142
137600799150,4cyclictest1557324-21latency_hist10:05:012
137600799150,3cyclictest1629798-21cron11:15:012
137600799150,3cyclictest1530335-21ntpq09:35:242
137600799150,3cyclictest1510637-21/usr/sbin/munin09:20:142
137600799150,3cyclictest1490296-21irqcore09:00:172
137600799150,2cyclictest2698871-21systemd-journal09:50:002
137600799150,2cyclictest1697310-21ntpq12:15:222
137600799150,2cyclictest1697310-21ntpq12:15:212
137600799150,2cyclictest1672372-21munin-plugin-st11:55:002
137600799150,2cyclictest1427377-21idleruntime-cro08:00:002
137600799150,2cyclictest1376586-21sed07:10:172
137600799150,1cyclictest1618623-21ntpq11:00:212
137600399151,4cyclictest1531145-21sh09:40:011
137600399151,3cyclictest1479241-21tr08:50:131
137600399151,2cyclictest1593777-21grep10:40:151
137600399150,5cyclictest1571798-21needreboot10:15:181
137600399150,4cyclictest1688265-21cut12:10:151
137600399150,4cyclictest1603699-21cron10:50:001
137600399150,4cyclictest1448456-21cut08:20:161
137600399150,3cyclictest1677135-21grep11:55:251
137600399150,3cyclictest1438113-21grep08:10:131
137600399150,3cyclictest1401692-21munin-plugin-st07:35:001
137600399150,3cyclictest1376008-21cut07:10:001
137599999158,5cyclictest1687854-21latency_hist12:10:000
1375999991513,2cyclictest60650irq/133-enp0s29f1:rx-011:05:500
1375999991513,1cyclictest60650irq/133-enp0s29f1:rx-010:50:200
137601099149,3cyclictest1375918-21kworker/u8:3+flush-259:007:15:273
137601099148,4cyclictest1531397-21/usr/sbin/munin09:40:163
1376010991411,2cyclictest1442854-21kworker/u8:0+flush-259:008:35:243
1376010991411,2cyclictest1414487-21kthreadcore07:45:183
1376010991410,3cyclictest1703296-21kworker/u8:1+flush-259:012:35:173
1376010991410,3cyclictest1332338-21kworker/u8:1+flush-259:007:15:003
1376010991410,2cyclictest1479218-21cpuspeed_turbos08:50:133
137600799149,3cyclictest1604093-21cut10:50:122
137600799141,3cyclictest0-21swapper/209:15:162
137600799141,2cyclictest1431628-21ntpq08:00:232
137600799141,10cyclictest0-21swapper/210:10:132
137600799140,4cyclictest1585464-21grep10:30:192
137600799140,3cyclictest1381766-21cut07:15:182
137600799140,2cyclictest1620009-21sort11:05:142
1375999991413,1cyclictest60650irq/133-enp0s29f1:rx-012:20:350
1375999991413,1cyclictest60650irq/133-enp0s29f1:rx-007:27:050
1375999991412,2cyclictest61450irq/134-enp0s29f1:tx-007:40:160
1375999991412,1cyclictest60650irq/133-enp0s29f1:rx-010:10:200
1375999991412,1cyclictest60650irq/133-enp0s29f1:rx-009:47:440
1375999991412,1cyclictest60650irq/133-enp0s29f1:rx-008:45:240
1375999991412,1cyclictest60650irq/133-enp0s29f1:rx-008:05:140
1375999991411,2cyclictest60650irq/133-enp0s29f1:rx-010:18:010
1375999991411,2cyclictest1432773-21kworker/u8:4+events_unbound08:25:240
1375999991411,2cyclictest0-21swapper/009:55:000
1375999991411,2cyclictest0-21swapper/009:55:000
137601099139,3cyclictest1573215-21uniq10:20:133
137601099137,4cyclictest1646527-21base6411:30:003
137601099131,2cyclictest14465732kthreadcore08:15:183
137600799130,2cyclictest1593662-21munin-node10:40:132
1375999991311,1cyclictest60650irq/133-enp0s29f1:rx-009:02:200
1375999991310,2cyclictest60650irq/133-enp0s29f1:rx-009:58:030
1375999991310,2cyclictest1311503-21kworker/u8:2+flush-259:007:45:250
137601099129,2cyclictest1645542-21unixbench_multi11:25:253
137601099128,3cyclictest1582635-21unixbench_multi10:25:243
1376010991210,1cyclictest0-21swapper/312:30:223
137600799120,2cyclictest1376016-21latency_hist07:10:002
137599999129,2cyclictest1624659-21kworker/u8:3+events_unbound11:13:220
137599999128,3cyclictest1624659-21kworker/u8:3+events_unbound11:24:570
137601099119,1cyclictest0-21swapper/308:10:133
137601099118,2cyclictest0-21swapper/312:05:293
137601099108,1cyclictest0-21swapper/309:00:163
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional