You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-07 - 21:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Mon Jul 07, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1289778994643,2cyclictest1284837-21kworker/u8:3+events_unbound09:05:181
1289778994441,2cyclictest1606738-21kworker/u8:3+events_unbound12:30:211
1289784994336,5cyclictest1424879-21latency_hist09:20:013
1289784994333,8cyclictest1440306-21latency_hist09:35:003
1289778994137,3cyclictest1316130-21kworker/u8:1+flush-259:007:35:161
1289778994136,4cyclictest1393612-21kworker/u8:0+flush-259:009:15:221
1289778994136,4cyclictest1383528-21/usr/sbin/munin08:40:151
1289778993935,3cyclictest1352605-21kworker/u8:4+events_unbound08:25:231
1289778993932,5cyclictest1362699-21latency_hist08:20:011
1289778993831,5cyclictest1570316-21cat11:40:021
1289784993734,2cyclictest1446102-21kworker/u8:3+flush-259:009:50:183
1289778993734,2cyclictest1507825-21kworker/u8:1+flush-259:011:00:221
1289776993731,4cyclictest1435596-21diskstats09:30:150
1289778993633,2cyclictest1554415-21kworker/u8:0+events_unbound11:25:231
1289778993530,4cyclictest1284837-21kworker/u8:3+flush-259:007:45:171
1289776993532,2cyclictest1352605-21kworker/u8:4+flush-259:008:45:000
1289784993428,4cyclictest1544095-21latency_hist11:15:003
1289778993428,4cyclictest1533332-21kworker/u8:2+flush-259:011:05:181
1289778993330,2cyclictest1606738-21kworker/u8:3+flush-259:012:25:191
1289778993330,2cyclictest1466561-21kworker/u8:0+events_unbound10:00:181
1289778993229,2cyclictest1393612-21kworker/u8:0+flush-259:009:10:231
1289784993125,4cyclictest1393592-21latency_hist08:50:003
1289784993124,5cyclictest1569883-21latency_hist11:40:003
1289778993128,2cyclictest1596470-21kworker/u8:2+flush-259:012:05:201
1289778993128,2cyclictest1487156-21kworker/u8:5+flush-259:010:20:151
1289778993128,2cyclictest1341643-21kworker/u8:0+events_unbound08:00:231
1289776993126,4cyclictest1502668-21latency_hist10:35:020
1289784993025,4cyclictest3754036-21systemd-journal12:25:003
1289778993026,3cyclictest1419441-21kworker/u8:2+flush-259:010:10:011
1289778993025,3cyclictest3754036-21systemd-journal12:21:211
1289778992926,2cyclictest1368092-21kworker/u8:1+flush-259:008:35:181
1289776992926,2cyclictest1419441-21kworker/u8:2+flush-259:009:25:180
1289776992926,2cyclictest1419441-21kworker/u8:2+flush-259:009:25:180
1289784992823,3cyclictest1627493-21/usr/sbin/munin12:35:133
1289784992822,4cyclictest1575444-21latency_hist11:45:013
1289778992824,3cyclictest1252858-21kworker/u8:4+flush-259:007:10:201
1289776992825,2cyclictest1476364-21kworker/u8:3+flush-259:010:10:160
1289776992824,3cyclictest1419441-21kworker/u8:2+flush-259:009:25:000
1289776992822,4cyclictest1415025-21diskstats09:10:140
1289784992722,3cyclictest3754036-21systemd-journal09:44:213
1289782992722,4cyclictest3754036-21systemd-journal09:15:002
1289778992724,2cyclictest1564958-21kworker/u8:3+events_unbound11:50:191
1289778992724,2cyclictest1305202-21kworker/u8:0+events_unbound07:30:141
1289782992616,9cyclictest1403905-21gpgv09:00:002
1289778992624,2cyclictest1503326-21kworker/u8:0+flush-259:010:55:001
1289778992623,2cyclictest1554415-21kworker/u8:0+events_unbound11:35:211
1289778992623,2cyclictest1548522-21kworker/u8:1+events_unbound11:45:231
1289778992623,2cyclictest1503326-21kworker/u8:0+events_unbound10:40:191
1289778992623,2cyclictest1477043-21kworker/u8:4+flush-259:010:25:211
1289778992623,2cyclictest1341643-21kworker/u8:0+events_unbound08:10:241
1289778992622,3cyclictest1477043-21kworker/u8:4+flush-259:010:30:271
1289778992622,3cyclictest1446102-21kworker/u8:3+events_unbound09:40:221
1289778992622,3cyclictest1352605-21kworker/u8:4+events_unbound08:30:191
1289778992620,4cyclictest3754036-21systemd-journal09:05:001
1289778992619,5cyclictest1403034-21ntpq08:55:251
1289784992521,3cyclictest1503326-21kworker/u8:0+flush-259:010:40:173
1289784992518,5cyclictest1622073-21latency_hist12:30:013
1289782992520,3cyclictest1321309-21apt-get07:40:132
1289782992519,4cyclictest3754036-21systemd-journal08:34:592
1289778992522,2cyclictest1419441-21kworker/u8:2+events_unbound09:20:251
1289778992522,2cyclictest1352605-21kworker/u8:4+flush-259:008:45:251
1289776992521,3cyclictest1622087-21kworker/u8:4+flush-259:012:35:170
1289776992521,3cyclictest1419441-21kworker/u8:2+flush-259:009:35:230
1289776992518,5cyclictest1554398-21latency_hist11:25:000
1289784992420,3cyclictest1279545-21kworker/u8:2+flush-259:007:15:193
1289778992422,1cyclictest1284837-21kworker/u8:3+events_unbound07:20:261
1289778992421,2cyclictest1622087-21kworker/u8:4+events_unbound12:35:181
1289778992421,2cyclictest1460974-21kworker/u8:4+events_unbound09:55:211
1289778992420,3cyclictest1284837-21kworker/u8:3+events_unbound08:50:141
1289778992418,2cyclictest1477043-21kworker/u8:4+events_unbound10:10:201
1289776992421,2cyclictest1612069-21kworker/u8:2+events_unbound12:20:130
1289776992421,2cyclictest1476817-21idleruntime-cro10:10:000
1289776992420,3cyclictest1548522-21kworker/u8:1+events_unbound11:45:150
1289776992419,3cyclictest1419441-21kworker/u8:2+events_unbound10:50:210
1289784992319,3cyclictest1331759-21diskstats07:50:143
1289784992316,5cyclictest1430446-21diskstats09:25:163
1289784992316,5cyclictest1430446-21diskstats09:25:153
1289782992317,4cyclictest3754036-21systemd-journal11:56:212
1289778992320,2cyclictest1548522-21kworker/u8:1+flush-259:011:55:201
1289778992320,2cyclictest1419441-21kworker/u8:2+flush-259:009:45:151
1289778992320,2cyclictest1284837-21kworker/u8:3+flush-259:007:40:211
1289778992319,3cyclictest1523588-21kworker/u8:3+events_unbound11:10:221
1289778992319,3cyclictest1347443-21kworker/u8:2+events_unbound08:20:181
1289778992319,3cyclictest1284837-21kworker/u8:3+events_unbound07:15:151
1289776992320,2cyclictest1393612-21kworker/u8:0+events_unbound08:55:200
1289776992319,3cyclictest1606738-21kworker/u8:3+events_unbound12:30:180
1289776992319,3cyclictest1446102-21kworker/u8:3+flush-259:009:45:270
1289776992319,3cyclictest1352605-21kworker/u8:4+events_unbound08:45:160
1289776992319,3cyclictest1347443-21kworker/u8:2+flush-259:008:20:270
1289776992318,4cyclictest1507825-21kworker/u8:1+events_unbound10:45:200
1289776992318,4cyclictest1315589-21gpgv07:35:000
1289776992317,4cyclictest1445453-21apt-get09:40:020
1289784992218,3cyclictest1523778-21diskstats10:55:153
1289784992217,3cyclictest1378146-21turbostat08:34:593
1289784992216,4cyclictest1430022-21latency_hist09:25:003
1289784992216,4cyclictest1352378-21latency_hist08:10:013
1289784992210,1cyclictest0-21swapper/307:45:003
1289782992217,3cyclictest1606724-21latency_hist12:15:012
128978299220,3cyclictest1299369-21seq07:15:262
1289778992219,2cyclictest1419441-21kworker/u8:2+events_unbound09:30:151
1289778992219,2cyclictest1419441-21kworker/u8:2+events_unbound09:25:201
1289778992219,2cyclictest1419441-21kworker/u8:2+events_unbound09:25:191
1289778992219,2cyclictest1316130-21kworker/u8:1+events_unbound08:05:231
1289778992215,5cyclictest1336515-21latency_hist07:55:011
1289776992218,3cyclictest1420006-21apt-get09:15:140
1289776992217,3cyclictest1284837-21kworker/u8:3+flush-259:008:50:190
1289776992215,5cyclictest1496604-21ntpq10:25:220
128978499218,5cyclictest1404540-21/usr/sbin/munin09:00:153
1289784992117,3cyclictest1497762-21/usr/sbin/munin10:30:143
1289782992117,3cyclictest1580609-21latency_hist11:50:002
1289782992117,2cyclictest1513224-21/usr/sbin/munin10:45:242
1289778992119,2cyclictest1477043-21kworker/u8:4+flush-259:010:15:251
1289778992118,2cyclictest1548522-21kworker/u8:1+flush-259:012:15:151
1289778992118,2cyclictest1548522-21kworker/u8:1+events_unbound11:20:281
1289778992118,2cyclictest1523588-21kworker/u8:3+flush-259:011:15:211
1289778992118,2cyclictest1507825-21kworker/u8:1+flush-259:010:45:171
1289776992118,2cyclictest1294835-21cron07:15:000
128977699211,18cyclictest1-21systemd11:09:200
1289784992016,3cyclictest1554415-21kworker/u8:0+flush-259:012:20:013
1289784992015,3cyclictest1575685-21/usr/sbin/munin11:45:213
1289782992017,2cyclictest1570117-21gpgv11:40:022
128978299200,2cyclictest1373012-21latency_hist08:30:012
1289776992017,2cyclictest1446102-21kworker/u8:3+events_unbound09:50:250
1289776992017,2cyclictest1305202-21kworker/u8:0+flush-259:007:25:160
1289776992016,3cyclictest1564958-21kworker/u8:3+events_unbound11:50:150
1289776992016,3cyclictest1554415-21kworker/u8:0+events_unbound12:00:160
1289776992016,3cyclictest1503326-21kworker/u8:0+flush-259:010:40:160
1289776992016,3cyclictest1252858-21kworker/u8:4+flush-259:007:45:210
128978299196,11cyclictest1398237-21grep08:50:252
1289778991916,2cyclictest1554415-21kworker/u8:0+events_unbound12:00:211
1289778991916,2cyclictest1305202-21kworker/u8:0+flush-259:007:25:191
1289778991912,5cyclictest1336935-21diskstats07:55:131
1289776991915,3cyclictest1533905-21kworker/u8:4+flush-259:011:25:160
1289776991915,3cyclictest1507825-21kworker/u8:1+events_unbound10:55:290
1289776991915,3cyclictest1487156-21kworker/u8:5+flush-259:010:20:210
1289784991815,2cyclictest1528755-21apt-get11:00:123
128978299189,8cyclictest0-21swapper/210:10:222
128978299186,10cyclictest1382404-21ntpq08:35:202
128978299185,10cyclictest0-21swapper/208:05:012
1289782991814,2cyclictest1358245-21irqcore08:15:182
1289782991812,4cyclictest3754036-21systemd-journal08:22:222
128978299180,2cyclictest1565826-21irqcore11:35:182
1289778991814,3cyclictest1596470-21kworker/u8:2+events_unbound12:11:231
1289776991815,2cyclictest1548522-21kworker/u8:1+events_unbound12:10:230
1289776991814,3cyclictest1554415-21kworker/u8:0+events_unbound11:35:260
1289776991814,3cyclictest1503326-21kworker/u8:0+flush-259:011:15:210
1289776991814,3cyclictest1466561-21kworker/u8:0+flush-259:010:00:210
1289776991814,3cyclictest1316130-21kworker/u8:1+events_unbound07:35:150
1289776991811,5cyclictest3754036-21systemd-journal08:10:000
1289784991714,2cyclictest1404145-21latency_hist09:00:003
1289784991713,3cyclictest1548522-21kworker/u8:1+flush-259:011:15:273
1289784991711,4cyclictest1586593-21diskstats11:55:143
1289784991711,4cyclictest1300255-21/usr/sbin/munin07:20:143
128978299179,7cyclictest0-21swapper/209:50:212
128978299171,4cyclictest1507826-21systemd-run10:40:012
1289782991713,3cyclictest3754036-21systemd-journal10:04:552
128978299170,4cyclictest1627242-21cron12:35:002
128978299170,4cyclictest1315472-21turbostat.cron07:35:002
128978299170,3cyclictest1451031-21turbostat.cron09:45:002
1289778991711,4cyclictest1506883-21ntpq10:35:201
1289776991714,2cyclictest60650irq/133-enp0s29f1:rx-008:30:170
1289776991714,2cyclictest60650irq/133-enp0s29f1:rx-008:28:030
1289776991713,3cyclictest1503326-21kworker/u8:0+flush-259:010:35:240
1289776991713,3cyclictest1341643-21kworker/u8:0+events_unbound08:10:170
1289776991713,3cyclictest1284837-21kworker/u8:3+events_unbound07:40:230
1289776991712,3cyclictest1466561-21kworker/u8:0+flush-259:010:20:010
128978299161,2cyclictest1500072-21ps10:30:172
1289782991611,4cyclictest1425122-21/usr/sbin/munin09:20:152
128978299160,4cyclictest3754036-21systemd-journal11:30:002
128978299160,4cyclictest1621570-21grep12:25:252
128978299160,4cyclictest1435262-21latency_hist09:30:002
128978299160,4cyclictest1435262-21latency_hist09:30:002
128978299160,3cyclictest1612252-21df12:20:132
128978299160,3cyclictest1610833-21tr12:15:212
128978299160,3cyclictest1549922-21tr11:20:172
128978299160,3cyclictest1544287-21latency_hist11:15:002
128978299160,3cyclictest1492448-21cat10:25:012
128978299160,3cyclictest1406869-21kthreadcore09:00:192
128978299160,3cyclictest1351375-21grep08:05:202
128978299160,3cyclictest1310298-21cron07:30:002
128978299160,3cyclictest1294847-21latency_hist07:14:592
128978299160,2cyclictest1456193-21latency_hist09:50:002
1289778991613,2cyclictest1419441-21kworker/u8:2+events_unbound09:35:211
1289778991611,3cyclictest1533905-21kworker/u8:4+flush-259:011:30:101
1289776991614,1cyclictest60650irq/133-enp0s29f1:rx-011:55:160
1289776991613,2cyclictest1279545-21kworker/u8:2+events_unbound07:15:190
1289776991612,3cyclictest1596470-21kworker/u8:2+events_unbound12:05:160
1289776991612,3cyclictest1347443-21kworker/u8:2+events_unbound08:15:170
1289784991512,2cyclictest1460974-21kworker/u8:4+flush-259:009:55:183
1289784991511,3cyclictest3754036-21systemd-journal07:14:553
128978299153,1cyclictest0-21swapper/207:20:172
128978299152,3cyclictest1462722-21tr09:55:182
128978299152,2cyclictest1409729-21apt-get09:05:142
128978299152,12cyclictest0-21swapper/207:55:002
128978299151,2cyclictest1340772-21ntpq07:55:212
128978299151,1cyclictest1527586-21ntpq10:55:222
128978299151,1cyclictest1424882-21sh09:20:012
128978299150,4cyclictest1632398-21munin-plugin-st12:40:012
128978299150,4cyclictest1389147-21sed08:45:172
128978299150,4cyclictest1326798-21if_enp0s29f107:45:162
128978299150,3cyclictest1581514-21fschecks_time11:50:132
128978299150,3cyclictest1560057-21cut11:30:162
128978299150,3cyclictest1548656-21timerwakeupswit11:15:272
128978299150,3cyclictest1528823-21cpuspeed_turbos11:00:132
128978299150,3cyclictest1510655-21grep10:40:192
128978299150,3cyclictest1383970-21ps08:40:172
128978299150,3cyclictest1321059-21systemd-run07:40:012
128978299150,2cyclictest1496622-21ntpq10:25:232
128978299150,2cyclictest1487125-21awk10:20:002
1289776991514,1cyclictest60650irq/133-enp0s29f1:rx-012:26:260
1289776991513,1cyclictest60650irq/133-enp0s29f1:rx-007:55:190
1289776991513,1cyclictest60650irq/133-enp0s29f1:rx-007:53:450
1289776991512,2cyclictest1284837-21kworker/u8:3+events_unbound09:05:160
1289776991511,3cyclictest1341643-21kworker/u8:0+events_unbound08:00:160
128978499145,7cyclictest1398756-21latency_hist08:55:003
1289784991411,2cyclictest1370423-21kthreadcore08:25:203
1289784991411,2cyclictest1347443-21kworker/u8:2+flush-259:008:20:213
1289784991411,2cyclictest0-21swapper/307:40:003
1289784991411,1cyclictest1445464-21turbostat09:40:003
1289784991410,3cyclictest1523588-21kworker/u8:3+flush-259:011:20:173
1289784991410,3cyclictest1352605-21kworker/u8:4+flush-259:009:11:263
1289784991410,3cyclictest1341643-21kworker/u8:0+flush-259:008:10:253
1289784991410,2cyclictest0-21swapper/309:05:183
128978299143,9cyclictest0-21swapper/209:30:002
128978299141,3cyclictest1534324-21cat11:05:162
128978299140,4cyclictest1596838-21grep12:05:132
128978299140,4cyclictest1440677-21expr09:35:142
128978299140,3cyclictest1591769-21fschecks_count12:00:152
128978299140,3cyclictest1518858-21grep10:50:192
128978299140,3cyclictest1473146-21grep10:05:192
128978299140,3cyclictest1352733-21grep08:10:132
1289778991410,3cyclictest1523588-21kworker/u8:3+events_unbound10:56:541
1289776991413,1cyclictest61450irq/134-enp0s29f1:tx-011:30:170
1289776991413,1cyclictest60650irq/133-enp0s29f1:rx-012:15:250
1289776991412,1cyclictest60650irq/133-enp0s29f1:rx-011:10:090
1289776991412,1cyclictest60650irq/133-enp0s29f1:rx-008:35:320
1289776991411,2cyclictest60650irq/133-enp0s29f1:rx-009:55:210
1289776991410,3cyclictest1570366-21gpgv11:40:020
1289776991410,3cyclictest1393612-21kworker/u8:0+events_unbound09:00:210
128978499139,3cyclictest1548522-21kworker/u8:1+flush-259:011:30:263
128978499138,3cyclictest1548522-21kworker/u8:1+flush-259:012:10:173
128978499138,3cyclictest1310303-21dump-pmu-power07:30:003
128978499137,4cyclictest1487803-21grep10:20:183
1289778991310,2cyclictest1460974-21kworker/u8:4+events_unbound09:54:241
1289776991312,1cyclictest60650irq/133-enp0s29f1:rx-007:21:360
128978499129,2cyclictest1507366-21unixbench_singl10:35:253
128978499129,2cyclictest1378832-21irqcore08:35:163
128978499129,1cyclictest0-21swapper/311:05:173
128978499126,4cyclictest1337043-21cut07:55:143
128978499124,6cyclictest1486274-21ntpq10:15:243
1289784991210,1cyclictest0-21swapper/312:30:173
1289784991210,1cyclictest0-21swapper/307:30:253
128978499119,1cyclictest0-21swapper/311:50:173
128978499117,3cyclictest1388470-21sh08:44:593
128978499116,3cyclictest214-21jbd2/nvme0n1p2-808:05:003
128978499112,7cyclictest1404176-21kworker/3:1+cgroup_destroy10:15:013
128978499107,2cyclictest1492806-21fschecks_time10:25:163
128978499107,2cyclictest0-21swapper/311:25:213
128978499103,5cyclictest1451463-21diskstats09:45:173
12897849991,6cyclictest1331326-21munin-plugin-st07:50:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional