You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-29 - 18:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Sat Nov 29, 2025 12:44:05)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1172609993733,3cyclictest1354001-21latency_hist10:05:001
1172609993529,4cyclictest2698871-21systemd-journal12:05:011
1172604993429,4cyclictest1317739-21base6409:30:080
1172604993427,5cyclictest1187944-21latency_hist07:25:000
1172609993330,2cyclictest1384420-21kworker/u8:1+flush-259:010:40:001
1172617993226,4cyclictest1462965-21latency_hist11:50:013
1172612993228,3cyclictest1452345-21kworker/u8:0+flush-259:011:55:182
1172612993226,4cyclictest1416452-21/usr/sbin/munin11:05:162
1172609993127,3cyclictest3000063-21rs:main1
1172612993027,2cyclictest1411090-21kworker/u8:2+flush-259:011:15:162
1172617992912,9cyclictest1412360-21kthreadcore11:00:183
1172609992924,3cyclictest1421351-21apt-get11:10:001
1172617992819,8cyclictest1349076-21apt-get10:00:123
1172612992824,3cyclictest503-21systemd-logind07:50:002
1172609992824,3cyclictest2698871-21systemd-journal12:15:201
1172612992724,2cyclictest1276266-21kworker/u8:1+flush-259:008:55:172
1172612992723,3cyclictest1294011-21kworker/u8:0+events_unbound09:20:132
1172609992724,2cyclictest1359368-21kworker/u8:4+events_unbound10:15:211
1172604992723,3cyclictest1276266-21kworker/u8:1+flush-259:008:55:230
1172612992621,4cyclictest1333919-21kworker/u8:0+flush-259:010:00:002
1172612992621,4cyclictest1129334-21kworker/u8:0+flush-259:007:20:162
1172609992623,2cyclictest503-21systemd-logind08:50:011
1172604992619,5cyclictest1374658-21apt-get10:25:010
1172612992522,2cyclictest1452345-21kworker/u8:0+flush-259:011:40:252
1172612992521,3cyclictest1338135-21kworker/u8:2+flush-259:010:25:202
1172609992520,4cyclictest1504412-21apt-get12:30:131
1172604992519,4cyclictest1353181-21ntpq10:00:230
1172612992418,4cyclictest1234710-21diskstats08:10:172
1172604992421,2cyclictest1229400-21apt-get08:05:140
1172617992320,2cyclictest1333919-21kworker/u8:0+flush-259:009:55:243
1172612992320,2cyclictest1276266-21kworker/u8:1+flush-259:008:50:202
1172612992319,3cyclictest1328388-21kworker/u8:3+flush-259:009:40:192
1172604992318,4cyclictest1452932-21/usr/sbin/munin11:40:230
1172617992219,2cyclictest1172821-21apt-get07:10:113
1172617992217,4cyclictest1509552-21apt-get12:35:133
1172612992219,2cyclictest1473259-21kworker/u8:2+flush-259:012:15:252
1172612992217,3cyclictest1270422-21apt-get08:45:002
1172609992220,2cyclictest1405593-21kworker/u8:1+flush-259:011:00:161
117260499226,4cyclictest1389568-21ntpq10:35:210
117261799212,2cyclictest1463719-21cut11:50:163
1172612992119,1cyclictest1212778-21kworker/u8:2+events_unbound08:30:232
1172612992117,3cyclictest1493606-21kworker/u8:0+events_unbound12:25:182
1172612992117,3cyclictest1348826-21kworker/u8:1+events_unbound10:15:002
1172609992118,2cyclictest1493606-21kworker/u8:0+events_unbound12:25:191
1172609992117,3cyclictest1411090-21kworker/u8:2+events_unbound11:20:141
117260499213,6cyclictest1265444-21sed08:40:000
1172604992116,4cyclictest2698871-21systemd-journal12:30:000
1172604992116,4cyclictest1375326-21/usr/sbin/munin10:25:160
1172617992016,3cyclictest492-21dbus-daemon10:25:013
1172617992016,3cyclictest1333919-21kworker/u8:0+flush-259:009:45:233
1172617992015,4cyclictest1375351-21apt-get10:25:143
1172617992015,4cyclictest1294011-21kworker/u8:0+flush-259:009:35:183
1172612992017,2cyclictest2307235-21kworker/u8:1+events_unbound07:35:202
1172612992017,2cyclictest1427147-21kworker/u8:3+flush-259:011:30:252
1172612992016,3cyclictest1338135-21kworker/u8:2+flush-259:010:00:182
1172609992017,2cyclictest1212778-21kworker/u8:2+events_unbound08:15:201
1172609992017,2cyclictest1212778-21kworker/u8:2+events_unbound08:15:191
117260499206,4cyclictest1438823-21grep11:25:210
117260499206,4cyclictest1437233-21/usr/sbin/munin11:25:010
1172604992016,3cyclictest1188217-21apt-get07:25:130
117261799196,11cyclictest1249784-21cron08:25:013
1172617991916,2cyclictest1473259-21kworker/u8:2+flush-259:012:10:183
1172617991915,3cyclictest3000063-21rs:main3
1172617991914,4cyclictest2698871-21systemd-journal11:25:003
1172617991913,4cyclictest1244633-21latency_hist08:20:003
1172617991913,4cyclictest1244633-21latency_hist08:20:003
1172609991916,2cyclictest1333919-21kworker/u8:0+flush-259:009:50:261
1172609991916,2cyclictest1276266-21kworker/u8:1+flush-259:009:05:171
1172609991915,3cyclictest1328388-21kworker/u8:3+events_unbound09:45:151
1172609991915,3cyclictest1244258-21kworker/u8:3+events_unbound08:35:181
1172609991914,4cyclictest1458095-21apt-get11:45:141
1172609991912,5cyclictest1182836-21latency_hist07:20:011
117260499195,5cyclictest1307088-21date09:20:000
117260499194,5cyclictest1411143-21wc11:00:010
117260499193,5cyclictest1474456-21cut12:00:190
117260499192,6cyclictest1259190-21modprobe08:30:200
1172604991916,2cyclictest1514409-21cron12:40:000
1172604991916,2cyclictest0-21swapper/009:30:000
1172617991816,1cyclictest0-21swapper/312:10:013
1172617991815,2cyclictest1339192-21irqcore09:50:173
1172617991814,3cyclictest1338135-21kworker/u8:2+flush-259:010:05:193
1172617991814,3cyclictest1193112-21kworker/u8:4+flush-259:007:50:173
1172617991812,4cyclictest1286338-21latency_hist09:00:003
1172617991810,7cyclictest1192586-21unixbench-2d07:25:253
1172617991810,6cyclictest0-21swapper/310:10:273
117261799180,3cyclictest1259756-21chrt08:31:133
1172612991815,2cyclictest1405593-21kworker/u8:1+events_unbound10:55:172
1172612991815,2cyclictest1359368-21kworker/u8:4+flush-259:010:15:252
1172612991815,2cyclictest1276266-21kworker/u8:1+flush-259:009:05:192
1172612991815,2cyclictest1276266-21kworker/u8:1+flush-259:009:05:012
1172612991815,2cyclictest1212778-21kworker/u8:2+flush-259:008:45:262
1172612991814,3cyclictest1338135-21kworker/u8:2+events_unbound10:50:232
1172609991816,2cyclictest1411090-21kworker/u8:2+events_unbound11:15:171
1172609991815,2cyclictest1294011-21kworker/u8:0+flush-259:009:35:261
117260499185,2cyclictest1358236-21ntpq10:05:230
117260499184,5cyclictest1204602-21sed07:40:170
117260499184,5cyclictest1180533-21cut07:15:180
117260499184,4cyclictest1265883-21cat08:40:180
117260499184,10cyclictest1426081-21ntpq11:10:240
117260499183,7cyclictest1479774-21cut12:05:190
117260499183,7cyclictest1243175-21cut08:15:200
117260499183,7cyclictest1243175-21cut08:15:190
117260499182,5cyclictest1254073-21ld08:25:230
1172604991813,3cyclictest1235567-21tail08:10:200
117261799173,12cyclictest1366416-21tr10:15:203
117261799172,12cyclictest0-21swapper/311:10:013
1172617991715,1cyclictest1265246-21cron08:40:013
1172617991714,2cyclictest1260096-21kworker/u8:0+flush-259:008:50:013
117261799171,2cyclictest1426070-21ntpq11:10:233
117261799171,2cyclictest1233360-21ntpq08:05:223
117261799171,14cyclictest2698871-21systemd-journal09:35:013
117261799170,3cyclictest1311947-21chrt09:21:543
1172612991715,2cyclictest1442592-21kworker/u8:4+flush-259:011:35:242
1172612991714,2cyclictest1338135-21kworker/u8:2+events_unbound10:40:152
1172612991714,2cyclictest1294011-21kworker/u8:0+events_unbound09:35:172
1172612991710,5cyclictest1343667-21latency_hist09:55:012
1172609991714,2cyclictest1338135-21kworker/u8:2+flush-259:010:20:201
1172609991714,2cyclictest1296889-21kworker/u8:2+events_unbound09:10:161
1172609991714,2cyclictest1254742-21kworker/u8:1+events_unbound08:40:241
1172609991714,2cyclictest1212778-21kworker/u8:2+events_unbound08:00:241
1172609991714,2cyclictest1212778-21kworker/u8:2+events_unbound07:55:241
1172609991714,2cyclictest1129334-21kworker/u8:0+events_unbound08:20:211
1172609991713,3cyclictest1338135-21kworker/u8:2+events_unbound10:25:141
1172609991712,3cyclictest1422007-21kworker/u8:0+flush-259:011:10:161
117260499174,5cyclictest1218280-21timerwakeupswit07:50:260
117260499174,4cyclictest1488192-21cat12:10:250
117260499174,4cyclictest1364105-21modprobe10:12:190
117260499174,2cyclictest1295735-21ntpq09:05:240
117260499173,5cyclictest1383841-21cut10:30:190
117260499173,5cyclictest1300366-21cut09:10:190
117260499173,4cyclictest1368853-21cat10:15:270
117260499173,4cyclictest1343597-21modprobe09:54:190
117260499172,5cyclictest1330586-21cut09:40:170
1172604991714,2cyclictest1395667-21turbostat10:45:000
117261799165,9cyclictest1291119-21vmstat09:00:273
117261799162,9cyclictest1395930-21apt-get10:45:153
117261799162,11cyclictest1406209-21apt-get10:55:123
117261799162,10cyclictest1224260-21apt-get08:00:143
117261799161,6cyclictest1306080-21ntpq09:15:213
117261799161,5cyclictest492-21dbus-daemon08:10:213
117261799161,5cyclictest1456968-21ntpq11:40:253
117261799161,5cyclictest1442353-21awk11:30:003
117261799161,4cyclictest2698871-21systemd-journal08:45:003
117261799161,3cyclictest1509270-21cron12:35:003
117261799161,3cyclictest1203380-21cron07:39:593
117261799161,3cyclictest1193764-21ps07:30:173
1172617991612,3cyclictest1493606-21kworker/u8:0+flush-259:012:25:193
117261799161,1cyclictest1212780-21ntpq07:45:213
117261799161,13cyclictest1489644-21sed12:15:193
117261799161,12cyclictest0-21swapper/309:10:123
117261799160,7cyclictest2698871-21systemd-journal09:45:013
117261799160,4cyclictest1452768-21sort11:40:003
117261799160,3cyclictest1296869-21date09:10:003
117261799160,3cyclictest1180234-21kthreadcore07:15:183
1172612991614,1cyclictest1296889-21kworker/u8:2+events_unbound09:10:142
1172612991613,2cyclictest1244258-21kworker/u8:3+flush-259:008:35:142
1172612991611,4cyclictest1494122-21apt-get12:20:142
1172609991613,2cyclictest1296889-21kworker/u8:2+events_unbound09:30:181
1172609991612,3cyclictest2307707-21kworker/u8:5+flush-259:007:10:171
1172609991612,3cyclictest1493606-21kworker/u8:0+flush-259:012:35:201
1172609991612,3cyclictest1338135-21kworker/u8:2+flush-259:010:10:251
1172609991612,3cyclictest1294011-21kworker/u8:0+flush-259:009:20:151
1172609991612,3cyclictest1276266-21kworker/u8:1+flush-259:008:55:241
1172609991612,3cyclictest1276266-21kworker/u8:1+events_unbound08:50:231
117260499165,2cyclictest1337589-21ntpq09:45:210
117260499164,2cyclictest1275193-21ntpq08:45:220
117260499163,4cyclictest1446479-21sshd11:30:190
117261799152,9cyclictest1390747-21apt-get10:40:143
117261799152,2cyclictest1380488-21apt-get10:30:123
117261799151,4cyclictest1204067-21irqcore07:40:173
117261799151,3cyclictest1494347-21cut12:20:163
117261799151,3cyclictest1401050-21chrt10:50:143
117261799151,3cyclictest1277327-21tr08:50:183
117261799151,2cyclictest1468397-21apt-get11:55:143
117261799151,2cyclictest14426302sleep311:30:133
117261799151,2cyclictest1389591-21ntpq10:35:233
117261799151,11cyclictest0-21swapper/312:00:143
117261799150,5cyclictest1312623-21tr09:25:143
117261799150,5cyclictest1219224-21cpuspeed_turbos07:55:133
117261799150,4cyclictest1250213-21tr08:25:153
117261799150,4cyclictest1183230-21grep07:20:153
117261299153,7cyclictest1182844-21munin-run07:20:012
1172612991512,3cyclictest1473259-21kworker/u8:2+flush-259:012:30:202
1172612991512,3cyclictest1473259-21kworker/u8:2+events_unbound12:10:232
1172612991512,2cyclictest1411090-21kworker/u8:2+flush-259:011:25:262
1172612991512,2cyclictest1250242-21fschecks_count08:25:162
1172612991511,3cyclictest1129334-21kworker/u8:0+flush-259:008:20:182
1172609991512,2cyclictest1390717-21kworker/u8:4+events_unbound10:50:141
1172609991511,3cyclictest1462669-21kworker/u8:1+events_unbound11:50:421
1172609991511,3cyclictest1427147-21kworker/u8:3+flush-259:011:55:211
1172609991511,3cyclictest1338135-21kworker/u8:2+events_unbound10:40:211
1172609991511,3cyclictest1212778-21kworker/u8:2+events_unbound08:30:151
117260499157,7cyclictest1203384-21awk07:39:590
117260499157,6cyclictest1213801-21latency_hist07:50:000
117260499154,2cyclictest1420533-21ntpq11:05:220
117260499154,10cyclictest1307235-21modprobe09:20:050
117260499153,2cyclictest1415348-21ntpq11:00:230
117260499152,4cyclictest1430080-21grep11:15:200
117260499152,4cyclictest1278084-21grep08:50:180
117260499151,4cyclictest1347962-21gcc09:55:240
1172604991513,1cyclictest60650irq/133-enp0s29f1:rx-011:38:110
117261799140,4cyclictest1172525-21latency_hist07:10:023
117261299149,3cyclictest2698871-21systemd-journal11:49:202
1172612991411,2cyclictest1338135-21kworker/u8:2+flush-259:010:35:262
1172612991411,2cyclictest0-21swapper/211:55:012
117261299141,11cyclictest1478429-21awk12:05:002
117260999148,5cyclictest0-21swapper/112:25:001
1172609991411,2cyclictest1483587-21idleruntime-cro12:10:011
1172609991411,2cyclictest1129334-21kworker/u8:0+flush-259:007:20:261
1172609991410,3cyclictest1473259-21kworker/u8:2+flush-259:012:10:261
1172609991410,3cyclictest1452345-21kworker/u8:0+flush-259:011:40:281
1172609991410,3cyclictest1442592-21kworker/u8:4+flush-259:011:35:191
1172609991410,3cyclictest1427147-21kworker/u8:3+events_unbound11:30:221
1172609991410,3cyclictest1302164-21cpuspeed_turbos09:15:131
1172609991410,3cyclictest1193112-21kworker/u8:4+flush-259:007:50:261
117260499149,3cyclictest1495379-21tr12:20:190
117260499148,5cyclictest1173103-21fschecks_time07:10:150
117260499146,6cyclictest1505138-21cut12:30:180
117260499143,2cyclictest1405088-21ntpq10:50:220
117260499143,2cyclictest1228256-21ntpq08:00:230
117260499143,2cyclictest1223175-21ntpq07:55:220
117260499142,7cyclictest1290599-21ntpq09:00:220
117260499142,2cyclictest1249803-21munin-run08:25:000
1172604991413,1cyclictest60650irq/133-enp0s29f1:rx-012:15:230
1172604991413,1cyclictest60650irq/133-enp0s29f1:rx-011:55:170
1172604991413,1cyclictest60650irq/133-enp0s29f1:rx-011:49:030
117261299139,3cyclictest2698871-21systemd-journal12:05:452
117261299139,3cyclictest1129334-21kworker/u8:0+events_unbound07:40:242
117261299137,5cyclictest1513543-21ntpq12:35:222
1172612991310,2cyclictest1338135-21kworker/u8:2+flush-259:010:05:222
1172612991310,2cyclictest1296889-21kworker/u8:2+flush-259:009:25:212
1172612991310,2cyclictest0-21swapper/210:35:002
117260999139,3cyclictest1390717-21kworker/u8:4+events_unbound10:55:261
117260999139,3cyclictest1328388-21kworker/u8:3+flush-259:009:40:131
117260999139,3cyclictest1244258-21kworker/u8:3+flush-259:008:25:211
117260999139,3cyclictest1212778-21kworker/u8:2+events_unbound08:05:151
117260999136,5cyclictest1359138-21latency_hist10:10:011
1172609991310,2cyclictest1212714-21needreboot07:45:191
117260499138,3cyclictest1197370-21ntpq07:30:200
117260499137,5cyclictest1324102-21cut09:35:190
117260499132,10cyclictest1395930-21apt-get10:45:140
117261299129,2cyclictest1411749-21irqcore11:00:182
117261299128,3cyclictest1338135-21kworker/u8:2+flush-259:010:20:222
117260999129,2cyclictest1208078-21unixbench_multi07:40:261
117260999128,3cyclictest2307235-21kworker/u8:1+flush-259:007:39:491
117260999128,3cyclictest1276266-21kworker/u8:1+flush-259:009:00:251
117260999128,3cyclictest1187990-21kworker/u8:3+flush-259:007:25:011
117261299118,2cyclictest1333919-21kworker/u8:0+flush-259:009:45:252
117261299118,2cyclictest1228704-21unixbench_singl08:00:262
117261299118,2cyclictest0-21swapper/209:15:252
117261299108,2cyclictest1193112-21kworker/u8:4+events_unbound07:50:152
11726129996,2cyclictest0-21swapper/210:45:222
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional