You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 11:05

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackeslot4s.osadl.org (updated Fri Nov 21, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1187609994438,4cyclictest1245474-21/usr/sbin/munin20:05:151
1187611994033,5cyclictest2698871-21systemd-journal20:54:592
1187605994037,2cyclictest1219020-21kworker/u8:1+flush-259:020:45:000
1187605993834,3cyclictest1400188-21kworker/u8:2+events_unbound22:50:140
1187605993834,3cyclictest1328181-21kworker/u8:1+events_unbound21:35:150
1187616993734,2cyclictest1219020-21kworker/u8:1+flush-259:019:40:223
1187605993734,2cyclictest1379588-21kworker/u8:1+events_unbound23:05:250
1187605993633,2cyclictest473571-21kworker/u8:2+events_unbound19:10:190
1187605993633,2cyclictest1338231-21kworker/u8:2+events_unbound21:45:260
1187605993633,2cyclictest1219020-21kworker/u8:1+flush-259:020:30:260
1187605993632,3cyclictest1219020-21kworker/u8:1+flush-259:019:55:000
1187609993529,4cyclictest1420908-21latency_hist22:55:011
1187605993532,2cyclictest1379588-21kworker/u8:1+flush-259:023:20:190
1187605993532,2cyclictest1353669-21kworker/u8:0+events_unbound22:20:180
1187605993532,2cyclictest1265897-21kworker/u8:3+flush-259:021:00:230
1187605993431,2cyclictest1379588-21kworker/u8:1+events_unbound22:25:180
1187605993431,2cyclictest1265897-21kworker/u8:3+flush-259:020:50:150
1187605993431,2cyclictest1235134-21kworker/u8:0+events_unbound19:55:240
1187605993428,4cyclictest1476667-21ntpq23:45:210
1187611993330,2cyclictest1219020-21kworker/u8:1+flush-259:019:50:162
1187605993331,1cyclictest1235134-21kworker/u8:0+events_unbound20:00:160
1187605993329,3cyclictest1265897-21kworker/u8:3+flush-259:021:30:190
1187605993329,3cyclictest1265897-21kworker/u8:3+events_unbound20:45:180
1187605993229,2cyclictest1353669-21kworker/u8:0+events_unbound21:55:170
1187605993228,3cyclictest1208932-21apt-get19:30:130
1187616993128,2cyclictest1338231-21kworker/u8:2+flush-259:022:00:243
1187609993126,3cyclictest1379588-21kworker/u8:1+flush-259:023:05:281
1187609993124,5cyclictest1207799-21ntpq19:25:221
1187605993128,2cyclictest1353669-21kworker/u8:0+events_unbound22:30:170
1187605993127,3cyclictest1410659-21kworker/u8:0+events_unbound23:56:560
1187605993127,3cyclictest1219020-21kworker/u8:1+events_unbound20:20:220
1187616993025,4cyclictest1166778-21kworker/u8:4+events_unbound19:15:173
1187611993024,4cyclictest1235330-21diskstats19:55:142
1187605993027,2cyclictest1296715-21kworker/u8:2+events_unbound21:20:260
1187611992922,5cyclictest1291535-21latency_hist20:50:012
1187605992921,6cyclictest1404223-21ntpq22:35:230
1187616992824,3cyclictest1456781-21kworker/u8:2+events_unbound00:10:003
1187616992824,3cyclictest1312706-21base6421:10:013
1187616992824,3cyclictest1265897-21kworker/u8:3+events_unbound21:05:003
1187609992824,3cyclictest1456781-21kworker/u8:2+flush-259:000:05:161
1187605992825,2cyclictest1456781-21kworker/u8:2+events_unbound00:00:220
1187605992825,2cyclictest1379588-21kworker/u8:1+events_unbound22:15:220
1187605992824,3cyclictest1493031-21kworker/u8:1+events_unbound00:07:270
1187605992821,5cyclictest1197889-21latency_hist19:20:010
1187616992723,3cyclictest1389658-21kworker/u8:3+flush-259:022:40:003
1187616992723,2cyclictest1400188-21kworker/u8:2+events_unbound22:55:013
1187611992719,5cyclictest1343341-21latency_hist21:40:012
1187605992725,2cyclictest1265897-21kworker/u8:3+events_unbound21:10:250
1187605992724,2cyclictest1410659-21kworker/u8:0+events_unbound23:15:180
1187616992623,2cyclictest1493031-21kworker/u8:1+flush-259:000:15:293
1187616992623,2cyclictest1353669-21kworker/u8:0+flush-259:022:30:163
1187611992622,3cyclictest1338231-21kworker/u8:2+flush-259:022:10:002
1187611992620,5cyclictest1384946-21diskstats22:20:162
1187609992623,2cyclictest1456781-21kworker/u8:2+flush-259:000:10:191
1187609992623,2cyclictest1265897-21kworker/u8:3+events_unbound20:40:001
1187609992622,3cyclictest1456781-21kworker/u8:2+flush-259:000:30:011
1187609992622,3cyclictest1456781-21kworker/u8:2+flush-259:000:30:001
1187609992620,4cyclictest1202651-21kworker/u8:0+flush-259:019:35:181
1187605992623,2cyclictest1456781-21kworker/u8:2+events_unbound23:30:190
1187605992623,2cyclictest1456781-21kworker/u8:2+events_unbound00:29:560
1187605992623,2cyclictest1456781-21kworker/u8:2+events_unbound00:29:560
1187605992623,2cyclictest1338231-21kworker/u8:2+events_unbound21:40:260
1187605992623,2cyclictest1202651-21kworker/u8:0+events_unbound19:35:160
1187605992621,4cyclictest1328190-21/usr/sbin/munin21:25:140
1187605992620,4cyclictest3000063-21rs:main0
1187616992521,3cyclictest1456781-21kworker/u8:2+flush-259:023:45:243
1187611992521,3cyclictest1198124-21/usr/sbin/munin19:20:142
1187611992520,4cyclictest1379588-21kworker/u8:1+flush-259:023:05:212
1187611992520,4cyclictest1379588-21kworker/u8:1+events_unbound23:20:232
1187611992519,4cyclictest1235134-21kworker/u8:0+flush-259:020:15:212
1187609992518,5cyclictest1378476-21ntpq22:10:261
1187605992522,2cyclictest1493031-21kworker/u8:1+events_unbound00:32:220
1187605992522,2cyclictest1493031-21kworker/u8:1+events_unbound00:32:210
1187605992522,2cyclictest1353669-21kworker/u8:0+events_unbound21:51:550
1187616992421,2cyclictest1353669-21kworker/u8:0+flush-259:022:20:163
1187616992421,2cyclictest1235134-21kworker/u8:0+events_unbound20:15:163
118761699240,23cyclictest1-21systemd23:05:013
1187611992420,3cyclictest1219020-21kworker/u8:1+flush-259:020:05:202
1187611992419,3cyclictest1219020-21kworker/u8:1+flush-259:019:40:142
1187611992418,4cyclictest1498387-21/usr/sbin/munin00:10:172
1187611992418,4cyclictest1208676-21latency_hist19:30:012
1187609992421,2cyclictest1219020-21kworker/u8:1+flush-259:019:45:001
1187609992420,3cyclictest1244885-21kworker/u8:2+flush-259:020:15:281
1187609992419,3cyclictest1166778-21kworker/u8:4+flush-259:019:15:011
1187605992419,4cyclictest1467492-21/usr/sbin/munin23:40:150
1187616992319,3cyclictest1410659-21kworker/u8:0+events_unbound23:38:563
1187616992319,3cyclictest1353669-21kworker/u8:0+flush-259:022:10:193
1187616992319,3cyclictest1322837-21kworker/u8:0+flush-259:021:20:143
1187616992318,3cyclictest1379588-21kworker/u8:1+events_unbound22:25:253
1187605992320,2cyclictest1456781-21kworker/u8:2+events_unbound00:20:170
1187605992320,2cyclictest1379588-21kworker/u8:1+events_unbound22:55:210
1187605992320,2cyclictest1244885-21kworker/u8:2+events_unbound20:05:150
1187605992319,3cyclictest1312144-21gpgv21:10:000
1187605992318,4cyclictest1235134-21kworker/u8:0+events_unbound20:16:580
1187605992316,5cyclictest1441508-21latency_hist23:15:010
1187616992219,2cyclictest1421142-21kworker/u8:3+flush-259:023:10:013
1187616992218,3cyclictest473571-21kworker/u8:2+flush-259:019:10:253
1187616992218,3cyclictest1456781-21kworker/u8:2+events_unbound23:30:133
1187616992218,3cyclictest1421142-21kworker/u8:3+events_unbound23:26:563
1187616992218,3cyclictest1244885-21kworker/u8:2+events_unbound20:05:023
1187616992218,3cyclictest1219020-21kworker/u8:1+flush-259:019:52:563
1187616992218,3cyclictest1202651-21kworker/u8:0+flush-259:019:30:213
1187616992217,4cyclictest1338231-21kworker/u8:2+events_unbound21:55:283
1187616992217,4cyclictest1265897-21kworker/u8:3+flush-259:021:05:193
1187609992210,6cyclictest1513517-21munin-run00:25:001
1187605992219,2cyclictest473571-21kworker/u8:2+events_unbound19:20:150
1187605992219,2cyclictest1379588-21kworker/u8:1+events_unbound23:00:210
1187605992219,2cyclictest1166778-21kworker/u8:4+events_unbound19:41:480
1187605992218,3cyclictest1296715-21kworker/u8:2+events_unbound20:56:550
1187605992217,3cyclictest3000063-21rs:main0
1187616992118,2cyclictest1244885-21kworker/u8:2+events_unbound20:25:193
1187616992117,3cyclictest1328181-21kworker/u8:1+flush-259:021:50:003
1187616992116,4cyclictest1493031-21kworker/u8:1+events_unbound00:25:243
1187616992116,4cyclictest1493031-21kworker/u8:1+events_unbound00:25:233
1187616992115,4cyclictest1219020-21kworker/u8:1+events_unbound20:40:303
1187611992115,4cyclictest2698871-21systemd-journal22:42:542
1187611992115,4cyclictest2698871-21systemd-journal00:25:002
1187609992116,4cyclictest1481774-21ntpq23:50:221
1187605992118,2cyclictest1244885-21kworker/u8:2+events_unbound20:25:540
1187605992116,3cyclictest1374878-21irqcore22:10:220
1187616992017,2cyclictest1410659-21kworker/u8:0+flush-259:023:15:163
1187611992017,2cyclictest1379588-21kworker/u8:1+flush-259:023:25:182
1187611992017,2cyclictest1265897-21kworker/u8:3+flush-259:020:40:002
1187611992016,3cyclictest1219020-21kworker/u8:1+events_unbound20:30:282
1187611992014,4cyclictest1348749-21apt-get21:45:122
1187609992013,5cyclictest2698871-21systemd-journal21:55:001
1187605992017,2cyclictest1493031-21kworker/u8:1+events_unbound00:15:290
1187605992013,5cyclictest1374197-21latency_hist22:10:010
1187616991915,3cyclictest1456781-21kworker/u8:2+events_unbound00:10:183
1187616991915,3cyclictest1400188-21kworker/u8:2+events_unbound22:48:563
1187616991915,3cyclictest1379588-21kworker/u8:1+events_unbound23:20:253
1187616991914,4cyclictest1219020-21kworker/u8:1+events_unbound20:35:243
1187616991912,5cyclictest1295812-21ntpq20:50:233
1187611991915,3cyclictest1379588-21kworker/u8:1+flush-259:022:45:212
1187611991915,3cyclictest1379588-21kworker/u8:1+events_unbound22:55:152
1187611991912,5cyclictest2698871-21systemd-journal20:45:002
1187609991913,4cyclictest1359023-21/usr/sbin/munin21:55:241
1187605991916,2cyclictest1265897-21kworker/u8:3+events_unbound20:38:350
1187605991915,3cyclictest1410659-21kworker/u8:0+flush-259:023:40:000
1187616991814,3cyclictest1456781-21kworker/u8:2+events_unbound23:55:233
1187616991814,3cyclictest1353669-21kworker/u8:0+events_unbound22:19:563
1187616991814,3cyclictest1328181-21kworker/u8:1+events_unbound21:35:173
1187616991814,3cyclictest1265897-21kworker/u8:3+flush-259:020:55:253
1187616991814,3cyclictest1244885-21kworker/u8:2+events_unbound20:11:553
1187616991814,3cyclictest1235134-21kworker/u8:0+events_unbound19:55:193
1187611991816,1cyclictest0-21swapper/223:40:002
1187611991814,3cyclictest1493031-21kworker/u8:1+events_unbound00:15:272
1187611991814,3cyclictest1265897-21kworker/u8:3+flush-259:021:25:192
1187611991814,3cyclictest1166778-21kworker/u8:4+events_unbound19:15:212
1187611991812,4cyclictest1518678-21latency_hist00:30:002
1187611991812,4cyclictest1487780-21latency_hist00:00:002
1187609991815,2cyclictest1307618-21irqcore21:05:171
1187609991812,4cyclictest1518945-21apt-get00:30:121
1187609991812,4cyclictest1518945-21apt-get00:30:121
1187609991812,4cyclictest1312773-21/usr/sbin/munin21:10:141
1187609991811,5cyclictest1456951-21latency_hist23:30:001
1187605991814,3cyclictest1493031-21kworker/u8:1+events_unbound00:38:050
1187605991814,3cyclictest1456781-21kworker/u8:2+events_unbound23:52:160
1187605991814,3cyclictest1296715-21kworker/u8:2+events_unbound21:15:240
1187616991714,2cyclictest1410659-21kworker/u8:0+flush-259:023:50:143
1187616991713,3cyclictest1379588-21kworker/u8:1+events_unbound23:10:183
1187616991713,3cyclictest1265897-21kworker/u8:3+flush-259:021:25:163
1187611991714,2cyclictest1456781-21kworker/u8:2+flush-259:023:30:172
1187611991713,3cyclictest1389658-21kworker/u8:3+flush-259:022:25:272
1187611991710,5cyclictest2698871-21systemd-journal22:35:002
1187611991710,5cyclictest2698871-21systemd-journal22:15:002
1187609991714,2cyclictest1410659-21kworker/u8:0+events_unbound23:12:271
1187609991713,3cyclictest1265897-21kworker/u8:3+flush-259:021:35:261
1187605991714,2cyclictest1328181-21kworker/u8:1+events_unbound22:00:290
1187605991713,3cyclictest1421142-21kworker/u8:3+events_unbound23:28:570
1187616991614,1cyclictest1219020-21kworker/u8:1+events_unbound19:45:143
1187616991613,2cyclictest1493031-21kworker/u8:1+flush-259:000:20:243
1187616991613,2cyclictest1328181-21kworker/u8:1+flush-259:021:50:213
1187611991612,3cyclictest1265897-21kworker/u8:3+flush-259:020:28:502
1187611991612,3cyclictest1219020-21kworker/u8:1+events_unbound19:45:182
1187611991611,4cyclictest1338231-21kworker/u8:2+events_unbound21:40:212
118760999169,5cyclictest1461272-21ntpq23:30:221
1187609991612,3cyclictest1338231-21kworker/u8:2+flush-259:021:40:181
1187609991611,4cyclictest1192984-21apt-get19:15:121
1187616991512,2cyclictest1296715-21kworker/u8:2+flush-259:021:15:213
1187616991511,3cyclictest1493031-21kworker/u8:1+events_unbound00:39:563
1187616991511,3cyclictest1456781-21kworker/u8:2+events_unbound00:04:563
1187616991511,3cyclictest1379588-21kworker/u8:1+events_unbound22:40:263
1187616991511,3cyclictest1338231-21kworker/u8:2+events_unbound21:40:153
1187616991511,3cyclictest1265897-21kworker/u8:3+flush-259:021:30:283
1187616991511,3cyclictest1203697-21kworker/u8:3+events_unbound19:25:253
1187616991511,3cyclictest1192725-21kworker/u8:1+events_unbound19:24:563
1187611991512,2cyclictest492-21dbus-daemon00:30:002
1187611991511,3cyclictest1405462-21latency_hist22:40:012
1187611991511,3cyclictest1353669-21kworker/u8:0+events_unbound21:55:202
118760999159,4cyclictest1379588-21kworker/u8:1+flush-259:022:30:061
1187609991512,2cyclictest0-21swapper/122:50:001
1187609991511,3cyclictest1503529-21apt-get00:15:141
1187605991513,1cyclictest60650irq/133-enp0s29f1:rx-020:10:230
1187605991511,3cyclictest1410902-21apt-get22:45:120
1187605991511,3cyclictest1203697-21kworker/u8:3+events_unbound19:26:250
1187616991411,2cyclictest1379588-21kworker/u8:1+flush-259:022:55:153
1187616991411,2cyclictest1263269-21kthreadcore20:20:203
1187616991410,3cyclictest1235134-21kworker/u8:0+events_unbound20:00:253
118761199142,7cyclictest1219003-21awk19:40:012
1187611991411,2cyclictest1522881-21ntp_kernel_err00:30:212
1187611991411,2cyclictest1522881-21ntp_kernel_err00:30:212
1187611991411,2cyclictest1311628-21unixbench_multi21:05:242
1187611991411,2cyclictest1235134-21kworker/u8:0+flush-259:020:00:272
1187611991410,3cyclictest1328181-21kworker/u8:1+events_unbound21:35:002
118760999149,4cyclictest1317959-21apt-get21:15:111
1187609991411,2cyclictest1390922-21kthreadcore22:25:201
1187609991410,3cyclictest1379588-21kworker/u8:1+flush-259:022:55:261
1187609991410,3cyclictest1338231-21kworker/u8:2+flush-259:022:00:171
1187605991411,2cyclictest1493031-21kworker/u8:1+events_unbound00:11:570
118761199139,3cyclictest1312544-21cat21:10:012
118761199139,3cyclictest1265897-21kworker/u8:3+flush-259:020:55:322
1187611991311,1cyclictest0-21swapper/221:00:222
1187611991310,2cyclictest1456781-21kworker/u8:2+flush-259:023:45:242
1187611991310,2cyclictest1328181-21kworker/u8:1+events_unbound21:50:182
1187611991310,2cyclictest1219020-21kworker/u8:1+flush-259:020:20:212
118760999139,2cyclictest0-21swapper/122:20:001
1187609991310,2cyclictest1410659-21kworker/u8:0+flush-259:023:15:241
1187609991310,2cyclictest1400288-21cpuspeed_turbos22:35:141
1187609991310,2cyclictest1296254-21unixbench_singl20:50:251
1187609991310,2cyclictest1285509-21ntp_kernel_err20:40:241
1187609991310,2cyclictest1271684-21irqcore20:30:221
1187609991310,2cyclictest1270109-21ntp_kernel_pll_20:25:231
118761199129,2cyclictest473571-21kworker/u8:2+events_unbound19:10:162
118761199129,2cyclictest1410659-21kworker/u8:0+events_unbound23:40:142
118761199128,3cyclictest1456781-21kworker/u8:2+flush-259:000:06:592
118761199128,3cyclictest1456781-21kworker/u8:2+events_unbound00:00:212
118761199128,3cyclictest1436658-21apt-get23:10:122
118760999128,3cyclictest1235366-21fschecks_count19:55:151
118760999128,3cyclictest1233675-21grep19:50:221
118760999126,4cyclictest2698871-21systemd-journal23:20:011
118760999124,1cyclictest12097262cut19:30:171
118761699119,1cyclictest0-21swapper/323:40:183
118761699118,2cyclictest1338231-21kworker/u8:2+events_unbound22:05:253
118761199118,2cyclictest1379588-21kworker/u8:1+flush-259:022:15:262
118760999117,3cyclictest2698871-21systemd-journal21:29:551
118760999117,3cyclictest1456781-21kworker/u8:2+flush-259:000:00:181
118760999115,4cyclictest1333332-21/usr/sbin/munin21:30:231
118761199106,3cyclictest1477919-21cut23:50:142
118761199106,3cyclictest1244885-21kworker/u8:2+events_unbound20:10:232
118761199105,3cyclictest1528969-21cron00:40:002
118761199101,8cyclictest1369078-21latency_hist22:05:012
118761199101,7cyclictest1323226-21df21:20:152
118760999108,1cyclictest0-21swapper/122:05:131
118760999107,2cyclictest1348921-21fschecks_count21:45:161
118760999107,2cyclictest1297086-21cpuspeed_turbos20:55:191
118760999107,2cyclictest1250879-21fschecks_count20:10:171
118760999106,2cyclictest1477524-21latency_hist23:50:011
118760999106,2cyclictest1427689-21kthreadcore23:00:171
118760999103,6cyclictest1387056-21kthreadcore22:20:211
11876169995,3cyclictest1166778-21kworker/u8:4+events_unbound19:35:273
11876119996,2cyclictest1209024-21cpuspeed_turbos19:30:132
11876099996,2cyclictest1379588-21kworker/u8:1+flush-259:022:40:171
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional