You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-09 - 07:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #e, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackeslot4s.osadl.org (updated Fri May 09, 2025 00:44:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1259640994526,14cyclictest1312716-21latency_hist20:00:012
1259633993630,4cyclictest1265105-21/usr/sbin/munin19:15:150
1259644993327,4cyclictest3000860-21systemd-journal21:25:003
1259644993226,4cyclictest1480421-21latency_hist22:40:013
1259644992925,3cyclictest1338771-21kworker/u8:1+flush-259:021:50:003
1259644992715,10cyclictest1338771-21kworker/u8:1+flush-259:020:55:003
1259644992715,10cyclictest1338771-21kworker/u8:1+flush-259:020:55:003
1259637992721,4cyclictest3000860-21systemd-journal19:27:011
1259633992723,3cyclictest1338771-21kworker/u8:1+events_unbound21:35:050
1259633992722,3cyclictest1-21systemd22:31:190
1259637992622,3cyclictest1485699-21gpgv22:45:001
1259644992518,5cyclictest1343948-21latency_hist20:29:593
1259644992418,4cyclictest1275354-21latency_hist19:25:013
1259637992418,4cyclictest3000860-21systemd-journal22:28:181
1259644992318,3cyclictest1543822-21kworker/u8:4+events_unbound00:40:003
1259644992317,4cyclictest1528262-21/usr/sbin/munin23:25:193
1259644992316,5cyclictest3000860-21systemd-journal00:05:183
1259640992317,4cyclictest1496388-21awk22:55:002
1259633992320,2cyclictest1391294-21kworker/u8:2+events_unbound21:44:180
1259644992217,3cyclictest1543822-21kworker/u8:4+flush-259:000:10:233
1259644992216,4cyclictest1318350-21diskstats20:05:153
125964099229,12cyclictest1474706-21unixbench_multi22:30:242
1259640992219,2cyclictest1391294-21kworker/u8:2+flush-259:021:30:002
1259640992216,4cyclictest1517590-21latency_hist23:15:012
1259637992217,3cyclictest1501574-21apt-get23:00:011
1259633992218,3cyclictest1543634-21kworker/u8:3+events_unbound00:39:550
1259633992215,5cyclictest1323154-21latency_hist20:09:590
1259644992117,3cyclictest1554075-21journal-offline23:50:003
1259644992115,5cyclictest1380281-21latency_hist21:05:013
1259640992116,3cyclictest1349114-21latency_hist20:35:002
1259633992118,2cyclictest1582813-21kworker/u8:0+events_unbound00:24:560
125964099205,6cyclictest1548865-21basename23:45:002
1259640992016,3cyclictest1318151-21kworker/u8:3+flush-259:020:10:152
1259633992017,2cyclictest1323428-21apt-get20:10:130
1259644991915,3cyclictest1364674-21idleruntime-cro20:50:003
1259644991915,3cyclictest1259816-21kworker/u8:1+flush-259:019:25:183
1259644991914,4cyclictest1297294-21kworker/u8:2+events_unbound20:05:013
1259644991912,5cyclictest3000860-21systemd-journal23:17:173
1259644991911,5cyclictest1585289-21latency_hist00:20:013
125964099191,6cyclictest1522779-21cron23:20:002
125964099190,10cyclictest1549919-21irqcore23:45:172
125963799196,11cyclictest3000860-21systemd-journal22:15:001
1259633991914,3cyclictest1297309-21/usr/sbin/munin19:45:160
1259633991913,4cyclictest1369057-21ntpq20:50:200
1259633991913,4cyclictest1369057-21ntpq20:50:200
125964499188,9cyclictest1354962-21irqcore20:40:163
1259644991813,3cyclictest1338771-21kworker/u8:1+flush-259:021:35:163
125964099185,5cyclictest1479550-21ntpq22:35:242
125964099181,5cyclictest1302556-21sshd19:50:112
125963799189,7cyclictest0-21swapper/100:20:201
1259637991814,3cyclictest1501698-21gpgv23:00:001
1259637991813,3cyclictest1517605-21kworker/u8:0+flush-259:000:05:161
1259637991812,4cyclictest3000860-21systemd-journal20:05:011
125963799180,16cyclictest1427918-21apt21:50:001
125963399181,3cyclictest1487030-21cut22:45:180
1259644991710,4cyclictest1417754-21/usr/sbin/munin21:40:153
125964099176,2cyclictest1564471-21latency_hist00:00:002
1259637991714,2cyclictest1254428-21kworker/u8:0+flush-259:020:00:011
125963799171,2cyclictest1442676-21ntpq22:00:241
1259637991711,4cyclictest1569668-21latency_hist00:05:011
1259637991710,5cyclictest0-21swapper/120:40:111
125963399173,10cyclictest1558670-21timerandwakeup23:50:230
1259633991714,1cyclictest191rcu_preempt20:40:200
125963399170,13cyclictest1333239-21runrttasks20:15:520
125964099163,4cyclictest1375092-21latency_hist21:00:012
1259640991613,2cyclictest1422733-21kworker/u8:0+flush-259:022:10:162
1259640991612,3cyclictest3000860-21systemd-journal20:50:002
125964099160,4cyclictest1427913-21awk21:50:002
125963799169,6cyclictest0-21swapper/120:45:271
125963799165,9cyclictest1260706-21irqcore19:10:161
125963799164,11cyclictest1401844-21cut21:25:191
125963799164,10cyclictest1328537-21cat20:15:001
125963799164,10cyclictest1302634-21cut19:50:121
125963799164,10cyclictest1-21systemd23:15:001
125963799161,6cyclictest1353446-21ntpq20:35:191
125963799161,2cyclictest1374280-21ntpq20:55:241
125963799161,2cyclictest1295769-21ntpq19:40:201
125963799161,1cyclictest1290537-21ntpq19:35:221
125963799161,13cyclictest1485581-21apt-get22:45:011
125963799160,3cyclictest1606188-21ls00:40:001
125963799160,2cyclictest1480421-21latency_hist22:40:001
125963399169,1cyclictest191rcu_preempt00:05:230
125963399163,3cyclictest1590738-21/usr/sbin/munin00:25:130
1259633991614,2cyclictest1543634-21kworker/u8:3+events_unbound23:49:570
1259633991614,1cyclictest60650irq/133-enp0s29f1:rx-023:30:260
1259633991614,1cyclictest60650irq/133-enp0s29f1:rx-021:05:190
1259633991614,1cyclictest1385498-21kworker/u8:3-ext4-rsv-conversion21:15:010
1259633991613,2cyclictest1507197-21kworker/u8:2+events_unbound23:14:550
1259633991613,2cyclictest1297294-21kworker/u8:2+events_unbound19:50:280
1259633991611,1cyclictest191rcu_preempt23:57:040
125963399160,3cyclictest16-21ksoftirqd/023:25:210
1259644991512,2cyclictest1411908-21kworker/u8:3+events_unbound22:15:193
1259644991512,2cyclictest1254428-21kworker/u8:0+events_unbound19:30:253
1259644991510,4cyclictest3000860-21systemd-journal21:17:003
125964499150,4cyclictest14419701kthreadcore22:00:203
1259640991512,2cyclictest1411908-21kworker/u8:3+flush-259:022:15:182
1259640991511,3cyclictest1595904-21/usr/sbin/munin00:30:212
1259640991510,3cyclictest3000063-21rs:main2
125963799154,9cyclictest0-21swapper/121:55:021
125963799153,11cyclictest1329621-21cut20:15:171
125963799153,10cyclictest1417518-21latency_hist21:40:011
125963799152,11cyclictest1580398-21grep00:15:161
125963799152,11cyclictest0-21swapper/100:35:011
125963799152,10cyclictest1538674-21cat23:35:131
125963799151,2cyclictest1528017-21apt23:25:011
125963799151,2cyclictest1395420-21ntpq21:15:211
125963799151,2cyclictest1389726-21ntpq21:10:221
125963799151,2cyclictest1375775-21ps21:00:181
125963799151,2cyclictest1337911-21ntpq20:20:191
125963799151,2cyclictest1301443-21ntpq19:45:231
125963799151,2cyclictest1274427-21ntpq19:20:171
125963799151,1cyclictest1410963-21ntpq21:30:211
125963799150,6cyclictest1281457-21/usr/sbin/munin19:30:151
125963799150,4cyclictest1538411-21sh23:35:001
125963799150,3cyclictest1590854-21cpuspeed_turbos00:25:141
125963799150,3cyclictest1396660-21cut21:20:151
125963799150,3cyclictest1385477-21latency_hist21:10:001
125963799150,3cyclictest1365375-21cut20:50:151
125963799150,3cyclictest1365375-21cut20:50:151
125963399159,4cyclictest1449591-21diskstats22:10:140
125963399154,7cyclictest1580709-21if_err_enp0s29f00:15:190
125963399153,2cyclictest1343031-21ntpq20:25:220
125963399152,6cyclictest835-21runrttasks20:45:060
125963399152,5cyclictest1566562-21grep00:00:220
125963399152,5cyclictest1528222-21cat23:25:010
125963399152,4cyclictest1274083-21cut19:20:160
125963399151,4cyclictest1521965-21ntpq23:15:210
125963399151,4cyclictest1431611-21grep21:50:220
1259633991513,2cyclictest1338771-21kworker/u8:1+events_unbound21:22:060
1259633991513,2cyclictest1259816-21kworker/u8:1+events_unbound19:34:350
1259633991513,1cyclictest60650irq/133-enp0s29f1:rx-023:35:250
1259633991513,1cyclictest60650irq/133-enp0s29f1:rx-022:50:200
1259633991513,1cyclictest60650irq/133-enp0s29f1:rx-022:25:210
1259633991513,1cyclictest60650irq/133-enp0s29f1:rx-020:21:250
1259633991513,1cyclictest60650irq/133-enp0s29f1:rx-020:00:190
1259633991513,1cyclictest60650irq/133-enp0s29f1:rx-019:10:180
1259633991513,1cyclictest60650irq/133-enp0s29f1:rx-000:30:210
1259633991513,1cyclictest60650irq/133-enp0s29f1:rx-000:10:250
1259633991512,2cyclictest61450irq/134-enp0s29f1:tx-021:00:180
1259633991512,2cyclictest60650irq/133-enp0s29f1:rx-023:40:180
1259633991512,2cyclictest60650irq/133-enp0s29f1:rx-021:15:210
1259633991512,2cyclictest1427933-21kworker/u8:5+events_unbound21:55:030
1259633991512,2cyclictest1338771-21kworker/u8:1+events_unbound21:31:520
1259633991512,1cyclictest191rcu_preempt20:30:170
1259633991511,2cyclictest60650irq/133-enp0s29f1:rx-020:35:100
125963399150,4cyclictest1403224-21grep21:25:230
1259644991411,2cyclictest1328406-21turbostat20:15:003
1259644991410,3cyclictest1496655-21apt-get22:55:133
1259644991410,2cyclictest0-21swapper/319:15:203
1259640991411,2cyclictest1507197-21kworker/u8:2+flush-259:023:30:012
1259640991411,2cyclictest1507197-21kworker/u8:2+events_unbound23:35:182
1259640991411,2cyclictest1497064-21grep22:55:172
125964099140,7cyclictest1449173-21munin-run22:10:002
125963799144,8cyclictest0-21swapper/122:20:131
125963799144,8cyclictest0-21swapper/100:10:201
125963799142,11cyclictest1491689-21sort22:50:151
125963799141,2cyclictest1474273-21ntpq22:30:201
125963799141,2cyclictest1421790-21ntpq21:40:211
125963799141,1cyclictest1558398-21ntpq23:50:201
125963799140,3cyclictest1559657-21cpuspeed_turbos23:55:141
125963799140,3cyclictest1549220-21cpuspeed_turbos23:45:141
125963799140,3cyclictest1544031-21grep23:40:151
125963799140,3cyclictest1518076-21sort23:15:131
125963799140,3cyclictest1512541-21ls23:10:011
125963799140,3cyclictest1512541-21ls23:10:001
125963799140,3cyclictest1448693-21unixbench_multi22:05:271
125963799140,3cyclictest1434226-21irqcore21:55:151
125963799140,3cyclictest1339253-21grep20:25:161
125963799140,3cyclictest1318602-21cut20:05:171
125963799140,3cyclictest1265236-21grep19:15:141
125963799140,2cyclictest835-21runrttasks22:15:191
125963799140,2cyclictest1528696-21ps23:25:201
125963399142,4cyclictest1502170-21find23:00:000
125963399142,3cyclictest1423903-21cut21:45:170
125963399142,2cyclictest16-21ksoftirqd/022:15:240
125963399142,2cyclictest1311868-21ntpq19:55:210
125963399141,3cyclictest1371725-21cut20:55:200
1259633991413,1cyclictest60650irq/133-enp0s29f1:rx-022:06:210
1259633991412,1cyclictest60650irq/133-enp0s29f1:rx-023:05:180
1259633991412,1cyclictest60650irq/133-enp0s29f1:rx-023:05:170
1259633991412,1cyclictest60650irq/133-enp0s29f1:rx-022:20:140
1259633991412,1cyclictest60650irq/133-enp0s29f1:rx-022:00:180
1259633991412,1cyclictest60650irq/133-enp0s29f1:rx-019:40:220
1259633991412,1cyclictest60650irq/133-enp0s29f1:rx-019:39:580
1259633991410,1cyclictest191rcu_preempt22:58:010
125963399140,9cyclictest1276328-21fw_forwarded_lo19:25:150
125964499137,4cyclictest3000860-21systemd-journal23:52:183
125964499137,4cyclictest3000860-21systemd-journal22:50:193
125964499137,4cyclictest3000063-21rs:main3
125964499137,4cyclictest1427933-21kworker/u8:5+flush-259:021:50:213
125964499136,5cyclictest1600853-21latency_hist00:35:013
1259644991310,1cyclictest0-21swapper/322:10:003
125964099139,2cyclictest0-21swapper/222:25:002
125964099133,5cyclictest1380287-21munin-run21:05:002
1259640991310,2cyclictest1467395-21tr22:25:192
1259640991310,2cyclictest1443553-21latency_hist22:05:012
1259640991310,2cyclictest1407355-21irqcore21:30:172
125963799130,2cyclictest1344700-21sed20:30:171
1259633991311,1cyclictest60650irq/133-enp0s29f1:rx-022:35:140
125964499129,2cyclictest503-21systemd-logind00:30:003
125964499129,2cyclictest1523123-21cpuspeed_turbos23:20:183
125964499129,2cyclictest1310087-21tr19:55:183
125964499127,3cyclictest1318151-21kworker/u8:3+events_unbound20:35:113
125964499126,4cyclictest1348207-21ntpq20:30:213
125964099129,2cyclictest1570349-21cut00:05:172
125964099129,2cyclictest1365360-21irqcore20:50:152
125964099129,2cyclictest1365360-21irqcore20:50:152
125964099128,3cyclictest1354543-21apt-get20:40:112
125964099127,4cyclictest1417781-21apt-get21:40:132
125964099126,5cyclictest3000860-21systemd-journal00:13:182
125964099125,6cyclictest1334012-21diskstats20:20:112
125964099121,3cyclictest1401463-21awk21:25:002
125964099120,4cyclictest1595195-21grep00:25:262
125964499119,1cyclictest481rcuc/319:35:183
125964499119,1cyclictest0-21swapper/323:45:013
125964499117,2cyclictest1539035-21if_enp0s29f123:35:163
125964499117,2cyclictest1474427-21ps22:30:233
125964499117,2cyclictest1464817-21cron22:24:593
125964499116,3cyclictest1305228-21cut19:50:173
125964499110,4cyclictest1338937-21cat20:25:003
125964099117,3cyclictest1281292-21apt-get19:30:122
125964099117,3cyclictest0-21swapper/221:10:172
125964099111,3cyclictest1538434-21awk23:35:002
125964099111,3cyclictest1270087-21awk19:19:592
125964499107,2cyclictest1534501-21kthreadcore23:30:193
125964499107,2cyclictest1502333-21cpuspeed_turbos23:00:133
125964499107,2cyclictest1470003-21idleruntime-cro22:29:593
125964499106,2cyclictest14911802sleep322:50:003
125964499101,6cyclictest1512395-21basename23:10:003
125964499101,6cyclictest1512395-21basename23:10:003
125964499100,2cyclictest1264869-21latency_hist19:15:013
125964099108,1cyclictest0-21swapper/221:05:152
125964099106,3cyclictest3000063-21rs:main2
125964099106,2cyclictest1275184-21sshd19:23:112
125964099106,2cyclictest0-21swapper/219:35:162
125964099106,2cyclictest0-21swapper/200:35:232
125964099104,4cyclictest516-21gdbus22:45:002
12596449996,1cyclictest0-21swapper/323:55:213
12596449994,4cyclictest1481134-21cut22:40:173
12596449991,6cyclictest3000860-21systemd-journal21:10:003
12596449991,6cyclictest1589636-21ntpq00:20:233
12596449991,6cyclictest1296633-21awk19:45:013
12596409996,2cyclictest1565153-21irqcore00:00:192
12596409996,1cyclictest0-21swapper/200:15:172
12596449981,5cyclictest1402726-21irqstats21:25:233
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional