You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-21 - 00:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot6s.osadl.org (updated Thu Nov 20, 2025 12:43:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5991993838,0cyclictest1180-21fwupd11:44:051
5999992018,2cyclictest66850irq/125-lan10:22:293
5999991919,0cyclictest66850irq/125-lan12:00:213
5999991918,1cyclictest66850irq/125-lan12:10:273
5999991918,1cyclictest66850irq/125-lan11:05:363
5999991918,1cyclictest66850irq/125-lan10:36:473
5999991918,1cyclictest66850irq/125-lan10:02:533
5999991918,1cyclictest66850irq/125-lan09:55:203
5999991918,1cyclictest66850irq/125-lan09:23:513
5999991917,2cyclictest66850irq/125-lan12:30:153
5999991917,2cyclictest66850irq/125-lan12:08:133
5999991917,2cyclictest66850irq/125-lan11:04:073
5999991917,2cyclictest66850irq/125-lan10:28:353
5999991916,3cyclictest66850irq/125-lan10:05:153
5999991818,0cyclictest66850irq/125-lan11:54:003
5999991818,0cyclictest66850irq/125-lan10:57:063
5999991818,0cyclictest66850irq/125-lan09:47:153
5999991817,1cyclictest66850irq/125-lan12:35:153
5999991817,1cyclictest66850irq/125-lan11:35:433
5999991817,1cyclictest66850irq/125-lan09:44:223
5999991817,1cyclictest66850irq/125-lan08:15:363
5999991817,1cyclictest66850irq/125-lan07:25:153
5999991816,2cyclictest66850irq/125-lan11:56:323
5999991816,2cyclictest66850irq/125-lan11:25:073
5999991816,2cyclictest66850irq/125-lan11:20:193
5999991816,2cyclictest66850irq/125-lan09:19:033
5999991816,2cyclictest66850irq/125-lan09:10:313
5999991815,3cyclictest66850irq/125-lan10:15:183
5999991717,0cyclictest66850irq/125-lan12:15:143
5999991716,1cyclictest66850irq/125-lan11:45:173
5999991716,1cyclictest66850irq/125-lan11:15:123
5999991716,1cyclictest66850irq/125-lan10:40:003
5999991716,1cyclictest66850irq/125-lan10:31:293
5999991716,1cyclictest66850irq/125-lan09:39:503
5999991716,1cyclictest66850irq/125-lan09:25:043
5999991716,1cyclictest66850irq/125-lan08:05:213
5999991715,2cyclictest66850irq/125-lan12:20:113
5999991715,2cyclictest66850irq/125-lan11:30:163
5999991715,2cyclictest66850irq/125-lan10:53:533
5999991715,2cyclictest66850irq/125-lan10:10:423
5999991715,2cyclictest66850irq/125-lan08:45:193
5999991715,2cyclictest66850irq/125-lan08:20:213
5999991714,3cyclictest66850irq/125-lan12:27:183
5999991714,3cyclictest66850irq/125-lan11:10:103
5996991717,0cyclictest771-21snmpd12:21:192
5996991717,0cyclictest771-21snmpd11:25:532
5996991717,0cyclictest771-21snmpd10:54:492
5996991717,0cyclictest771-21snmpd09:51:132
5996991717,0cyclictest771-21snmpd08:30:212
5996991717,0cyclictest771-21snmpd08:22:412
5996991717,0cyclictest771-21snmpd07:57:042
5996991717,0cyclictest771-21snmpd07:31:482
5996991717,0cyclictest12303-21cat11:55:132
5999991616,0cyclictest66850irq/125-lan09:07:393
5999991616,0cyclictest66850irq/125-lan07:30:173
5999991616,0cyclictest66850irq/125-lan07:20:203
5999991615,1cyclictest66850irq/125-lan11:40:203
5999991615,1cyclictest66850irq/125-lan10:45:133
5999991615,1cyclictest66850irq/125-lan09:30:093
5999991615,1cyclictest66850irq/125-lan09:00:203
5999991615,1cyclictest66850irq/125-lan08:25:113
5999991615,1cyclictest66850irq/125-lan08:10:163
5999991615,1cyclictest66850irq/125-lan08:00:113
5999991615,1cyclictest66850irq/125-lan07:55:133
5999991615,1cyclictest66850irq/125-lan07:40:173
5999991615,1cyclictest66850irq/125-lan07:13:323
5996991616,0cyclictest771-21snmpd12:03:542
5996991616,0cyclictest771-21snmpd07:43:102
5996991616,0cyclictest32077-21cat09:20:122
5996991616,0cyclictest17222-21cat10:15:142
5999991514,1cyclictest66850irq/125-lan09:50:473
5999991514,1cyclictest66850irq/125-lan08:30:093
5999991514,1cyclictest66850irq/125-lan07:15:173
5999991513,2cyclictest66850irq/125-lan08:35:173
5999991512,3cyclictest66850irq/125-lan07:45:143
5996991414,0cyclictest771-21snmpd11:13:292
5999991312,1cyclictest66850irq/125-lan08:59:373
5999991312,1cyclictest66850irq/125-lan08:43:423
5999991312,1cyclictest66850irq/125-lan07:36:363
5996991313,0cyclictest771-21snmpd09:27:282
5999991111,0cyclictest66850irq/125-lan08:50:133
59969998,1cyclictest771-21snmpd11:37:002
59969998,1cyclictest771-21snmpd10:24:322
59969998,1cyclictest771-21snmpd08:47:032
59919998,1cyclictest771-21snmpd12:15:101
59919998,1cyclictest771-21snmpd09:17:461
59919998,1cyclictest2949-21cut09:25:001
59879998,1cyclictest524-21avahi-daemon12:15:510
59999986,2cyclictest749-21ntpd07:50:293
59969988,0cyclictest771-21snmpd11:15:052
59969988,0cyclictest769-21lldpd11:41:122
59969988,0cyclictest769-21lldpd10:42:012
59969988,0cyclictest769-21lldpd09:01:552
59969987,1cyclictest771-21snmpd07:53:462
59969987,1cyclictest524-21avahi-daemon12:12:372
59969987,1cyclictest524-21avahi-daemon12:07:072
59969987,1cyclictest524-21avahi-daemon10:59:382
59969986,2cyclictest771-21snmpd08:52:302
59969986,2cyclictest749-21ntpd12:37:562
59969986,2cyclictest749-21ntpd12:30:302
59969986,2cyclictest749-21ntpd11:54:462
59969986,2cyclictest749-21ntpd11:05:572
59969986,2cyclictest749-21ntpd10:39:102
59969986,2cyclictest749-21ntpd10:33:422
59969986,2cyclictest749-21ntpd10:10:512
59969986,2cyclictest749-21ntpd10:06:122
59969986,2cyclictest749-21ntpd09:40:432
59969986,2cyclictest749-21ntpd09:31:372
59969986,2cyclictest749-21ntpd09:15:242
59969986,2cyclictest749-21ntpd09:14:502
59919988,0cyclictest771-21snmpd11:18:261
59919988,0cyclictest771-21snmpd09:55:261
59919988,0cyclictest769-21lldpd11:22:391
59919988,0cyclictest769-21lldpd11:03:131
59919988,0cyclictest769-21lldpd10:58:091
59919988,0cyclictest769-21lldpd08:14:111
59919987,1cyclictest771-21snmpd10:50:101
59919987,1cyclictest524-21avahi-daemon10:22:151
59879988,0cyclictest771-21snmpd11:18:410
59879988,0cyclictest771-21snmpd07:37:450
59879988,0cyclictest769-21lldpd10:46:490
59879988,0cyclictest769-21lldpd08:51:380
59879987,1cyclictest771-21snmpd12:35:010
59879987,1cyclictest771-21snmpd12:05:270
59879987,1cyclictest771-21snmpd11:58:480
59879987,1cyclictest771-21snmpd08:11:110
59879987,1cyclictest771-21snmpd07:46:040
59879986,2cyclictest771-21snmpd12:32:400
59879986,2cyclictest771-21snmpd10:19:270
59879986,2cyclictest771-21snmpd08:00:560
59879986,2cyclictest771-21snmpd07:14:180
59969977,0cyclictest7930-21sh07:15:012
59969977,0cyclictest771-21snmpd11:45:452
59969977,0cyclictest771-21snmpd08:27:292
59969977,0cyclictest769-21lldpd11:00:072
59969977,0cyclictest769-21lldpd08:18:592
59969976,1cyclictest749-21ntpd08:40:162
59969976,1cyclictest749-21ntpd08:35:162
59969976,1cyclictest517-21systemd-resolve10:27:222
59969976,1cyclictest21699-21nvmesmart_nvme009:05:152
59969975,2cyclictest771-21snmpd11:30:472
59969975,2cyclictest749-21ntpd10:00:432
59969975,2cyclictest749-21ntpd08:55:182
59919977,0cyclictest771-21snmpd12:10:071
59919977,0cyclictest771-21snmpd10:39:551
59919977,0cyclictest769-21lldpd08:55:231
59919977,0cyclictest769-21lldpd08:40:551
59919977,0cyclictest517-21systemd-resolve10:00:171
59919976,1cyclictest771-21snmpd10:11:511
59919976,1cyclictest771-21snmpd09:38:461
59919976,1cyclictest524-21avahi-daemon12:35:371
59919976,1cyclictest524-21avahi-daemon07:16:411
59919975,2cyclictest771-21snmpd11:10:471
59919975,2cyclictest771-21snmpd09:54:281
59919975,2cyclictest771-21snmpd09:26:491
59919975,2cyclictest771-21snmpd09:03:551
59919975,2cyclictest771-21snmpd08:03:021
59919975,2cyclictest771-21snmpd07:24:481
59919973,3cyclictest8797-21ssh12:24:471
59879977,0cyclictest771-21snmpd12:29:010
59879977,0cyclictest771-21snmpd11:14:260
59879977,0cyclictest771-21snmpd07:42:010
59879977,0cyclictest771-21snmpd07:24:050
59879977,0cyclictest771-21snmpd07:18:090
59879977,0cyclictest769-21lldpd12:20:460
59879977,0cyclictest769-21lldpd11:38:380
59879977,0cyclictest769-21lldpd10:32:110
59879977,0cyclictest769-21lldpd10:08:420
59879977,0cyclictest769-21lldpd09:44:120
59879977,0cyclictest769-21lldpd08:41:390
59879977,0cyclictest524-21avahi-daemon12:11:330
59879976,1cyclictest771-21snmpd10:56:100
59879976,1cyclictest771-21snmpd09:05:210
59879976,1cyclictest771-21snmpd08:36:590
59879976,1cyclictest771-21snmpd08:26:020
59879976,1cyclictest771-21snmpd08:20:040
59879976,1cyclictest769-21lldpd11:33:060
59879976,1cyclictest769-21lldpd09:23:090
59879976,1cyclictest524-21avahi-daemon10:28:220
59879975,2cyclictest771-21snmpd12:00:270
59879975,2cyclictest771-21snmpd11:50:120
59879975,2cyclictest771-21snmpd09:59:230
59879975,2cyclictest771-21snmpd09:49:590
59879975,2cyclictest771-21snmpd09:27:430
59879975,2cyclictest771-21snmpd08:57:460
59879975,2cyclictest771-21snmpd08:48:540
59879975,2cyclictest771-21snmpd07:33:340
59879975,2cyclictest17137-21cat10:50:150
59879974,3cyclictest550-21NetworkManager10:44:340
59879973,1cyclictest55050NetworkManager10:35:250
59969966,0cyclictest769-21lldpd08:08:412
59969966,0cyclictest769-21lldpd07:37:382
59969964,2cyclictest771-21snmpd07:49:542
59969963,2cyclictest31834-21ssh12:15:142
59969962,3cyclictest9216-21ssh12:25:012
59969961,4cyclictest16532-21cat10:50:002
59919966,0cyclictest771-21snmpd09:33:471
59919966,0cyclictest771-21snmpd07:25:541
59919966,0cyclictest769-21lldpd11:31:511
59919966,0cyclictest769-21lldpd08:30:431
59919965,1cyclictest771-21snmpd10:33:431
59919965,1cyclictest771-21snmpd09:46:321
59919965,1cyclictest769-21lldpd10:25:571
59919965,1cyclictest524-21avahi-daemon10:08:571
59919965,1cyclictest524-21avahi-daemon08:53:431
59919964,2cyclictest771-21snmpd12:08:251
59919963,3cyclictest550-21NetworkManager11:39:381
59919963,2cyclictest25165-21ssh09:11:571
59919963,2cyclictest20210-21ssh10:18:361
59919962,3cyclictest9754-21sed12:25:111
59919962,2cyclictest26737-21cat08:00:011
59879966,0cyclictest769-21lldpd11:42:140
59879966,0cyclictest769-21lldpd11:20:370
59879965,1cyclictest771-21snmpd07:54:480
59879965,1cyclictest771-21snmpd07:25:270
59879965,1cyclictest19083-21fschecks_time09:00:130
59879964,2cyclictest771-21snmpd09:37:020
59879964,2cyclictest771-21snmpd08:16:470
59879963,2cyclictest586-21rs:main0
59879963,2cyclictest255-20systemd-journal09:52:090
59879963,2cyclictest19001-21ssh11:27:200
59879962,3cyclictest4299-21wc10:00:200
59879962,3cyclictest255-20systemd-journal11:02:120
59879962,3cyclictest10693-21ssh09:33:300
59969955,0cyclictest769-21lldpd07:25:502
59969954,1cyclictest769-21lldpd08:02:492
59969953,2cyclictest255-20systemd-journal11:20:032
59969952,3cyclictest287-21systemd-udevd07:20:212
59969952,2cyclictest8951-21sed07:15:182
59969952,2cyclictest31310-21ntpq09:55:162
59969952,2cyclictest14458-21ssh09:36:212
59969951,3cyclictest22997-21users09:45:252
59919955,0cyclictest769-21lldpd08:21:301
59919955,0cyclictest769-21lldpd08:17:511
59919955,0cyclictest524-21avahi-daemon12:00:471
59919955,0cyclictest524-21avahi-daemon07:48:091
59919954,1cyclictest517-21systemd-resolve10:40:161
59919953,2cyclictest771-21snmpd08:35:531
59919952,3cyclictest5106-21sed08:25:121
59919952,2cyclictest7369-21sed07:10:211
59919952,2cyclictest4105-21ssh11:46:161
59919952,2cyclictest31969-21grep11:05:211
59919952,2cyclictest255-20systemd-journal11:55:171
59919952,2cyclictest255-20systemd-journal09:09:371
59919952,2cyclictest23670-21ntp_states07:50:171
59919952,2cyclictest17331-21sed07:35:151
59919952,2cyclictest16937-21ssh12:31:071
59919952,2cyclictest12492-21latency_hist08:45:011
59919952,2cyclictest10565-21rm11:54:191
59919951,3cyclictest19127-21grep07:40:141
59919951,3cyclictest14231-21ssh10:46:171
59919950,5cyclictest550-21NetworkManager09:43:151
59919950,2cyclictest1870850ssh11:26:511
59879955,0cyclictest769-21lldpd07:58:040
59879954,1cyclictest171ktimers/009:11:550
59879953,2cyclictest771-21snmpd08:05:010
59879952,2cyclictest7465-21grep08:30:160
59879952,2cyclictest5083-21ssh11:47:530
59879952,2cyclictest30337-21dpkg11:05:000
59969941,2cyclictest31497-21sed08:10:162
59919941,2cyclictest28875-21https08:05:101
59919941,2cyclictest14470-21gpgv07:30:011
59879943,1cyclictest171ktimers/009:17:190
59879941,2cyclictest21759-21grep10:20:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional