You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-15 - 09:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot6s.osadl.org (updated Sat Nov 15, 2025 00:43:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24847991918,1cyclictest66850irq/125-lan22:25:012
24847991918,1cyclictest66850irq/125-lan21:55:002
24847991918,1cyclictest66850irq/125-lan21:48:482
24847991918,1cyclictest66850irq/125-lan19:35:002
24847991818,0cyclictest66850irq/125-lan23:40:202
24847991818,0cyclictest66850irq/125-lan23:33:572
24847991818,0cyclictest66850irq/125-lan23:17:402
24847991818,0cyclictest66850irq/125-lan23:09:042
24847991818,0cyclictest66850irq/125-lan22:18:512
24847991818,0cyclictest66850irq/125-lan21:25:002
24847991818,0cyclictest66850irq/125-lan00:25:382
24847991818,0cyclictest66850irq/125-lan00:12:302
24847991817,1cyclictest66850irq/125-lan23:49:572
24847991817,1cyclictest66850irq/125-lan23:25:182
24847991817,1cyclictest66850irq/125-lan23:10:222
24847991817,1cyclictest66850irq/125-lan22:57:532
24847991817,1cyclictest66850irq/125-lan22:46:202
24847991817,1cyclictest66850irq/125-lan22:36:282
24847991817,1cyclictest66850irq/125-lan22:12:322
24847991817,1cyclictest66850irq/125-lan22:00:342
24847991817,1cyclictest66850irq/125-lan21:40:152
24847991817,1cyclictest66850irq/125-lan21:35:162
24847991817,1cyclictest66850irq/125-lan21:15:152
24847991817,1cyclictest66850irq/125-lan00:30:152
24847991817,1cyclictest66850irq/125-lan00:21:122
24847991817,1cyclictest66850irq/125-lan00:02:152
24847991717,0cyclictest66850irq/125-lan23:50:382
24847991717,0cyclictest66850irq/125-lan23:22:492
24847991717,0cyclictest66850irq/125-lan22:30:142
24847991717,0cyclictest66850irq/125-lan21:53:262
24847991717,0cyclictest66850irq/125-lan21:33:282
24847991716,1cyclictest66850irq/125-lan23:58:512
24847991716,1cyclictest66850irq/125-lan22:50:132
24847991716,1cyclictest66850irq/125-lan22:41:452
24847991716,1cyclictest66850irq/125-lan22:27:062
24847991716,1cyclictest66850irq/125-lan22:07:302
24847991716,1cyclictest66850irq/125-lan21:21:512
24847991716,1cyclictest66850irq/125-lan21:10:212
24847991716,1cyclictest66850irq/125-lan20:00:202
24847991716,1cyclictest66850irq/125-lan19:25:132
24847991716,1cyclictest66850irq/125-lan19:20:242
24847991716,1cyclictest66850irq/125-lan00:17:532
24847991616,0cyclictest66850irq/125-lan21:05:202
24847991616,0cyclictest66850irq/125-lan20:56:492
24847991615,1cyclictest66850irq/125-lan23:35:022
24847991615,1cyclictest66850irq/125-lan23:00:142
24847991615,1cyclictest66850irq/125-lan21:00:152
24847991615,1cyclictest66850irq/125-lan19:55:132
24847991615,1cyclictest66850irq/125-lan19:40:152
24847991615,1cyclictest66850irq/125-lan00:36:302
24847991515,0cyclictest66850irq/125-lan20:10:202
24847991515,0cyclictest66850irq/125-lan00:05:212
24847991514,1cyclictest66850irq/125-lan19:45:202
24847991414,0cyclictest66850irq/125-lan20:53:142
24847991313,0cyclictest66850irq/125-lan20:35:182
24847991313,0cyclictest66850irq/125-lan20:33:452
24847991313,0cyclictest66850irq/125-lan19:50:152
24847991212,0cyclictest66850irq/125-lan20:29:442
24847991211,1cyclictest66850irq/125-lan20:40:212
24847991211,1cyclictest66850irq/125-lan19:10:132
24847991111,0cyclictest66850irq/125-lan20:45:192
24847991111,0cyclictest66850irq/125-lan19:35:172
2484799109,1cyclictest66850irq/125-lan20:20:172
24847991010,0cyclictest66850irq/125-lan20:15:212
24847991010,0cyclictest66850irq/125-lan19:18:152
248509998,1cyclictest771-21snmpd21:34:313
248479998,1cyclictest66850irq/125-lan20:05:202
248429998,1cyclictest771-21snmpd23:22:581
248429997,2cyclictest29175-21head23:35:171
248379998,1cyclictest769-21lldpd23:56:380
248379998,1cyclictest749-21ntpd00:20:170
248509988,0cyclictest771-21snmpd22:40:473
248509988,0cyclictest771-21snmpd20:57:583
248509988,0cyclictest769-21lldpd23:54:073
248509988,0cyclictest769-21lldpd23:16:193
248509987,1cyclictest769-21lldpd00:35:523
248509987,1cyclictest524-21avahi-daemon22:22:213
248509986,2cyclictest771-21snmpd23:43:373
248509986,2cyclictest771-21snmpd21:15:543
248509986,2cyclictest771-21snmpd00:03:273
248429988,0cyclictest771-21snmpd23:25:521
248429988,0cyclictest771-21snmpd22:47:291
248429988,0cyclictest771-21snmpd20:38:531
248429988,0cyclictest771-21snmpd19:20:101
248429988,0cyclictest769-21lldpd23:43:421
248429988,0cyclictest769-21lldpd23:30:521
248429988,0cyclictest769-21lldpd21:20:191
248429987,1cyclictest771-21snmpd21:57:091
248429987,1cyclictest771-21snmpd21:45:211
248429987,1cyclictest771-21snmpd20:10:091
248429987,1cyclictest771-21snmpd19:25:211
248429987,1cyclictest771-21snmpd00:30:541
248429987,1cyclictest769-21lldpd20:47:481
248429987,1cyclictest524-21avahi-daemon23:05:201
248429987,1cyclictest524-21avahi-daemon21:54:571
248429987,1cyclictest517-21systemd-resolve23:02:491
248429986,2cyclictest771-21snmpd23:18:181
248429986,2cyclictest771-21snmpd19:10:561
248429986,2cyclictest5553-21cat21:00:151
248429985,2cyclictest255-20systemd-journal21:18:271
248379988,0cyclictest771-21snmpd23:09:100
248379988,0cyclictest771-21snmpd22:48:440
248379988,0cyclictest771-21snmpd21:19:440
248379988,0cyclictest769-21lldpd23:11:570
248379988,0cyclictest769-21lldpd21:51:540
248379988,0cyclictest769-21lldpd21:31:200
248379988,0cyclictest769-21lldpd20:29:510
248379987,1cyclictest771-21snmpd21:00:550
248379987,1cyclictest771-21snmpd20:24:040
248379987,1cyclictest771-21snmpd20:06:040
248379987,1cyclictest771-21snmpd19:58:190
248379987,1cyclictest771-21snmpd19:23:070
248379987,1cyclictest769-21lldpd23:33:080
248379987,1cyclictest769-21lldpd20:12:490
248379986,2cyclictest771-21snmpd23:17:390
248379985,3cyclictest27830-21cat20:35:170
248509977,0cyclictest771-21snmpd22:28:173
248509977,0cyclictest771-21snmpd21:13:473
248509977,0cyclictest771-21snmpd20:08:333
248509977,0cyclictest771-21snmpd19:41:003
248509977,0cyclictest769-21lldpd22:54:343
248509977,0cyclictest769-21lldpd21:28:323
248509977,0cyclictest769-21lldpd20:02:023
248509977,0cyclictest769-21lldpd19:59:053
248509976,1cyclictest771-21snmpd22:56:483
248509976,1cyclictest771-21snmpd21:00:113
248509976,1cyclictest771-21snmpd20:39:493
248509976,1cyclictest771-21snmpd19:31:193
248509976,1cyclictest771-21snmpd00:32:163
248509976,1cyclictest769-21lldpd21:52:023
248509976,1cyclictest524-21avahi-daemon21:35:513
248509976,1cyclictest27456-21fschecks_count00:05:153
248509976,1cyclictest10139-21/usr/sbin/munin00:20:193
248509975,2cyclictest771-21snmpd23:47:173
248509975,2cyclictest771-21snmpd23:11:033
248509975,2cyclictest771-21snmpd20:54:343
248509975,1cyclictest771-21snmpd23:01:253
248509972,5cyclictest550-21NetworkManager00:28:183
248429977,0cyclictest771-21snmpd22:25:041
248429977,0cyclictest771-21snmpd22:17:461
248429977,0cyclictest771-21snmpd20:25:121
248429977,0cyclictest769-21lldpd22:30:211
248429977,0cyclictest769-21lldpd21:13:321
248429977,0cyclictest769-21lldpd20:44:211
248429976,1cyclictest771-21snmpd22:40:021
248429976,1cyclictest771-21snmpd19:16:361
248429976,1cyclictest771-21snmpd00:11:171
248429976,1cyclictest771-21snmpd00:03:141
248429976,1cyclictest524-21avahi-daemon22:35:371
248429975,2cyclictest771-21snmpd21:41:001
248429975,2cyclictest771-21snmpd20:08:401
248429975,2cyclictest771-21snmpd19:54:131
248429975,2cyclictest771-21snmpd00:37:041
248429975,2cyclictest749-21ntpd22:54:431
248379977,0cyclictest771-21snmpd22:57:070
248379977,0cyclictest771-21snmpd22:10:100
248379977,0cyclictest771-21snmpd19:27:570
248379977,0cyclictest771-21snmpd00:08:090
248379977,0cyclictest771-21snmpd00:04:530
248379977,0cyclictest769-21lldpd22:34:330
248379977,0cyclictest769-21lldpd21:58:090
248379976,1cyclictest771-21snmpd19:36:550
248379976,1cyclictest769-21lldpd20:32:470
248379976,1cyclictest255-20systemd-journal00:18:020
248379976,1cyclictest21775-21gltestperf22:25:140
248379973,3cyclictest21142-21ssh23:27:470
248509966,0cyclictest771-21snmpd23:30:293
248509966,0cyclictest771-21snmpd23:27:463
248509966,0cyclictest771-21snmpd23:21:253
248509966,0cyclictest771-21snmpd19:39:243
248509965,1cyclictest771-21snmpd23:08:253
248509965,1cyclictest771-21snmpd21:46:553
248509965,1cyclictest771-21snmpd21:41:393
248509965,1cyclictest524-21avahi-daemon23:55:053
248509965,1cyclictest524-21avahi-daemon22:02:353
248509965,1cyclictest524-21avahi-daemon19:50:323
248509965,1cyclictest27116-21cpuspeed_turbos19:15:133
248509964,2cyclictest771-21snmpd22:38:273
248509964,2cyclictest771-21snmpd20:20:093
248509964,2cyclictest771-21snmpd20:12:153
248509964,2cyclictest771-21snmpd19:29:243
248509964,2cyclictest29231-21cat23:35:173
248509963,2cyclictest586-21rs:main3
248509963,2cyclictest255-20systemd-journal00:15:253
248509963,2cyclictest23028-21apt-get21:55:113
248509962,4cyclictest550-21NetworkManager20:31:153
248509962,3cyclictest26638-21grep22:30:143
248509962,3cyclictest10177-21ssh22:14:083
248429966,0cyclictest771-21snmpd23:47:281
248429966,0cyclictest771-21snmpd21:33:271
248429966,0cyclictest769-21lldpd00:19:051
248429965,1cyclictest771-21snmpd20:21:161
248429965,1cyclictest771-21snmpd19:45:061
248429965,1cyclictest771-21snmpd00:20:301
248429965,1cyclictest524-21avahi-daemon22:05:311
248429965,1cyclictest524-21avahi-daemon21:25:511
248429965,1cyclictest25567-21fschecks_count20:30:131
248429964,2cyclictest771-21snmpd21:06:241
248429964,2cyclictest771-21snmpd20:51:051
248429963,3cyclictest550-21NetworkManager23:11:251
248429963,3cyclictest550-21NetworkManager00:25:121
248429963,2cyclictest24056-21systemctl23:00:001
248429963,2cyclictest14065-21ntpq20:00:181
248429962,3cyclictest27573-21fw_forwarded_lo00:05:151
248429962,3cyclictest19086-21ssh23:56:111
248429962,3cyclictest12084-21df23:50:121
248379966,0cyclictest771-21snmpd22:04:460
248379966,0cyclictest769-21lldpd21:47:060
248379966,0cyclictest769-21lldpd20:16:190
248379966,0cyclictest769-21lldpd19:10:340
248379966,0cyclictest27711-21needreboot19:15:150
248379965,1cyclictest9378-21cpuspeed_turbos21:10:130
248379965,1cyclictest769-21lldpd23:22:290
248379965,1cyclictest31971-21cat22:35:160
248379965,1cyclictest171ktimers/022:05:130
248379965,1cyclictest171ktimers/000:34:210
248379964,2cyclictest771-21snmpd00:14:000
248379963,3cyclictest550-21NetworkManager23:42:030
248379963,3cyclictest32159-21ntp_192.168.11420:45:200
248379963,2cyclictest255-20systemd-journal23:49:390
248379963,2cyclictest14512-21ssh23:51:590
248379962,3cyclictest23239-21ssh21:23:450
248379962,3cyclictest19374-21ssh22:22:430
248379961,5cyclictest550-21NetworkManager23:36:100
248509955,0cyclictest771-21snmpd22:07:443
248509955,0cyclictest771-21snmpd20:43:133
248509955,0cyclictest771-21snmpd19:45:163
248509952,3cyclictest550-21NetworkManager19:24:013
248509952,2cyclictest7146-21cpu21:05:163
248509952,2cyclictest25886-21ntpq19:10:183
248509952,2cyclictest20248-21ntpq21:20:203
248509952,2cyclictest14371-21ssh22:17:513
248509951,3cyclictest32432-21df_inode00:10:133
248509951,3cyclictest23912-21sed20:25:153
248429953,2cyclictest771-21snmpd21:35:071
248429953,2cyclictest771-21snmpd19:31:211
248429953,2cyclictest0-21swapper/122:20:201
248429952,2cyclictest6163-21grep22:10:121
248429952,2cyclictest3514-21cat20:55:141
248429951,3cyclictest30479-21sh22:01:331
248429951,3cyclictest255-20systemd-journal19:41:061
248429951,3cyclictest11696-21irqstats19:55:151
248429950,3cyclictest4147-21runrttasks19:35:511
248379955,0cyclictest769-21lldpd22:15:050
248379955,0cyclictest171ktimers/020:40:210
248379954,1cyclictest771-21snmpd21:27:230
248379954,1cyclictest771-21snmpd19:44:000
248379954,1cyclictest769-21lldpd19:30:260
248379953,2cyclictest550-21NetworkManager23:00:550
248379952,3cyclictest550-21NetworkManager21:40:130
248379952,3cyclictest255-20systemd-journal22:50:090
248379952,2cyclictest7301-21df21:05:170
248379952,2cyclictest7054-21cut19:45:170
248379952,2cyclictest3315-21fschecks_time20:55:130
248379952,2cyclictest255-20systemd-journal22:42:320
248379952,2cyclictest1805-21ntpq20:50:170
248379952,2cyclictest17509-21ssh00:26:250
248379951,3cyclictest2660-21grep21:35:160
248379951,3cyclictest10089-21awk19:50:210
248379950,3cyclictest15118-21cat20:04:590
248509942,2cyclictest31455-21dpkg20:45:173
248509941,3cyclictest550-21NetworkManager20:15:073
248429940,3cyclictest19479-21diskstats20:15:121
248379943,1cyclictest25390-21cut00:35:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional