You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-03 - 03:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack8slot6s.osadl.org (updated Mon Nov 03, 2025 00:43:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23053991918,1cyclictest66850irq/125-lan23:10:162
23053991918,1cyclictest66850irq/125-lan22:17:112
23053991818,0cyclictest771-21snmpd23:44:462
23053991818,0cyclictest771-21snmpd21:50:182
23053991818,0cyclictest771-21snmpd20:03:082
23053991818,0cyclictest66850irq/125-lan23:35:182
23053991818,0cyclictest66850irq/125-lan23:28:262
23053991818,0cyclictest66850irq/125-lan22:50:182
23053991818,0cyclictest66850irq/125-lan22:07:532
23053991818,0cyclictest66850irq/125-lan21:39:542
23053991818,0cyclictest66850irq/125-lan21:25:142
23053991818,0cyclictest66850irq/125-lan21:10:202
23053991818,0cyclictest66850irq/125-lan20:47:412
23053991818,0cyclictest66850irq/125-lan19:35:112
23053991818,0cyclictest66850irq/125-lan00:35:212
23053991817,1cyclictest66850irq/125-lan23:45:182
23053991817,1cyclictest66850irq/125-lan22:47:172
23053991817,1cyclictest66850irq/125-lan22:41:342
23053991817,1cyclictest66850irq/125-lan22:10:442
23053991817,1cyclictest66850irq/125-lan21:58:222
23053991817,1cyclictest66850irq/125-lan19:40:512
23053991717,0cyclictest66850irq/125-lan23:55:162
23053991717,0cyclictest66850irq/125-lan23:30:132
23053991717,0cyclictest66850irq/125-lan23:20:182
23053991717,0cyclictest66850irq/125-lan23:00:192
23053991717,0cyclictest66850irq/125-lan22:57:472
23053991717,0cyclictest66850irq/125-lan22:26:152
23053991717,0cyclictest66850irq/125-lan22:20:172
23053991717,0cyclictest66850irq/125-lan21:45:232
23053991717,0cyclictest66850irq/125-lan21:16:052
23053991717,0cyclictest66850irq/125-lan20:50:142
23053991717,0cyclictest66850irq/125-lan19:48:142
23053991717,0cyclictest66850irq/125-lan19:22:232
23053991717,0cyclictest66850irq/125-lan00:25:112
23053991717,0cyclictest66850irq/125-lan00:21:122
23053991717,0cyclictest66850irq/125-lan00:14:402
23053991717,0cyclictest66850irq/125-lan00:09:482
23053991716,1cyclictest66850irq/125-lan23:52:012
23053991716,1cyclictest66850irq/125-lan23:15:292
23053991716,1cyclictest66850irq/125-lan23:05:182
23053991716,1cyclictest66850irq/125-lan22:35:212
23053991716,1cyclictest66850irq/125-lan22:30:482
23053991716,1cyclictest66850irq/125-lan22:00:562
23053991716,1cyclictest66850irq/125-lan21:40:172
23053991716,1cyclictest66850irq/125-lan21:20:212
23053991716,1cyclictest66850irq/125-lan20:15:162
23053991716,1cyclictest66850irq/125-lan00:30:142
23053991716,1cyclictest66850irq/125-lan00:00:032
23053991616,0cyclictest66850irq/125-lan21:30:122
23053991616,0cyclictest66850irq/125-lan20:35:142
23053991616,0cyclictest66850irq/125-lan00:15:232
23053991615,1cyclictest66850irq/125-lan19:30:132
23053991615,1cyclictest66850irq/125-lan19:25:152
23053991515,0cyclictest66850irq/125-lan20:27:382
23053991515,0cyclictest66850irq/125-lan20:10:192
23053991515,0cyclictest66850irq/125-lan19:55:222
23053991414,0cyclictest66850irq/125-lan21:05:242
23053991414,0cyclictest66850irq/125-lan20:30:162
23053991414,0cyclictest66850irq/125-lan20:20:232
23053991414,0cyclictest66850irq/125-lan19:18:502
23053991313,0cyclictest66850irq/125-lan21:04:412
23053991313,0cyclictest66850irq/125-lan20:40:132
23053991313,0cyclictest66850irq/125-lan19:50:542
23053991212,0cyclictest66850irq/125-lan19:14:492
23053991111,0cyclictest66850irq/125-lan20:58:402
23053991111,0cyclictest66850irq/125-lan20:05:142
230579998,1cyclictest517-21systemd-resolve23:31:353
230579996,3cyclictest771-21snmpd20:37:013
230579996,3cyclictest771-21snmpd19:49:473
230509998,1cyclictest524-21avahi-daemon23:25:011
230509998,1cyclictest524-21avahi-daemon22:25:361
230509998,1cyclictest524-21avahi-daemon00:17:421
230509997,2cyclictest13746-21cron23:50:001
230479998,1cyclictest771-21snmpd21:04:070
230579988,0cyclictest771-21snmpd00:29:473
230579988,0cyclictest769-21lldpd22:58:143
230579988,0cyclictest769-21lldpd22:19:413
230579988,0cyclictest769-21lldpd20:23:003
230579988,0cyclictest769-21lldpd19:36:533
230579988,0cyclictest769-21lldpd00:13:053
230579987,1cyclictest771-21snmpd22:12:223
230579987,1cyclictest771-21snmpd21:43:243
230579987,1cyclictest771-21snmpd20:11:483
230579987,1cyclictest771-21snmpd19:34:293
230579987,1cyclictest769-21lldpd00:15:413
230579987,1cyclictest524-21avahi-daemon23:22:283
230579987,1cyclictest24551-21needreboot00:00:163
230579987,1cyclictest13103-21fschecks_count21:45:143
230579986,2cyclictest771-21snmpd21:05:073
230579986,2cyclictest771-21snmpd20:44:583
230579986,2cyclictest771-21snmpd00:22:453
230579986,2cyclictest30065-21cat21:30:163
230509988,0cyclictest771-21snmpd23:09:491
230509988,0cyclictest771-21snmpd22:52:151
230509988,0cyclictest771-21snmpd22:47:381
230509988,0cyclictest771-21snmpd22:37:361
230509988,0cyclictest771-21snmpd22:30:311
230509988,0cyclictest771-21snmpd20:30:151
230509988,0cyclictest771-21snmpd20:14:571
230509988,0cyclictest771-21snmpd20:07:151
230509988,0cyclictest769-21lldpd23:33:061
230509988,0cyclictest769-21lldpd21:38:391
230509988,0cyclictest769-21lldpd00:31:171
230509988,0cyclictest769-21lldpd00:24:271
230509987,1cyclictest820-21fschecks_count00:10:171
230509987,1cyclictest771-21snmpd21:59:511
230509987,1cyclictest771-21snmpd20:27:241
230509987,1cyclictest771-21snmpd20:17:421
230509987,1cyclictest524-21avahi-daemon23:58:241
230509987,1cyclictest524-21avahi-daemon22:44:091
230509987,1cyclictest517-21systemd-resolve23:18:051
230509987,1cyclictest18461-21tail21:50:131
230509986,2cyclictest771-21snmpd20:36:101
230509986,2cyclictest771-21snmpd19:17:071
230509986,2cyclictest771-21snmpd00:01:341
230479988,0cyclictest771-21snmpd22:38:000
230479988,0cyclictest771-21snmpd22:04:250
230479988,0cyclictest771-21snmpd21:08:290
230479988,0cyclictest771-21snmpd19:34:050
230479988,0cyclictest769-21lldpd23:39:420
230479988,0cyclictest769-21lldpd22:17:470
230479988,0cyclictest769-21lldpd21:31:190
230479988,0cyclictest769-21lldpd19:25:090
230479987,1cyclictest771-21snmpd20:57:400
230479987,1cyclictest771-21snmpd19:57:260
230479987,1cyclictest771-21snmpd19:53:440
230479987,1cyclictest771-21snmpd19:39:290
230479987,1cyclictest769-21lldpd20:08:070
230479987,1cyclictest524-21avahi-daemon00:06:420
230479987,1cyclictest524-21avahi-daemon00:04:520
230479987,1cyclictest517-21systemd-resolve23:29:490
230479987,1cyclictest255-20systemd-journal23:34:530
230479986,2cyclictest771-21snmpd23:51:160
230479986,2cyclictest771-21snmpd23:18:240
230479986,2cyclictest771-21snmpd23:04:300
230479986,2cyclictest771-21snmpd21:48:090
230479986,2cyclictest771-21snmpd20:03:590
230479986,2cyclictest771-21snmpd19:19:160
230479986,2cyclictest771-21snmpd19:10:550
230579977,0cyclictest771-21snmpd21:21:383
230579977,0cyclictest769-21lldpd22:38:063
230579977,0cyclictest769-21lldpd21:28:393
230579976,1cyclictest771-21snmpd20:33:153
230579976,1cyclictest771-21snmpd19:23:013
230579976,1cyclictest5452-21latency_hist21:05:003
230579976,1cyclictest524-21avahi-daemon21:50:423
230579976,1cyclictest524-21avahi-daemon21:11:293
230579976,1cyclictest19535-21cat23:55:163
230579975,2cyclictest771-21snmpd20:48:013
230579975,2cyclictest771-21snmpd00:35:403
230579975,2cyclictest511ktimers/319:15:183
230579973,3cyclictest13922-21ssh22:47:333
230509977,0cyclictest771-21snmpd19:20:521
230509977,0cyclictest771-21snmpd19:14:011
230509977,0cyclictest769-21lldpd21:15:101
230509977,0cyclictest769-21lldpd20:03:391
230509976,1cyclictest584-21in:imuxsock21:32:281
230509976,1cyclictest21925-21needreboot22:55:181
230509975,2cyclictest771-21snmpd19:42:471
230509973,3cyclictest467-21ssh23:35:311
230509973,3cyclictest31465-21ssh00:08:201
230509973,3cyclictest255-20systemd-journal22:24:021
230479977,0cyclictest771-21snmpd00:20:350
230479977,0cyclictest769-21lldpd23:14:360
230479977,0cyclictest769-21lldpd22:51:570
230479977,0cyclictest769-21lldpd22:14:050
230479977,0cyclictest769-21lldpd22:08:570
230479977,0cyclictest769-21lldpd19:43:470
230479977,0cyclictest769-21lldpd19:24:470
230479976,1cyclictest771-21snmpd22:55:080
230479976,1cyclictest769-21lldpd20:48:440
230479976,1cyclictest524-21avahi-daemon21:38:360
230479976,1cyclictest524-21avahi-daemon20:35:130
230479976,1cyclictest21408-21gltestperf20:25:140
230479975,2cyclictest771-21snmpd21:56:060
230479975,2cyclictest771-21snmpd20:40:550
230479975,2cyclictest771-21snmpd20:20:390
230479970,7cyclictest0-21swapper/000:25:200
230579966,0cyclictest769-21lldpd23:39:193
230579966,0cyclictest769-21lldpd21:35:453
230579966,0cyclictest524-21avahi-daemon23:08:553
230579965,1cyclictest771-21snmpd22:54:413
230579965,1cyclictest771-21snmpd22:29:493
230579964,2cyclictest771-21snmpd20:53:103
230579964,2cyclictest771-21snmpd19:56:503
230579963,2cyclictest586-21rs:main3
230579963,2cyclictest29308-21awk22:00:183
230579963,2cyclictest255-20systemd-journal23:00:233
230579963,2cyclictest24194-21ssh23:27:543
230509966,0cyclictest769-21lldpd22:00:311
230509966,0cyclictest769-21lldpd21:13:001
230509966,0cyclictest769-21lldpd21:08:341
230509966,0cyclictest769-21lldpd20:51:121
230509965,1cyclictest771-21snmpd23:12:241
230509965,1cyclictest771-21snmpd22:06:071
230509965,1cyclictest771-21snmpd21:44:451
230509965,1cyclictest524-21avahi-daemon21:03:391
230509965,1cyclictest517-21systemd-resolve19:45:441
230509965,1cyclictest27265-21nvmesmart_nvme023:00:191
230509964,2cyclictest771-21snmpd19:26:521
230509963,3cyclictest771-21snmpd00:38:161
230509963,3cyclictest550-21NetworkManager21:25:051
230509963,2cyclictest19956-21ntp_states20:20:211
230509963,2cyclictest11030-21ssh22:14:321
230509962,3cyclictest17594-21ssh23:21:051
230509962,3cyclictest15458-21grep23:50:231
230509960,6cyclictest2038-21grep20:55:141
230479966,0cyclictest771-21snmpd23:49:190
230479966,0cyclictest771-21snmpd20:32:030
230479966,0cyclictest769-21lldpd22:30:150
230479965,1cyclictest771-21snmpd21:10:470
230479965,1cyclictest22464-21cpuspeed_turbos22:25:150
230479965,1cyclictest14880-21pmu-power21:15:210
230479964,2cyclictest771-21snmpd20:16:570
230479964,0cyclictest0-21swapper/021:43:450
230479963,3cyclictest550-21NetworkManager23:08:530
230479963,2cyclictest5730-21ssh23:40:360
230479963,2cyclictest21873-21ssh22:24:560
230479963,2cyclictest20756-21ssh21:52:020
230479963,0cyclictest13213-21ssh22:46:330
230479962,3cyclictest18508-21ssh23:22:200
230479961,4cyclictest23587-21cat00:00:000
230579954,1cyclictest771-21snmpd20:15:033
230579954,1cyclictest7602-21needreboot19:50:163
230579954,1cyclictest23629-21gltestperf21:55:153
230579953,2cyclictest771-21snmpd19:43:053
230579953,2cyclictest771-21snmpd19:13:163
230579952,3cyclictest550-21NetworkManager20:55:213
230579952,2cyclictest7306-21ssh23:11:513
230579952,2cyclictest586-21rs:main3
230579952,2cyclictest586-21rs:main3
230579952,2cyclictest3809-21ssh22:06:463
230579952,2cyclictest27608-21apt-get22:30:183
230579952,2cyclictest22600-21kworker/u16:3+nfsiod00:08:223
230579952,2cyclictest14272-21ssh23:18:413
230579952,2cyclictest11111-21grep20:00:133
230579951,3cyclictest6251-21ls22:40:163
230579951,3cyclictest255-20systemd-journal21:16:073
230579951,3cyclictest10923-21sh23:45:523
230579950,4cyclictest17983-21ntp_states22:20:183
230509953,2cyclictest7744-21ntpq19:50:171
230509953,2cyclictest771-21snmpd21:45:121
230509953,2cyclictest771-21snmpd00:27:201
230509952,3cyclictest9790-21grep19:55:211
230509952,2cyclictest5663-21ssh23:40:301
230509952,2cyclictest255-20systemd-journal20:46:291
230509952,2cyclictest21526-21ssh21:21:381
230509952,2cyclictest12383-21expr22:15:161
230509951,3cyclictest1394-21uname19:35:141
230479954,1cyclictest517-21systemd-resolve00:33:160
230479953,2cyclictest5824-21cat00:15:140
230479952,2cyclictest6148-21ssh22:40:160
230479952,2cyclictest472-21ssh00:10:140
230479952,2cyclictest32168-21sed20:50:160
230479952,2cyclictest25264-21ssh00:36:000
230479952,2cyclictest24271-21ssh21:25:060
230479952,2cyclictest19910-21sh21:20:200
230479951,3cyclictest5130-21df19:45:140
230579941,2cyclictest21944-21cat20:25:183
230579941,2cyclictest14134-21cpu23:50:143
230579941,2cyclictest13750-21sh20:05:173
230509941,3cyclictest550-21NetworkManager20:40:131
230509941,2cyclictest31767-21uname19:30:151
230479940,4cyclictest15888-21ntpq20:10:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional