You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-01 - 04:30

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack8slot6s.osadl.org (updated Mon Dec 01, 2025 00:43:44)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
18871993333,0cyclictest31291-21fwupd22:38:191
18871991919,0cyclictest66850irq/125-lan23:38:121
18871991918,1cyclictest66850irq/125-lan23:11:161
18871991918,1cyclictest66850irq/125-lan21:41:551
18871991918,1cyclictest66850irq/125-lan19:50:171
18871991918,1cyclictest66850irq/125-lan00:18:521
18871991918,1cyclictest66850irq/125-lan00:04:411
18871991818,0cyclictest66850irq/125-lan23:30:551
18871991818,0cyclictest66850irq/125-lan23:00:161
18871991818,0cyclictest66850irq/125-lan22:50:181
18871991818,0cyclictest66850irq/125-lan22:40:331
18871991818,0cyclictest66850irq/125-lan21:54:011
18871991818,0cyclictest66850irq/125-lan21:46:441
18871991818,0cyclictest66850irq/125-lan21:27:541
18871991818,0cyclictest66850irq/125-lan21:10:211
18871991818,0cyclictest66850irq/125-lan00:37:051
18871991818,0cyclictest66850irq/125-lan00:10:161
18871991817,1cyclictest66850irq/125-lan23:55:291
18871991817,1cyclictest66850irq/125-lan23:45:181
18871991817,1cyclictest66850irq/125-lan23:20:461
18871991817,1cyclictest66850irq/125-lan22:55:171
18871991817,1cyclictest66850irq/125-lan22:45:141
18871991817,1cyclictest66850irq/125-lan22:15:211
18871991817,1cyclictest66850irq/125-lan21:55:161
18871991817,1cyclictest66850irq/125-lan00:08:491
18871991717,0cyclictest771-21snmpd22:33:171
18871991717,0cyclictest66850irq/125-lan23:15:411
18871991717,0cyclictest66850irq/125-lan22:25:171
18871991717,0cyclictest66850irq/125-lan22:00:161
18871991717,0cyclictest66850irq/125-lan21:20:351
18871991717,0cyclictest66850irq/125-lan21:18:031
18871991717,0cyclictest66850irq/125-lan20:30:221
18871991717,0cyclictest66850irq/125-lan00:28:161
18871991716,1cyclictest66850irq/125-lan23:50:141
18871991716,1cyclictest66850irq/125-lan23:42:431
18871991716,1cyclictest66850irq/125-lan23:25:371
18871991716,1cyclictest66850irq/125-lan23:06:541
18871991716,1cyclictest66850irq/125-lan22:21:571
18871991716,1cyclictest66850irq/125-lan22:10:191
18871991716,1cyclictest66850irq/125-lan22:08:431
18871991716,1cyclictest66850irq/125-lan21:35:131
18871991716,1cyclictest66850irq/125-lan21:31:181
18871991716,1cyclictest66850irq/125-lan20:25:191
18871991716,1cyclictest66850irq/125-lan19:22:081
18871991616,0cyclictest771-21snmpd20:16:191
18871991616,0cyclictest771-21snmpd19:35:471
18871991616,0cyclictest66850irq/125-lan20:35:151
18871991616,0cyclictest66850irq/125-lan20:02:081
18871991616,0cyclictest66850irq/125-lan19:25:141
18871991615,1cyclictest66850irq/125-lan21:06:521
18871991615,1cyclictest66850irq/125-lan20:10:141
18871991615,1cyclictest66850irq/125-lan19:30:161
18871991615,1cyclictest66850irq/125-lan00:30:191
18871991615,1cyclictest66850irq/125-lan00:20:211
18871991515,0cyclictest66850irq/125-lan20:50:141
18871991515,0cyclictest66850irq/125-lan20:45:191
18871991515,0cyclictest66850irq/125-lan20:40:231
18871991515,0cyclictest66850irq/125-lan20:21:081
18871991515,0cyclictest66850irq/125-lan20:05:181
18871991515,0cyclictest66850irq/125-lan19:40:191
18871991515,0cyclictest66850irq/125-lan19:15:201
18871991414,0cyclictest771-21snmpd19:57:121
18871991414,0cyclictest66850irq/125-lan20:55:221
18871991413,1cyclictest66850irq/125-lan19:45:161
18871991413,1cyclictest66850irq/125-lan19:13:251
18871991313,0cyclictest66850irq/125-lan21:00:171
1887699107,3cyclictest771-21snmpd00:01:182
188769998,1cyclictest769-21lldpd23:39:172
188769998,1cyclictest524-21avahi-daemon23:50:412
188769998,1cyclictest255-20systemd-journal22:59:202
188769996,3cyclictest771-21snmpd23:56:482
188679998,1cyclictest771-21snmpd22:19:260
188679998,1cyclictest771-21snmpd21:40:400
188679998,1cyclictest771-21snmpd21:05:520
188809988,0cyclictest771-21snmpd22:33:293
188809988,0cyclictest771-21snmpd22:25:383
188809988,0cyclictest771-21snmpd22:09:473
188809988,0cyclictest771-21snmpd21:48:063
188809988,0cyclictest771-21snmpd19:24:563
188809988,0cyclictest771-21snmpd00:36:253
188809988,0cyclictest771-21snmpd00:07:183
188809988,0cyclictest769-21lldpd21:11:373
188809988,0cyclictest769-21lldpd00:22:423
188809987,1cyclictest771-21snmpd20:50:563
188809987,1cyclictest524-21avahi-daemon22:54:063
188809987,1cyclictest517-21systemd-resolve23:29:463
188809986,2cyclictest771-21snmpd22:24:293
188809986,2cyclictest771-21snmpd21:59:133
188809986,2cyclictest771-21snmpd19:58:573
188769988,0cyclictest771-21snmpd23:07:312
188769988,0cyclictest771-21snmpd19:10:502
188769988,0cyclictest769-21lldpd23:49:562
188769988,0cyclictest769-21lldpd23:33:492
188769988,0cyclictest769-21lldpd22:46:512
188769988,0cyclictest769-21lldpd22:38:282
188769988,0cyclictest769-21lldpd21:26:392
188769987,1cyclictest771-21snmpd19:49:532
188769987,1cyclictest524-21avahi-daemon21:56:092
188769987,1cyclictest524-21avahi-daemon21:19:582
188769986,2cyclictest771-21snmpd23:13:132
188769986,2cyclictest771-21snmpd22:08:442
188769986,2cyclictest771-21snmpd20:33:362
188679988,0cyclictest771-21snmpd20:51:350
188679988,0cyclictest769-21lldpd21:58:000
188679988,0cyclictest769-21lldpd19:33:320
188679988,0cyclictest769-21lldpd00:16:460
188679988,0cyclictest769-21lldpd00:04:220
188679987,1cyclictest771-21snmpd22:57:510
188679987,1cyclictest771-21snmpd20:58:590
188679987,1cyclictest771-21snmpd20:37:520
188679987,1cyclictest524-21avahi-daemon23:28:360
188679987,1cyclictest517-21systemd-resolve23:51:490
188679987,1cyclictest517-21systemd-resolve21:48:270
188679986,2cyclictest771-21snmpd23:15:010
188679986,2cyclictest771-21snmpd22:20:170
188679986,2cyclictest771-21snmpd19:54:000
188679985,2cyclictest771-21snmpd20:33:280
188809977,0cyclictest771-21snmpd23:47:173
188809977,0cyclictest771-21snmpd21:15:353
188809977,0cyclictest771-21snmpd19:30:073
188809977,0cyclictest769-21lldpd23:00:553
188809977,0cyclictest769-21lldpd19:12:593
188809977,0cyclictest524-21avahi-daemon20:19:223
188809976,1cyclictest524-21avahi-daemon23:36:403
188809976,1cyclictest524-21avahi-daemon21:51:273
188809976,1cyclictest517-21systemd-resolve00:01:253
188809976,1cyclictest511ktimers/322:10:113
188809975,2cyclictest771-21snmpd23:05:343
188809975,2cyclictest771-21snmpd20:32:153
188809975,2cyclictest771-21snmpd20:11:093
188809975,2cyclictest771-21snmpd19:38:363
188809974,2cyclictest586-21rs:main3
188809973,3cyclictest19259-21fw_forwarded_lo21:25:143
188769977,0cyclictest771-21snmpd23:02:462
188769977,0cyclictest771-21snmpd21:01:022
188769977,0cyclictest771-21snmpd20:17:062
188769977,0cyclictest771-21snmpd19:37:242
188769977,0cyclictest769-21lldpd22:52:252
188769977,0cyclictest769-21lldpd21:36:442
188769977,0cyclictest769-21lldpd21:23:212
188769977,0cyclictest769-21lldpd21:09:192
188769977,0cyclictest769-21lldpd20:42:352
188769977,0cyclictest769-21lldpd20:22:262
188769977,0cyclictest769-21lldpd19:53:402
188769977,0cyclictest769-21lldpd00:07:302
188769976,1cyclictest9494-21needreboot20:05:152
188769975,2cyclictest771-21snmpd22:28:112
188769975,2cyclictest771-21snmpd20:55:082
188769975,2cyclictest771-21snmpd20:46:562
188769975,2cyclictest23791-21grep22:00:142
188769974,2cyclictest586-21rs:main2
188769973,3cyclictest9927-21ntp_states23:20:172
188679977,0cyclictest771-21snmpd23:11:190
188679977,0cyclictest771-21snmpd23:06:070
188679977,0cyclictest771-21snmpd20:28:250
188679977,0cyclictest769-21lldpd23:35:110
188679977,0cyclictest769-21lldpd21:53:240
188679977,0cyclictest769-21lldpd19:10:090
188679976,1cyclictest771-21snmpd20:04:300
188679976,1cyclictest771-21snmpd19:35:050
188679976,1cyclictest771-21snmpd00:25:010
188679975,2cyclictest771-21snmpd21:11:500
188679975,2cyclictest771-21snmpd19:24:110
188679974,2cyclictest255-20systemd-journal21:37:510
188679973,3cyclictest690-21ssh22:09:410
188809966,0cyclictest771-21snmpd20:09:573
188809966,0cyclictest771-21snmpd00:28:583
188809966,0cyclictest769-21lldpd22:43:093
188809966,0cyclictest769-21lldpd21:40:303
188809966,0cyclictest524-21avahi-daemon21:01:153
188809966,0cyclictest517-21systemd-resolve21:06:283
188809966,0cyclictest17723-21gltestperf20:25:153
188809965,1cyclictest771-21snmpd22:40:003
188809965,1cyclictest255-20systemd-journal22:45:183
188809965,1cyclictest24852-21turbostat19:29:563
188809963,3cyclictest550-21NetworkManager23:53:193
188809963,3cyclictest26175-21grep20:45:143
188809963,2cyclictest3576-21ssh00:19:143
188809963,2cyclictest255-20systemd-journal21:23:523
188809963,2cyclictest13406-21ntpq23:55:173
188809962,4cyclictest31280-21cat21:35:213
188769966,0cyclictest771-21snmpd23:19:462
188769966,0cyclictest769-21lldpd23:29:192
188769966,0cyclictest769-21lldpd20:25:362
188769965,1cyclictest771-21snmpd21:52:342
188769965,1cyclictest769-21lldpd22:44:152
188769965,1cyclictest524-21avahi-daemon21:41:312
188769965,1cyclictest524-21avahi-daemon20:11:492
188769965,1cyclictest14882-21gltestperf00:30:162
188769964,2cyclictest771-21snmpd21:45:372
188769964,2cyclictest771-21snmpd19:24:202
188769964,2cyclictest12444-21cat22:20:132
188769963,2cyclictest255-20systemd-journal21:30:362
188769962,3cyclictest6753-21ssh00:21:292
188679966,0cyclictest771-21snmpd22:48:420
188679966,0cyclictest771-21snmpd19:25:290
188679966,0cyclictest771-21snmpd00:20:520
188679966,0cyclictest769-21lldpd22:50:510
188679966,0cyclictest769-21lldpd00:38:180
188679966,0cyclictest517-21systemd-resolve20:19:090
188679965,1cyclictest771-21snmpd23:03:100
188679965,1cyclictest524-21avahi-daemon22:41:440
188679965,1cyclictest255-20systemd-journal22:36:130
188679964,2cyclictest771-21snmpd22:34:450
188679964,2cyclictest550-21NetworkManager21:31:560
188679963,3cyclictest550-21NetworkManager00:13:410
188679963,2cyclictest586-21rs:main0
188679963,2cyclictest11026-21ssh21:16:320
188679962,3cyclictest26726-21ntp_states20:45:190
188679962,3cyclictest22463-21ssh23:32:360
188679962,3cyclictest19921-21ssh22:26:360
188809955,0cyclictest24408-21latency20:40:173
188809954,1cyclictest771-21snmpd22:04:343
188809954,1cyclictest771-21snmpd19:46:183
188809954,1cyclictest749-21ntpd19:15:133
188809953,2cyclictest771-21snmpd20:58:323
188809952,2cyclictest31794-21memory23:10:163
188809952,2cyclictest26109-21ssh21:30:303
188809952,2cyclictest16476-21ssh00:31:133
188809951,4cyclictest7713-21sed22:15:193
188809951,4cyclictest550-21NetworkManager20:36:583
188809951,3cyclictest517-21systemd-resolve23:32:443
188809950,4cyclictest30496-21sed23:40:163
188809950,4cyclictest26818-21diskstats00:10:093
188769955,0cyclictest771-21snmpd19:25:562
188769955,0cyclictest584-21in:imuxsock20:38:362
188769954,1cyclictest771-21snmpd19:15:012
188769954,1cyclictest524-21avahi-daemon22:31:312
188769954,1cyclictest4530-21ntpq21:10:172
188769953,2cyclictest771-21snmpd20:03:082
188769953,2cyclictest771-21snmpd00:15:392
188769953,2cyclictest749-21ntpd22:10:282
188769952,3cyclictest550-21NetworkManager00:25:472
188769952,2cyclictest6508-21dpkg22:15:002
188769952,2cyclictest31239-21ssh23:40:262
188769952,2cyclictest19163-21ssh00:35:142
188769951,3cyclictest31986-21ntpq19:40:182
188679955,0cyclictest771-21snmpd19:55:180
188679955,0cyclictest769-21lldpd22:12:400
188679954,1cyclictest23710-21sh21:30:000
188679953,2cyclictest771-21snmpd19:49:240
188679953,2cyclictest771-21snmpd00:32:010
188679952,3cyclictest550-21NetworkManager20:42:040
188679952,2cyclictest32677-21ntp_kernel_pll_21:00:170
188679952,2cyclictest30112-21grep23:40:140
188679952,2cyclictest26152-21ssh22:02:060
188679952,2cyclictest25494-21seq00:08:490
188679952,2cyclictest21047-21sort19:15:130
188679952,2cyclictest15424-21grep20:20:140
188679952,2cyclictest14391-21ssh23:55:530
188679951,3cyclictest9042-21diskstats20:05:120
188679951,3cyclictest15557-21ssh21:20:440
188679951,3cyclictest12979-21ssh23:23:550
188679950,4cyclictest31344-21diskstats19:40:140
188809943,1cyclictest9526-21irqstats23:20:153
188809941,2cyclictest7248-21smartctl20:00:133
188809941,2cyclictest3764-21grep19:50:213
188809940,4cyclictest16447-21ntpq22:55:183
188809940,3cyclictest31626-21interrupts19:40:163
188809940,3cyclictest16197-21ntp_states20:20:193
188769942,2cyclictest586-21rs:main2
188769942,2cyclictest550-21NetworkManager20:50:322
188679942,2cyclictest771-21snmpd20:10:090
188769930,3cyclictest27285-21df19:30:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional