You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-01 - 06:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot6s.osadl.org (updated Tue Sep 01, 2026 00:43:40)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
167089910511049,2cyclictest17354-21kworker/2:2+events00:30:212
167119910501049,1cyclictest30491-21kworker/3:0+events20:11:143
167119910491048,1cyclictest504-21kworker/3:1+events19:55:443
167089910491047,2cyclictest29749-21kworker/2:0+events21:37:192
167119910481046,2cyclictest5101-21kworker/3:2+events20:23:583
167039910481047,1cyclictest23944-21kworker/1:0+events22:04:071
167119910461045,1cyclictest22935-21kworker/3:2+events19:33:013
167089910461045,1cyclictest29749-21kworker/2:0+events22:23:052
167119910441043,1cyclictest30491-21kworker/3:0+events20:27:303
167119910411040,1cyclictest3166-21kworker/3:2+events21:12:223
167119910411040,1cyclictest20013-21kworker/3:1+events00:27:573
167039910381037,1cyclictest32370-21kworker/1:2+events22:14:541
167119910371035,2cyclictest10540-21kworker/3:1+events23:09:063
167119910351033,2cyclictest20013-21kworker/3:1+events00:20:233
167039910341033,1cyclictest6885-21kworker/1:0+events22:18:471
167039910311030,1cyclictest18601-21kworker/1:1+events21:48:051
166959910311029,2cyclictest22946-21turbostat23:39:550
167119910301029,1cyclictest32620-21kworker/3:2+events23:15:433
167119910281027,1cyclictest22935-21kworker/3:2+events19:36:483
167119910261024,1cyclictest504-21kworker/3:1+events19:50:203
167089910261024,2cyclictest29749-21kworker/2:0+events22:34:322
167119910241021,3cyclictest20961-21turbostat00:39:563
167119910241021,3cyclictest20961-21turbostat00:39:563
167119910241020,4cyclictest27517-21turbostat20:59:553
167039910221021,1cyclictest23944-21kworker/1:0+events22:05:441
167039910221021,1cyclictest12857-21kworker/1:0+events21:50:531
166959910221020,2cyclictest23724-21kworker/0:2+events20:52:350
167119910211020,1cyclictest5101-21kworker/3:2+events20:09:303
167119910211018,3cyclictest27247-21turbostat21:39:553
167039910211020,1cyclictest14949-21kworker/1:1+events23:58:551
167119910201019,1cyclictest22935-21kworker/3:2+events19:29:233
167089910191018,1cyclictest23202-21kworker/2:1+events21:25:212
167119910171016,1cyclictest22935-21kworker/3:2+events19:46:483
166959910071005,2cyclictest17976-21kworker/0:0+events20:34:140
167119910041002,2cyclictest20013-21kworker/3:1+events00:06:293
166959910031001,2cyclictest17976-21kworker/0:0+events20:55:210
167119910021000,2cyclictest8804-21kworker/3:1+events21:18:293
166959910021000,2cyclictest23724-21kworker/0:2+events20:47:550
1671199996995,1cyclictest30491-21kworker/3:0+events21:09:423
1671199995993,2cyclictest12232-21turbostat23:59:553
1671199995993,2cyclictest10540-21kworker/3:1+events22:54:373
1670899992991,1cyclictest17354-21kworker/2:2+events00:02:432
1670399991990,1cyclictest6885-21kworker/1:0+events23:48:511
1670399991990,1cyclictest18353-21kworker/1:1+events21:59:091
1671199986982,4cyclictest4511-21turbostat22:19:553
1670899985984,1cyclictest23202-21kworker/2:1+events21:30:442
1670899983980,3cyclictest30782-21turbostat19:49:552
1671199980980,0cyclictest3963-21kworker/3:3+events23:23:423
1669599979977,2cyclictest3330-21turbostat22:49:550
1670399977976,1cyclictest32370-21kworker/1:2+events21:44:531
1671199973972,1cyclictest10540-21kworker/3:1+events23:04:333
1670399971970,1cyclictest31635-21kworker/1:2+events19:18:041
1671199969968,1cyclictest3882-21kworker/3:0+events22:49:153
1671199969968,1cyclictest10540-21kworker/3:1+events23:10:433
1671199968968,0cyclictest20013-21kworker/3:1+events23:31:283
1670899968967,1cyclictest12193-21kworker/2:2+events21:20:432
1670399968967,1cyclictest28960-21kworker/1:2+events23:52:441
1670399966965,1cyclictest31635-21kworker/1:2+events19:14:051
1671199964963,1cyclictest32620-21kworker/3:2+events00:10:443
1671199964961,3cyclictest15331-21turbostat00:34:553
1671199962961,1cyclictest5101-21kworker/3:2+events20:18:493
1671199962960,2cyclictest30454-21kworker/3:2+events22:40:423
1669599961959,2cyclictest1194-21turbostat21:14:550
1671199960958,2cyclictest32620-21kworker/3:2+events00:15:233
1671199958957,1cyclictest32620-21kworker/3:2+events23:37:133
1671199958957,1cyclictest30491-21kworker/3:0+events20:03:493
1671199958957,1cyclictest22935-21kworker/3:2+events19:44:273
1671199957956,1cyclictest30491-21kworker/3:0+events21:02:293
1671199957956,1cyclictest16431-21kworker/3:0+events19:23:033
1671199956955,1cyclictest15970-21kworker/3:0+events22:57:003
1670399955954,1cyclictest6885-21kworker/1:0+events00:35:221
1670399955954,1cyclictest6885-21kworker/1:0+events00:35:221
1671199953952,1cyclictest32620-21kworker/3:2+events23:26:283
1670399953950,3cyclictest12556-21turbostat23:29:551
1670399952951,1cyclictest28960-21kworker/1:2+events23:41:281
1670899944943,1cyclictest21140-21kworker/2:0+events20:36:502
1670899943942,1cyclictest19102-21kworker/2:2+events20:40:442
1670899943942,1cyclictest10105-21kworker/2:1+events22:28:172
1669599942939,3cyclictest28745-21turbostat19:44:560
1670899940937,3cyclictest6423-21turbostat21:19:552
1670899932930,2cyclictest27966-21kworker/2:1+events22:38:292
1669599910908,2cyclictest9335-21turbostat20:14:540
1669599901899,2cyclictest25328-21turbostat22:39:550
1670399885883,2cyclictest24720-21turbostat19:34:551
1671199870868,2cyclictest17428-21turbostat20:34:543
1670899868866,2cyclictest31576-21turbostat21:09:542
1669599866864,2cyclictest22698-21turbostat19:29:560
1669599866864,2cyclictest18643-21turbostat19:19:550
1671199854852,2cyclictest21450-21turbostat20:44:543
1669599668665,3cyclictest21450-21turbostat20:44:550
1669599630628,2cyclictest15396-21turbostat20:29:540
1670899614611,3cyclictest17428-21turbostat20:34:542
1671199598596,2cyclictest25328-21turbostat22:39:543
1670399584581,3cyclictest27247-21turbostat21:39:551
1671199561560,1cyclictest18643-21turbostat19:19:553
1670899537536,1cyclictest12232-21turbostat23:59:552
1670399534533,1cyclictest22946-21turbostat23:39:541
1670899533530,3cyclictest10124-21turbostat00:29:562
1670899526525,1cyclictest4511-21turbostat22:19:552
1670399525524,1cyclictest15331-21turbostat00:34:551
1671199517515,2cyclictest17432-21turbostat00:04:563
16711992019,1cyclictest67750irq/123-lan23:50:013
16711991918,1cyclictest67750irq/123-lan20:35:153
16711991818,0cyclictest67750irq/123-lan23:40:103
16711991818,0cyclictest67750irq/123-lan21:47:533
16711991817,1cyclictest67750irq/123-lan23:47:233
16711991817,1cyclictest67750irq/123-lan21:55:233
16711991816,1cyclictest67750irq/123-lan22:25:113
16711991717,0cyclictest67750irq/123-lan22:10:533
16711991717,0cyclictest67750irq/123-lan22:05:203
16711991716,1cyclictest67750irq/123-lan22:31:553
16711991716,1cyclictest67750irq/123-lan22:00:103
16711991716,1cyclictest67750irq/123-lan21:53:233
16711991716,1cyclictest67750irq/123-lan21:30:133
16711991716,1cyclictest67750irq/123-lan21:25:503
16711991716,1cyclictest67750irq/123-lan21:23:333
16711991616,0cyclictest67750irq/123-lan21:43:543
16711991615,1cyclictest67750irq/123-lan22:22:033
16711991615,1cyclictest67750irq/123-lan20:45:183
16711991413,1cyclictest67750irq/123-lan19:10:003
16711991212,0cyclictest67750irq/123-lan20:50:133
1669599114,4cyclictest526-21avahi-daemon23:01:050
1669599108,2cyclictest782-21lldpd19:59:430
1669599102,6cyclictest27379-21cat22:05:150
1669599100,8cyclictest7735-21ntpq21:15:180
1669599100,7cyclictest25294-21ntpq23:05:160
1669599100,7cyclictest1193-21diskmemload21:25:100
1669599100,6cyclictest25743-21ntpq19:30:150
1669599100,6cyclictest21712-21ntpq19:20:160
1669599100,6cyclictest19118-21latency_hist23:00:010
1669599100,6cyclictest13546-21ssh22:54:470
1669599100,6cyclictest1193-21diskmemload22:00:100
1669599100,5cyclictest3424-21ssh22:13:300
1669599100,5cyclictest12839-21ntpq21:20:160
167039998,1cyclictest784-21snmpd20:58:241
167039996,3cyclictest784-21snmpd23:20:191
166959996,2cyclictest10971-21cat00:25:150
166959992,5cyclictest26968-21kworker/u16:0+rpciod23:30:020
166959990,7cyclictest30626-21ntpq23:10:170
166959990,7cyclictest29396-21ssh00:11:140
166959990,7cyclictest1193-21diskmemload22:28:230
166959990,6cyclictest6197-21ntpq00:20:170
166959990,6cyclictest29862-21ntpq23:40:160
166959990,6cyclictest22757-21taskset22:31:260
166959990,5cyclictest30545-21ntpq21:00:180
166959990,5cyclictest27187-21cut19:35:130
166959990,5cyclictest17070-21ntpq00:30:180
166959990,5cyclictest1193-21diskmemload23:59:290
166959990,5cyclictest1193-21diskmemload22:43:290
166959990,5cyclictest1193-21diskmemload21:48:380
166959990,5cyclictest1193-21diskmemload21:35:030
166959990,5cyclictest1193-21diskmemload00:35:410
166959990,5cyclictest1193-21diskmemload00:35:410
166959990,5cyclictest1193-21diskmemload00:18:410
166959990,5cyclictest11199-21ssh21:50:070
166959990,5cyclictest10870-21/usr/sbin/munin22:20:160
166959990,4cyclictest8554-21ntpq23:20:140
166959990,4cyclictest20463-21ntpq20:35:170
167089988,0cyclictest784-21snmpd22:42:082
167089988,0cyclictest782-21lldpd22:10:202
167089988,0cyclictest782-21lldpd22:05:472
167089988,0cyclictest782-21lldpd20:10:412
167089988,0cyclictest782-21lldpd00:12:442
167089987,1cyclictest784-21snmpd21:04:142
167089987,1cyclictest784-21snmpd20:07:392
167089987,1cyclictest782-21lldpd20:49:562
167089986,2cyclictest784-21snmpd19:40:212
167089986,2cyclictest2971-21cat23:15:142
167039988,0cyclictest784-21snmpd21:17:011
167039988,0cyclictest782-21lldpd20:25:191
167039987,1cyclictest782-21lldpd19:43:501
167039987,1cyclictest526-21avahi-daemon22:56:081
167039986,2cyclictest784-21snmpd23:08:041
167039986,2cyclictest3961-21cat19:55:151
166959988,0cyclictest784-21snmpd19:51:060
166959988,0cyclictest782-21lldpd21:30:340
166959987,1cyclictest784-21snmpd19:11:040
166959986,2cyclictest526-21avahi-daemon23:49:340
166959986,2cyclictest526-21avahi-daemon20:23:350
166959980,8cyclictest17346-21ntpq21:55:180
166959980,7cyclictest6666-21ssh22:15:470
166959980,7cyclictest18772-21ntpq00:00:170
166959980,6cyclictest9061-21ssh23:50:370
166959980,6cyclictest791-21runrttasks20:19:460
166959980,5cyclictest376-21sendmail_mailst21:05:180
166959980,5cyclictest3065-21sed23:15:140
166959980,5cyclictest23451-21taskset00:05:470
166959980,5cyclictest1925-21sh21:40:210
167089977,0cyclictest784-21snmpd23:12:082
167089977,0cyclictest784-21snmpd23:06:222
167089976,1cyclictest784-21snmpd21:59:022
167089976,1cyclictest782-21lldpd22:55:222
167089976,1cyclictest782-21lldpd22:02:152
167089976,1cyclictest782-21lldpd19:52:452
167089976,1cyclictest526-21avahi-daemon19:26:102
167089976,1cyclictest421ktimers/223:00:102
167089975,2cyclictest784-21snmpd20:25:042
167089975,2cyclictest784-21snmpd19:30:562
167089975,2cyclictest784-21snmpd00:36:462
167089975,2cyclictest784-21snmpd00:36:462
167089975,2cyclictest784-21snmpd00:24:102
167089975,2cyclictest784-21snmpd00:07:422
167039977,0cyclictest782-21lldpd00:20:321
167039976,1cyclictest507-21systemd-resolve22:53:361
167039976,1cyclictest22683-21fschecks_count21:30:141
167039975,2cyclictest784-21snmpd21:11:001
167039975,2cyclictest784-21snmpd21:01:511
167039975,2cyclictest784-21snmpd20:31:191
167039975,2cyclictest784-21snmpd00:16:151
167039973,3cyclictest29866-21df23:10:121
167039973,3cyclictest12343-21fw_forwarded_lo21:20:121
166959970,4cyclictest9020-21taskset20:06:190
166959970,4cyclictest14724-21taskset23:25:480
166959970,4cyclictest0-21swapper/020:02:070
167089966,0cyclictest784-21snmpd20:18:102
167089966,0cyclictest782-21lldpd23:23:392
167089966,0cyclictest782-21lldpd00:15:592
167089965,1cyclictest784-21snmpd23:52:482
167089965,1cyclictest526-21avahi-daemon20:23:252
167089964,2cyclictest784-21snmpd23:40:502
167089964,2cyclictest784-21snmpd21:48:522
167089964,2cyclictest784-21snmpd19:21:502
167089964,1cyclictest784-21snmpd19:37:302
167089963,2cyclictest3910-21ssh21:11:332
167089963,2cyclictest19705-21ssh23:30:302
167089962,3cyclictest5960-21smartctl20:00:152
167039966,0cyclictest784-21snmpd22:20:021
167039966,0cyclictest784-21snmpd20:43:291
167039966,0cyclictest784-21snmpd20:01:461
167039966,0cyclictest782-21lldpd00:28:241
167039965,1cyclictest526-21avahi-daemon22:46:581
167039965,1cyclictest526-21avahi-daemon19:47:321
167039965,1cyclictest24627-21nvmesmart_nvme020:45:181
167039964,2cyclictest784-21snmpd19:25:051
167039964,2cyclictest784-21snmpd19:23:291
167039964,2cyclictest784-21snmpd00:09:271
167039964,2cyclictest784-21snmpd00:00:361
167039964,2cyclictest245-20systemd-journal20:20:171
167039963,3cyclictest547-21NetworkManager23:02:171
167039962,3cyclictest537-21ssh22:41:111
167039962,3cyclictest25723-21ssh22:35:131
167039962,3cyclictest245-20systemd-journal20:10:221
167039962,3cyclictest20554-21ntp_states20:35:181
167039960,5cyclictest19493-21sed23:30:211
166959965,1cyclictest31543-21cpuspeed_turbos19:45:120
167089955,0cyclictest782-21lldpd23:45:132
167089953,2cyclictest547-21NetworkManager21:40:262
167089952,2cyclictest5317-21ssh22:45:322
167089952,2cyclictest13241-21sh23:25:142
167089952,2cyclictest10276-21sed22:50:212
167089950,4cyclictest19698-21ntpq19:15:172
167039954,1cyclictest11680-21cpuspeed_turbos20:15:131
167039953,2cyclictest762-21ntpd19:51:161
167039952,2cyclictest7874-21fw_packets20:05:131
167039952,2cyclictest6668-21ssh23:19:051
167039952,2cyclictest666-21sed21:05:201
167039952,2cyclictest28076-21wc19:35:211
167039952,2cyclictest27581-21gltestperf00:10:101
167039952,2cyclictest22323-21ssh22:30:501
167039952,2cyclictest16604-21sed22:25:201
167089943,1cyclictest0-21swapper/220:50:122
167089942,2cyclictest762-21ntpd21:50:022
167089942,2cyclictest23578-21uniq23:35:122
167089941,2cyclictest3929-21sort19:55:152
167089941,2cyclictest28108-21grep20:55:142
167089940,3cyclictest17183-21fw_conntrack19:10:132
167039943,1cyclictest25839-21cpuspeed_turbos20:50:111
167039941,2cyclictest17119-21sh21:25:041
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional