You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-09-06 - 08:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack8slot6s.osadl.org (updated Sun Sep 06, 2026 00:43:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
198789910531050,3cyclictest23910-21kworker/3:0+events21:37:423
198789910521050,2cyclictest14930-21kworker/3:0+events00:31:373
198659910521050,2cyclictest20373-21kworker/0:0+events20:30:440
198659910521049,3cyclictest20553-21kworker/0:1+events23:33:300
198659910521049,2cyclictest19039-21kworker/0:0+events22:07:440
198789910511049,2cyclictest7503-21kworker/3:0+events22:32:293
198789910511048,2cyclictest23910-21kworker/3:0+events21:33:213
198659910511049,2cyclictest18053-21kworker/0:2+events20:36:390
198789910501049,1cyclictest12958-21kworker/3:2+events00:26:583
198789910501048,2cyclictest7503-21kworker/3:0+events22:36:493
198789910501048,2cyclictest23910-21kworker/3:0+events21:46:063
198789910501048,2cyclictest19016-21kworker/3:1+events00:35:223
198789910501048,2cyclictest18093-21kworker/3:2+events20:27:023
198789910501048,2cyclictest1805-21kworker/3:2+events22:15:223
198699910501048,2cyclictest11630-21kworker/1:0+events00:06:471
198699910501047,3cyclictest11630-21kworker/1:0+events00:03:491
198659910501049,1cyclictest19039-21kworker/0:0+events21:59:210
198659910501048,2cyclictest9841-21kworker/0:0+events20:09:350
198659910501048,2cyclictest5521-21kworker/0:0+events19:15:230
198659910501048,2cyclictest23960-21kworker/0:1+events19:30:220
198659910501048,2cyclictest15983-21kworker/0:1+events20:24:360
198789910491047,2cyclictest7503-21kworker/3:0+events22:24:483
198789910491045,3cyclictest7503-21kworker/3:0+events22:40:453
198699910491047,2cyclictest607-21kworker/1:0+events23:44:101
198659910491047,2cyclictest19039-21kworker/0:0+events22:03:490
198659910491046,3cyclictest9841-21kworker/0:0+events20:18:380
198659910491045,3cyclictest5521-21kworker/0:0+events19:48:190
198789910481046,2cyclictest6837-21kworker/3:2+events21:21:403
198789910481046,2cyclictest3578-21kworker/3:1+events21:12:363
198789910481046,2cyclictest25397-21kworker/3:2+events22:45:443
198789910481045,3cyclictest5110-21kworker/3:1+events21:51:393
198659910481046,2cyclictest1769-21kworker/0:2+events19:55:440
198659910481045,3cyclictest3984-21turbostat19:54:550
198789910471045,2cyclictest3753-21kworker/3:1+events20:55:433
198789910471045,2cyclictest19115-21kworker/3:1+events22:59:353
198789910471045,2cyclictest1533-21kworker/3:2+events21:02:403
198789910471044,3cyclictest3753-21kworker/3:1+events20:54:543
198699910471045,2cyclictest25743-21kworker/1:1+events23:54:511
198659910471045,2cyclictest23960-21kworker/0:1+events19:40:220
198659910471045,2cyclictest11872-21kworker/0:2+events20:11:060
198659910471044,3cyclictest5521-21kworker/0:0+events19:25:210
198789910461044,2cyclictest7503-21kworker/3:0+events22:28:403
198789910461044,2cyclictest23910-21kworker/3:0+events21:43:403
198789910461043,3cyclictest26173-21kworker/3:0+events20:48:133
198789910451043,2cyclictest14930-21kworker/3:0+events23:25:453
198699910451042,3cyclictest24136-21kworker/1:2+events23:39:191
198659910451043,2cyclictest23960-21kworker/0:1+events19:24:030
198659910451042,3cyclictest5521-21kworker/0:0+events19:37:340
198659910441042,2cyclictest5884-21kworker/0:1+events20:00:440
198699910431041,2cyclictest31829-21kworker/1:1+events00:14:321
198659910431041,2cyclictest5521-21kworker/0:0+events19:12:580
198699910421040,2cyclictest31829-21kworker/1:1+events00:24:411
198659910421038,3cyclictest5842-21turbostat22:49:550
198789910411039,2cyclictest3578-21kworker/3:1+events21:08:283
198699910401038,2cyclictest17100-21kworker/1:2+events23:57:551
198699910391037,2cyclictest24136-21kworker/1:2+events20:40:431
198699910381036,2cyclictest6096-21kworker/1:2+events23:47:021
198699910341031,2cyclictest31829-21kworker/1:1+events00:17:361
198789910221021,1cyclictest7503-21kworker/3:0+events22:52:133
198789910191016,3cyclictest20138-21turbostat20:34:553
198749910171016,1cyclictest28609-21kworker/2:2+events23:06:492
198749910171016,1cyclictest21792-21kworker/2:1+events23:01:272
198659910151012,3cyclictest4594-21turbostat23:19:540
198789910121008,4cyclictest24258-21turbostat19:24:543
198789910031001,2cyclictest10031-21turbostat20:09:553
1987499996995,1cyclictest21792-21kworker/2:1+events23:14:062
1987499993991,2cyclictest26206-21turbostat20:49:562
1987899984983,1cyclictest6837-21kworker/3:2+events21:18:363
1987499977976,1cyclictest28609-21kworker/2:2+events23:21:282
1987499977976,1cyclictest21792-21kworker/2:1+events23:18:352
1987899967965,2cyclictest22150-21turbostat20:39:563
1986999963960,3cyclictest9779-21turbostat23:24:551
1986599959957,2cyclictest30217-21turbostat20:59:560
1987899950948,2cyclictest25819-21turbostat23:39:553
1987499925923,2cyclictest20615-21turbostat23:34:542
1987899906905,1cyclictest1805-21kworker/3:2+events22:14:243
1986999903900,3cyclictest13496-21turbostat21:54:561
1986599896893,3cyclictest14314-21turbostat21:24:550
1987899886885,1cyclictest6837-21kworker/3:2+events21:29:483
1986999681678,3cyclictest22150-21turbostat20:39:561
1986599666663,3cyclictest13496-21turbostat21:54:560
1987899633630,3cyclictest16086-21turbostat20:24:543
1986599630628,2cyclictest18113-21turbostat20:29:540
1987499610609,1cyclictest16207-21turbostat22:59:552
1987899588585,3cyclictest7836-21turbostat00:24:553
1986599587584,3cyclictest14994-21turbostat23:29:550
1987899578574,4cyclictest29089-21turbostat22:09:563
1987899558556,2cyclictest9779-21turbostat23:24:553
1986999553550,3cyclictest20615-21turbostat23:34:541
1987899546545,1cyclictest24158-21turbostat20:44:553
19878991818,0cyclictest67750irq/123-lan23:52:293
19878991818,0cyclictest67750irq/123-lan20:10:213
19878991817,1cyclictest67750irq/123-lan23:30:123
19878991817,1cyclictest67750irq/123-lan23:11:073
19878991817,1cyclictest67750irq/123-lan21:59:553
19878991817,1cyclictest67750irq/123-lan00:05:133
19878991717,0cyclictest67750irq/123-lan23:02:343
19878991717,0cyclictest67750irq/123-lan22:04:453
19878991716,1cyclictest67750irq/123-lan23:48:193
19878991716,1cyclictest67750irq/123-lan23:41:593
19878991716,1cyclictest67750irq/123-lan23:05:143
19878991616,0cyclictest67750irq/123-lan19:45:163
19878991615,1cyclictest67750irq/123-lan23:55:123
19878991615,1cyclictest67750irq/123-lan23:15:133
19878991615,1cyclictest67750irq/123-lan19:55:223
19878991615,1cyclictest67750irq/123-lan19:50:043
19878991615,1cyclictest67750irq/123-lan19:19:053
19878991615,1cyclictest67750irq/123-lan00:15:123
19878991615,1cyclictest67750irq/123-lan00:00:143
19878991515,0cyclictest67750irq/123-lan19:12:063
19878991514,1cyclictest67750irq/123-lan20:00:103
19878991514,1cyclictest67750irq/123-lan19:30:133
19878991514,1cyclictest67750irq/123-lan00:13:283
19878991313,0cyclictest67750irq/123-lan20:15:143
19878991312,1cyclictest67750irq/123-lan19:40:113
19878991312,1cyclictest67750irq/123-lan19:35:193
19874991312,1cyclictest762-21ntpd22:02:072
1986999119,2cyclictest784-21snmpd21:21:061
198749998,1cyclictest782-21lldpd22:39:092
198749998,1cyclictest782-21lldpd20:32:192
198749998,1cyclictest782-21lldpd00:19:422
198749998,1cyclictest507-21systemd-resolve22:22:462
198699998,1cyclictest784-21snmpd22:32:461
198699998,1cyclictest784-21snmpd00:38:441
198659998,1cyclictest784-21snmpd21:39:320
198659998,1cyclictest782-21lldpd20:43:100
198659998,1cyclictest526-21avahi-daemon22:56:450
198789987,1cyclictest67750irq/123-lan19:25:123
198749988,0cyclictest782-21lldpd21:36:112
198749988,0cyclictest782-21lldpd20:19:092
198749987,1cyclictest784-21snmpd19:46:572
198749987,1cyclictest784-21snmpd19:28:352
198749987,1cyclictest507-21systemd-resolve00:39:082
198749987,1cyclictest14293-21latency_hist21:20:002
198749986,2cyclictest784-21snmpd23:36:482
198749986,2cyclictest784-21snmpd22:31:312
198749986,2cyclictest784-21snmpd20:44:292
198699988,0cyclictest784-21snmpd19:10:061
198699988,0cyclictest782-21lldpd22:18:141
198699988,0cyclictest782-21lldpd20:27:501
198699987,1cyclictest935-21ntp_kernel_err19:40:161
198699987,1cyclictest784-21snmpd23:12:231
198699987,1cyclictest784-21snmpd19:50:251
198699987,1cyclictest782-21lldpd19:22:181
198699986,2cyclictest784-21snmpd19:15:151
198699986,2cyclictest784-21snmpd00:28:411
198659988,0cyclictest782-21lldpd00:00:520
198659987,1cyclictest784-21snmpd22:39:150
198659987,1cyclictest14987-21cron23:25:000
198659983,4cyclictest28906-21ssh21:33:050
198749977,0cyclictest782-21lldpd23:45:182
198749977,0cyclictest782-21lldpd20:35:092
198749977,0cyclictest782-21lldpd00:20:182
198749977,0cyclictest782-21lldpd00:13:002
198749976,1cyclictest784-21snmpd22:54:232
198749976,1cyclictest784-21snmpd21:45:082
198749976,1cyclictest784-21snmpd19:10:332
198749976,1cyclictest782-21lldpd22:05:212
198749976,1cyclictest526-21avahi-daemon23:55:252
198749976,1cyclictest507-21systemd-resolve00:34:482
198749976,1cyclictest507-21systemd-resolve00:07:332
198749976,1cyclictest16437-21cpuspeed_turbos20:20:112
198749976,1cyclictest1501-21cat22:40:152
198749975,2cyclictest784-21snmpd21:50:472
198749975,2cyclictest784-21snmpd21:28:592
198749975,2cyclictest784-21snmpd21:14:372
198749975,2cyclictest762-21ntpd21:58:502
198749975,2cyclictest762-21ntpd19:20:172
198749973,3cyclictest970-21ntp_states21:00:182
198749973,3cyclictest31811-21ssh23:40:132
198749973,3cyclictest23429-21awk19:15:192
198699977,0cyclictest782-21lldpd22:50:071
198699977,0cyclictest782-21lldpd22:02:081
198699977,0cyclictest782-21lldpd21:42:381
198699977,0cyclictest782-21lldpd20:16:591
198699976,1cyclictest784-21snmpd22:24:311
198699976,1cyclictest784-21snmpd21:59:081
198699976,1cyclictest782-21lldpd21:15:561
198699976,1cyclictest7052-21idleruntime-cro22:15:001
198699976,1cyclictest1882-21idleruntime-cro22:10:001
198699973,3cyclictest2689-21memory21:05:171
198659977,0cyclictest784-21snmpd23:41:510
198659977,0cyclictest784-21snmpd22:25:550
198659977,0cyclictest784-21snmpd21:04:030
198659977,0cyclictest784-21snmpd00:16:280
198659977,0cyclictest782-21lldpd22:51:590
198659977,0cyclictest782-21lldpd21:26:580
198659977,0cyclictest782-21lldpd21:18:460
198659976,1cyclictest784-21snmpd21:46:380
198659976,1cyclictest784-21snmpd21:44:250
198659976,1cyclictest526-21avahi-daemon23:35:550
198659976,1cyclictest526-21avahi-daemon00:33:360
198749966,0cyclictest526-21avahi-daemon20:26:482
198749965,1cyclictest784-21snmpd20:02:222
198749965,1cyclictest526-21avahi-daemon22:26:022
198749965,1cyclictest507-21systemd-resolve21:40:312
198749965,1cyclictest4668-21cat19:50:152
198749965,1cyclictest30293-21sh21:35:012
198749964,2cyclictest784-21snmpd19:36:002
198749964,2cyclictest784-21snmpd00:02:192
198749964,2cyclictest762-21ntpd20:10:172
198749962,3cyclictest14170-21fw_forwarded_lo00:25:142
198699966,0cyclictest782-21lldpd19:47:081
198699966,0cyclictest526-21avahi-daemon20:31:381
198699965,1cyclictest784-21snmpd22:39:061
198699965,1cyclictest0-21swapper/122:40:161
198699963,2cyclictest574-21rs:main1
198699963,2cyclictest27172-21ssh21:30:391
198699962,3cyclictest9312-21mailstats20:00:191
198699962,3cyclictest784-21snmpd21:45:561
198699962,3cyclictest18795-21ssh22:56:221
198659965,1cyclictest784-21snmpd21:10:430
198659965,1cyclictest784-21snmpd00:10:090
198659965,1cyclictest526-21avahi-daemon22:16:270
198659964,2cyclictest784-21snmpd23:52:410
198659964,0cyclictest18241-21ssh00:29:270
198659963,2cyclictest784-21snmpd20:46:300
198659963,2cyclictest6649-21ssh22:14:270
198659963,2cyclictest16335-21ssh23:55:450
198659962,4cyclictest547-21NetworkManager23:45:150
198659962,3cyclictest29231-21ssh00:39:400
198659962,3cyclictest24436-21ssh23:01:580
198659962,3cyclictest2374-21ssh23:11:550
198659962,3cyclictest14173-21ssh22:20:260
198659962,3cyclictest1359-21fw_forwarded_lo22:40:140
198749955,0cyclictest784-21snmpd20:59:022
198749955,0cyclictest526-21avahi-daemon19:56:042
198749953,2cyclictest762-21ntpd22:10:132
198749952,3cyclictest547-21NetworkManager23:26:532
198749952,2cyclictest9363-21ssh23:50:112
198749952,2cyclictest8254-21ntpq22:15:162
198749952,2cyclictest7469-21sed22:45:232
198749952,2cyclictest574-21rs:main2
198749952,2cyclictest29060-21wc19:30:152
198749952,2cyclictest28651-21fschecks_count20:50:152
198749952,2cyclictest16231-21grep21:20:222
198749952,2cyclictest11506-21ls20:05:192
198749950,4cyclictest983-21ntpq19:40:172
198699955,0cyclictest784-21snmpd20:53:121
198699955,0cyclictest782-21lldpd20:10:151
198699955,0cyclictest507-21systemd-resolve23:02:441
198699953,2cyclictest784-21snmpd19:56:551
198699953,2cyclictest784-21snmpd19:25:231
198699953,2cyclictest16364-21apt-get20:20:111
198699952,2cyclictest8789-21ssh21:14:371
198699952,2cyclictest4857-21ssh23:15:031
198699952,2cyclictest31257-21ssh23:09:151
198699952,2cyclictest30648-21entropy20:55:121
198699952,2cyclictest22143-21ssh21:25:531
198699952,2cyclictest18942-21ssh00:30:061
198699952,2cyclictest15859-21cpu23:25:121
198699950,5cyclictest547-21NetworkManager19:32:311
198699950,4cyclictest6417-21df_inode22:45:141
198659953,2cyclictest784-21snmpd23:05:170
198659952,2cyclictest28346-21ssh00:07:560
198659952,2cyclictest23999-21ssh22:30:210
198659952,2cyclictest11334-21ssh00:22:390
198699942,2cyclictest547-21NetworkManager21:00:121
198699941,2cyclictest31109-21cat19:35:161
198699940,4cyclictest27238-21ntpq20:45:161
198699940,3cyclictest18023-21diskstats22:25:121
198699940,3cyclictest10592-21fw_forwarded_lo20:05:121
198659941,2cyclictest2230-21tr21:05:140
198659940,3cyclictest29091-21/usr/sbin/munin20:50:180
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional