You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-28 - 12:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot6s.osadl.org (updated Fri Aug 28, 2026 00:43:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
323569910541049,4cyclictest16970-21kworker/3:2+events21:58:173
323569910531048,4cyclictest18441-21kworker/3:0+events19:58:403
323569910521050,2cyclictest14160-21kworker/3:0+events23:31:063
323569910521049,3cyclictest7104-21kworker/3:2+events23:43:083
323569910521049,3cyclictest10808-21kworker/3:0+events21:02:313
323569910521048,4cyclictest7104-21kworker/3:2+events23:47:123
323569910511049,2cyclictest7104-21kworker/3:2+events00:22:423
323569910511049,2cyclictest21348-21kworker/3:1+events20:11:503
323569910511047,2cyclictest24572-21kworker/3:2+events20:16:173
323569910511047,2cyclictest21348-21kworker/3:1+events20:00:453
323569910511047,2cyclictest17023-21kworker/3:1+events21:40:433
323569910511047,2cyclictest10808-21kworker/3:0+events21:07:003
323569910501048,2cyclictest1871-21kworker/3:1+events19:15:433
323569910501048,2cyclictest16970-21kworker/3:2+events22:01:173
323569910501047,3cyclictest5349-21kworker/3:1+events20:40:433
323569910501047,3cyclictest10403-21kworker/3:0+events19:21:033
323569910501047,2cyclictest5349-21kworker/3:1+events20:54:023
323569910501047,2cyclictest29089-21kworker/3:1+events22:06:513
323569910491048,1cyclictest16970-21kworker/3:2+events21:48:323
323569910491046,2cyclictest7104-21kworker/3:2+events23:55:443
323569910491046,2cyclictest4171-21kworker/3:2+events19:31:483
323569910491046,2cyclictest14160-21kworker/3:0+events22:13:293
323569910481045,2cyclictest5349-21kworker/3:1+events20:48:173
323409910481047,1cyclictest2293-21kworker/0:0+events22:35:450
323569910471045,2cyclictest24572-21kworker/3:2+events20:22:453
323569910471043,3cyclictest1481-21kworker/3:1+events23:54:203
323409910471044,3cyclictest21943-21turbostat00:24:550
323569910461044,2cyclictest5349-21kworker/3:1+events20:56:473
323569910461044,2cyclictest28750-21turbostat23:29:553
323569910461044,2cyclictest24572-21kworker/3:2+events20:07:063
323569910441042,2cyclictest3029-21kworker/3:1+events00:02:573
323409910431040,3cyclictest29984-21turbostat22:59:560
323569910421039,3cyclictest16970-21kworker/3:2+events21:52:553
323529910341033,1cyclictest25602-21kworker/2:2+events21:13:472
323409910331031,2cyclictest4806-21kworker/0:2+events23:06:490
323409910331031,2cyclictest20370-21kworker/0:1+events00:31:490
323409910311029,2cyclictest20370-21kworker/0:1+events00:26:500
323409910311027,4cyclictest2293-21kworker/0:0+events20:31:040
323409910301028,2cyclictest663-21kworker/0:2+events22:29:140
323409910301028,2cyclictest663-21kworker/0:2+events22:29:140
323409910301028,2cyclictest4806-21kworker/0:2+events23:22:200
323409910291027,2cyclictest20985-21kworker/0:1+events22:22:360
323409910281026,2cyclictest21897-21kworker/0:0+events22:46:290
323569910271023,3cyclictest1481-21kworker/3:1+events23:39:003
323409910271025,2cyclictest31246-21kworker/0:2+events20:27:110
323409910251023,2cyclictest20370-21kworker/0:1+events23:18:200
323409910241022,2cyclictest2293-21kworker/0:0+events22:17:350
323529910231022,1cyclictest8549-21kworker/2:0+events00:08:452
323529910231021,2cyclictest17745-21kworker/2:1+events21:20:232
323409910231021,2cyclictest8991-21kworker/0:1+events19:41:270
323529910221020,2cyclictest8549-21kworker/2:0+events00:15:202
323529910221020,2cyclictest17745-21kworker/2:1+events21:25:442
323409910221020,2cyclictest4806-21kworker/0:2+events23:28:060
323409910211019,2cyclictest8991-21kworker/0:1+events19:35:230
323529910191018,1cyclictest2995-21kworker/2:2+events21:37:122
323409910191017,2cyclictest2293-21kworker/0:0+events20:37:560
323409910181015,3cyclictest27574-21kworker/0:2+events00:35:450
323409910181015,3cyclictest21897-21kworker/0:0+events23:02:570
323529910171016,1cyclictest2995-21kworker/2:2+events21:32:432
323409910131011,2cyclictest21897-21kworker/0:0+events22:53:480
323409910121010,2cyclictest8991-21kworker/0:1+events19:48:370
323409910121010,2cyclictest6255-21kworker/0:1+events22:32:460
323479910101009,1cyclictest11940-21turbostat23:44:551
323409910101006,2cyclictest23615-21kworker/0:0+events19:10:220
323409910091006,2cyclictest6255-21kworker/0:1+events22:42:150
323409910091005,2cyclictest4806-21kworker/0:2+events23:14:370
323479910011000,1cyclictest5923-21kworker/1:2+events19:27:441
3235699994992,2cyclictest16732-21turbostat00:19:553
3235299994991,3cyclictest5949-21turbostat19:29:552
3235699993991,2cyclictest13169-21turbostat23:14:553
3235299988987,1cyclictest25602-21kworker/2:2+events21:17:242
3235299985984,1cyclictest25602-21kworker/2:2+events19:53:372
3235299972971,1cyclictest8549-21kworker/2:0+events00:10:212
3234799971970,1cyclictest28975-21turbostat20:24:551
3234099956954,2cyclictest21651-21turbostat21:49:540
32352999320,931cyclictest19176-21/usr/sbin/munin23:45:192
3234099932929,3cyclictest16028-21turbostat19:54:540
3235299930928,2cyclictest1902-21turbostat19:19:552
3234799928925,3cyclictest2652-21turbostat20:39:551
3235699926924,2cyclictest24815-21turbostat22:54:563
3235699921919,2cyclictest14016-21turbostat19:49:543
3235699910908,2cyclictest3619-21turbostat22:34:563
3234099870868,2cyclictest10842-21turbostat21:39:550
3235699665663,2cyclictest10842-21turbostat21:39:553
3234799655652,3cyclictest3919-21turbostat19:24:551
3235699638635,3cyclictest5949-21turbostat19:29:543
3235299614610,3cyclictest729-21turbostat00:04:552
3234099614611,3cyclictest7963-21turbostat19:34:540
3235299586583,3cyclictest14016-21turbostat19:49:542
3234099544540,4cyclictest28975-21turbostat20:24:550
3235299539538,1cyclictest15190-21turbostat21:09:542
3235699537535,2cyclictest2652-21turbostat20:39:553
3235699525524,1cyclictest16028-21turbostat19:54:553
3234099521517,4cyclictest15236-21turbostat22:14:560
3235699518517,1cyclictest32235-21turbostat19:14:543
32356991918,1cyclictest67750irq/123-lan22:35:143
32356991918,1cyclictest67750irq/123-lan21:29:003
32356991818,0cyclictest67750irq/123-lan23:07:183
32356991818,0cyclictest67750irq/123-lan22:48:573
32356991818,0cyclictest67750irq/123-lan22:43:413
32356991817,1cyclictest67750irq/123-lan23:01:143
32356991817,1cyclictest67750irq/123-lan21:33:323
32356991817,1cyclictest67750irq/123-lan00:05:173
32356991717,0cyclictest67750irq/123-lan23:22:063
32356991717,0cyclictest67750irq/123-lan22:55:453
32356991717,0cyclictest67750irq/123-lan22:15:103
32356991717,0cyclictest67750irq/123-lan21:13:313
32356991716,1cyclictest67750irq/123-lan23:17:433
32356991716,1cyclictest67750irq/123-lan22:20:223
32356991716,1cyclictest67750irq/123-lan00:30:153
32356991716,1cyclictest67750irq/123-lan00:25:583
32356991716,1cyclictest67750irq/123-lan00:11:463
32356991616,0cyclictest67750irq/123-lan19:36:563
32356991615,1cyclictest67750irq/123-lan22:25:153
32356991615,1cyclictest67750irq/123-lan22:25:153
32356991615,1cyclictest67750irq/123-lan21:20:143
32356991615,1cyclictest67750irq/123-lan21:16:243
32356991615,1cyclictest67750irq/123-lan19:40:213
32356991615,1cyclictest67750irq/123-lan00:35:123
32356991414,0cyclictest67750irq/123-lan20:30:213
32356991111,0cyclictest67750irq/123-lan20:25:163
3234099118,3cyclictest782-21lldpd23:35:450
3234099106,1cyclictest784-21snmpd19:25:050
3234099105,2cyclictest784-21snmpd20:15:520
323529998,1cyclictest784-21snmpd20:23:522
323529998,1cyclictest782-21lldpd20:08:272
323479998,1cyclictest784-21snmpd20:05:431
323479998,1cyclictest782-21lldpd23:16:351
323479993,3cyclictest0-21swapper/122:45:431
323409998,1cyclictest784-21snmpd21:02:020
323529988,0cyclictest784-21snmpd23:27:032
323529988,0cyclictest784-21snmpd19:44:452
323529988,0cyclictest782-21lldpd20:01:282
323529988,0cyclictest782-21lldpd00:21:082
323529987,1cyclictest507-21systemd-resolve22:46:292
323529986,2cyclictest784-21snmpd23:33:132
323479988,0cyclictest784-21snmpd22:35:231
323479988,0cyclictest784-21snmpd22:25:471
323479988,0cyclictest784-21snmpd22:25:471
323479988,0cyclictest784-21snmpd21:33:461
323479988,0cyclictest784-21snmpd20:55:411
323479988,0cyclictest782-21lldpd23:57:401
323479988,0cyclictest782-21lldpd23:54:451
323479988,0cyclictest782-21lldpd23:49:451
323479988,0cyclictest782-21lldpd22:34:331
323479988,0cyclictest782-21lldpd19:31:551
323479987,1cyclictest784-21snmpd22:40:331
323479987,1cyclictest784-21snmpd21:49:581
323479987,1cyclictest784-21snmpd20:47:101
323479987,1cyclictest784-21snmpd19:52:351
323479986,2cyclictest784-21snmpd19:41:571
323479986,1cyclictest34-21ksoftirqd/120:40:161
323479985,3cyclictest784-21snmpd23:00:241
323479984,3cyclictest784-21snmpd22:03:121
323409988,0cyclictest784-21snmpd21:53:580
323409987,1cyclictest784-21snmpd21:11:000
323409986,2cyclictest784-21snmpd20:54:550
323409986,2cyclictest784-21snmpd19:55:060
323529977,0cyclictest784-21snmpd23:15:042
323529977,0cyclictest784-21snmpd21:42:042
323529977,0cyclictest784-21snmpd19:12:372
323529977,0cyclictest782-21lldpd20:51:472
323529977,0cyclictest507-21systemd-resolve19:20:102
323529976,1cyclictest784-21snmpd23:55:082
323529976,1cyclictest784-21snmpd19:34:592
323529975,2cyclictest784-21snmpd22:56:542
323529975,2cyclictest784-21snmpd22:16:292
323529975,2cyclictest784-21snmpd21:56:162
323529975,2cyclictest784-21snmpd21:53:042
323529975,2cyclictest784-21snmpd21:48:462
323529975,2cyclictest762-21ntpd22:25:202
323529975,2cyclictest762-21ntpd22:25:202
323529973,3cyclictest18013-21ssh22:42:462
323479977,0cyclictest784-21snmpd23:05:151
323479977,0cyclictest784-21snmpd20:19:251
323479977,0cyclictest784-21snmpd19:59:241
323479977,0cyclictest784-21snmpd00:09:041
323479977,0cyclictest782-21lldpd22:09:081
323479977,0cyclictest782-21lldpd19:49:001
323479977,0cyclictest526-21avahi-daemon00:30:271
323479976,1cyclictest784-21snmpd23:39:131
323479976,1cyclictest784-21snmpd21:22:121
323479976,1cyclictest507-21systemd-resolve00:01:291
323479976,1cyclictest32497-21cpuspeed_turbos21:55:121
323479975,0cyclictest191rcu_preempt00:35:301
323479973,3cyclictest27666-21ssh23:23:301
323479973,3cyclictest22286-21ssh22:15:231
323479973,2cyclictest22240-21sh21:14:461
323479970,5cyclictest2652-21turbostat20:34:591
323409977,0cyclictest784-21snmpd19:15:400
323409977,0cyclictest782-21lldpd23:54:270
323409977,0cyclictest782-21lldpd20:48:350
323409976,1cyclictest784-21snmpd20:55:170
323409976,1cyclictest782-21lldpd22:01:020
323409976,1cyclictest526-21avahi-daemon23:33:360
323409976,1cyclictest507-21systemd-resolve23:41:320
323409976,1cyclictest11614-21switchtime22:05:220
323409975,2cyclictest784-21snmpd21:18:090
323409975,2cyclictest784-21snmpd21:07:500
323409973,3cyclictest7728-21ssh00:05:180
323529966,0cyclictest784-21snmpd19:55:482
323529966,0cyclictest782-21lldpd23:05:352
323529966,0cyclictest12057-21latency_hist23:40:002
323529965,1cyclictest507-21systemd-resolve23:20:242
323529965,1cyclictest4243-21fschecks_count22:30:122
323529964,2cyclictest784-21snmpd20:26:122
323529964,2cyclictest784-21snmpd00:27:032
323529964,2cyclictest547-21NetworkManager23:04:262
323529963,3cyclictest547-21NetworkManager22:06:292
323529963,2cyclictest28372-21ssh22:52:452
323529963,2cyclictest27137-21ssh22:20:212
323529963,2cyclictest17788-21ssh23:14:132
323529962,3cyclictest8331-21ssh22:02:382
323529962,3cyclictest3211-21fw_forwarded_lo20:35:142
323529962,3cyclictest10532-21fw_packets19:35:142
323479966,0cyclictest782-21lldpd20:00:521
323479966,0cyclictest782-21lldpd00:15:361
323479965,1cyclictest23525-21needreboot21:15:171
323479965,0cyclictest191rcu_preempt23:30:181
323479965,0cyclictest191rcu_preempt22:55:211
323479965,0cyclictest191rcu_preempt21:40:301
323479965,0cyclictest191rcu_preempt00:20:331
323479965,0cyclictest191rcu_preempt00:10:541
323479964,2cyclictest784-21snmpd21:35:041
323479964,2cyclictest32334-21cyclictest20:25:131
323479964,1cyclictest34-21ksoftirqd/122:10:131
323479963,3cyclictest784-21snmpd20:10:001
323479963,1cyclictest34-21ksoftirqd/100:25:161
323479962,2cyclictest34-21ksoftirqd/123:10:161
323479962,2cyclictest34-21ksoftirqd/121:50:141
323479962,2cyclictest34-21ksoftirqd/119:15:181
323479962,2cyclictest34-21ksoftirqd/119:10:171
323479962,2cyclictest27482-21users22:20:231
323479961,1cyclictest34-21ksoftirqd/121:25:161
323409966,0cyclictest784-21snmpd20:40:170
323409965,1cyclictest526-21avahi-daemon00:02:030
323409965,1cyclictest23549-21gltestperf20:05:140
323409964,2cyclictest784-21snmpd20:14:210
323409964,2cyclictest784-21snmpd20:04:100
323409963,2cyclictest18990-21ssh00:15:550
323409962,3cyclictest7197-21sed21:30:210
323409962,3cyclictest3617-21ssh21:58:190
323409962,3cyclictest31631-21ssh21:23:200
323409962,3cyclictest18737-21ssh23:45:160
323529955,0cyclictest507-21systemd-resolve20:33:322
323529954,1cyclictest26060-21ntp_states20:10:162
323529952,3cyclictest13443-21apt-get21:00:112
323529952,2cyclictest9074-21seq20:49:232
323529952,2cyclictest2682-21ssh00:31:382
323529952,2cyclictest16148-21sed22:10:142
323529952,2cyclictest12725-21dpkg23:40:092
323529951,3cyclictest9012-21ssh00:38:142
323529951,3cyclictest11390-21find20:55:112
323529950,4cyclictest6076-21ntp_192.168.11420:40:162
323479952,2cyclictest34-21ksoftirqd/123:25:131
323479952,2cyclictest16732-21grep21:05:211
323479950,3cyclictest34-21ksoftirqd/120:50:121
323479950,2cyclictest34-21ksoftirqd/121:00:111
323479950,1cyclictest34-21ksoftirqd/122:50:141
323409954,1cyclictest1705-21ntpq21:25:160
323409954,1cyclictest12704-21ntpq00:10:160
323409953,2cyclictest0-21swapper/023:55:090
323409952,3cyclictest17768-21ntpq21:40:170
323409952,2cyclictest4640-21cat19:20:130
323529942,2cyclictest762-21ntpd23:50:002
323529941,2cyclictest9141-21sh22:35:092
323529941,2cyclictest28349-21timerandwakeup20:15:202
323479943,1cyclictest0-21swapper/119:35:121
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional