You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-31 - 20:50

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri Jul 31, 2026 12:45:18)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24016215797,24sleep20-21swapper/207:05:492
242382143104,26sleep30-21swapper/307:08:443
240962142105,26sleep00-21swapper/007:06:520
242162140104,25sleep10-21swapper/107:08:251
120672640,3sleep012071-21gltestperf11:10:170
292792630,1sleep10-21swapper/110:10:221
21892600,2sleep20-21swapper/211:26:292
5452590,2sleep30-21swapper/309:25:173
77452550,2sleep30-21swapper/311:06:383
69562510,4sleep26955-21/usr/sbin/munin11:30:172
96062500,2sleep00-21swapper/010:20:170
96062500,2sleep00-21swapper/010:20:160
87012500,2sleep10-21swapper/109:00:231
42102500,2sleep00-21swapper/009:51:590
188392480,1chrt0-21swapper/011:39:460
92562470,2sleep122-21ksoftirqd/109:31:261
151582460,7sleep122-21ksoftirqd/111:36:021
37212440,2chrt0-21swapper/109:27:361
158352440,2chrt0-21swapper/210:49:122
149872430,1sleep10-21swapper/109:11:391
6102400,1sleep00-21swapper/012:14:250
300432330,2chrt0-21swapper/111:23:381
5422290,9sleep012-21ksoftirqd/011:25:220
312592290,10sleep122-21ksoftirqd/107:25:131
2461199271,25cyclictest0-21swapper/007:12:420
2461399261,24cyclictest0-21swapper/207:13:062
15372260,3sleep30-21swapper/310:38:213
152342260,2sleep30-21swapper/310:00:173
2461499251,23cyclictest0-21swapper/307:12:453
24612992523,1cyclictest0-21swapper/107:12:241
24611992510,3cyclictest141rcu_preempt10:35:240
224182250,12sleep122-21ksoftirqd/112:30:151
201152250,2sleep30-21swapper/310:27:583
109072250,2sleep30-21swapper/310:45:163
2461499247,4cyclictest141rcu_preempt10:30:233
2461399240,9cyclictest64050snmpd08:30:152
24612992416,6cyclictest28890-21ssh11:46:231
2461199245,6cyclictest32676-21timerandwakeup08:40:250
75622230,2sleep00-21swapper/009:55:080
60262230,2sleep00-21swapper/011:30:070
2461499233,3cyclictest141rcu_preempt09:15:153
2461499231,19cyclictest18108-21ntpq10:50:203
24613992312,8cyclictest2667-21users09:50:272
24613992312,8cyclictest12769-21interrupts09:10:172
2461399231,10cyclictest30202sh11:51:152
24612992317,4cyclictest22-21ksoftirqd/112:30:001
24612992311,4cyclictest31852-21latency_hist09:25:001
2461199231,6cyclictest28890-21perf10:35:010
24611992314,7cyclictest14544-21hddtemp_smartct12:00:160
24611992313,7cyclictest5896-21ntp_states11:05:220
24611992313,7cyclictest13009-21python307:55:240
24611992311,9cyclictest30264-21cat07:20:250
2461199231,18cyclictest53050irq/142-enp3s0-09:35:120
173372230,2sleep10-21swapper/112:02:071
159172230,2sleep30-21swapper/311:36:483
2461499222,9cyclictest11610-21diskmemload11:52:433
2461499222,15cyclictest26671-21ssh09:45:123
24614992216,3cyclictest11610-21diskmemload12:00:123
24614992213,4cyclictest52850irq/140-enp3s0-12:30:173
2461499221,10cyclictest141rcu_preempt12:09:533
24613992213,7cyclictest30895-21df11:00:152
24613992212,7cyclictest7896-21df07:45:132
24613992212,7cyclictest27043-21cut08:30:012
2461299224,11cyclictest19578-21ssh11:40:151
24612992216,4cyclictest22-21ksoftirqd/110:20:001
24612992216,4cyclictest22-21ksoftirqd/109:00:001
2461299221,5cyclictest0-21swapper/112:06:551
24612992213,7cyclictest13356-21users07:55:261
24612992213,6cyclictest21274-21sensors_temp09:40:221
2461199227,3cyclictest141rcu_preempt12:05:150
2461199227,12cyclictest8669-21wc11:55:240
2461199222,15cyclictest11610-21diskmemload09:30:140
165082221,2sleep10-21swapper/109:37:001
51652210,2sleep00-21swapper/012:17:140
2461499216,5cyclictest13239-21/usr/sbin/munin11:10:273
2461499213,13cyclictest640-21snmpd10:20:133
2461499213,13cyclictest640-21snmpd10:20:133
2461499213,13cyclictest28555-21ssh12:35:133
2461499213,12cyclictest1637-21apt-get12:15:133
2461499212,14cyclictest21296-21ssh10:05:123
24614992113,6cyclictest5187-21pluginstate08:55:003
24614992113,3cyclictest52850irq/140-enp3s0-12:25:253
24614992113,3cyclictest52850irq/140-enp3s0-12:20:343
24614992113,3cyclictest52850irq/140-enp3s0-11:40:043
24614992112,6cyclictest52850irq/140-enp3s0-10:15:183
2461399218,8cyclictest52750irq/139-enp3s0-10:00:132
2461399212,13cyclictest7157-21apt-get11:55:102
2461399211,8cyclictest9114-21runrttasks10:44:122
24613992115,4cyclictest13874-21grep09:35:172
24613992113,6cyclictest32758-21fw_packets09:25:152
24613992113,6cyclictest16697-21diskstats08:05:172
24613992113,6cyclictest14956-21ntpq12:00:202
2461399211,11cyclictest11610-21diskmemload12:25:502
24612992112,7cyclictest3550-21seq09:51:191
24612992112,6cyclictest26840-21seq08:29:451
24612992112,6cyclictest10462-21cat09:05:181
2461199219,8cyclictest27945-21users11:45:250
2461199213,5cyclictest12-21ksoftirqd/007:53:370
2461199212,16cyclictest32196-21grep11:00:270
2461199211,6cyclictest3485-21pmu-power10:15:210
24611992114,4cyclictest31759-21sh12:36:580
24611992113,6cyclictest24580-21seq08:23:150
24611992113,6cyclictest2046-21ntpq08:45:220
24611992113,6cyclictest2046-21ntpq08:45:210
24611992113,6cyclictest19997-21users08:10:260
24611992112,6cyclictest29758-21seq11:23:210
24611992112,6cyclictest27953-21sensors_temp07:15:240
24611992111,8cyclictest18802-21df11:15:150
2461199211,12cyclictest53050irq/142-enp3s0-09:10:130
145642210,2sleep20-21swapper/212:24:562
126562210,10sleep334-21ksoftirqd/311:59:123
2461499206,10cyclictest52850irq/140-enp3s0-09:55:173
2461499205,10cyclictest52850irq/140-enp3s0-10:55:573
2461499202,5cyclictest34-21ksoftirqd/311:28:163
2461499202,12cyclictest26216-21apt-get11:45:133
2461499202,11cyclictest31103-21apt-get07:25:123
2461499201,3cyclictest141rcu_preempt09:20:243
24614992012,6cyclictest1741-21sed09:50:183
2461499201,16cyclictest14319-21ssh09:35:223
24614992011,6cyclictest1-21systemd07:31:153
2461399205,6cyclictest28-21ksoftirqd/211:06:212
2461399201,8cyclictest11610-21diskmemload10:29:392
24613992014,5cyclictest28-21ksoftirqd/211:39:592
24613992014,4cyclictest14999-21sh11:12:132
24613992012,6cyclictest29972-21ls12:35:252
24613992012,6cyclictest23115-21df08:20:142
24613992012,6cyclictest19967-21diskstats11:40:182
24613992011,7cyclictest2903-21fw_packets10:15:152
24613992011,7cyclictest27262-21perl07:15:152
2461399201,10cyclictest11610-21diskmemload11:23:582
2461399201,10cyclictest11610-21diskmemload10:54:062
24613992010,8cyclictest28961-21unixbench_multi12:10:242
24613992010,6cyclictest52750irq/139-enp3s0-10:30:112
24613992010,6cyclictest52750irq/139-enp3s0-10:20:202
24613992010,6cyclictest52750irq/139-enp3s0-10:20:202
2461299208,9cyclictest31141-21sed11:00:171
2461299206,8cyclictest52950irq/141-enp3s0-09:15:121
24612992013,5cyclictest2901-21seq08:48:281
24612992013,5cyclictest2901-21seq08:48:271
24612992012,5cyclictest29020-21grep12:10:251
24612992011,7cyclictest15504-21date10:25:011
24612992011,7cyclictest15504-21date10:25:001
24612992011,7cyclictest11085-21seq11:57:361
24612992011,6cyclictest8810-21which09:55:221
24612992011,6cyclictest29915-21pluginstate07:20:231
24612992011,6cyclictest27128-21df_inode07:15:141
24612992011,6cyclictest22654-21switchtime10:05:241
2461299201,15cyclictest52950irq/141-enp3s0-10:40:121
2461199208,6cyclictest6345-21ntp_offset07:40:220
2461199208,4cyclictest53050irq/142-enp3s0-10:58:230
2461199203,3cyclictest217342rm09:40:290
2461199203,3cyclictest11610-21diskmemload10:49:250
2461199201,5cyclictest0-21swapper/009:18:200
2461199201,4cyclictest4379-21irqstats10:40:170
24611992013,4cyclictest20376-21sh10:04:330
24611992011,7cyclictest22416-21seq08:18:530
24611992011,7cyclictest19853-21expr11:40:180
176532200,2sleep00-21swapper/010:25:300
2461499194,3cyclictest141rcu_preempt10:44:113
2461499192,3cyclictest11610-21diskmemload09:10:053
2461499192,14cyclictest28026-21apt-get10:10:123
2461499192,12cyclictest24986-21apt-get08:25:123
2461499191,3cyclictest141rcu_preempt09:31:593
24614991913,3cyclictest24737-21turbostat08:25:013
24614991912,5cyclictest20160-21expr09:40:123
24614991912,5cyclictest11610-21diskmemload11:30:163
24614991911,6cyclictest25786-21sed11:20:173
24614991911,6cyclictest13358-21who07:55:263
24614991911,6cyclictest10515-21ls07:50:213
2461499191,15cyclictest3926-21ls07:35:193
24614991911,5cyclictest19588-21grep11:15:243
24614991911,5cyclictest1695-21sed08:45:163
24614991911,5cyclictest1695-21sed08:45:163
2461399194,8cyclictest636-21runrttasks10:56:192
2461399191,9cyclictest11610-21diskmemload10:39:432
2461399191,8cyclictest2507-21memory12:15:192
24613991914,3cyclictest20378-21munin-plugin-st08:15:002
24613991914,3cyclictest12183-21sh09:34:232
24613991913,4cyclictest23540-21sh09:42:182
24613991912,5cyclictest8059-21tune2fs09:55:152
24613991912,5cyclictest7130-21seq07:42:422
24613991912,4cyclictest23487-21sh11:18:492
24613991912,4cyclictest11610-21diskmemload12:30:342
24613991911,6cyclictest3421-21cat07:35:142
24613991911,6cyclictest29323-21ls08:35:132
24613991911,6cyclictest21560-21expr10:05:142
24613991911,6cyclictest1903-21ls08:45:192
24613991911,6cyclictest1903-21ls08:45:192
24613991911,5cyclictest31055-21cut07:25:012
24613991910,7cyclictest3689-21tune2fs08:50:152
24613991910,6cyclictest27006-21grep12:10:012
24613991910,6cyclictest13187-21cat07:55:252
2461399190,6cyclictest11610-21diskmemload10:10:152
2461299192,12cyclictest7575-21apt-get07:45:131
2461299192,12cyclictest3290-21apt-get08:50:121
2461299192,11cyclictest5438-21apt-get07:40:131
2461299191,5cyclictest0-21swapper/111:33:461
24612991915,3cyclictest19575-21ssh10:51:131
2461299191,4cyclictest913-21fschecks_count11:50:151
24612991912,5cyclictest18239-21seq10:02:231
24612991912,4cyclictest11610-21diskmemload09:45:201
24612991911,6cyclictest772-21cat11:25:231
24612991911,6cyclictest31408-21find08:40:111
24612991911,6cyclictest29665-21cut12:35:231
24612991911,6cyclictest28742-21seq08:31:411
24612991911,6cyclictest22674-21seq11:17:591
24612991911,6cyclictest22177-21seq08:15:481
24612991911,6cyclictest16749-21cat10:25:191
24612991911,6cyclictest12815-21seq10:46:091
24612991911,6cyclictest12637-21cut11:10:231
24612991911,6cyclictest11664-21grep07:55:001
24612991910,6cyclictest6196-21wc11:05:241
2461199198,3cyclictest53050irq/142-enp3s0-10:51:470
2461199193,3cyclictest11610-21diskmemload10:06:100
2461199191,7cyclictest28141-21timerandwakeup09:45:240
24611991914,3cyclictest8826-21threads09:00:250
24611991914,3cyclictest10969-21timerandwakeup09:05:240
24611991913,4cyclictest29089-21ntp_states10:10:200
24611991913,4cyclictest26995-21sh12:33:550
24611991913,4cyclictest17250-21sh12:25:530
24611991913,4cyclictest10358-21sh12:20:420
24611991913,3cyclictest2725-21sh09:26:370
24611991912,5cyclictest9391-21seq07:49:160
24611991912,5cyclictest30346-21rt-features08:35:230
24611991912,5cyclictest24956-21seq08:25:070
24611991912,5cyclictest1111-21smartctl11:50:160
24611991910,6cyclictest28132-21which08:30:230
2461199190,16cyclictest5673-21expr08:55:150
324502180,2sleep10-21swapper/110:36:511
2461499189,7cyclictest8959-21date09:00:273
2461499182,11cyclictest9723-21apt-get09:05:133
24614991813,3cyclictest636-21runrttasks11:00:143
24614991811,5cyclictest22371-21seq08:18:173
2461499181,14cyclictest14317-21cut08:00:133
2461499181,14cyclictest14317-21cut08:00:133
2461499181,12cyclictest31430-21apt-get08:40:123
24614991810,6cyclictest8395-21sed07:45:193
24614991810,6cyclictest17665-21date08:05:263
2461399181,6cyclictest11610-21diskmemload09:24:072
24613991813,3cyclictest22323-21sh09:17:142
24613991811,4cyclictest9942-21cat09:05:142
24613991811,4cyclictest26127-21sh11:45:062
24613991810,6cyclictest32696-21cat08:40:252
24613991810,6cyclictest27447-21expr09:45:182
24613991810,6cyclictest20749-21expr08:15:142
24613991810,5cyclictest7838-21expr09:00:142
24613991810,5cyclictest1257-21cat07:30:142
24612991813,3cyclictest4129-21sh12:16:121
24612991813,3cyclictest26240-21sh10:56:091
24612991812,4cyclictest11141-21sh12:21:281
24612991811,5cyclictest30024-21load08:35:191
24612991811,5cyclictest22655-21cat10:30:121
2461199189,6cyclictest16704-21cat08:05:170
24611991811,5cyclictest15291-21rt-features08:00:220
24611991811,5cyclictest15291-21rt-features08:00:220
24611991810,6cyclictest3626-21diskstats08:50:150
24611991810,6cyclictest31649-21expr07:25:160
24611991810,6cyclictest26072-21fw_packets09:20:160
24611991810,6cyclictest1353-21expr07:30:150
2461499179,5cyclictest29876-21ntp_offset07:20:213
2461499179,5cyclictest28713-21switchtime12:10:243
2461499175,9cyclictest27492-21wc07:15:183
2461499171,13cyclictest19893-21sed08:10:263
24613991712,3cyclictest5401-21/usr/sbin/munin08:55:112
24613991712,3cyclictest31077-21/usr/sbin/munin07:25:182
24613991712,3cyclictest14192-21/usr/sbin/munin08:00:132
24613991712,3cyclictest14192-21/usr/sbin/munin08:00:132
24612991712,3cyclictest16333-21/usr/sbin/munin08:05:161
2461299171,11cyclictest3177-21apt-get07:35:131
24612991710,5cyclictest23917-21which08:20:231
24612991710,5cyclictest18679-21expr08:10:151
24612991710,5cyclictest15961-21seq08:03:211
24612991710,5cyclictest15961-21seq08:03:201
24611991710,5cyclictest4752-21seq07:36:120
2461499169,5cyclictest5435-21awk07:40:113
2461499168,6cyclictest6871-21awk08:55:263
2461499162,8cyclictest29270-21apt-get08:35:143
2461499162,8cyclictest27129-21apt-get08:30:133
2461399169,5cyclictest9879-21cat07:50:152
24612991611,3cyclictest1004-21/usr/sbin/munin07:30:201
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional