You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-25 - 02:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun May 24, 2026 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
42232206104,27sleep20-21swapper/207:06:312
42222184170,8sleep10-21swapper/107:06:291
41942172101,25sleep00-21swapper/007:06:080
44892171106,28sleep30-21swapper/307:09:563
1335021240,2sleep10-21swapper/112:15:241
200872810,1sleep10-21swapper/107:45:001
276172640,1sleep30-21swapper/310:00:183
11732580,2sleep334-21ksoftirqd/310:54:113
210572550,2sleep10-21swapper/111:57:071
210572550,2sleep10-21swapper/111:57:061
212152530,1sleep20-21swapper/209:00:212
257282520,2sleep00-21swapper/011:35:580
307332460,1sleep00-21swapper/012:29:150
212472460,3sleep1211rcuc/112:21:211
124412460,2sleep10-21swapper/109:25:051
91942450,1sleep10-21swapper/108:35:011
82312410,1sleep20-21swapper/212:11:402
303612400,3sleep122-21ksoftirqd/112:04:451
274362320,3sleep10-21swapper/108:00:181
232772300,1sleep20-21swapper/211:10:192
235402290,2sleep10-21swapper/110:21:281
4766992617,7cyclictest17557-21basename07:40:002
476599261,24cyclictest0-21swapper/112:11:541
4765992611,9cyclictest24336-21diskmemload11:05:121
4764992624,1cyclictest0-21swapper/012:05:530
217482260,4sleep30-21swapper/309:00:273
56442250,2sleep10-21swapper/109:44:141
476799251,1cyclictest0-21swapper/312:05:553
476699259,7cyclictest28-21ksoftirqd/211:40:012
4766992522,1cyclictest0-21swapper/212:05:192
476599258,4cyclictest141rcu_preempt09:50:111
476599252,16cyclictest31639-21ssh09:15:131
4764992516,7cyclictest16031-21rm10:15:330
4764992510,3cyclictest232822missed_timers11:10:190
476499250,24cyclictest0-21swapper/012:11:550
17482250,3sleep20-21swapper/211:42:462
147862250,2sleep10-21swapper/108:45:191
99082240,1sleep00-21swapper/012:37:310
476699240,14cyclictest0-21swapper/210:47:562
476499241,4cyclictest0-21swapper/011:49:190
262582240,2sleep334-21ksoftirqd/309:35:143
207642240,2sleep10-21swapper/111:32:421
102122240,2sleep30-21swapper/312:13:423
476799232,14cyclictest23453-21ssh11:35:113
476699234,17cyclictest20389-21munin-run10:19:592
476699233,10cyclictest24336-21diskmemload11:17:382
476699231,7cyclictest903-21grep09:15:252
476699230,9cyclictest28972-21ls10:25:252
116152230,11sleep012-21ksoftirqd/011:01:140
53672220,2sleep20-21swapper/208:25:152
4767992215,5cyclictest34-21ksoftirqd/309:50:013
4767992215,5cyclictest34-21ksoftirqd/309:50:003
4767992214,6cyclictest23794-21ls07:50:253
4767992213,7cyclictest10955-21cat11:25:183
4767992213,6cyclictest14196-21df08:45:133
4766992213,6cyclictest278-21systemd-journal08:20:012
4766992210,4cyclictest5740-21ssh10:32:552
4766992210,4cyclictest5740-21ssh10:32:542
476599224,4cyclictest24336-21diskmemload10:44:491
4765992213,7cyclictest15032-21seq07:32:031
4765992213,6cyclictest52850irq/140-enp3s0-12:25:151
476499227,3cyclictest44402ssh10:07:150
4764992217,3cyclictest12-21ksoftirqd/008:40:000
4764992213,7cyclictest30427-21vmstat08:05:260
221602220,4chrt22158-21ssh10:45:173
8112210,1sleep20-21swapper/210:05:062
476799219,7cyclictest52950irq/141-enp3s0-09:55:133
4767992116,3cyclictest8632-21sh11:00:003
4767992114,5cyclictest34-21ksoftirqd/308:00:003
4767992113,6cyclictest18950irq/143-ahci[0007:40:153
4767992112,7cyclictest5653-21smartctl08:25:173
4767992112,7cyclictest23005-21seq10:20:533
4767992111,6cyclictest52950irq/141-enp3s0-11:45:123
476799210,9cyclictest3154-21cat11:20:003
4766992114,5cyclictest2941-21ssh09:41:292
4766992114,4cyclictest7446-21sshd09:45:162
4766992114,4cyclictest7446-21sshd09:45:152
4766992114,4cyclictest24336-21diskmemload10:14:342
4766992113,6cyclictest31808-21ls08:10:172
4766992112,6cyclictest27040-21awk09:35:212
476599218,3cyclictest141rcu_preempt11:25:251
476599218,11cyclictest27770-21munin-run10:50:001
476599216,3cyclictest141rcu_preempt09:20:141
476599215,3cyclictest24336-21diskmemload11:03:091
476599214,3cyclictest24336-21diskmemload10:19:121
476599212,5cyclictest4042-21sort11:44:591
476599211,6cyclictest13948-21ssh10:39:301
476499212,6cyclictest115852rm09:48:340
476499212,6cyclictest115852rm09:48:330
476499212,14cyclictest17699-21apt-get11:55:130
476499212,14cyclictest17699-21apt-get11:55:120
4764992115,4cyclictest12-21ksoftirqd/012:35:000
4764992113,6cyclictest28870-21seq11:15:030
4764992113,6cyclictest23952-21users09:05:250
4764992113,6cyclictest19621-21cron09:55:010
4764992113,5cyclictest18118-21seq08:55:050
4764992111,8cyclictest29650-21sh10:01:410
476499211,14cyclictest31474-21sh09:15:030
476799202,13cyclictest8253-21ssh10:35:133
476799202,13cyclictest3503-21ssh11:20:123
476799201,5cyclictest1169-21apt10:30:003
476799201,3cyclictest141rcu_preempt09:25:003
4767992012,6cyclictest9127-21grep10:10:233
4767992012,6cyclictest30925-21sed08:10:003
4767992012,6cyclictest18950-21ls08:55:193
4767992012,6cyclictest15841-21expr07:35:143
4767992012,6cyclictest13451-21cut09:25:193
4767992012,6cyclictest1032-21cat12:30:243
476799201,14cyclictest31475-21ssh09:15:033
476699209,6cyclictest53050irq/142-enp3s0-12:35:102
476699208,4cyclictest53050irq/142-enp3s0-11:05:162
476699204,13cyclictest14014-21cpuspeed_turbos08:45:132
476699203,4cyclictest28-21ksoftirqd/210:55:072
4766992015,3cyclictest23036-21dump-pmu-power11:35:002
4766992014,4cyclictest9105-21cron08:35:002
4766992012,6cyclictest8640-21forks10:35:152
4766992012,6cyclictest27282-21cut10:00:162
4766992012,6cyclictest24014-21sed12:00:012
4766992012,6cyclictest24014-21sed12:00:002
4766992012,6cyclictest20584-21tune2fs09:55:162
4766992012,6cyclictest15388-21seq12:17:112
4766992012,6cyclictest11101-21ls11:25:202
4766992011,7cyclictest26132-21who07:55:262
4766992011,7cyclictest24227-21ssh10:22:102
4766992011,7cyclictest13618-21cat09:50:122
4766992011,6cyclictest12757-21vmstat11:50:272
4766992010,9cyclictest7081-21ssh11:22:182
476599204,3cyclictest24336-21diskmemload11:46:331
476599204,14cyclictest30304-21grep11:15:221
476599203,15cyclictest3413-21ntpq10:55:191
476599202,11cyclictest22358-21ssh11:10:121
4765992017,2cyclictest0-21swapper/112:05:201
476599201,5cyclictest2622-21ntp_states10:30:201
476599201,5cyclictest2622-21ntp_states10:30:191
476599201,5cyclictest1975-21ntp_states10:05:211
4765992014,3cyclictest9674-21hwlatdetect07:20:161
4765992012,6cyclictest9637-21seq12:37:151
4765992012,6cyclictest5893-21sed08:25:191
4765992012,6cyclictest20274-21ntp_kernel_pll_09:30:191
4765992012,5cyclictest52850irq/140-enp3s0-10:10:111
4765992011,7cyclictest32279-21cat08:10:241
4765992011,6cyclictest9889-21cat11:25:011
4765992011,6cyclictest2349-21seq08:16:051
476499202,4cyclictest12-21ksoftirqd/008:40:130
4764992014,4cyclictest28534-21tr10:25:220
4764992014,4cyclictest12-21ksoftirqd/009:15:010
4764992012,6cyclictest7705-21memory07:15:170
4764992012,6cyclictest23426-21open_inodes07:50:220
4764992012,6cyclictest22815-21systemctl10:45:240
4764992012,6cyclictest15037-21seq07:32:080
4764992012,6cyclictest14794-21seq09:25:550
4764992010,7cyclictest278-21systemd-journal07:50:010
476799199,6cyclictest52950irq/141-enp3s0-10:30:123
476799199,6cyclictest52950irq/141-enp3s0-10:30:123
4767991914,3cyclictest24734-21sh12:00:143
4767991913,4cyclictest20634-21sh11:08:403
4767991912,5cyclictest4539-21seq10:07:223
4767991912,5cyclictest19092-21cat11:55:253
4767991912,5cyclictest19092-21cat11:55:243
4767991912,5cyclictest12758-21vmstat07:25:263
4767991912,5cyclictest12260-21sed08:40:183
4767991912,4cyclictest21484-21sh12:21:383
4767991911,6cyclictest8043-21date08:30:173
4767991911,6cyclictest13943-21smartctl07:30:153
4767991911,6cyclictest10101-21sed08:35:173
4767991910,7cyclictest23191-21ls11:10:183
4767991910,7cyclictest11917-21cat11:50:193
4767991910,6cyclictest823-21ls09:40:163
476699194,3cyclictest141rcu_preempt09:30:122
476699192,11cyclictest13388-21apt-get07:30:132
476699191,9cyclictest0-21swapper/209:24:152
476699191,6cyclictest0-21swapper/212:24:592
476699191,5cyclictest0-21swapper/211:49:142
4766991912,5cyclictest28482-21expr10:50:132
4766991912,5cyclictest20352-21cat07:45:122
4766991912,5cyclictest16782-21expr08:50:182
4766991911,6cyclictest26886-21grep08:00:132
4766991911,6cyclictest12778-21cat09:25:132
4766991911,5cyclictest53050irq/142-enp3s0-10:40:132
476699191,10cyclictest0-21swapper/212:31:052
4766991910,6cyclictest12144-21cat08:40:162
476699190,10cyclictest0-21swapper/212:26:012
476599199,8cyclictest25765-21ssh09:35:051
476599199,7cyclictest5149-21cat07:10:131
476599199,7cyclictest25288-21gltestperf09:10:151
476599199,7cyclictest19118-21cut08:55:231
476599193,3cyclictest24336-21diskmemload11:51:521
476599192,12cyclictest20137-21apt-get07:45:121
4765991914,3cyclictest21363-21sendmail_mailtr09:55:241
4765991912,5cyclictest4387-21awk08:20:261
4765991912,5cyclictest28022-21sensors10:00:211
4765991912,4cyclictest24336-21diskmemload12:30:551
4765991911,6cyclictest22419-21seq09:05:101
4765991911,6cyclictest16272-21sed07:35:191
4765991911,6cyclictest11300-21wc08:40:001
4765991911,5cyclictest23981-21grep11:35:151
476499199,8cyclictest278-21systemd-journal10:00:000
476499199,6cyclictest52750irq/139-enp3s0-12:20:120
476499198,8cyclictest20446-21processes09:30:210
476499192,11cyclictest24362-21apt-get12:00:120
4764991914,3cyclictest9680-21hwlatdetect07:20:170
4764991912,5cyclictest8377-21mailstats08:30:200
4764991912,4cyclictest30499-21sh11:40:140
4764991912,4cyclictest13526-21grep12:15:250
4764991911,6cyclictest3136-21cat10:30:250
4764991911,6cyclictest3136-21cat10:30:240
4764991911,6cyclictest31219-21expr08:10:120
4764991911,6cyclictest29433-21which10:50:220
4764991911,6cyclictest26381-21seq07:58:330
4764991911,6cyclictest20647-21diskstats09:00:170
476799189,7cyclictest15996-21pluginstate10:40:213
476799189,6cyclictest6945-21sed12:35:173
476799185,3cyclictest24336-21diskmemload11:02:293
476799184,10cyclictest24339-21awk09:10:003
476799181,8cyclictest19300-21ssh11:31:133
4767991813,3cyclictest23891-21sh09:33:253
4767991811,5cyclictest26991-21vmstat12:25:283
4767991810,6cyclictest14568-21forks10:15:143
4767991810,6cyclictest13847-21cut09:50:153
4767991810,5cyclictest6243-21grep07:10:243
476699189,7cyclictest25682-21cat09:10:192
476699187,9cyclictest52950irq/141-enp3s0-08:35:222
476699182,14cyclictest24783-21fschecks_count12:00:142
476699182,11cyclictest4805-21apt-get07:10:092
476699181,8cyclictest8915-21rm11:00:022
4766991810,6cyclictest30031-21tr08:05:232
4766991810,5cyclictest23076-21idleruntime07:50:182
476599186,10cyclictest32108-21sh10:28:301
476599182,11cyclictest6948-21apt-get07:15:121
476599181,2cyclictest0-21swapper/110:50:091
4765991811,5cyclictest30670-21seq08:08:271
4765991811,5cyclictest22634-21expr07:50:151
4765991811,5cyclictest11731-21expr07:25:161
4765991810,6cyclictest25404-21cat07:55:191
4765991810,6cyclictest16634-21cat08:50:161
4765991810,6cyclictest13302-21seq08:43:461
4765991810,5cyclictest52850irq/140-enp3s0-09:45:111
4765991810,5cyclictest52850irq/140-enp3s0-09:45:111
476499189,6cyclictest15617-21cut07:35:130
476499189,5cyclictest15738-21apt-get11:05:130
476499182,11cyclictest2390-21apt-get10:55:130
476499182,11cyclictest10153-21apt-get11:25:120
476499181,6cyclictest0-21swapper/010:43:280
476499181,5cyclictest0-21swapper/010:35:010
4764991813,3cyclictest5870-21/usr/sbin/munin09:20:220
4764991811,5cyclictest18928-21open_files07:40:200
4764991811,5cyclictest11233-21expr11:50:140
4764991811,4cyclictest22050-21processes10:20:210
4764991810,6cyclictest788-21smartctl09:40:160
4764991810,6cyclictest17607-21seq08:51:250
251282180,2sleep30-21swapper/309:10:143
210262180,11sleep334-21ksoftirqd/307:45:203
476799178,7cyclictest1907-21which08:15:233
4767991712,3cyclictest32438-21unixbench-2d08:10:243
4767991712,3cyclictest30084-21/usr/sbin/munin11:40:183
4767991710,5cyclictest7549-21basename07:15:153
4767991710,5cyclictest52950irq/141-enp3s0-12:15:143
4767991710,5cyclictest4084-21tr08:20:223
476699179,6cyclictest19385-21date08:55:262
476699173,9cyclictest17999-21apt-get07:40:132
4766991710,6cyclictest11682-21fschecks_time07:25:152
4766991710,5cyclictest4904-21ls08:25:002
476599179,6cyclictest20821-21grep09:00:171
476499178,6cyclictest17203-21cat11:30:150
476499172,10cyclictest7549-21apt-get10:10:120
476499171,7cyclictest5484-21grep08:25:170
4764991711,4cyclictest24336-21diskmemload11:20:070
4764991710,5cyclictest3297-21grep08:20:150
4764991710,5cyclictest26128-21expr09:35:130
4764991710,5cyclictest1917-21tr08:15:240
4764991710,5cyclictest12963-21seq07:27:580
476799168,6cyclictest560-21dbus-daemon08:04:593
476799167,7cyclictest18003-21grep08:55:013
4767991610,4cyclictest9400-21expr07:20:143
476699168,6cyclictest8309-21cat07:15:232
476699165,9cyclictest52950irq/141-enp3s0-09:05:162
4766991610,4cyclictest9657-21expr07:20:152
476499169,5cyclictest5827-21ps07:10:200
476499161,13cyclictest636-21runrttasks08:45:240
4764991510,3cyclictest27371-21idleruntime08:00:170
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional