You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-11 - 02:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Fri Jul 10, 2026 12:45:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1752821470,1sleep30-21swapper/311:36:183
202972143106,26sleep30-21swapper/307:08:333
203292140102,26sleep00-21swapper/007:08:570
201392140104,25sleep10-21swapper/107:06:301
201082140105,24sleep20-21swapper/207:06:072
106472780,1sleep20-21swapper/212:18:342
319252700,2sleep20-21swapper/208:50:152
192132680,2sleep20-21swapper/210:50:502
71042610,1sleep10-21swapper/112:15:241
48522570,3sleep334-21ksoftirqd/310:40:353
287102570,3chrt0-21swapper/010:35:150
158412570,7sleep0131rcuc/010:02:280
140512570,3sleep1211rcuc/110:47:451
140512570,3sleep1211rcuc/110:47:441
116452550,1sleep00-21swapper/011:32:170
116452550,1sleep00-21swapper/011:32:170
275362540,1sleep00-21swapper/012:06:570
80182530,1sleep20-21swapper/210:20:172
246432520,1sleep00-21swapper/012:28:440
254542510,4sleep3331rcuc/311:42:163
293492500,2sleep20-21swapper/210:58:522
66002470,4sleep07421-21diskmemload11:05:210
203352470,1sleep20-21swapper/209:42:242
49902460,2chrt0-21swapper/112:14:481
5202440,2sleep30-21swapper/309:28:163
189282440,2chrt0-21swapper/110:28:131
296072420,2sleep00-21swapper/009:25:250
80522400,1chrt0-21swapper/311:53:233
80052400,1sleep00-21swapper/010:43:370
190492390,2sleep00-21swapper/012:00:270
244952370,3chrt0-21swapper/312:28:353
196072360,3chrt0-21swapper/010:51:130
208562310,2sleep00-21swapper/009:42:540
265362300,2sleep3341ksoftirqd/312:30:113
310472290,2chrt0-21swapper/111:45:591
9152270,2sleep3331rcuc/310:15:153
319122270,1sleep10-21swapper/110:14:331
226792270,1sleep10-21swapper/111:16:571
139102270,2chrt0-21swapper/210:00:372
2068499261,13cyclictest17782ssh09:29:271
2068399261,24cyclictest0-21swapper/007:15:350
67122250,12sleep012-21ksoftirqd/010:19:500
20686992512,7cyclictest21239-21ssh10:30:113
2068599251,10cyclictest2923-21apt-get10:40:122
20685992422,1cyclictest0-21swapper/207:15:122
61072230,2chrt0-21swapper/211:28:382
267782230,20sleep37421-21diskmemload10:56:263
20686992316,5cyclictest19200-21ssh09:18:103
20686992316,5cyclictest19200-21ssh09:18:103
2068699231,10cyclictest74212diskmemload09:21:153
2068599234,11cyclictest21544-21ssh09:20:112
2068499232,15cyclictest30478-21ssh09:50:131
20684992321,1cyclictest0-21swapper/107:15:011
2068399235,16cyclictest12-21ksoftirqd/011:19:360
20683992316,5cyclictest3478-21ssh11:26:060
20683992314,7cyclictest278-21systemd-journal07:35:000
20683992313,6cyclictest0-21swapper/011:20:030
85812220,3sleep30-21swapper/311:06:443
2068699229,10cyclictest0-21swapper/309:30:113
20686992213,6cyclictest14541-21grep11:35:003
20686992213,6cyclictest14541-21grep11:35:003
20685992214,4cyclictest0-21swapper/209:30:042
20685992210,7cyclictest7895-21ssh09:10:132
20684992216,4cyclictest26532-21grep12:30:121
20684992214,6cyclictest3760-21fw_packets11:50:151
20684992213,7cyclictest560-21dbus-daemon07:45:001
2068499221,14cyclictest9804-21ssh09:35:121
2068399226,3cyclictest141rcu_preempt11:43:020
2068399221,5cyclictest0-21swapper/009:50:360
2068399221,3cyclictest141rcu_preempt09:24:320
153732220,9sleep122-21ksoftirqd/110:02:011
104462221,6sleep310461-21ssh12:18:243
298112210,2sleep30-21swapper/307:30:153
20686992116,3cyclictest331rcuc/311:15:263
20686992112,7cyclictest30995-21perl09:50:173
20686992112,7cyclictest21289-21smartctl07:10:163
2068699210,7cyclictest12524-21tr11:10:133
2068599212,16cyclictest20105-21sed08:20:272
20685992112,7cyclictest17791-21perl12:00:142
20685992110,8cyclictest22260-21sh11:16:332
2068499212,13cyclictest24223-21ssh12:05:121
2068499212,11cyclictest18658-21taskset08:20:131
20684992112,7cyclictest8749-21ls07:55:201
20684992112,7cyclictest30279-21load07:30:201
20684992112,6cyclictest18899-21cat10:05:101
20684992111,7cyclictest14346-21grep09:15:001
2068499210,18cyclictest26586-21sort11:20:121
2068399218,11cyclictest3082-21expr09:30:120
2068399217,10cyclictest6005-21df_inode12:15:140
20683992114,5cyclictest30026-21sh09:50:000
20683992114,5cyclictest12-21ksoftirqd/010:25:000
2068399211,13cyclictest141rcu_preempt10:26:180
18142210,2sleep30-21swapper/308:55:183
2068699207,3cyclictest141rcu_preempt11:25:153
20686992013,4cyclictest7421-21diskmemload10:39:123
20686992012,6cyclictest32576-21ntp_states07:35:213
20686992012,6cyclictest32576-21ntp_states07:35:213
20686992012,6cyclictest32342-21ls08:50:193
20686992012,6cyclictest10205-21entropy09:35:163
2068599209,8cyclictest15606-21cat09:15:192
2068599209,8cyclictest15606-21cat09:15:192
20685992014,4cyclictest28733-21sh12:08:072
20685992013,5cyclictest7421-21diskmemload09:36:362
20685992012,6cyclictest32421-21memory07:35:192
20685992012,6cyclictest32421-21memory07:35:192
20685992012,6cyclictest32049-21cat12:10:182
20685992012,6cyclictest26841-21sed10:10:192
20685992012,6cyclictest16243-21seq11:59:382
20685992011,7cyclictest6823-21cat09:05:242
20685992011,7cyclictest25705-21ssh11:42:302
20685992011,7cyclictest19483-21head10:05:152
20685992011,6cyclictest11233-21sensors_temp08:00:232
2068499204,10cyclictest52950irq/141-enp3s0-12:25:131
20684992013,4cyclictest16309-21sh12:22:221
20684992012,6cyclictest3240-21sed09:00:001
20684992012,6cyclictest278-21systemd-journal07:30:001
20684992012,6cyclictest26670-21seq08:36:211
20684992012,6cyclictest22573-21seq08:28:421
2068499201,17cyclictest6241-21ssh10:41:561
20684992011,6cyclictest5248-21seq09:30:581
20684992011,6cyclictest24776-21seq08:34:281
20684992011,6cyclictest2442-21timerwakeupswit12:35:241
2068499201,14cyclictest7379-21ssh10:20:121
20684992010,7cyclictest25815-21cat07:20:191
20684992010,6cyclictest52950irq/141-enp3s0-09:55:111
2068399209,3cyclictest12-21ksoftirqd/009:00:220
2068399208,10cyclictest22379-21proc_pri10:30:210
2068399207,3cyclictest12-21ksoftirqd/011:12:570
2068399201,6cyclictest131rcuc/009:55:110
2068399201,6cyclictest0-21swapper/011:53:150
20683992014,4cyclictest12-21ksoftirqd/012:00:010
2068399201,3cyclictest7421-21diskmemload11:35:220
20683992012,6cyclictest29751-21ntpq11:45:200
20683992012,6cyclictest25035-21apt-get07:20:110
20683992012,6cyclictest20334-21seq08:22:320
20683992012,6cyclictest13000-21cut12:20:140
20683992011,7cyclictest24552-21seq08:31:350
56462190,1sleep30-21swapper/309:05:133
2068699194,12cyclictest17412-21cat10:50:133
2068699192,12cyclictest20268-21ssh10:05:233
20686991913,4cyclictest14422-21cpuspeed_turbos10:25:123
20686991912,5cyclictest3957-21sh12:36:433
20686991912,4cyclictest10521-21df11:55:143
20686991911,6cyclictest3889-21expr07:45:153
20686991911,6cyclictest24927-21sed08:35:003
20686991911,6cyclictest14443-21date09:15:003
20686991911,5cyclictest2606-21run-parts07:40:233
20686991910,7cyclictest23828-21pmu-power07:15:233
2068599198,9cyclictest278-21systemd-journal11:40:002
20685991914,3cyclictest26921-21sh07:25:002
20685991913,4cyclictest9614-21sh09:58:122
20685991913,4cyclictest25309-21sh09:45:302
20685991913,4cyclictest24081-21sh12:28:112
20685991912,5cyclictest4155-21sh12:36:542
20685991912,5cyclictest21483-21load07:10:182
20685991912,5cyclictest21013-21sed10:30:012
20685991912,5cyclictest15450-21sh11:11:402
20685991911,6cyclictest3677-21expr09:00:142
2068599191,12cyclictest12538-21ssh12:20:102
20685991910,7cyclictest8307-21df11:30:132
20685991910,7cyclictest8307-21df11:30:132
20685991910,6cyclictest6138-21grep07:50:152
20685991910,6cyclictest27565-21df08:40:142
20685991910,5cyclictest53050irq/142-enp3s0-10:30:112
2068499192,12cyclictest7945-21apt-get11:30:121
2068499192,12cyclictest7945-21apt-get11:30:111
2068499192,12cyclictest21833-21runrttasks11:40:121
20684991913,4cyclictest4778-21sh11:04:451
20684991913,4cyclictest3280-21sh10:16:301
20684991913,4cyclictest11429-21grep11:55:231
20684991913,3cyclictest21872-21hwlatdetect10:30:161
20684991912,5cyclictest7308-21seq09:08:431
20684991912,5cyclictest28421-21expr10:35:121
20684991911,6cyclictest21641-21cat09:20:131
20684991911,5cyclictest28994-21seq08:43:271
20684991910,6cyclictest4193-21grep09:00:201
20684991910,6cyclictest23958-21cat12:05:001
20684991910,6cyclictest22-21ksoftirqd/108:50:121
20684991910,6cyclictest16588-21acpi09:40:111
20684991910,6cyclictest16030-21grep11:35:211
2068399197,9cyclictest12-21ksoftirqd/012:30:090
2068399197,3cyclictest64652awk09:05:210
2068399197,2cyclictest12-21ksoftirqd/009:15:170
2068399197,2cyclictest12-21ksoftirqd/009:15:170
2068399195,3cyclictest141rcu_preempt10:56:180
20683991914,3cyclictest5010-21sh12:37:450
20683991913,4cyclictest21715-21ntp_offset08:25:220
20683991913,4cyclictest12-21ksoftirqd/008:10:000
20683991912,5cyclictest8896-21ntp_kernel_err09:10:210
20683991912,5cyclictest560-21dbus-daemon11:05:000
20683991912,5cyclictest27947-21sed08:40:180
20683991912,5cyclictest16930-21perl08:15:160
20683991911,6cyclictest804-21seq08:52:540
20683991911,6cyclictest5969-21expr07:50:130
20683991911,6cyclictest1424-21expr08:55:140
298592180,4sleep329837-21ssh11:45:223
2068699189,6cyclictest7997-21cat07:55:143
2068699189,6cyclictest25963-21ntp_kernel_err08:35:203
2068699189,6cyclictest12126-21cut08:05:013
2068699182,13cyclictest17123-21ssh12:00:043
2068699182,11cyclictest31100-21apt-get12:10:133
2068699181,8cyclictest15963-21ssh10:02:353
20686991812,4cyclictest9723-21sh10:45:033
20686991812,4cyclictest9723-21sh10:45:033
20686991812,4cyclictest30961-21apt-get11:00:093
20686991811,5cyclictest5814-21cat09:55:183
20686991811,5cyclictest25223-21expr07:20:133
20686991811,5cyclictest12846-21expr12:20:123
2068699181,12cyclictest26354-21sh11:20:033
20686991810,6cyclictest8964-21sed10:20:243
20686991810,6cyclictest31349-21cut08:50:013
20686991810,6cyclictest27708-21fw_packets08:40:163
20686991810,6cyclictest24442-21expr12:05:133
20686991810,6cyclictest17638-21ntpq09:40:203
20686991810,6cyclictest14780-21tune2fs08:10:183
2068599182,11cyclictest22943-21apt-get08:30:142
2068599181,8cyclictest29452-21ntp_states09:25:232
20685991813,3cyclictest9856-21/usr/sbin/munin10:45:192
20685991813,3cyclictest9856-21/usr/sbin/munin10:45:192
20685991813,3cyclictest6503-21sh11:51:542
20685991813,3cyclictest2758-21vmstat08:55:272
20685991812,4cyclictest8228-21sh11:06:242
20685991812,4cyclictest29266-21idleruntime-cro07:30:002
20685991812,4cyclictest28331-21sh11:20:302
20685991812,4cyclictest26204-21sh12:30:022
2068599181,15cyclictest780-21ssh10:15:142
20685991811,5cyclictest29459-21awk11:45:162
20685991811,5cyclictest28448-21expr10:35:122
20685991811,5cyclictest25068-21find08:35:112
20685991811,5cyclictest16992-21expr08:15:172
2068599181,12cyclictest30981-21sh11:00:102
20685991810,6cyclictest31022-21expr09:50:162
2068499189,6cyclictest29795-21cut08:45:161
2068499188,8cyclictest560-21dbus-daemon07:15:001
2068499187,3cyclictest273842sh09:47:291
2068499186,10cyclictest0-21swapper/111:05:081
20684991813,3cyclictest16345-21sh11:12:311
20684991813,3cyclictest14710-21fschecks_count08:10:171
20684991812,4cyclictest828-21sh11:25:011
2068499181,11cyclictest16427-21apt-get10:50:031
20684991810,6cyclictest25242-21cut10:55:241
20684991810,6cyclictest16998-21expr08:15:171
20684991810,6cyclictest15085-21awk09:15:151
20684991810,6cyclictest15085-21awk09:15:151
20684991810,5cyclictest10290-21expr08:00:131
2068399185,4cyclictest10096-21cstates09:35:140
20683991813,3cyclictest7421-21diskmemload10:45:190
20683991813,3cyclictest7421-21diskmemload10:45:190
20683991813,3cyclictest20482-21timerwakeupswit10:05:250
20683991811,5cyclictest5001-21cat07:45:270
20683991811,5cyclictest31433-21expr12:10:130
20683991811,5cyclictest29378-21awk08:45:110
20683991810,6cyclictest25417-21df08:35:140
2068399180,3cyclictest141rcu_preempt10:11:350
20686991712,3cyclictest25845-21sh09:46:013
2068699171,11cyclictest12220-21apt-get08:05:143
20686991710,5cyclictest30301-21seq10:12:593
2068599178,7cyclictest15274-21sh08:10:232
20685991712,3cyclictest30030-21gltestperf07:30:172
20685991712,3cyclictest20778-21/usr/sbin/munin08:25:172
2068599171,12cyclictest29382-21apt-get08:45:122
2068599171,11cyclictest1509-21apt-get07:40:132
20685991710,5cyclictest3841-21expr07:45:152
20685991710,5cyclictest13639-21awk08:05:272
2068499179,6cyclictest31634-21seq07:35:091
2068499179,6cyclictest31634-21seq07:35:091
2068499171,11cyclictest5784-21apt-get07:50:121
20684991710,5cyclictest3823-21ls07:45:151
20684991710,5cyclictest12284-21cat08:05:141
2068399179,6cyclictest15999-21seq08:12:050
2068399175,2cyclictest75-21kswapd007:40:130
20683991712,3cyclictest28214-21needreboot07:25:220
2068399171,11cyclictest10081-21apt-get08:00:120
20683991710,5cyclictest9140-21sed07:55:250
20683991710,5cyclictest22251-21seq07:10:300
20683991710,3cyclictest12-21ksoftirqd/007:35:120
20683991710,3cyclictest12-21ksoftirqd/007:35:120
2068699167,7cyclictest28308-21ntp_states07:25:243
2068699166,8cyclictest3740-21entropy09:00:163
2068699161,11cyclictest16493-21apt-get08:15:133
2068699161,10cyclictest20803-21apt-get08:25:143
20686991610,4cyclictest34-21ksoftirqd/307:55:003
2068699157,6cyclictest18630-21/usr/sbin/munin08:20:243
2068599159,4cyclictest7981-21wc07:55:142
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional