You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-07-06 - 01:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Sun Jul 05, 2026 12:45:02)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
40272179101,28sleep20-21swapper/207:09:002
40922166114,29sleep30-21swapper/307:09:493
243821640,12sleep10-21swapper/107:05:131
38862158102,28sleep00-21swapper/007:07:090
187542710,2sleep334-21ksoftirqd/310:40:193
259852620,1sleep325916-21latency_hist08:00:003
26682610,2sleep10-21swapper/112:04:301
273662600,2sleep00-21swapper/009:59:560
110302580,1sleep10-21swapper/109:47:281
290862560,3sleep229089-21proc_pri10:00:212
172732550,2sleep10-21swapper/110:39:491
115502550,4sleep223589-21diskmemload12:10:232
121852540,2sleep20-21swapper/207:25:242
197712520,2sleep10-21swapper/111:52:421
190602520,2sleep10-21swapper/109:30:071
437999516,17cyclictest0-21swapper/009:30:120
291342510,1sleep20-21swapper/210:24:532
132092510,3sleep228-21ksoftirqd/208:45:132
215062500,4sleep221499-21ssh09:55:152
27242480,1sleep10-21swapper/109:17:421
206932480,2sleep00-21swapper/009:00:240
155632480,2sleep20-21swapper/212:37:132
120992440,1sleep30-21swapper/311:46:453
256782370,1sleep10-21swapper/110:21:281
4379993113,12cyclictest3640-21runrttasks12:05:130
4379993113,12cyclictest3640-21runrttasks12:05:120
437999295,21cyclictest12747-21turbostat07:30:000
437999294,17cyclictest12-21ksoftirqd/009:21:130
437999293,6cyclictest29445-21timerwakeupswit11:35:260
437999291,18cyclictest12228-21rm11:23:320
49282280,2sleep10-21swapper/109:43:121
438299281,24cyclictest9487-21memory07:20:193
438099281,25cyclictest0-21swapper/107:34:271
438099281,25cyclictest0-21swapper/107:21:081
437999282,9cyclictest16578-21ssh12:38:130
437999282,6cyclictest24401-21sed10:20:240
437999282,6cyclictest10740-21apt-key10:35:010
4379992822,3cyclictest6539-21/usr/sbin/munin07:15:190
314672280,9sleep228-21ksoftirqd/209:15:162
438099271,24cyclictest0-21swapper/107:26:271
437999276,11cyclictest12-21ksoftirqd/010:56:490
437999274,6cyclictest17098-21sh11:50:220
437999274,10cyclictest12-21ksoftirqd/009:40:210
437999273,8cyclictest22524-21ssh11:55:110
437999273,6cyclictest12255-21sed08:40:260
437999272,9cyclictest21473-21ssh10:19:040
437999272,8cyclictest8588-21/usr/sbin/munin09:45:240
438299261,23cyclictest0-21swapper/307:34:143
438299261,23cyclictest0-21swapper/307:25:373
438299261,23cyclictest0-21swapper/307:15:003
438099260,24cyclictest0-21swapper/107:19:381
437999264,6cyclictest20209-21tune2fs07:45:150
437999264,6cyclictest14332-21ssh10:13:550
437999264,6cyclictest13830-21ssh09:25:250
437999263,7cyclictest14208-21ssh12:12:350
437999263,7cyclictest10014-21cut11:45:190
437999263,6cyclictest2914-21ssh10:29:180
437999263,6cyclictest22803-21ssh11:07:030
4379992623,2cyclictest131rcuc/011:44:100
437999261,7cyclictest12-21ksoftirqd/011:10:220
437999261,16cyclictest5983-21ssh10:07:240
437999261,10cyclictest957-21/usr/sbin/munin11:15:150
437999260,24cyclictest0-21swapper/007:20:510
438299252,7cyclictest15468-21apt-get10:15:043
438199251,23cyclictest0-21swapper/207:20:532
438199250,8cyclictest259842cat08:00:002
438199250,23cyclictest0-21swapper/207:19:112
4380992523,1cyclictest0-21swapper/107:51:401
437999254,8cyclictest12-21ksoftirqd/009:35:440
437999254,6cyclictest24854-21ssh11:32:250
437999253,8cyclictest6847-21ssh10:31:260
437999253,3cyclictest12-21ksoftirqd/008:25:010
437999252,7cyclictest12-21ksoftirqd/012:20:140
437999252,7cyclictest12-21ksoftirqd/008:35:160
437999251,6cyclictest14497-21fschecks_time09:50:150
437999250,9cyclictest17082-21find07:38:030
4380992418,4cyclictest22-21ksoftirqd/109:00:001
437999246,7cyclictest12-21ksoftirqd/007:30:550
437999244,4cyclictest24749-21ssh10:45:130
437999243,7cyclictest28502-21head10:00:150
437999243,6cyclictest0-21swapper/012:25:120
437999243,6cyclictest0-21swapper/010:40:130
437999242,7cyclictest12-21ksoftirqd/012:15:210
437999242,6cyclictest21831-21cpuspeed_turbos09:05:140
437999242,15cyclictest24028-21ssh09:10:120
437999241,8cyclictest12-21ksoftirqd/010:50:160
437999241,6cyclictest32745-21grep08:15:120
437999241,6cyclictest12-21ksoftirqd/012:03:550
437999241,13cyclictest16672-21ssh11:26:130
437999240,5cyclictest0-21swapper/011:00:250
4382992312,7cyclictest29457-21grep10:25:003
438199239,4cyclictest141rcu_preempt11:30:002
438099231,10cyclictest0-21swapper/110:00:301
437999234,16cyclictest18934-21grep07:40:250
437999231,7cyclictest7179-21ssh12:30:430
437999231,6cyclictest23635-21seq07:53:230
437999231,6cyclictest12-21ksoftirqd/008:50:060
437999230,7cyclictest12-21ksoftirqd/009:15:150
438299221,7cyclictest25899-21ssh12:20:323
4382992214,6cyclictest17758-21ssh11:27:173
4382992214,5cyclictest12448-21sh10:12:033
4382992213,6cyclictest6617-21kernelversion10:55:183
438299221,13cyclictest23589-21diskmemload11:37:533
4381992220,1cyclictest0-21swapper/207:33:282
4381992215,5cyclictest10723-21ls08:39:592
4381992214,6cyclictest28-21ksoftirqd/209:55:002
437999223,7cyclictest12-21ksoftirqd/008:01:280
4379992218,3cyclictest131rcuc/008:10:470
437999221,6cyclictest636-21runrttasks08:09:400
437999220,7cyclictest5248-21sed08:25:170
437999220,5cyclictest0-21swapper/008:55:210
166932220,3sleep116696-21latency_hist12:15:001
74322211,3sleep37444-21ssh12:31:003
63372210,2sleep20-21swapper/209:20:162
438299217,11cyclictest15794-21ssh11:50:123
4382992112,7cyclictest560-21dbus-daemon12:20:003
4382992112,7cyclictest24713-21cat10:45:123
438199218,11cyclictest21183-21ssh11:05:262
438199215,5cyclictest636-21runrttasks10:57:012
438199212,16cyclictest18551-21tail10:40:172
438199212,14cyclictest3628-21ssh10:05:232
4381992115,4cyclictest28-21ksoftirqd/210:50:002
4381992114,4cyclictest636-21runrttasks11:30:032
4381992114,4cyclictest23979-21/usr/sbin/munin09:10:162
4381992113,6cyclictest6299-21cron07:15:002
4381992112,7cyclictest8925-21users09:45:252
4381992112,5cyclictest23589-21diskmemload11:15:122
4380992115,4cyclictest22-21ksoftirqd/111:25:001
4380992112,7cyclictest1850-21basename11:40:001
4380992110,7cyclictest53050irq/142-enp3s0-10:05:111
437999214,6cyclictest12-21ksoftirqd/008:30:260
438299209,7cyclictest52950irq/141-enp3s0-11:30:123
4382992015,3cyclictest30621-21python312:00:223
4382992013,5cyclictest18153-21memory08:55:183
4382992013,5cyclictest14010-21sed09:25:263
4382992012,6cyclictest5359-21which10:30:213
4382992012,6cyclictest20825-21sed09:55:013
4382992012,6cyclictest1005-21ls08:15:193
4382992011,7cyclictest21123-21date07:45:253
438199203,12cyclictest8238-21ssh11:20:182
438199202,14cyclictest17132-21ssh12:15:042
438199201,7cyclictest32346-21sh08:15:002
4381992014,4cyclictest636-21runrttasks12:24:532
4381992013,4cyclictest31439-21cut12:25:172
4381992012,6cyclictest18836-21sed07:40:242
4381992012,4cyclictest23589-21diskmemload12:05:142
4381992012,4cyclictest23589-21diskmemload12:05:142
438099205,3cyclictest69572expr08:30:131
438099201,5cyclictest0-21swapper/112:26:201
4380992015,3cyclictest32543-21sh11:15:001
4380992014,4cyclictest22-21ksoftirqd/111:20:011
4380992012,6cyclictest6232-21seq07:14:141
4380992012,6cyclictest21329-21seq07:46:131
4380992011,7cyclictest27481-21latency_hist11:35:001
437999203,6cyclictest12-21ksoftirqd/007:56:310
4379992012,6cyclictest6197-21seq07:13:460
232012200,1sleep10-21swapper/111:07:261
438299197,7cyclictest522-21apt-get11:15:123
438299196,3cyclictest141rcu_preempt12:35:043
438299191,8cyclictest23589-21diskmemload11:06:063
438299191,7cyclictest23589-21diskmemload11:03:253
438299191,7cyclictest23589-21diskmemload10:27:133
438299191,7cyclictest23589-21diskmemload10:07:163
438299191,7cyclictest23589-21diskmemload09:49:153
438299191,6cyclictest23589-21diskmemload11:59:343
438299191,4cyclictest31678-21cut10:50:143
4382991913,4cyclictest19294-21tr09:30:143
4382991913,4cyclictest11869-21sh09:24:593
4382991912,5cyclictest9172-21forks08:35:163
4382991912,5cyclictest22677-21awk09:05:223
4382991911,6cyclictest5676-21ls07:10:243
4382991911,6cyclictest32589-21ls09:15:243
4382991911,6cyclictest31227-21awk12:25:153
4382991911,6cyclictest19625-21cat09:00:153
4382991911,6cyclictest17626-21grep07:40:143
4382991911,6cyclictest14008-21ntpq08:45:203
4382991911,6cyclictest11716-21smartctl12:10:263
4382991911,5cyclictest6668-21acpi08:30:123
438299191,10cyclictest12314-21/usr/sbin/munin10:35:233
4382991910,6cyclictest11318-21entropy08:40:153
438199199,8cyclictest5307-21wc08:25:172
438199197,9cyclictest0-21swapper/211:35:182
438199197,3cyclictest141rcu_preempt11:02:502
438199196,3cyclictest141rcu_preempt10:30:002
438199193,5cyclictest636-21runrttasks09:29:012
438199192,12cyclictest15923-21apt-get10:15:042
438199192,12cyclictest15770-21apt-get11:50:112
438199191,5cyclictest10740-21apt-key10:35:002
438199191,4cyclictest0-21swapper/211:47:542
4381991914,3cyclictest21254-21uptime07:45:252
4381991910,6cyclictest1033-21cut08:15:192
4381991910,5cyclictest0-21swapper/211:10:122
438099195,10cyclictest17139-21apt-get12:15:031
438099192,12cyclictest3610-21apt-get12:05:111
438099192,12cyclictest3610-21apt-get12:05:101
438099191,4cyclictest0-21swapper/110:12:421
4380991913,4cyclictest29462-21sh09:37:101
4380991913,4cyclictest29077-21sh10:48:001
4380991913,4cyclictest24751-21sh09:57:211
4380991913,4cyclictest22901-21sh11:55:141
4380991913,4cyclictest1685-21sh10:28:041
4380991913,4cyclictest13854-21sh11:48:301
4380991912,5cyclictest8514-21seq12:32:061
4380991912,5cyclictest29939-21sh09:14:311
4380991912,5cyclictest17159-31updatedb.mlocat07:38:041
4380991912,5cyclictest16955-21seq08:52:451
4380991912,5cyclictest1334-21sensors08:15:231
4380991912,4cyclictest23589-21diskmemload10:33:391
4380991912,4cyclictest16322-21sh09:50:591
4380991911,6cyclictest3588-21cat08:20:221
4380991911,6cyclictest30798-21entropy08:10:141
4380991911,6cyclictest25468-21ls12:20:221
4380991911,6cyclictest18339-21ls07:40:191
4380991911,6cyclictest15470-21seq12:37:061
4380991910,7cyclictest9838-21seq10:57:381
4380991910,7cyclictest6656-21tr09:20:211
4380991910,6cyclictest53050irq/142-enp3s0-09:25:131
4380991910,6cyclictest17199-21cut10:15:211
437999190,7cyclictest12-21ksoftirqd/008:45:110
171602190,5sleep20-21swapper/207:38:052
438299182,11cyclictest30460-21apt-get08:10:123
438299181,6cyclictest23589-21diskmemload11:40:443
438299181,6cyclictest23589-21diskmemload11:20:423
438299181,6cyclictest23589-21diskmemload09:40:083
4382991813,3cyclictest4710-21sh12:05:213
4382991813,3cyclictest4710-21sh12:05:203
4382991812,4cyclictest27344-21switchtime09:35:233
4382991811,5cyclictest4709-21cat08:25:123
4382991811,5cyclictest28194-21expr10:00:133
4382991811,5cyclictest27005-21ls11:10:173
4382991811,5cyclictest24954-21cat09:10:193
4382991811,5cyclictest22142-21cat07:50:113
4382991810,6cyclictest624-21gdbus08:25:003
4382991810,6cyclictest15603-21forks08:50:153
4382991810,5cyclictest15552-21cat07:35:153
438199187,7cyclictest52850irq/140-enp3s0-10:10:122
438199182,12cyclictest25993-21ssh09:35:132
438199181,8cyclictest640-21snmpd12:03:332
4381991813,3cyclictest3411-21proc_pri08:20:212
4381991812,4cyclictest2593-21sh10:52:112
4381991812,4cyclictest25198-21sh11:56:222
4381991811,5cyclictest30272-21wc08:10:002
4381991811,5cyclictest23650-21latency_hist09:10:012
438199181,14cyclictest3831-21ssh11:40:242
4381991811,4cyclictest25643-21sed09:35:002
4381991811,4cyclictest10474-21apt-get10:35:002
4381991810,5cyclictest0-21swapper/212:30:202
438099182,11cyclictest21692-21apt-get09:05:131
438099181,4cyclictest0-21swapper/110:54:371
438099181,3cyclictest0-21swapper/111:25:031
4380991813,3cyclictest4370-21sh11:40:471
4380991813,3cyclictest29272-21open_inodes08:05:211
4380991812,4cyclictest12604-21sh11:00:101
4380991811,5cyclictest12685-21seq08:42:501
4380991811,4cyclictest23589-21diskmemload10:41:311
4380991810,6cyclictest9839-21which08:35:221
4380991810,6cyclictest21250-21seq09:02:451
438299172,11cyclictest26186-21apt-get08:00:133
438299171,5cyclictest15096-21ntp_states09:50:213
438199179,6cyclictest7104-21tune2fs08:30:152
438199172,11cyclictest10980-21apt-get08:40:132
438199171,14cyclictest18335-21/usr/sbin/munin08:55:202
4381991710,5cyclictest15478-21expr08:50:132
4381991710,4cyclictest0-21swapper/209:40:112
438099172,10cyclictest4543-21apt-get08:25:121
438099171,11cyclictest13121-21apt-get08:45:131
4380991710,5cyclictest27033-21strings08:00:191
438299169,5cyclictest28417-21expr08:05:123
438199162,10cyclictest20318-21runrttasks09:00:202
4381991610,4cyclictest22674-21missed_timers07:50:172
438099169,5cyclictest24726-21sed07:55:171
438199157,6cyclictest26650-21gltestperf08:00:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional