You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-24 - 21:43

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Tue Mar 24, 2026 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
317625140,5sleep20-21swapper/207:05:232
785021140,2sleep00-21swapper/012:14:000
2744321050,2sleep20-21swapper/210:40:222
306312910,4sleep09-21ksoftirqd/010:05:090
44622890,2sleep20-21swapper/209:34:152
299832880,2sleep10-21swapper/109:25:321
87622870,2sleep00-21swapper/009:39:160
87622870,2sleep00-21swapper/009:39:160
11812850,11sleep27950irq/56-dwc_otg_12:05:292
492499732,47cyclictest0-21swapper/011:20:210
284412730,2sleep30-21swapper/312:38:433
72682690,1sleep20-21swapper/211:34:132
283722690,2sleep20-21swapper/211:59:412
201512690,5sleep020149-21uniq11:10:270
4924996824,9cyclictest0-21swapper/010:25:220
492499671,23cyclictest0-21swapper/007:50:220
244702670,3sleep121-21rcuc/112:34:021
492599638,21cyclictest0-21swapper/110:05:211
4924996336,11cyclictest24421-21grep08:00:390
4924996336,11cyclictest24421-21grep08:00:390
4924996220,4cyclictest0-21swapper/012:25:220
4924996220,4cyclictest0-21swapper/012:25:220
4924996218,10cyclictest9-21ksoftirqd/008:50:200
492499621,44cyclictest1115-21wget09:30:220
4924996141,5cyclictest7850irq/56-dwc_otg_10:35:090
4924996044,6cyclictest7850irq/56-dwc_otg_11:50:360
4924996028,15cyclictest6213-21unixbench_singl07:10:360
4924996026,23cyclictest10961-21grep09:40:380
492499602,40cyclictest3631-21fschecks_time10:50:280
4924995935,12cyclictest1091-21grep11:25:390
4924995935,12cyclictest1091-21grep11:25:390
492499593,37cyclictest21868-21unixbench_multi09:15:370
492499593,37cyclictest21868-21unixbench_multi09:15:370
4924995735,12cyclictest19619-21ls11:10:090
492599569,32cyclictest15026-21ssh10:25:221
4924995641,5cyclictest7850irq/56-dwc_otg_09:05:240
4924995636,11cyclictest25063-21/usr/sbin/munin09:20:230
492499562,33cyclictest0-21swapper/008:30:100
492499561,44cyclictest0-21swapper/011:40:240
492499554,27cyclictest0-21swapper/012:00:090
4924995524,7cyclictest17919-21aten_rbpower_cu07:45:220
492499552,41cyclictest29275-21fschecks_count12:00:260
492499552,40cyclictest0-21swapper/010:35:270
492499552,33cyclictest0-21swapper/009:10:130
4924995523,21cyclictest0-21swapper/010:05:370
45312553,43sleep00-21swapper/007:08:420
4924995432,6cyclictest10727-21latency_hist10:20:090
492499542,41cyclictest0-21swapper/012:05:120
130892540,1sleep00-21swapper/010:22:160
492499532,44cyclictest0-21swapper/010:12:350
492499531,34cyclictest17298-21ntp_kernel_err11:45:300
4925995219,18cyclictest23-21ksoftirqd/109:20:201
492499526,19cyclictest0-21swapper/011:30:220
4924995226,6cyclictest11624-21wget11:00:230
4924995225,16cyclictest0-21swapper/012:30:220
4924995214,30cyclictest7713-21grep10:55:270
492499521,42cyclictest8935-21runrttasks12:15:230
492499521,40cyclictest0-21swapper/007:55:340
492499521,39cyclictest0-21swapper/008:45:360
492499521,37cyclictest0-21swapper/009:45:350
492499521,33cyclictest0-21swapper/012:20:200
492499521,28cyclictest25369-21ssh12:35:160
492499516,34cyclictest27162-21/usr/sbin/munin08:10:270
4924995122,14cyclictest0-21swapper/008:20:270
492499511,41cyclictest0-21swapper/008:34:240
492499511,39cyclictest9-21ksoftirqd/007:35:270
492499511,36cyclictest22268-21latency_hist09:55:090
4924995031,4cyclictest7750irq/56-dwc_otg07:30:240
4924995031,4cyclictest7750irq/56-dwc_otg07:30:240
492499502,31cyclictest0-21swapper/009:00:320
492499501,31cyclictest0-21swapper/010:40:270
492499499,37cyclictest23605-21timerandwakeup09:55:360
492499491,5cyclictest10237-21idleruntime-cro07:25:080
492499491,35cyclictest0-21swapper/009:25:150
492799487,5cyclictest11-21rcu_preempt08:50:203
4924994832,7cyclictest0-21swapper/011:36:270
492499482,43cyclictest0-21swapper/007:15:360
492499482,37cyclictest0-21swapper/008:40:220
4924994822,3cyclictest0-21swapper/007:40:190
492499481,33cyclictest32750-21unixbench-2d10:45:400
492499481,33cyclictest27910-21latency_hist11:20:090
492699473,9cyclictest0-21swapper/212:10:202
492599471,13cyclictest0-21swapper/111:20:201
492599469,4cyclictest0-21swapper/110:02:141
492499465,38cyclictest29552-21switchtime08:15:370
492499462,32cyclictest0-21swapper/008:35:370
492499462,32cyclictest0-21swapper/008:35:370
4927994512,12cyclictest39-21ksoftirqd/311:15:083
492599457,23cyclictest0-21swapper/107:35:161
492599456,24cyclictest0-21swapper/108:15:251
492599455,27cyclictest7750irq/56-dwc_otg11:30:091
492599455,27cyclictest7750irq/56-dwc_otg11:30:091
4925994511,21cyclictest17156-21ssh12:25:221
4925994511,21cyclictest17156-21ssh12:25:221
4924994520,9cyclictest0-21swapper/008:08:440
194592450,19sleep27950irq/56-dwc_otg_11:48:342
4927994432,5cyclictest13070-21ntp_states07:30:313
4927994432,5cyclictest13070-21ntp_states07:30:313
4926994414,6cyclictest22847-21fake-hwclock09:17:092
4926994414,6cyclictest22847-21fake-hwclock09:17:092
4926994410,5cyclictest7950irq/56-dwc_otg_12:00:212
4924994430,4cyclictest7750irq/56-dwc_otg08:59:290
4925994315,6cyclictest14124-21latency_hist07:35:101
4925994315,6cyclictest14124-21latency_hist07:35:101
492699427,7cyclictest7950irq/56-dwc_otg_09:10:222
492499423,33cyclictest11170-21cat07:25:300
492799413,9cyclictest7850irq/56-dwc_otg_10:10:203
4927994113,5cyclictest24273-21sh12:33:413
492799411,12cyclictest7850irq/56-dwc_otg_07:55:373
492699416,4cyclictest7850irq/56-dwc_otg_10:20:222
492599417,24cyclictest0-21swapper/108:25:141
28212410,1sleep20-21swapper/211:28:412
28212410,1sleep20-21swapper/211:28:412
4927994017,12cyclictest0-21swapper/307:40:363
4927994013,6cyclictest3634-21fschecks_time10:50:283
492699409,7cyclictest7850irq/56-dwc_otg_08:45:082
492599407,22cyclictest0-21swapper/111:50:211
492599406,24cyclictest7850irq/56-dwc_otg_10:20:271
492599406,22cyclictest0-21swapper/111:45:141
492599401,30cyclictest0-21swapper/107:16:221
201132401,4sleep120238-21cat10:30:361
492799392,3cyclictest0-21swapper/309:20:243
492699391,4cyclictest7850irq/56-dwc_otg_09:25:212
492599397,23cyclictest0-21swapper/111:15:231
492599397,21cyclictest16013-21diskmemload10:15:211
67642381,34sleep20-21swapper/209:35:492
67642381,34sleep20-21swapper/209:35:492
4926993818,8cyclictest20636-21/usr/sbin/munin11:50:352
492699381,25cyclictest25132-21ls09:20:242
4926993811,5cyclictest0-21swapper/211:35:382
492599381,23cyclictest31847-21grep10:45:291
225732380,2sleep337-21rcuc/311:51:413
4927993719,11cyclictest15688-21ssh12:23:093
4927993716,5cyclictest14439-21sh10:24:443
4926993717,5cyclictest31008-21basename10:05:232
4926993711,5cyclictest22974-21basename10:35:092
492599377,25cyclictest0-21swapper/112:00:211
492599376,3cyclictest0-21swapper/109:30:211
4925993715,6cyclictest23409-21sh09:18:061
4925993715,6cyclictest23409-21sh09:18:061
4925993715,4cyclictest14021-21sh11:02:441
95012361,2sleep30-21swapper/311:35:443
4927993617,12cyclictest8465-21ssh09:38:453
4927993617,12cyclictest8465-21ssh09:38:453
492699367,24cyclictest0-21swapper/208:20:222
4926993617,7cyclictest16006-21sh11:44:572
4926993615,5cyclictest16013-21diskmemload12:37:032
4925993618,5cyclictest264762idleruntime-cro10:00:091
492599361,25cyclictest0-21swapper/110:10:221
135642360,1sleep30-21swapper/309:44:593
4927993519,9cyclictest30361-21ssh12:00:453
4927993516,12cyclictest9617-21ssh10:57:163
4927993515,5cyclictest31420-21latency_hist10:45:093
4927993513,12cyclictest8793-21modprobe08:50:083
4927993512,5cyclictest12270-21latency_hist07:30:103
492699359,4cyclictest0-21swapper/210:15:252
492699358,4cyclictest0-21swapper/210:58:132
492699355,5cyclictest0-21swapper/209:54:472
4926993516,5cyclictest908-21runrttasks10:51:042
4926993516,4cyclictest908-21runrttasks09:46:382
4926993511,4cyclictest7850irq/56-dwc_otg_11:09:512
492599354,23cyclictest14681-21ls09:45:301
492599354,15cyclictest7950irq/56-dwc_otg_09:35:271
492599354,15cyclictest7950irq/56-dwc_otg_09:35:271
492799348,14cyclictest0-21swapper/310:30:373
492799345,4cyclictest0-21swapper/312:05:363
4927993419,9cyclictest15088-21ssh11:43:173
492699348,7cyclictest0-21swapper/212:30:252
492699348,4cyclictest0-21swapper/210:10:152
492699345,4cyclictest0-21swapper/210:35:272
492699344,13cyclictest7950irq/56-dwc_otg_10:00:262
4926993420,4cyclictest22490-21sh09:55:172
4926993416,4cyclictest13127-21aten_rbpower_vo12:20:242
4926993414,9cyclictest31-21ksoftirqd/211:15:082
492599349,13cyclictest0-21swapper/110:36:271
492599346,15cyclictest0-21swapper/110:40:271
492599345,5cyclictest7750irq/56-dwc_otg08:05:371
492599343,6cyclictest3989-21ssh11:30:221
4925993419,5cyclictest22027-21sh09:54:521
492599341,27cyclictest12685-21ls11:40:241
492799336,4cyclictest7750irq/56-dwc_otg09:05:253
492799334,12cyclictest8823-21ssh12:15:203
4927993315,12cyclictest20613-21/usr/sbin/munin09:15:353
4927993315,12cyclictest20613-21/usr/sbin/munin09:15:353
4927993315,10cyclictest16013-21diskmemload11:02:083
4927993312,8cyclictest8124-21seq07:16:383
4927993310,13cyclictest0-21swapper/309:57:273
492699337,5cyclictest7950irq/56-dwc_otg_12:15:222
492699337,4cyclictest0-21swapper/211:15:242
492699337,24cyclictest0-21swapper/208:45:212
492699332,4cyclictest7950irq/56-dwc_otg_07:30:362
492699332,4cyclictest7950irq/56-dwc_otg_07:30:362
4926993314,5cyclictest11387-21load08:55:292
4926993314,5cyclictest10132-21idleruntime09:40:272
4926993314,4cyclictest140192rm11:02:442
492699331,25cyclictest0-21swapper/210:29:582
492599338,5cyclictest7950irq/56-dwc_otg_11:05:281
492599337,13cyclictest7950irq/56-dwc_otg_11:58:451
492599334,11cyclictest0-21swapper/107:40:191
492599332,19cyclictest0-21swapper/112:15:261
4925993314,5cyclictest22866-21sh11:13:481
181802330,2sleep10-21swapper/107:45:261
128392330,3sleep24923-21cyclictest09:00:252
492799327,4cyclictest0-21swapper/310:05:203
492799325,9cyclictest7850irq/56-dwc_otg_07:50:313
492799322,17cyclictest7950irq/56-dwc_otg_11:46:083
4927993215,8cyclictest7950irq/56-dwc_otg_09:00:213
4927993215,4cyclictest16013-21diskmemload12:27:203
4927993215,4cyclictest16013-21diskmemload12:27:203
4927993215,10cyclictest8597-21sh10:16:393
4927993211,5cyclictest19057-21ntpq09:50:323
4927993210,4cyclictest30299-21timerandwakeup09:25:373
492699328,4cyclictest0-21swapper/211:22:122
4926993212,6cyclictest908-21runrttasks07:22:012
4926993211,8cyclictest19574-21ssh12:27:402
4926993211,8cyclictest19574-21ssh12:27:402
492599327,6cyclictest23-21ksoftirqd/109:10:231
492599326,13cyclictest7950irq/56-dwc_otg_10:55:391
492599324,12cyclictest7750irq/56-dwc_otg08:49:491
492599323,18cyclictest7950irq/56-dwc_otg_10:55:091
492599323,15cyclictest7750irq/56-dwc_otg09:05:081
492599322,4cyclictest9286-21fschecks_count08:50:261
492599321,15cyclictest7750irq/56-dwc_otg07:10:341
492599321,14cyclictest26008-21ssh12:35:291
4925993211,4cyclictest0-21swapper/112:05:221
492799319,14cyclictest30023-1kworker/u9:111:25:283
492799319,14cyclictest30023-1kworker/u9:111:25:283
492799318,5cyclictest7850irq/56-dwc_otg_08:00:393
492799318,5cyclictest7850irq/56-dwc_otg_08:00:393
492799317,4cyclictest0-21swapper/310:03:433
492799315,17cyclictest4774-21aten_rbpower_cu12:10:233
4927993118,4cyclictest23931-21/usr/sbin/munin11:15:203
4927993113,5cyclictest28670-21cat11:20:283
492799311,23cyclictest4285-21uniq11:30:263
492799311,18cyclictest8949-21grep07:20:253
4927993111,5cyclictest6775-21sshd08:42:413
492799311,10cyclictest0-21swapper/311:57:273
4927993110,3cyclictest0-21swapper/310:45:313
492699318,8cyclictest7850irq/56-dwc_otg_08:00:102
492699315,10cyclictest7950irq/56-dwc_otg_08:15:252
4926993116,5cyclictest14539-21/usr/sbin/munin09:05:242
492699311,4cyclictest31580-21ssh10:45:232
4926993112,8cyclictest4706-21http07:10:102
492599319,12cyclictest7950irq/56-dwc_otg_09:40:331
492599316,13cyclictest7950irq/56-dwc_otg_09:10:091
492599311,21cyclictest0-21swapper/111:35:241
492799307,6cyclictest7750irq/56-dwc_otg11:10:093
492799303,6cyclictest32435-21apt-get08:25:233
492799301,6cyclictest7950irq/56-dwc_otg_08:30:343
4927993013,10cyclictest16301-21ssh09:10:163
4927993011,5cyclictest6060-21sed07:10:353
492799301,15cyclictest15360-21ssh09:45:413
492699309,4cyclictest26661-21cron08:10:082
492699307,14cyclictest0-21swapper/207:15:222
492699302,12cyclictest7950irq/56-dwc_otg_08:25:392
492699301,25cyclictest18742-21ls07:45:342
492599307,3cyclictest0-21swapper/108:30:331
492599307,21cyclictest0-21swapper/112:20:231
492599306,4cyclictest7950irq/56-dwc_otg_12:13:331
492599305,7cyclictest7750irq/56-dwc_otg08:40:271
492599303,14cyclictest7850irq/56-dwc_otg_07:25:081
492599301,18cyclictest0-21swapper/107:25:211
40882305,15sleep34185-21unixbench_multi07:05:373
492799299,4cyclictest0-21swapper/308:10:343
492799297,5cyclictest0-21swapper/307:45:223
492799292,8cyclictest381ktimersoftd/310:25:243
4927992921,6cyclictest32316-21latency_hist08:25:093
4927992920,7cyclictest24087-21ntpq10:35:333
4927992910,12cyclictest2192-21cat09:30:363
492699296,15cyclictest0-21swapper/207:25:152
492699295,4cyclictest7750irq/56-dwc_otg08:31:132
492699291,2cyclictest0-21swapper/207:40:212
492599295,14cyclictest21-21rcuc/108:20:341
492599294,4cyclictest0-21swapper/108:40:101
492599294,4cyclictest0-21swapper/108:40:101
492599291,6cyclictest20033-21grep07:50:261
492799285,4cyclictest0-21swapper/308:55:293
4927992816,6cyclictest381ktimersoftd/307:35:153
492699283,5cyclictest0-21swapper/208:35:202
492699283,5cyclictest0-21swapper/208:35:202
492699282,4cyclictest27951-21grep08:10:372
492699281,6cyclictest15833-21modprobe07:40:082
492699281,5cyclictest9013-21/usr/sbin/munin08:50:312
492599286,14cyclictest7950irq/56-dwc_otg_08:00:221
492599286,14cyclictest7950irq/56-dwc_otg_08:00:221
492799276,4cyclictest7950irq/56-dwc_otg_08:35:213
492799276,4cyclictest7950irq/56-dwc_otg_08:35:213
492799271,6cyclictest30100-21taskset08:17:583
492799271,10cyclictest0-21swapper/308:09:423
492699274,10cyclictest0-21swapper/207:50:192
492699271,4cyclictest23306-21apt-get08:00:212
492699271,4cyclictest23306-21apt-get08:00:212
492599271,22cyclictest0-21swapper/107:55:231
43352273,14sleep10-21swapper/107:06:081
492599261,4cyclictest0-21swapper/108:10:251
492599255,4cyclictest7750irq/56-dwc_otg08:55:211
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional