You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-10 - 22:19

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Mon Aug 10, 2026 12:43:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1969121070,2sleep10-21swapper/107:33:141
2473721040,2sleep31093499cyclictest11:15:283
49972940,2sleep30-21swapper/309:29:273
49972940,2sleep30-21swapper/309:29:273
188802930,2sleep00-21swapper/012:30:240
1093199798,42cyclictest0-21swapper/012:15:090
288002780,4sleep10-21swapper/112:01:201
10931997719,5cyclictest5295-21cron09:30:020
10931997719,5cyclictest5295-21cron09:30:020
156252740,3sleep215602-21timerandwakeup07:20:312
98002730,3sleep10-21swapper/107:05:261
149922730,2sleep20-21swapper/210:22:252
1093499691,12cyclictest0-21swapper/312:01:153
1093499681,23cyclictest0-21swapper/311:42:353
10931996830,13cyclictest7950irq/56-dwc_otg_07:20:170
10931996826,16cyclictest7750irq/56-dwc_otg09:35:320
1093199682,12cyclictest14127-21grep08:45:310
82012670,2sleep00-21swapper/008:30:250
1093199671,47cyclictest28176-21fschecks_time11:20:220
16452660,2sleep10-21swapper/110:05:271
1093499651,22cyclictest0-21swapper/311:12:003
10931996510,43cyclictest0-21swapper/011:00:290
1093499641,20cyclictest30587-21chrt11:23:223
251822630,3sleep025181-21load10:35:240
213212630,6sleep21093399cyclictest11:52:062
10931996315,7cyclictest1180-21runrttasks12:13:150
10931996248,3cyclictest0-21swapper/011:25:130
10931996248,3cyclictest0-21swapper/011:25:130
10931996211,29cyclictest0-21swapper/009:55:020
1093199611,26cyclictest31240-21uniq08:05:220
1093499591,17cyclictest0-21swapper/309:21:573
1093199597,33cyclictest3721-21kworker/0:209:55:340
10931995824,5cyclictest17415-21ntpq08:55:290
1093199582,34cyclictest29374-21fschecks_count08:00:210
1093199576,11cyclictest0-21swapper/008:20:140
10931995737,5cyclictest7950irq/56-dwc_otg_12:00:030
10931995737,5cyclictest7950irq/56-dwc_otg_12:00:030
10931995731,6cyclictest15048-21apt-key11:45:030
10931995727,12cyclictest7850irq/56-dwc_otg_10:50:030
10931995724,8cyclictest18022-21latency_hist12:30:020
1093199572,29cyclictest2298-21latency_hist12:10:030
10931995712,31cyclictest9661-21/usr/sbin/munin10:15:280
10931995711,33cyclictest0-21swapper/008:45:020
10932995628,10cyclictest31807-21ntp_states12:05:281
10931995626,5cyclictest6908-21unixbench_multi08:25:310
10931995626,5cyclictest6908-21unixbench_multi08:25:310
10931995624,18cyclictest27206-21tr12:00:190
10931995612,31cyclictest9136-21unixbench_multi10:55:330
10931995612,31cyclictest9136-21unixbench_multi10:55:330
1093499551,26cyclictest17444-21ls07:25:323
10931995530,12cyclictest7950irq/56-dwc_otg_12:35:260
1093199552,8cyclictest4486-21latency_hist10:10:020
10931995524,18cyclictest0-21swapper/010:50:350
10931995522,9cyclictest27399-21aten_rbpower_po07:55:150
1093199551,5cyclictest20681-21ssh11:10:270
1093499541,42cyclictest7850irq/56-dwc_otg_11:35:173
1093299543,41cyclictest7750irq/56-dwc_otg07:45:111
1093199545,22cyclictest7251-21ssh11:35:100
10931995438,8cyclictest0-21swapper/011:45:150
10931995434,5cyclictest0-21swapper/007:35:140
10931995428,16cyclictest2464150ssh11:15:260
10931995426,6cyclictest9500-21tr08:35:170
1093299532,5cyclictest22305-21ssh12:35:161
1093199532,39cyclictest0-21swapper/011:05:210
10931995320,23cyclictest0-21swapper/012:20:210
1093299528,27cyclictest11896-21ssh11:00:101
1093199528,25cyclictest10930-21cyclictest09:45:230
10931995242,6cyclictest26605-21grep07:50:310
10931995224,6cyclictest22380-21aten_rbpower_vo09:10:180
10931995224,6cyclictest22380-21aten_rbpower_vo09:10:180
1093199522,12cyclictest17953-21unixbench_singl10:25:330
1093199521,38cyclictest0-21swapper/010:40:310
1093199521,38cyclictest0-21swapper/010:40:310
1093399518,29cyclictest0-21swapper/211:10:152
1093299517,35cyclictest7750irq/56-dwc_otg12:10:341
1093299512,39cyclictest7750irq/56-dwc_otg11:23:381
1093199519,25cyclictest0-21swapper/009:05:140
1093199518,9cyclictest7950irq/56-dwc_otg_09:20:150
10931995126,5cyclictest13454-21nfs4_client10:20:250
10931995123,9cyclictest18329-21tr07:30:150
10931995122,8cyclictest23815-21cut07:45:150
10931995119,7cyclictest7850irq/56-dwc_otg_09:30:280
10931995119,22cyclictest0-21swapper/011:32:290
1093499508,25cyclictest0-21swapper/307:30:153
1093299504,33cyclictest7750irq/56-dwc_otg08:40:221
10931995027,5cyclictest7750irq/56-dwc_otg08:15:130
10931995025,16cyclictest0-21swapper/007:40:140
10931995020,3cyclictest0-21swapper/010:30:140
1093199501,29cyclictest20139-21ssh11:50:300
318622490,5sleep31093499cyclictest12:05:293
1093399499,29cyclictest0-21swapper/207:30:152
1093299498,32cyclictest0-21swapper/111:10:151
10931994922,3cyclictest0-21swapper/010:00:240
10931994921,14cyclictest15434-21sh09:41:400
1093199491,8cyclictest6098-21unixbench_singl10:10:330
1093199491,28cyclictest26764-21kernelversion09:15:250
166442480,2sleep20-21swapper/211:45:372
10934994815,21cyclictest13592-21basename09:40:163
10932994818,6cyclictest1180-21runrttasks10:51:301
10932994814,15cyclictest0-21swapper/111:20:041
1093199486,24cyclictest0-21swapper/008:50:340
1093199482,25cyclictest0-21swapper/009:00:290
10931994822,11cyclictest7850irq/56-dwc_otg_07:25:150
1093199482,11cyclictest1374-21grep08:10:320
106022483,37sleep00-21swapper/007:09:180
1093499471,33cyclictest24467-21unixbench_singl11:55:343
1093499471,33cyclictest24467-21unixbench_singl11:55:343
1093399479,21cyclictest7850irq/56-dwc_otg_09:23:162
1093399472,23cyclictest7950irq/56-dwc_otg_08:40:232
1093299475,31cyclictest7750irq/56-dwc_otg10:45:031
1093299475,31cyclictest7750irq/56-dwc_otg10:45:031
10931994719,10cyclictest13334-21grep07:15:230
10931994719,10cyclictest13334-21grep07:15:230
1093199471,31cyclictest12134-21/usr/sbin/munin07:10:290
1093499462,29cyclictest0-21swapper/308:00:173
10934994620,7cyclictest16148-21sh11:05:193
1093499461,12cyclictest7191-21missed_timers12:15:233
1093399461,37cyclictest0-21swapper/211:58:372
1093399461,37cyclictest0-21swapper/211:58:372
10933994610,16cyclictest7850irq/56-dwc_otg_12:35:232
1093299467,20cyclictest7750irq/56-dwc_otg09:40:201
10932994611,18cyclictest23-21ksoftirqd/110:25:151
239772450,4sleep223979-21fschecks_count07:45:182
10933994517,5cyclictest16825-21grep08:55:202
1093299458,28cyclictest0-21swapper/112:15:141
1093299458,14cyclictest10907-21sh11:40:031
10932994523,6cyclictest1180-21runrttasks08:35:291
1093399445,24cyclictest0-21swapper/211:28:162
1093399445,24cyclictest0-21swapper/211:28:162
10933994426,8cyclictest7950irq/56-dwc_otg_12:07:172
10933994422,7cyclictest10930-21cyclictest09:35:142
10933994418,5cyclictest11514-21sh10:18:202
1093299446,20cyclictest7850irq/56-dwc_otg_09:30:181
1093299442,28cyclictest7850irq/56-dwc_otg_09:10:031
1093299441,6cyclictest20839-21rm10:30:201
10932994416,6cyclictest1180-21runrttasks09:19:171
83612430,3sleep310930-21cyclictest09:33:183
10934994320,11cyclictest7850irq/56-dwc_otg_10:31:553
10933994321,5cyclictest20617-21ls07:35:242
1093299433,6cyclictest7850irq/56-dwc_otg_09:40:021
1093299433,29cyclictest14080-21modprobe12:25:031
1093299432,22cyclictest7750irq/56-dwc_otg12:30:311
10932994316,5cyclictest1570-21sh10:46:031
1093299431,32cyclictest0-21swapper/111:40:241
42332420,2sleep20-21swapper/212:11:202
1093499422,17cyclictest0-21swapper/311:50:243
10934994215,12cyclictest4789-21/usr/sbin/munin10:50:293
10933994222,10cyclictest17654-21ssh10:25:312
10933994219,13cyclictest22458-21ssh09:50:312
1093299426,25cyclictest7750irq/56-dwc_otg10:15:241
1093299422,7cyclictest5314-21munin-plugin-st09:30:031
1093299422,7cyclictest5314-21munin-plugin-st09:30:031
10932994224,5cyclictest17832-21fschecks_time09:45:211
1093299421,9cyclictest16425-21/usr/sbin/munin11:05:241
10932994219,5cyclictest28164-21latency_hist10:40:031
10932994210,13cyclictest7750irq/56-dwc_otg11:50:271
10934994123,9cyclictest7950irq/56-dwc_otg_11:45:193
1093399419,20cyclictest0-21swapper/209:28:072
1093399419,20cyclictest0-21swapper/209:28:072
1093399419,16cyclictest7750irq/56-dwc_otg12:15:262
1093399418,25cyclictest0-21swapper/211:00:102
1093399415,16cyclictest0-21swapper/212:00:352
1093399413,6cyclictest3989-21/usr/sbin/munin08:20:202
1093399411,5cyclictest32132-21ssh10:45:032
1093399411,5cyclictest32132-21ssh10:45:032
10933994112,5cyclictest7850irq/56-dwc_otg_10:56:142
10933994112,5cyclictest7850irq/56-dwc_otg_10:56:142
10933994111,20cyclictest7850irq/56-dwc_otg_09:49:202
10933994110,6cyclictest1180-21runrttasks10:46:382
1093299416,19cyclictest7750irq/56-dwc_otg08:10:321
1093299412,6cyclictest0-21swapper/111:25:161
1093299412,6cyclictest0-21swapper/111:25:161
1093299412,27cyclictest7750irq/56-dwc_otg11:55:221
1093299412,27cyclictest7750irq/56-dwc_otg11:55:221
10932994119,6cyclictest24057-21sh09:53:071
10932994116,5cyclictest22662-21/usr/sbin/munin09:10:221
10932994116,5cyclictest22662-21/usr/sbin/munin09:10:221
82142400,2sleep20-21swapper/210:14:392
266072400,1sleep20-21swapper/211:18:172
265332400,2sleep30-21swapper/307:50:313
1093499407,28cyclictest0-21swapper/308:30:173
1093499401,18cyclictest9689-21aten_rbpower_vo09:35:163
1093499401,16cyclictest0-21swapper/312:29:553
1093399408,18cyclictest7850irq/56-dwc_otg_09:10:202
1093399408,18cyclictest7850irq/56-dwc_otg_09:10:202
1093399406,18cyclictest7850irq/56-dwc_otg_09:15:352
1093399403,6cyclictest14158-21unixbench_singl08:45:322
10933994027,10cyclictest0-21swapper/211:45:032
1093299406,14cyclictest7850irq/56-dwc_otg_10:10:141
1093299404,6cyclictest13784-21proc_pri08:45:271
1093299404,14cyclictest0-21swapper/111:33:161
1093499399,18cyclictest22653-21processes07:40:273
1093499395,18cyclictest25544-21grep09:55:193
10934993913,16cyclictest0-21swapper/310:02:503
1093499391,26cyclictest7950irq/56-dwc_otg_10:10:333
1093499391,24cyclictest0-21swapper/309:05:153
1093499391,18cyclictest0-21swapper/309:53:203
1093499391,16cyclictest4434-21ssh10:09:583
10934993911,18cyclictest18057-21latency09:45:243
1093499390,28cyclictest0-21swapper/310:20:313
1093399399,5cyclictest0-21swapper/211:38:362
1093399398,21cyclictest7850irq/56-dwc_otg_10:10:032
1093399397,19cyclictest7850irq/56-dwc_otg_10:50:192
1093399392,30cyclictest0-21swapper/209:01:342
1093299398,18cyclictest14674-21/usr/sbin/munin07:20:331
1093299396,25cyclictest7750irq/56-dwc_otg09:55:191
1093299393,19cyclictest7750irq/56-dwc_otg10:25:011
1093299391,7cyclictest17041-21fschecks_time08:55:231
1093299391,5cyclictest9194-21grep10:55:331
1093299391,5cyclictest9194-21grep10:55:331
10932993911,5cyclictest0-21swapper/112:25:191
1093499384,30cyclictest0-21swapper/308:20:163
1093499383,22cyclictest7850irq/56-dwc_otg_08:45:303
1093499382,23cyclictest0-21swapper/312:30:483
1093499381,18cyclictest0-21swapper/311:00:143
1093399382,16cyclictest0-21swapper/210:34:022
10933993819,8cyclictest7850irq/56-dwc_otg_12:31:542
10933993812,6cyclictest31210-21cut08:05:212
1093399381,26cyclictest28525-21ls11:20:272
10933993812,5cyclictest7156-21latency_hist11:35:032
10933993811,15cyclictest0-21swapper/209:40:532
1093299384,27cyclictest7850irq/56-dwc_otg_08:25:281
1093299384,27cyclictest7850irq/56-dwc_otg_08:25:281
1093299383,20cyclictest7750irq/56-dwc_otg11:45:331
10932993812,5cyclictest7950irq/56-dwc_otg_08:15:101
1093499379,16cyclictest7950irq/56-dwc_otg_10:40:263
1093499379,16cyclictest7950irq/56-dwc_otg_10:40:263
1093499372,27cyclictest4494-21/usr/sbin/munin11:30:343
10934993720,10cyclictest7928-21aten_rbpower_cu10:55:163
10934993720,10cyclictest7928-21aten_rbpower_cu10:55:163
10934993716,14cyclictest0-21swapper/308:55:253
1093499371,3cyclictest7850irq/56-dwc_otg_09:19:003
1093499371,21cyclictest0-21swapper/312:22:453
1093499371,16cyclictest0-21swapper/310:25:293
1093499371,11cyclictest9133-21grep10:15:203
1093399377,25cyclictest30357-21ls10:00:312
1093399375,17cyclictest7850irq/56-dwc_otg_12:20:172
1093399373,7cyclictest25837-21grep10:35:332
1093299376,17cyclictest7750irq/56-dwc_otg07:55:301
1093299373,4cyclictest0-21swapper/110:00:351
10932993718,10cyclictest16481-21/usr/sbin/munin07:25:301
10932993717,5cyclictest30164-21/usr/sbin/munin09:20:211
1093499366,18cyclictest32121-21fschecks_time11:25:203
1093499366,18cyclictest32121-21fschecks_time11:25:203
1093499366,18cyclictest0-21swapper/308:25:303
1093499366,18cyclictest0-21swapper/308:25:303
1093499363,26cyclictest0-21swapper/309:10:253
1093499363,26cyclictest0-21swapper/309:10:253
1093499362,23cyclictest0-21swapper/310:45:353
1093499362,19cyclictest14461-21latency_hist07:20:033
1093499362,19cyclictest14461-21latency_hist07:20:033
10934993621,4cyclictest381ktimersoftd/308:40:193
1093499361,16cyclictest22554-21forks12:35:203
1093399368,5cyclictest7850irq/56-dwc_otg_07:50:142
1093399365,16cyclictest7850irq/56-dwc_otg_11:05:272
1093399362,11cyclictest0-21swapper/209:55:322
10933993616,12cyclictest1353-21unixbench_multi08:10:322
1093399361,5cyclictest15200-21ssh12:25:282
10932993619,8cyclictest7950irq/56-dwc_otg_08:30:231
109812350,5sleep310984-21apt08:40:023
1093499355,6cyclictest7850irq/56-dwc_otg_08:10:323
10934993520,6cyclictest2561-21aten_rbpower_en12:10:173
1093499351,4cyclictest7850irq/56-dwc_otg_08:15:103
1093399351,7cyclictest6538-21/usr/sbin/munin08:25:272
1093399351,7cyclictest6538-21/usr/sbin/munin08:25:272
10933993515,5cyclictest28870-21latency_hist08:00:032
1093399351,23cyclictest0-21swapper/209:30:332
1093299351,5cyclictest7750irq/56-dwc_otg08:20:201
1093499342,21cyclictest0-21swapper/310:36:153
1093399348,21cyclictest3744-21cron08:20:022
1093399347,12cyclictest7950irq/56-dwc_otg_09:05:242
1093399345,6cyclictest9469-21wget08:35:172
1093299347,13cyclictest19292-21ps09:00:291
1093299343,7cyclictest7750irq/56-dwc_otg07:35:141
1093299341,6cyclictest0-21swapper/107:17:391
1093299341,6cyclictest0-21swapper/107:17:391
1093499338,15cyclictest24787-21unixbench_multi07:45:303
10934993312,9cyclictest28156-21switchtime07:55:273
1093499331,20cyclictest0-21swapper/307:20:163
1093399331,24cyclictest0-21swapper/208:53:192
10933993311,6cyclictest140-21systemd-journal08:02:492
1093399331,14cyclictest7750irq/56-dwc_otg07:30:032
1093299331,5cyclictest7750irq/56-dwc_otg07:10:291
10932993310,9cyclictest0-21swapper/107:50:271
1093499326,12cyclictest31312-21/usr/sbin/munin08:05:233
1093499321,22cyclictest0-21swapper/309:00:173
1093399321,7cyclictest13043-21grep07:15:192
1093399321,7cyclictest13043-21grep07:15:192
1093299322,6cyclictest31477-21memory08:05:251
10932993211,5cyclictest160942chrt08:51:311
1093499312,18cyclictest0-21swapper/308:50:303
1093499311,19cyclictest0-21swapper/307:10:153
1093299315,9cyclictest0-21swapper/108:00:211
1093399309,17cyclictest0-21swapper/208:30:252
1093399301,11cyclictest7850irq/56-dwc_otg_07:40:312
1093499291,18cyclictest0-21swapper/307:35:073
1093399292,5cyclictest7850irq/56-dwc_otg_07:10:292
1093299291,15cyclictest22777-21ls07:40:281
104842272,14sleep30-21swapper/307:07:473
104202278,12sleep20-21swapper/207:06:572
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional