You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-25 - 14:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Thu Jun 25, 2026 00:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2890621750,6sleep1205399cyclictest22:05:321
94321110,7sleep0205299cyclictest20:35:260
149802950,9sleep3381ktimersoftd/321:50:083
222432900,2sleep00-21swapper/020:05:230
82192860,3sleep38218-21df00:15:233
82192860,3sleep38218-21df00:15:233
137712840,5sleep1205399cyclictest23:42:231
265782820,5sleep3205599cyclictest21:25:243
299232770,2sleep20-21swapper/200:01:412
87942750,2sleep30-21swapper/321:42:113
31172740,6sleep0205299cyclictest21:35:320
197392730,2sleep10-21swapper/123:10:351
36612690,6sleep2205499cyclictest22:15:082
257812690,2sleep30-21swapper/321:25:003
88362680,2sleep30-21swapper/320:55:333
2052996721,31cyclictest11982-21sh23:40:270
49852650,2sleep10-21swapper/123:31:401
205399657,4cyclictest8050irq/56-dwc_otg_00:10:361
205299651,23cyclictest0-21swapper/022:25:190
287702630,2sleep20-21swapper/223:21:022
205399630,44cyclictest20983-21unixbench_multi23:50:361
205399630,44cyclictest20983-21unixbench_multi23:50:361
2052996318,24cyclictest505-21cron00:30:070
205399625,5cyclictest7950irq/56-dwc_otg_22:05:071
2052996231,6cyclictest8050irq/56-dwc_otg_21:10:200
2052996225,9cyclictest7950irq/56-dwc_otg_23:10:180
205399617,5cyclictest8050irq/56-dwc_otg_21:27:431
205399617,42cyclictest7850irq/56-dwc_otg20:20:291
205399617,42cyclictest7850irq/56-dwc_otg20:20:291
205399612,5cyclictest27701-21irqstats23:20:251
205299612,4cyclictest17654-21ssh21:15:200
205299612,16cyclictest16818-21ssh23:45:360
205399609,4cyclictest7950irq/56-dwc_otg_21:13:311
205299607,15cyclictest0-21swapper/019:15:210
205299602,38cyclictest8847-21unixbench_multi20:55:330
205299601,32cyclictest4498-21ls20:45:260
205299601,32cyclictest4498-21ls20:45:260
205299601,30cyclictest22659-21ntp_kernel_err21:20:270
205399595,4cyclictest7850irq/56-dwc_otg21:30:251
205399595,4cyclictest7850irq/56-dwc_otg21:30:251
205399595,4cyclictest0-21swapper/122:23:271
205399592,46cyclictest0-21swapper/100:25:251
205399591,13cyclictest2891-21grep22:50:371
2053995910,35cyclictest0-21swapper/121:39:321
205299597,4cyclictest1343-21latency_hist22:50:070
205299592,33cyclictest7426-21wget23:35:220
205399587,4cyclictest10840-21ssh21:45:201
205399586,40cyclictest0-21swapper/121:18:381
2053995815,40cyclictest30958-21latency_hist20:30:081
205399581,47cyclictest0-21swapper/123:27:451
2053995811,4cyclictest8050irq/56-dwc_otg_19:30:421
205299581,38cyclictest6861-21grep19:20:340
205399578,4cyclictest8050irq/56-dwc_otg_19:50:101
205399576,38cyclictest8050irq/56-dwc_otg_21:51:161
205399573,4cyclictest14356-21sh23:05:141
205399573,3cyclictest7850irq/56-dwc_otg22:35:301
205399572,4cyclictest8050irq/56-dwc_otg_00:19:311
205399572,4cyclictest8050irq/56-dwc_otg_00:19:311
205399572,47cyclictest12736-21aten_rbpower_vo22:25:231
205299573,32cyclictest0-21swapper/022:35:240
2052995719,25cyclictest11226-21latency_hist21:05:080
205299571,7cyclictest19095-21ntpq19:55:290
205299571,36cyclictest15691-21grep23:05:350
205399565,7cyclictest8050irq/56-dwc_otg_23:56:201
205399565,43cyclictest0-21swapper/122:14:531
205399561,5cyclictest21321-21/usr/sbin/munin00:30:331
2053995610,42cyclictest0-21swapper/100:20:361
205299565,32cyclictest0-21swapper/000:35:190
205399552,4cyclictest26244-21sshd00:36:171
205399552,10cyclictest0-21swapper/122:48:291
205399551,3cyclictest0-21swapper/121:55:301
2053995511,4cyclictest1863-21taskset20:37:211
2052995530,4cyclictest9550irq/48-DMA15:26:470
2052995526,11cyclictest8050irq/56-dwc_otg_19:15:070
205299552,36cyclictest11-21rcu_preempt21:50:320
205299551,35cyclictest11143-21fschecks_count21:45:250
205399547,3cyclictest0-21swapper/123:03:171
205399545,36cyclictest0-21swapper/123:45:241
205399541,3cyclictest0-21swapper/122:55:341
205399541,13cyclictest0-21swapper/119:27:371
2052995429,6cyclictest7950irq/56-dwc_otg_00:15:210
2052995429,6cyclictest7950irq/56-dwc_otg_00:15:210
205299541,33cyclictest23326-21latency_hist22:00:080
49422530,2sleep00-21swapper/023:31:350
205399533,9cyclictest7950irq/56-dwc_otg_23:19:111
205399533,8cyclictest8050irq/56-dwc_otg_19:36:141
205399532,9cyclictest3263-21wc20:40:341
205399532,5cyclictest7659-21rm22:19:421
205399531,7cyclictest25754-21aten_rbpower_vo20:15:231
205399531,3cyclictest0-21swapper/121:21:001
2052995333,6cyclictest7850irq/56-dwc_otg22:15:070
2052995331,8cyclictest0-21swapper/000:01:570
205299532,32cyclictest0-21swapper/000:20:290
205299532,25cyclictest30827-21apt-get20:30:150
205299530,35cyclictest8370-21ssh21:41:270
205399526,7cyclictest8050irq/56-dwc_otg_22:33:541
205399525,4cyclictest0-21swapper/100:03:481
205399522,9cyclictest0-21swapper/121:44:521
205399522,6cyclictest9549-21/usr/sbin/munin21:00:231
205299524,26cyclictest9-21ksoftirqd/021:30:240
205299524,26cyclictest9-21ksoftirqd/021:30:240
205299523,25cyclictest22942-21ssh23:15:210
2052995225,4cyclictest27218-21latency_hist20:20:080
205299521,8cyclictest2387-21aten_rbpower_en20:40:220
205299521,34cyclictest0-21swapper/020:50:210
205299521,14cyclictest31880-21latency_hist22:10:080
205399517,6cyclictest7950irq/56-dwc_otg_21:05:331
205399514,7cyclictest0-21swapper/119:49:411
205399512,7cyclictest0-21swapper/119:45:081
205399512,5cyclictest7850irq/56-dwc_otg19:55:211
2053995111,5cyclictest12940-21diskmemload00:07:431
205299511,5cyclictest0-21swapper/022:20:360
205299511,24cyclictest589850rm22:55:200
205399506,3cyclictest7950irq/56-dwc_otg_22:42:261
205399506,3cyclictest7950irq/56-dwc_otg_22:42:261
205399503,7cyclictest0-21swapper/120:10:351
205399502,5cyclictest10590-21ssh23:39:101
205399501,9cyclictest3942-21cut19:15:211
205299502,6cyclictest0-21swapper/019:25:310
2052995024,5cyclictest7850irq/56-dwc_otg22:30:200
205299502,25cyclictest28041-21wc23:20:300
2052995015,6cyclictest0-21swapper/023:00:190
2052995010,22cyclictest4542-21latency22:15:290
205399491,5cyclictest6009-21aten_rbpower_en20:50:211
2052994926,3cyclictest0-21swapper/019:30:320
205599488,26cyclictest0-21swapper/319:45:153
205399482,6cyclictest21315-21unixbench_multi20:00:371
205399481,5cyclictest22014-21/usr/sbin/munin20:05:311
2053994812,33cyclictest8698-21timerandwakeup20:55:321
205399481,10cyclictest4435-21cut20:45:261
205399481,10cyclictest4435-21cut20:45:261
2052994824,14cyclictest0-21swapper/019:40:300
205299481,26cyclictest23962-21basename23:55:200
205299481,15cyclictest0-21swapper/000:10:080
205299481,11cyclictest0-21swapper/000:34:410
205499478,26cyclictest0-21swapper/219:40:212
205399472,3cyclictest0-21swapper/120:30:211
205299473,23cyclictest0-21swapper/020:10:190
205299472,24cyclictest0-21swapper/022:40:320
205299472,24cyclictest0-21swapper/022:40:320
205299471,24cyclictest20968-21proc_pri20:00:320
238192460,1sleep30-21swapper/323:55:143
205599468,23cyclictest0-21swapper/320:10:213
205499467,24cyclictest0-21swapper/219:10:202
205399462,8cyclictest5946-21fschecks_count19:20:231
205399462,4cyclictest0-21swapper/119:13:481
205299461,31cyclictest125-21systemd-journal21:10:070
205299461,25cyclictest11606-21cat19:35:290
2054994532,11cyclictest8050irq/56-dwc_otg_22:50:212
2052994511,18cyclictest15378-21missed_timers19:45:280
205599447,26cyclictest0-21swapper/319:40:213
205599447,23cyclictest0-21swapper/320:05:203
205499447,23cyclictest0-21swapper/221:10:132
2052994434,3cyclictest8050irq/56-dwc_otg_22:55:080
2052994421,4cyclictest17366-21ps19:50:320
205299442,13cyclictest25185-21rm22:01:000
2052994421,13cyclictest0-21swapper/023:53:390
2052994421,13cyclictest0-21swapper/023:53:390
156862440,2sleep20-21swapper/222:28:402
144742440,5sleep2205499cyclictest23:43:412
205599436,18cyclictest0-21swapper/320:35:203
205299432,25cyclictest0-21swapper/000:13:360
205499422,3cyclictest7950irq/56-dwc_otg_23:15:202
205299423,15cyclictest31365-21ssh23:25:190
16442422,12sleep00-21swapper/019:08:310
219042410,1sleep20-21swapper/221:20:162
205499416,23cyclictest7850irq/56-dwc_otg21:34:042
205499416,23cyclictest7850irq/56-dwc_otg21:34:042
205499413,4cyclictest8050irq/56-dwc_otg_00:35:182
2054994116,15cyclictest0-21swapper/221:25:542
205299417,8cyclictest7850irq/56-dwc_otg20:25:180
238492401,35sleep20-21swapper/222:38:032
205599402,13cyclictest0-21swapper/322:25:383
205499406,21cyclictest0-21swapper/220:35:202
2054994018,6cyclictest7740-21/usr/sbin/munin20:55:162
205499401,27cyclictest7850irq/56-dwc_otg22:00:342
205299401,6cyclictest29152-21latency_hist20:25:090
205299401,6cyclictest29152-21latency_hist20:25:090
178032400,1sleep20-21swapper/221:53:012
91252390,1sleep30-21swapper/319:30:083
2055993923,4cyclictest3920-21idleruntime-cro20:45:083
2054993923,6cyclictest5820-21wget19:20:202
205499391,27cyclictest9728-21ls21:00:222
205599387,22cyclictest0-21swapper/319:55:213
205599387,17cyclictest0-21swapper/319:10:203
2054993815,4cyclictest7796-21tail21:40:362
110312380,1sleep30-21swapper/322:22:583
2055993713,8cyclictest39-21ksoftirqd/321:15:263
205499371,25cyclictest25682-21ls22:40:212
205499371,25cyclictest25682-21ls22:40:212
205499371,15cyclictest19965-21kworker/u8:122:45:352
205599367,23cyclictest0-21swapper/320:30:213
2055993619,5cyclictest4240-21ssh22:15:253
2055993612,14cyclictest37-21rcuc/321:10:223
205499366,25cyclictest0-21swapper/220:00:212
205499366,24cyclictest0-21swapper/219:35:222
205499364,8cyclictest0-21swapper/222:55:192
205499363,22cyclictest7850irq/56-dwc_otg20:54:592
166982360,1sleep30-21swapper/323:07:183
205599351,5cyclictest8050irq/56-dwc_otg_23:35:213
205599351,5cyclictest10595-21ssh23:00:263
205599351,24cyclictest8050irq/56-dwc_otg_22:30:183
205499356,24cyclictest0-21swapper/220:30:142
205499356,19cyclictest0-21swapper/223:55:192
205499353,5cyclictest7850irq/56-dwc_otg22:05:172
205499352,27cyclictest7950irq/56-dwc_otg_00:25:212
2054993512,6cyclictest3540-21idleruntime-cro00:10:082
198262350,3sleep219829-21aten_rbpower_po23:50:212
198262350,3sleep219829-21aten_rbpower_po23:50:212
205599348,12cyclictest8050irq/56-dwc_otg_22:53:303
2055993419,4cyclictest27990-21aten_rbpower_po22:05:203
2055993413,5cyclictest37-21rcuc/321:30:263
2055993413,5cyclictest37-21rcuc/321:30:263
2055993412,4cyclictest7850irq/56-dwc_otg00:30:203
205499345,5cyclictest7950irq/56-dwc_otg_22:15:302
205499344,21cyclictest7950irq/56-dwc_otg_23:45:252
205599337,14cyclictest8050irq/56-dwc_otg_22:00:253
205599336,13cyclictest7850irq/56-dwc_otg20:20:363
205599336,13cyclictest7850irq/56-dwc_otg20:20:363
205599334,4cyclictest0-21swapper/300:00:253
2055993316,8cyclictest3312-21ssh22:14:353
205599331,13cyclictest7850irq/56-dwc_otg22:55:363
205499337,4cyclictest0-21swapper/200:30:182
205499336,20cyclictest14755-21cut23:05:242
205499333,5cyclictest12934-21sh21:10:072
2054993314,5cyclictest25624-21/usr/sbin/munin20:15:292
2054993314,11cyclictest0-21swapper/223:35:122
68152320,2sleep20-21swapper/200:13:362
205599323,7cyclictest4261-21wget20:45:233
205599323,7cyclictest4261-21wget20:45:233
205599323,14cyclictest8050irq/56-dwc_otg_19:35:253
2055993223,7cyclictest18953-21ssh23:49:183
2055993222,7cyclictest19216-21latency_hist21:55:083
2055993222,4cyclictest8050irq/56-dwc_otg_23:29:193
2055993219,4cyclictest8050irq/56-dwc_otg_00:25:323
205599321,6cyclictest29749-21/usr/sbin/munin22:45:353
205599321,19cyclictest26345-21ssh23:19:153
205599320,22cyclictest3099-21ssh21:35:323
205499329,15cyclictest7950irq/56-dwc_otg_22:20:372
205499327,5cyclictest7950irq/56-dwc_otg_00:20:272
205499327,4cyclictest7950irq/56-dwc_otg_21:55:352
205499327,16cyclictest0-21swapper/223:00:212
205499322,6cyclictest0-21swapper/219:55:292
205499322,4cyclictest7850irq/56-dwc_otg00:15:242
205499322,4cyclictest7850irq/56-dwc_otg00:15:242
205499321,28cyclictest9251-21ls19:30:082
2054993210,5cyclictest12940-21diskmemload21:49:532
205599319,6cyclictest0-21swapper/300:21:323
205599319,4cyclictest0-21swapper/300:13:383
205599313,13cyclictest8050irq/56-dwc_otg_20:00:273
205599311,3cyclictest0-21swapper/323:10:543
205499314,6cyclictest17297-21ntpq19:50:302
205499313,4cyclictest0-21swapper/220:10:322
2054993125,2cyclictest0-21swapper/219:45:102
2054993124,5cyclictest17677-21ssh21:15:222
205499311,7cyclictest7850irq/56-dwc_otg21:35:262
2054993113,5cyclictest4487-21fschecks_time20:45:262
2054993113,5cyclictest4487-21fschecks_time20:45:262
205499311,18cyclictest7850irq/56-dwc_otg23:35:082
205599308,11cyclictest8050irq/56-dwc_otg_22:35:203
205599307,14cyclictest0-21swapper/319:30:213
205599306,7cyclictest7850irq/56-dwc_otg00:05:313
205599306,4cyclictest7950irq/56-dwc_otg_00:36:323
205599306,13cyclictest19986-21df_inode23:50:233
205599306,13cyclictest19986-21df_inode23:50:233
2055993025,3cyclictest381ktimersoftd/323:30:173
205599302,13cyclictest8050irq/56-dwc_otg_21:56:403
2055993015,4cyclictest727-21runrttasks23:23:473
2055993015,11cyclictest0-21swapper/320:50:253
205499303,5cyclictest22778-21proc_pri20:05:312
205499302,8cyclictest0-21swapper/220:40:212
2054993011,6cyclictest0-21swapper/223:10:182
120882300,2sleep312086-21ssh23:40:283
205599299,4cyclictest0-21swapper/322:40:323
205599299,4cyclictest0-21swapper/322:40:323
205599295,6cyclictest7950irq/56-dwc_otg_21:00:213
205599291,6cyclictest13104-21ssh21:10:083
205499294,17cyclictest7850irq/56-dwc_otg23:25:302
205499293,16cyclictest7850irq/56-dwc_otg22:33:062
9302285,8sleep30-21swapper/319:05:313
205499286,4cyclictest9880-21cat19:30:292
205499281,24cyclictest0-21swapper/219:15:212
16602284,14sleep20-21swapper/219:08:382
8592277,12sleep10-21swapper/119:05:291
205599273,16cyclictest0-21swapper/320:16:033
205599272,6cyclictest8050irq/56-dwc_otg_19:15:213
205599271,16cyclictest0-21swapper/319:20:283
205499272,17cyclictest0-21swapper/220:28:332
205499272,15cyclictest0-21swapper/220:22:062
205499272,15cyclictest0-21swapper/220:22:062
205599262,4cyclictest0-21swapper/320:30:083
205599261,22cyclictest0-21swapper/319:50:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional