You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-29 - 13:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Mon Dec 29, 2025 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2740599871,47cyclictest0-21swapper/022:35:230
63562860,2sleep00-21swapper/019:40:320
2740599862,63cyclictest27945-21wget22:10:260
2740599843,59cyclictest0-21swapper/023:30:280
2740599831,21cyclictest25783-21latency_hist22:05:090
2741299822,8cyclictest25264-21kworker/u8:120:05:151
178392820,6sleep1221ktimersoftd/123:10:321
27405997931,37cyclictest7850irq/56-dwc_otg_23:25:090
27405997926,42cyclictest7850irq/56-dwc_otg_00:35:350
2740599791,17cyclictest0-21swapper/019:45:160
27405997831,33cyclictest7950irq/56-dwc_otg_22:55:400
27405997827,33cyclictest7850irq/56-dwc_otg_20:30:090
2740599782,64cyclictest0-21swapper/020:25:090
2740599782,64cyclictest0-21swapper/020:25:090
270922770,4sleep127089-21ls19:10:101
270922770,4sleep127089-21ls19:10:101
27412997512,29cyclictest27403-21cyclictest20:30:201
27405997522,42cyclictest10194-21cat19:50:320
32892730,3sleep22742099cyclictest00:00:292
27405997224,4cyclictest7750irq/56-dwc_otg00:30:260
2740599721,20cyclictest9753-21/usr/sbin/munin21:20:330
117022720,3sleep013-21rcuc/000:25:100
2742099715,54cyclictest10471-21open_files21:20:342
2742099711,55cyclictest10159-21memory19:50:312
2740599712,25cyclictest0-21swapper/022:21:230
27405996921,6cyclictest25986-21aten_rbpower_cu20:35:250
2740599691,25cyclictest5729-21grep00:05:380
2740599681,25cyclictest0-21swapper/000:00:100
2741299671,6cyclictest9984-21aten_rbpower_po22:50:221
27405996738,17cyclictest15914-21grep21:35:350
27405996722,5cyclictest0-21swapper/020:40:420
27405996719,5cyclictest1124-21snmpd20:55:310
2740599662,5cyclictest0-21swapper/020:30:260
2740599662,47cyclictest0-21swapper/021:55:370
2740599662,47cyclictest0-21swapper/021:55:370
27405996620,5cyclictest0-21swapper/022:05:300
27405996617,26cyclictest19249-21fschecks_count23:15:270
108572660,3sleep210859-21rt-features00:20:372
2742099651,52cyclictest11992-21if_wlan021:25:302
27405996535,6cyclictest7850irq/56-dwc_otg_23:00:280
27405996522,3cyclictest0-21swapper/021:25:320
2740599651,22cyclictest0-21swapper/021:41:420
27405996416,35cyclictest1125-21runrttasks23:40:360
2740599641,24cyclictest0-21swapper/019:29:280
2740599641,21cyclictest9898-21seq22:50:190
2740599641,20cyclictest0-21swapper/000:00:310
2742399633,14cyclictest7850irq/56-dwc_otg_19:40:233
2742099631,49cyclictest9900-21apt-get22:50:192
2740599632,26cyclictest0-21swapper/020:53:520
2740599632,19cyclictest22897-21fschecks_count23:25:270
2740599631,19cyclictest0-21swapper/022:18:260
2740599631,18cyclictest0-21swapper/023:06:200
2740599631,18cyclictest0-21swapper/023:06:200
2742099621,49cyclictest25852-21cut22:05:092
2742099621,49cyclictest22155-21latency_hist20:25:092
2742099621,49cyclictest22155-21latency_hist20:25:092
27405996224,20cyclictest0-21swapper/022:44:270
27405996220,3cyclictest128082sleep019:58:270
27405996220,3cyclictest128082sleep019:58:270
27405996219,8cyclictest0-21swapper/023:52:230
27405996219,8cyclictest0-21swapper/023:52:230
2742099616,44cyclictest0-21swapper/219:45:352
2742099611,49cyclictest24651-21df_inode23:30:292
2740599613,49cyclictest0-21swapper/020:00:210
27405996131,13cyclictest18351-21grep23:10:390
27405996121,37cyclictest19044-21fschecks_time20:15:300
27405996112,8cyclictest0-21swapper/020:06:220
2742099601,48cyclictest19997-21seq20:16:552
2742099601,47cyclictest1981-21chrt20:56:292
27405996018,7cyclictest0-21swapper/023:46:380
2740599601,24cyclictest0-21swapper/019:24:130
2740599601,24cyclictest0-21swapper/019:24:130
2742099593,43cyclictest0-21swapper/220:30:272
2742099591,8cyclictest2729-21latency_hist00:00:102
2742099591,10cyclictest12417-21/usr/sbin/munin22:55:382
27405995918,36cyclictest101ktimersoftd/021:50:220
27405995915,10cyclictest0-21swapper/021:17:560
27405995912,5cyclictest6743-21fschecks_time00:10:280
2740599591,22cyclictest0-21swapper/021:49:370
2740599591,14cyclictest0-21swapper/022:25:240
2742099586,43cyclictest7750irq/56-dwc_otg00:30:252
2742099585,40cyclictest7850irq/56-dwc_otg_22:00:112
2742099585,40cyclictest7850irq/56-dwc_otg_22:00:112
2742099584,46cyclictest0-21swapper/223:12:032
2742099584,44cyclictest0-21swapper/200:38:412
2742099582,42cyclictest0-21swapper/220:30:092
2740599582,20cyclictest0-21swapper/019:10:270
27405995814,4cyclictest468-21rs:main0
2740599581,20cyclictest0-21swapper/022:34:210
2740599581,20cyclictest0-21swapper/022:34:210
2740599581,20cyclictest0-21swapper/000:25:230
2740599581,17cyclictest0-21swapper/021:09:410
27405995733,10cyclictest0-21swapper/023:38:510
2740599572,19cyclictest0-21swapper/020:11:370
2740599571,37cyclictest31123-21idleruntime-cro20:50:090
2740599571,23cyclictest0-21swapper/021:12:180
2740599571,23cyclictest0-21swapper/021:12:180
2742099566,40cyclictest7750irq/56-dwc_otg22:05:282
2742099561,41cyclictest0-21swapper/222:31:412
2742099561,41cyclictest0-21swapper/222:31:412
27405995620,27cyclictest0-21swapper/022:47:590
2740599561,15cyclictest0-21swapper/019:16:200
2742099557,39cyclictest7850irq/56-dwc_otg_23:47:002
2742099554,42cyclictest7850irq/56-dwc_otg_21:30:512
2742099553,5cyclictest26039-21grep20:35:252
2742099551,44cyclictest7929-21chrt21:15:172
2742099551,44cyclictest0-21swapper/220:51:142
2742099551,40cyclictest0-21swapper/221:36:042
2740599551,22cyclictest0-21swapper/021:31:080
27405995512,10cyclictest0-21swapper/019:35:040
2742099543,6cyclictest7950irq/56-dwc_otg_00:27:462
2742099543,43cyclictest0-21swapper/223:25:272
2742099541,8cyclictest0-21swapper/221:11:192
2742099541,8cyclictest0-21swapper/221:11:192
2742099541,42cyclictest305322chrt19:16:392
27405995432,9cyclictest5067-21timerwakeupswit19:35:350
2742099533,7cyclictest184322chrt21:44:032
2742099533,42cyclictest0-21swapper/200:16:512
2742099533,41cyclictest0-21swapper/223:44:142
2742099531,8cyclictest0-21swapper/219:26:492
2742099531,42cyclictest28009-21aten_rbpower_vo22:10:262
2742099531,3cyclictest0-21swapper/219:30:262
27420995311,38cyclictest6711-21if_eth022:40:282
2742099524,37cyclictest0-21swapper/200:06:302
2742099522,43cyclictest7850irq/56-dwc_otg_21:07:212
2742099521,42cyclictest0-21swapper/220:09:002
2742099521,40cyclictest0-21swapper/220:45:572
2742099521,38cyclictest0-21swapper/222:29:212
2742099511,9cyclictest4565-21basename22:35:242
2742099511,8cyclictest31318-21fschecks_count19:20:262
2742099511,8cyclictest31318-21fschecks_count19:20:262
2742099508,34cyclictest7850irq/56-dwc_otg_20:14:192
2742099502,5cyclictest0-21swapper/223:15:212
2742099502,2cyclictest0-21swapper/222:45:242
2742099501,5cyclictest21984-21unixbench_singl23:20:392
2742099501,4cyclictest0-21swapper/220:43:182
2742099501,37cyclictest0-21swapper/221:49:162
2742099498,26cyclictest0-21swapper/200:10:252
2742099491,9cyclictest0-21swapper/219:45:102
2742099491,7cyclictest3646-21grep21:00:402
2742099491,7cyclictest13605-21/usr/sbin/munin23:00:282
2742099491,6cyclictest32627-21grep22:20:382
125432490,27sleep27950irq/56-dwc_otg_19:55:392
125432490,27sleep27950irq/56-dwc_otg_19:55:392
2742099485,35cyclictest7850irq/56-dwc_otg_23:35:532
2742099482,37cyclictest0-21swapper/223:52:012
2742099482,37cyclictest0-21swapper/223:52:012
2742099482,24cyclictest0-21swapper/223:05:212
2742099482,24cyclictest0-21swapper/223:05:212
2742099481,6cyclictest14780-21http20:05:092
27412994824,7cyclictest23-21ksoftirqd/122:55:261
2742399477,25cyclictest0-21swapper/323:00:273
2742099477,4cyclictest28089-21grep19:10:312
2742099471,7cyclictest21722-21sed21:50:402
2740599472,27cyclictest0-21swapper/021:00:310
2742399467,25cyclictest0-21swapper/320:55:253
2742399467,24cyclictest0-21swapper/319:35:223
2742099466,25cyclictest0-21swapper/222:15:242
2742099462,4cyclictest0-21swapper/219:35:222
269792463,36sleep00-21swapper/019:09:040
269792463,36sleep00-21swapper/019:09:040
2742399457,23cyclictest0-21swapper/323:55:243
2742399457,23cyclictest0-21swapper/322:10:153
2742399457,23cyclictest0-21swapper/320:05:153
2742399456,25cyclictest0-21swapper/321:25:253
2742399456,24cyclictest0-21swapper/319:50:233
2742399447,23cyclictest0-21swapper/322:55:283
2742399447,23cyclictest0-21swapper/322:25:243
2742399447,23cyclictest0-21swapper/300:25:243
2742399447,22cyclictest0-21swapper/323:05:243
2742399447,22cyclictest0-21swapper/323:05:243
2742399447,22cyclictest0-21swapper/300:15:233
2741299448,4cyclictest0-21swapper/100:00:201
2742399437,23cyclictest0-21swapper/323:20:243
2742399437,22cyclictest0-21swapper/321:00:253
2742399437,22cyclictest0-21swapper/300:10:253
2742399433,10cyclictest0-21swapper/320:20:263
2742399433,10cyclictest0-21swapper/320:20:263
27412994325,6cyclictest7950irq/56-dwc_otg_20:05:091
2742399427,23cyclictest0-21swapper/320:00:243
2742399411,25cyclictest24462-21uniq20:30:273
27412994129,3cyclictest0-21swapper/119:10:111
2742399407,21cyclictest0-21swapper/321:10:253
2742399407,21cyclictest0-21swapper/321:10:253
27423994017,9cyclictest381ktimersoftd/319:55:393
27423994017,9cyclictest381ktimersoftd/319:55:393
177682400,2sleep30-21swapper/321:40:343
2742399398,18cyclictest37-21rcuc/319:24:263
2742399398,18cyclictest37-21rcuc/319:24:263
2742399397,5cyclictest7750irq/56-dwc_otg21:20:383
27423993913,11cyclictest7750irq/56-dwc_otg22:15:353
27412993917,4cyclictest24779-21fschecks_time23:30:311
2742399384,21cyclictest39-21ksoftirqd/300:35:283
2742399383,6cyclictest37-21rcuc/300:05:293
2742399382,7cyclictest0-21swapper/321:50:103
2742399382,22cyclictest6586-21cut22:40:273
27423993817,9cyclictest7750irq/56-dwc_otg20:40:343
2741299382,7cyclictest7950irq/56-dwc_otg_21:30:231
2742399377,20cyclictest0-21swapper/323:30:273
2742399377,18cyclictest0-21swapper/319:25:233
2741299375,7cyclictest13905-21fschecks_time23:00:311
2741299372,13cyclictest7750irq/56-dwc_otg00:30:101
27412993716,5cyclictest15158-21acpi21:35:211
27412993715,6cyclictest1032-21fschecks_time20:55:291
27412993714,10cyclictest1549-21unixbench_multi19:25:371
2742399369,6cyclictest7850irq/56-dwc_otg_21:30:383
2742399363,6cyclictest7850irq/56-dwc_otg_19:15:403
2742399361,26cyclictest8149-21ls19:45:293
2742399361,26cyclictest32476-21timerandwakeup22:20:363
2742399361,25cyclictest17534-21cut23:10:293
27412993613,5cyclictest7850irq/56-dwc_otg_19:35:091
2742399358,17cyclictest0-21swapper/319:33:383
2742399352,8cyclictest11166-21grep00:20:403
2742399351,22cyclictest22143-21latency_hist21:55:103
2742399351,10cyclictest19812-21sed20:15:403
2741299358,6cyclictest7950irq/56-dwc_otg_21:20:381
27412993515,4cyclictest24448-21fschecks_count22:00:281
2742399349,10cyclictest37-21rcuc/300:00:203
2741299347,6cyclictest7950irq/56-dwc_otg_21:45:411
2741299343,13cyclictest7950irq/56-dwc_otg_22:20:381
27412993413,10cyclictest26164-21df_inode20:35:271
2741299341,19cyclictest0-21swapper/122:30:101
2742399335,4cyclictest0-21swapper/319:10:273
2742399334,6cyclictest37-21rcuc/322:50:103
2742399334,15cyclictest381ktimersoftd/300:30:253
2742399332,6cyclictest0-21swapper/320:10:243
27423993311,9cyclictest7950irq/56-dwc_otg_21:20:103
2741299336,6cyclictest11160-21idleruntime-cro19:55:101
2741299336,10cyclictest7950irq/56-dwc_otg_20:20:401
2741299336,10cyclictest7950irq/56-dwc_otg_20:20:401
2741299334,5cyclictest7850irq/56-dwc_otg_20:45:301
2741299333,7cyclictest0-21swapper/119:15:211
2741299332,10cyclictest0-21swapper/123:40:101
2741299331,25cyclictest21715-21ls23:20:371
2741299331,13cyclictest7750irq/56-dwc_otg20:20:111
2742399325,13cyclictest39-21ksoftirqd/322:30:203
2742399325,13cyclictest39-21ksoftirqd/322:30:203
2742399325,10cyclictest37-21rcuc/322:05:023
2742399324,7cyclictest7850irq/56-dwc_otg_23:20:093
2742399324,14cyclictest0-21swapper/321:05:273
2742399321,7cyclictest27339-21grep23:35:383
27423993215,5cyclictest22628-21/usr/sbin/munin23:25:253
2742399321,14cyclictest11-21rcu_preempt22:05:223
2742399321,11cyclictest7950irq/56-dwc_otg_22:50:323
27423993210,9cyclictest7950irq/56-dwc_otg_20:35:413
2741299328,5cyclictest7950irq/56-dwc_otg_21:50:251
2741299327,8cyclictest7850irq/56-dwc_otg_23:25:311
2741299324,6cyclictest7850irq/56-dwc_otg_21:40:231
2741299321,18cyclictest9656-21cron22:50:091
2741299321,12cyclictest7850irq/56-dwc_otg_19:20:371
2741299321,12cyclictest7850irq/56-dwc_otg_19:20:371
261262325,16sleep326182-21/usr/sbin/munin19:05:313
261262325,16sleep326182-21/usr/sbin/munin19:05:313
2742399317,15cyclictest0-21swapper/321:55:233
2742399317,15cyclictest0-21swapper/321:55:233
2742399316,13cyclictest7850irq/56-dwc_otg_23:40:283
2742399315,5cyclictest7850irq/56-dwc_otg_21:40:093
2742399311,3cyclictest0-21swapper/320:50:213
2741299318,5cyclictest7950irq/56-dwc_otg_23:50:301
2741299318,5cyclictest7950irq/56-dwc_otg_23:50:301
2741299318,4cyclictest7950irq/56-dwc_otg_23:10:101
2741299318,4cyclictest7950irq/56-dwc_otg_23:10:101
2741299316,6cyclictest7750irq/56-dwc_otg20:10:401
2741299315,5cyclictest7850irq/56-dwc_otg_23:15:321
2741299311,6cyclictest7750irq/56-dwc_otg00:20:251
27412993110,7cyclictest8750-21latency00:15:301
2742399304,9cyclictest381ktimersoftd/320:27:393
2742399302,4cyclictest7750irq/56-dwc_otg23:55:113
2742399302,4cyclictest7750irq/56-dwc_otg23:55:113
2741299309,7cyclictest28380-21fschecks_time23:40:281
2741299309,5cyclictest22407-21basename20:25:241
2741299306,8cyclictest0-21swapper/122:30:321
2741299306,8cyclictest0-21swapper/122:30:321
2741299306,6cyclictest7950irq/56-dwc_otg_21:15:331
2741299306,4cyclictest7850irq/56-dwc_otg_21:05:311
2741299303,11cyclictest468-21rs:main1
2741299301,8cyclictest0-21swapper/100:30:291
27412993017,5cyclictest6298-21if_eth019:40:311
2741299301,6cyclictest7750irq/56-dwc_otg23:48:391
2741299301,5cyclictest7950irq/56-dwc_otg_21:25:401
27412993015,5cyclictest28147-21fschecks_count22:10:291
2741299301,4cyclictest7750irq/56-dwc_otg19:45:391
2741299301,23cyclictest7031-21missed_timers00:10:321
2742399297,15cyclictest0-21swapper/323:45:263
2741299299,7cyclictest304372pluginstate22:15:351
2741299299,5cyclictest4749-21fschecks_count22:35:271
2741299293,9cyclictest7950irq/56-dwc_otg_20:50:341
2741299292,5cyclictest6849-21ntp_states21:10:351
2741299292,5cyclictest6849-21ntp_states21:10:351
2741299291,13cyclictest0-21swapper/100:07:511
70042280,2sleep10-21swapper/122:40:331
2741299286,6cyclictest0-21swapper/123:55:231
2741299283,6cyclictest29332-21idleruntime-cro20:45:091
2741299283,10cyclictest7950irq/56-dwc_otg_21:55:301
2741299283,10cyclictest7950irq/56-dwc_otg_21:55:301
2741299282,3cyclictest0-21swapper/119:35:221
2741299281,15cyclictest0-21swapper/119:55:361
2741299281,15cyclictest0-21swapper/119:55:361
2690822810,12sleep20-21swapper/219:08:092
2690822810,12sleep20-21swapper/219:08:092
2741299272,11cyclictest7750irq/56-dwc_otg00:35:391
2742399264,4cyclictest39-21ksoftirqd/322:37:153
2742399261,17cyclictest0-21swapper/320:45:243
2741299261,12cyclictest1544-21ntpd21:00:401
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional