You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-08-22 - 10:34

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Sat Aug 22, 2026 00:43:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
30277991623,139cyclictest0-21swapper/021:50:410
30277991604,126cyclictest8761-21apt-config21:10:090
30277991592,127cyclictest5202-21latency_hist21:45:110
30277991592,127cyclictest5202-21latency_hist21:45:110
30277991512,142cyclictest0-21swapper/021:20:300
30277991512,142cyclictest0-21swapper/021:20:300
30277991392,104cyclictest11758-21/usr/sbin/munin23:10:340
30277991322,106cyclictest0-21swapper/022:00:270
30277991312,107cyclictest16164-21cat23:15:370
30277991303,106cyclictest0-21swapper/021:10:110
30277991262,103cyclictest5441-21ssh21:45:130
30277991244,99cyclictest0-21swapper/021:25:250
30277991243,97cyclictest21006-21grep00:00:390
30277991242,58cyclictest7950irq/56-dwc_otg_23:45:200
30277991223,84cyclictest0-21swapper/023:30:230
30277991203,94cyclictest31516-21rm21:36:300
30277991143,90cyclictest0-21swapper/023:55:240
30277991072,82cyclictest12771-21ssh23:50:410
30277991072,82cyclictest12771-21ssh23:50:410
30279991062,74cyclictest9611-21/usr/sbin/munin21:50:402
1352121050,8sleep33028099cyclictest00:30:383
30279991047,74cyclictest5761-21sh21:45:252
302789910238,31cyclictest18089-21netstat21:20:311
302789910238,31cyclictest18089-21netstat21:20:311
30279991013,80cyclictest0-21swapper/221:20:352
30279991013,80cyclictest0-21swapper/221:20:352
302789910032,25cyclictest10947-21ssh21:50:411
302789910030,23cyclictest8702-21apt-config21:10:081
3027999992,70cyclictest7850irq/56-dwc_otg_21:45:102
3027999992,70cyclictest7850irq/56-dwc_otg_21:45:102
3027999973,52cyclictest9846-21ntp_states21:10:312
30278999354,11cyclictest30382-21missed_timers21:35:311
3027799938,70cyclictest0-21swapper/019:25:220
3027999923,64cyclictest8924-21cat21:10:112
30277999210,68cyclictest14001-21/usr/sbin/munin21:15:330
3027999913,62cyclictest27401-21ssh23:30:232
70292870,3sleep13027899cyclictest21:46:011
3027999873,60cyclictest18293-21df_inode22:00:252
3027799872,6cyclictest5406-21fschecks_time19:30:260
3027999863,76cyclictest7302-21apt-get23:45:222
30279998633,48cyclictest12139-21netstat23:50:312
30279998633,48cyclictest12139-21netstat23:50:312
3027999862,77cyclictest15796-21/usr/sbin/munin23:55:252
3027999853,75cyclictest11776-21taskset23:10:342
3027999853,74cyclictest16146-21ssh23:15:372
30279998512,67cyclictest20989-21unixbench_multi00:00:392
3027799852,5cyclictest0-21swapper/023:05:080
3027899835,45cyclictest0-21swapper/121:10:321
3027799831,62cyclictest3139-21cat22:20:360
3027899823,43cyclictest0-21swapper/123:50:301
3027899823,43cyclictest0-21swapper/123:50:301
3027799821,6cyclictest23049-21sort22:45:260
3027799821,5cyclictest6988-21munin-run23:45:090
3027999813,71cyclictest30376-21ssh21:35:292
30278998128,26cyclictest15789-21ssh23:55:251
30278998126,29cyclictest5425-21sed21:45:111
3027799812,8cyclictest17778-21aten_rbpower_en20:05:230
3027799811,67cyclictest0-21swapper/019:20:210
3027899804,19cyclictest18318-21ssh22:00:271
30278998026,35cyclictest0-21swapper/100:00:401
3027799801,65cyclictest15670-21cron20:00:080
3027999792,70cyclictest21722-21ssh21:25:242
3027799792,66cyclictest0-21swapper/023:24:360
3027799792,5cyclictest0-21swapper/000:25:200
3027799781,8cyclictest0-21swapper/022:59:300
30277997812,53cyclictest13890-21idleruntime-cro19:55:080
3027799772,6cyclictest7678-21ps23:05:340
227752770,2sleep10-21swapper/122:05:301
227752770,2sleep10-21swapper/122:05:301
3027799761,67cyclictest0-21swapper/022:14:020
3027799752,3cyclictest0-21swapper/022:15:240
3027899744,16cyclictest0-21swapper/123:15:401
3027799741,6cyclictest0-21swapper/000:11:010
3027799741,63cyclictest0-21swapper/022:30:240
3027799741,5cyclictest2398-21rm23:39:200
30280997348,18cyclictest15808-21ssh23:55:263
30280997310,22cyclictest7850irq/56-dwc_otg_23:40:183
3027799738,51cyclictest30508-21aten_rbpower_cu20:40:230
3027799731,6cyclictest28426-21ssh22:51:030
3027799731,5cyclictest0-21swapper/000:39:360
30278997212,34cyclictest5234-21latency_hist21:45:091
3027799716,54cyclictest15511-21ssh21:56:170
3027799716,50cyclictest13832-21sh00:31:110
3027799711,62cyclictest9548-21ssh22:29:220
3027799711,53cyclictest27160-21rm00:09:140
3027799711,53cyclictest27160-21rm00:09:140
30277997110,52cyclictest994-1kworker/u9:100:16:120
30277997110,45cyclictest23335-21aten_rbpower_en23:25:210
3027899703,13cyclictest0-21swapper/123:10:341
3027799707,59cyclictest0-21swapper/020:00:270
3027799704,9cyclictest25656-21ssh21:30:240
3027799702,4cyclictest0-21swapper/022:40:090
3027799701,3cyclictest0-21swapper/020:45:350
3028099699,43cyclictest520-21rs:main3
3027799692,5cyclictest0-21swapper/021:04:200
30277996910,49cyclictest28846-21/usr/sbin/munin20:35:210
30280996817,28cyclictest10219-21timerwakeupswit21:10:373
30278996815,26cyclictest7750irq/56-dwc_otg21:25:241
3027799681,47cyclictest25623-21ssh22:09:370
3027799681,47cyclictest25623-21ssh22:09:370
3027799681,37cyclictest0-21swapper/020:25:200
30279996713,31cyclictest20294-1kworker/u9:021:30:362
3027799678,50cyclictest31612-21sshd19:12:040
3027799672,8cyclictest0-21swapper/022:42:390
3028099668,23cyclictest37-21rcuc/321:45:093
3028099668,23cyclictest37-21rcuc/321:45:093
3028099663,13cyclictest10657-21switchtime21:50:363
30279996625,37cyclictest17178-21latency_hist21:20:092
3027899664,35cyclictest0-21swapper/123:45:211
3028099654,41cyclictest0-21swapper/321:45:263
3027799651,51cyclictest0-21swapper/000:23:590
3027999646,46cyclictest7950irq/56-dwc_otg_20:05:342
3027899643,12cyclictest27388-21ssh23:30:211
3028099635,28cyclictest0-21swapper/321:35:313
3028099634,16cyclictest21734-21diskstats21:25:253
3028099631,28cyclictest7750irq/56-dwc_otg23:15:423
30278996324,25cyclictest31403-21/usr/sbin/munin20:40:351
3027799632,5cyclictest0-21swapper/020:32:340
3027799631,51cyclictest20961-21taskset20:13:300
3027799631,45cyclictest0-21swapper/019:35:190
30280996218,22cyclictest7950irq/56-dwc_otg_21:20:303
30280996218,22cyclictest7950irq/56-dwc_otg_21:20:303
30280996213,25cyclictest7950irq/56-dwc_otg_21:10:093
30279996214,39cyclictest12285-21sh22:31:502
30278996217,27cyclictest13739-21ls21:15:291
3027799623,41cyclictest0-21swapper/020:55:190
3027799623,41cyclictest0-21swapper/020:55:190
3027999614,4cyclictest0-21swapper/200:06:442
3027999614,4cyclictest0-21swapper/200:06:442
3028099606,32cyclictest18549-21if_wlan022:00:303
3027999604,37cyclictest7850irq/56-dwc_otg_22:37:282
3027999602,4cyclictest7750irq/56-dwc_otg22:45:332
30278996018,26cyclictest16043-21cpu20:00:231
3027799601,6cyclictest31517-21systemd-journal20:16:110
3027799601,5cyclictest1998-21grep20:50:270
3027799592,5cyclictest0-21swapper/019:45:220
3027799591,54cyclictest23223-21wget20:20:230
3027799591,54cyclictest23223-21wget20:20:230
30277995914,33cyclictest11852sleep019:19:440
3028099585,33cyclictest0-21swapper/323:50:393
3028099585,33cyclictest0-21swapper/323:50:393
30280995812,22cyclictest7850irq/56-dwc_otg_00:00:413
30280995810,26cyclictest7750irq/56-dwc_otg23:45:193
3027999582,3cyclictest0-21swapper/222:55:172
30278995822,21cyclictest0-21swapper/123:04:221
3027799581,7cyclictest0-21swapper/019:41:090
3027799581,7cyclictest0-21swapper/019:41:090
3027999573,43cyclictest21321-21sh23:22:122
3027999572,4cyclictest0-21swapper/222:51:302
30278995717,22cyclictest28442-21ssh22:51:051
30278995714,27cyclictest8344-21taskset19:38:511
3027999563,3cyclictest7850irq/56-dwc_otg_00:25:282
3027999561,5cyclictest31517-21systemd-journal22:15:082
3027999561,3cyclictest7850irq/56-dwc_otg_21:58:242
30278995625,12cyclictest22935-21fschecks_count22:45:251
30278995615,25cyclictest21176-21ssh22:43:021
3027999556,39cyclictest7850irq/56-dwc_otg_00:10:202
30279995515,36cyclictest6179-21/usr/sbin/munin22:25:352
3027999551,50cyclictest3289-21grep22:20:372
3027999551,4cyclictest30675-21fschecks_count22:15:262
3028099543,11cyclictest27380-21apt-get23:30:223
3027999542,5cyclictest0-21swapper/222:43:022
3027999541,6cyclictest0-21swapper/223:05:382
30278995420,12cyclictest15244-21rm21:55:471
30278995414,24cyclictest25597-21ssh23:27:301
30278995412,24cyclictest3562-21ssh22:20:561
3028099535,20cyclictest24634-1kworker/u9:200:10:223
3027999539,41cyclictest16959-21sed20:00:362
3027999536,4cyclictest0-21swapper/222:09:542
3027999536,4cyclictest0-21swapper/222:09:542
3027999532,6cyclictest0-21swapper/220:00:092
30278995321,16cyclictest30276-21cyclictest22:15:211
30278995314,23cyclictest9455-21ssh23:08:041
3028099521,9cyclictest0-21swapper/320:15:223
3027999529,40cyclictest31052-21missed_timers20:40:302
3027999522,3cyclictest0-21swapper/200:20:362
3027999521,7cyclictest0-21swapper/219:35:192
3027999521,6cyclictest5689-21grep19:30:292
30278995218,22cyclictest1624-21ssh00:16:451
30278995217,17cyclictest28688-21taskset20:35:191
3027999514,4cyclictest7950irq/56-dwc_otg_23:41:202
3027999511,5cyclictest23252-21apt-get23:25:192
30278995119,17cyclictest2193-21ssh22:59:521
30278995118,14cyclictest24978-21/usr/sbin/munin00:05:361
30278995118,14cyclictest24978-21/usr/sbin/munin00:05:361
30278995114,21cyclictest18817-21kworker/u8:400:37:111
3027999507,26cyclictest0-21swapper/219:45:212
3027999506,4cyclictest7750irq/56-dwc_otg00:36:492
3027999503,3cyclictest7950irq/56-dwc_otg_00:15:272
3027999502,3cyclictest0-21swapper/200:32:562
3027999501,8cyclictest7950irq/56-dwc_otg_23:36:182
30278995017,13cyclictest18198-21ssh22:40:091
30278995013,23cyclictest10723-21taskset00:28:141
3027999497,26cyclictest0-21swapper/219:10:212
3027999495,3cyclictest7950irq/56-dwc_otg_23:01:282
3027999492,4cyclictest0-21swapper/219:54:492
3027999491,8cyclictest6221-21wc21:00:342
3027999491,7cyclictest27210-21fschecks_time20:30:272
3027999491,7cyclictest1895-21fschecks_count20:50:262
3027999491,6cyclictest32354-21cut20:45:212
30278994926,7cyclictest16205-21latency_hist00:35:091
30278994911,21cyclictest28613-21ls00:10:291
3027999482,7cyclictest5024-21sh21:00:092
3027999482,7cyclictest5024-21sh21:00:092
3027999482,7cyclictest23759-21cat20:20:302
3027999482,7cyclictest23759-21cat20:20:302
3027999481,6cyclictest7850irq/56-dwc_otg_19:25:092
3027999481,3cyclictest7950irq/56-dwc_otg_20:14:542
30278994821,13cyclictest21469-21ssh23:22:281
30278994814,21cyclictest2611-21ssh23:39:431
30278994812,24cyclictest27625-21ssh21:31:551
30278994810,22cyclictest0-21swapper/122:29:021
3027999472,7cyclictest24998-21aten_rbpower_cu20:25:212
3027999472,5cyclictest0-21swapper/219:25:222
3027999472,41cyclictest9192-21cat19:40:272
3027999472,41cyclictest9192-21cat19:40:272
3027999471,6cyclictest30429-21cat20:40:092
30278994711,20cyclictest3706-21awk19:25:241
30278994711,15cyclictest20985-21taskset20:13:501
3027999462,5cyclictest0-21swapper/220:15:272
3027999461,6cyclictest712-21unixbench-2d19:15:372
30278994620,11cyclictest3173-21apt-get23:40:181
30278994615,23cyclictest6333-21ssh00:22:461
30278994612,14cyclictest17737-21taskset20:05:211
30278994610,24cyclictest0-21swapper/122:30:191
300042463,36sleep00-21swapper/019:10:070
15572460,2sleep10-21swapper/120:50:091
102712460,5sleep110272-21apt19:45:091
102712460,5sleep110272-21apt19:45:091
325912450,2sleep30-21swapper/320:45:243
3028099457,26cyclictest0-21swapper/321:30:263
3027899455,28cyclictest26267-21runrttasks20:27:021
30278994514,19cyclictest28200-21taskset20:32:181
30278994511,23cyclictest14062-21sort19:55:091
30278994510,18cyclictest22801-21taskset20:18:471
3028099448,22cyclictest0-21swapper/319:55:213
3028099437,22cyclictest0-21swapper/321:00:203
3028099436,22cyclictest31523-21ssh23:35:223
3027899436,25cyclictest26273-21ssh22:10:221
30278994313,17cyclictest7850irq/56-dwc_otg_20:54:041
30278994313,12cyclictest6508-21taskset19:33:241
30278994214,11cyclictest7750irq/56-dwc_otg20:00:091
3028099417,24cyclictest0-21swapper/322:50:223
30280994128,8cyclictest0-21swapper/322:20:223
30280994112,20cyclictest19178-21ssh23:20:223
3027899419,21cyclictest1359-21latency_hist19:20:091
3027899417,21cyclictest0-21swapper/120:55:211
3027899417,21cyclictest0-21swapper/120:55:211
3028099407,24cyclictest0-21swapper/319:50:213
3028099403,6cyclictest7950irq/56-dwc_otg_23:25:183
3027899408,22cyclictest3196-21cat19:25:091
30278994030,7cyclictest30276-21cyclictest19:10:091
30278994012,15cyclictest24417-21seq20:21:261
30278994012,15cyclictest24417-21seq20:21:261
6982391,2sleep30-21swapper/319:15:373
3028099394,12cyclictest0-21swapper/319:20:183
90202380,2sleep30-21swapper/323:07:163
3028099381,17cyclictest29796-21grep20:35:343
3028099377,21cyclictest0-21swapper/322:45:213
3028099377,17cyclictest7950irq/56-dwc_otg_00:05:373
3028099377,17cyclictest7950irq/56-dwc_otg_00:05:373
30280993710,6cyclictest7950irq/56-dwc_otg_22:10:263
3027899376,24cyclictest6765-21runrttasks21:04:311
3028099368,17cyclictest13835-21ssh21:55:193
3028099367,6cyclictest7750irq/56-dwc_otg22:05:273
3028099367,6cyclictest7750irq/56-dwc_otg22:05:273
3028099366,16cyclictest7950irq/56-dwc_otg_00:15:213
3028099362,6cyclictest7950irq/56-dwc_otg_20:20:203
3028099362,6cyclictest7950irq/56-dwc_otg_20:20:203
3028099361,25cyclictest3789-21ls19:25:263
30278993611,11cyclictest11995-21taskset19:49:191
3028099351,5cyclictest8801-21ssh00:25:323
30280993512,4cyclictest0-21swapper/322:35:093
3028099344,19cyclictest60-21kswapd000:20:193
3028099344,16cyclictest7950irq/56-dwc_otg_22:40:093
30280993425,5cyclictest381ktimersoftd/322:25:193
3028099342,18cyclictest31924-21latency_hist19:15:083
3028099341,21cyclictest27161-21cut20:30:263
184442340,5sleep33028099cyclictest22:40:103
3028099335,6cyclictest7750irq/56-dwc_otg20:00:223
3028099335,14cyclictest7950irq/56-dwc_otg_19:40:253
3028099335,14cyclictest7950irq/56-dwc_otg_19:40:253
3028099334,13cyclictest7850irq/56-dwc_otg_20:40:353
3028099331,14cyclictest7750irq/56-dwc_otg22:15:233
3028099327,5cyclictest7950irq/56-dwc_otg_20:55:343
3028099327,5cyclictest7950irq/56-dwc_otg_20:55:343
3028099321,22cyclictest18019-21uniq20:05:273
3028099321,20cyclictest6818-21kworker/u8:123:00:243
3028099317,15cyclictest0-21swapper/319:30:223
3028099316,15cyclictest0-21swapper/320:50:233
3028099315,7cyclictest12140-21latency_hist19:50:083
3028099314,13cyclictest7950irq/56-dwc_otg_00:35:253
3028099312,12cyclictest17011-21apt-config21:20:093
3028099311,27cyclictest7216-21cut19:35:243
297582313,20sleep10-21swapper/119:07:001
3028099308,11cyclictest0-21swapper/322:58:483
3028099307,5cyclictest7950irq/56-dwc_otg_20:10:333
3028099295,11cyclictest7950irq/56-dwc_otg_20:25:363
298372293,17sleep30-21swapper/319:08:033
288052296,15sleep228858-21/usr/sbin/munin19:05:282
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional