You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-13 - 15:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Sat Jun 13, 2026 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
719221800,5sleep03259099cyclictest08:59:330
102582850,5sleep13259199cyclictest07:35:291
34482790,2sleep10-21swapper/110:16:071
32593997521,31cyclictest9821-21sh12:21:283
304762750,3sleep230477-21modprobe10:50:022
3259399717,49cyclictest25713-21seq12:01:333
80732700,2sleep30-21swapper/309:44:333
3259399696,52cyclictest1399-21sed09:35:303
3259399695,50cyclictest17054-21latency_hist09:55:033
3259399694,47cyclictest11036-21apt-get07:40:163
3259399671,5cyclictest8050irq/56-dwc_otg_12:30:023
3259199677,10cyclictest8050irq/56-dwc_otg_12:10:211
3259199671,4cyclictest7950irq/56-dwc_otg_09:05:021
32590996720,28cyclictest9650irq/80-mmc012:30:030
320582670,2sleep10-21swapper/110:11:521
117642670,2sleep20-21swapper/209:48:342
102372670,2sleep10-21swapper/109:05:301
3259399661,48cyclictest5403-21ssh12:15:563
3259399651,55cyclictest21989-21ssh09:21:583
3259399651,50cyclictest20925-21taskset10:37:413
32590996512,32cyclictest27042-21modprobe08:25:010
3259399636,42cyclictest8050irq/56-dwc_otg_09:18:513
3259399632,49cyclictest8841-21taskset10:23:143
3259399631,54cyclictest32547-21/usr/sbin/munin12:10:193
3259399631,4cyclictest7850irq/56-dwc_otg08:10:123
3259399631,49cyclictest9507-21df_inode09:05:193
3259099633,51cyclictest0-21swapper/007:45:160
3259399622,52cyclictest18180-21sh11:14:003
32593996210,31cyclictest0-21swapper/311:22:073
32590996242,4cyclictest8050irq/56-dwc_otg_10:00:150
3259099621,13cyclictest0-21swapper/012:00:150
3259399616,42cyclictest7950irq/56-dwc_otg_10:40:313
3259399615,44cyclictest31151-21ssh09:33:053
3259399615,44cyclictest31151-21ssh09:33:053
3259399612,46cyclictest12061-21/usr/sbin/munin11:05:313
3259399611,52cyclictest16832-21ls12:30:223
3259399611,46cyclictest29872-21ssh10:48:573
3259399611,46cyclictest0-21swapper/309:57:513
3259399611,43cyclictest11649-21/usr/sbin/munin11:45:173
3259199612,5cyclictest8050irq/56-dwc_otg_11:45:211
3259399607,34cyclictest7850irq/56-dwc_otg11:40:293
3259399606,36cyclictest27751-21ssh10:06:453
3259399604,42cyclictest8050irq/56-dwc_otg_09:10:323
3259399604,42cyclictest26571-21ssh09:27:323
3259399602,50cyclictest12274-21ssh09:49:273
3259399601,49cyclictest21604-21ssh11:56:343
3259399594,44cyclictest8050irq/56-dwc_otg_11:29:553
3259399594,44cyclictest16950-21ssh11:50:353
3259399592,48cyclictest22937-21ssh11:20:003
3259399592,48cyclictest22937-21ssh11:20:003
3259399591,5cyclictest22582-21grep10:00:303
3259399591,46cyclictest0-21swapper/308:40:153
3259199597,4cyclictest0-21swapper/109:24:191
3259199593,10cyclictest8050irq/56-dwc_otg_12:08:011
3259199592,6cyclictest0-21swapper/109:30:161
3259199592,6cyclictest0-21swapper/109:30:161
3259199592,4cyclictest7850irq/56-dwc_otg09:25:221
3259199591,6cyclictest6392-21ssh10:20:211
3259099594,8cyclictest32621-21runrttasks07:10:130
32590995941,5cyclictest19243-21cut11:15:150
32590995941,5cyclictest19243-21cut11:15:150
3259099592,32cyclictest0-21swapper/009:30:030
3259399585,42cyclictest1262-21sh10:14:463
3259399582,46cyclictest1692-21ssh10:15:073
3259399581,45cyclictest5100-21ssh11:36:313
3259199584,4cyclictest6644-21ssh11:00:151
3259199583,4cyclictest0-21swapper/112:03:491
3259199581,5cyclictest7850irq/56-dwc_otg11:25:191
3259099582,29cyclictest32252-21cron09:35:020
3259099582,29cyclictest32252-21cron09:35:020
3259099581,30cyclictest3919-21aten_rbpower_en08:50:130
3259099581,30cyclictest3919-21aten_rbpower_en08:50:130
3259399574,43cyclictest4682-21sh10:57:053
3259399571,43cyclictest9044-21apt09:05:013
3259399571,42cyclictest7145-21taskset07:29:493
3259399571,3cyclictest0-21swapper/311:00:133
3259199575,11cyclictest10849-21diskmemload12:27:341
3259199573,4cyclictest7850irq/56-dwc_otg10:55:161
3259199571,7cyclictest0-21swapper/108:40:151
3259199571,12cyclictest0-21swapper/107:40:171
32591995710,4cyclictest0-21swapper/110:37:341
3259099573,30cyclictest14674-21ssh11:10:020
32590995732,4cyclictest0-21swapper/011:40:150
3259099572,30cyclictest3728-21unixbench_multi10:55:300
32590995715,4cyclictest0-21swapper/008:15:150
42692560,3sleep20-21swapper/212:15:202
3259399561,47cyclictest0-21swapper/310:54:153
3259399561,45cyclictest19814-21pluginstate08:05:023
3259399561,45cyclictest19814-21pluginstate08:05:023
3259199567,4cyclictest0-21swapper/111:38:231
3259199562,5cyclictest0-21swapper/112:19:351
3259199562,3cyclictest0-21swapper/111:30:341
3259099562,34cyclictest7272-21ssh10:20:310
3259099561,43cyclictest0-21swapper/009:20:020
3259399554,41cyclictest21710-21ssh12:35:363
3259399554,40cyclictest8050irq/56-dwc_otg_11:33:413
3259399553,38cyclictest8050irq/56-dwc_otg_07:18:433
3259399551,44cyclictest0-21swapper/310:26:033
3259399551,44cyclictest0-21swapper/310:26:033
3259199553,4cyclictest8050irq/56-dwc_otg_09:57:541
3259199552,6cyclictest26413-21ssh11:23:491
3259199552,3cyclictest0-21swapper/111:10:311
3259199551,6cyclictest26172-21ntp_states08:20:251
3259199551,6cyclictest1493-21vmstat09:35:311
3259199551,4cyclictest4419-21/usr/sbin/munin09:40:251
32591995513,4cyclictest31206-21fschecks_time08:35:191
32590995531,5cyclictest5078-21unixbench_multi12:15:300
32590995529,15cyclictest0-21swapper/010:45:220
32590995523,14cyclictest22086-21fschecks_count08:10:180
3259099552,14cyclictest0-21swapper/009:45:020
3259099551,34cyclictest14533-21apt-get11:10:080
3259099551,31cyclictest22607-21aten_rbpower_cu10:40:160
3259099551,31cyclictest11753-21ssh12:25:020
32590995511,4cyclictest3512-21grep11:35:190
3259399544,38cyclictest287912sleep308:29:373
3259399542,48cyclictest768-21wc07:10:233
3259399541,42cyclictest28809-21ssh12:05:253
3259399541,34cyclictest0-21swapper/308:18:193
3259199541,5cyclictest23335-21ntp_states10:40:261
3259199541,10cyclictest7950irq/56-dwc_otg_08:10:181
3259099548,26cyclictest0-21swapper/008:45:160
3259099541,31cyclictest5645-21wget07:25:160
3259099541,31cyclictest30232-21grep08:30:290
32590995413,13cyclictest0-21swapper/009:35:150
61942530,1sleep20-21swapper/210:59:522
3259399531,41cyclictest14261-21chrt07:48:233
3259399531,39cyclictest0-21swapper/310:34:303
3259199534,10cyclictest9877-21taskset12:21:341
3259199533,5cyclictest0-21swapper/112:30:201
3259199532,10cyclictest0-21swapper/111:05:221
3259199531,9cyclictest8050irq/56-dwc_otg_11:20:021
3259199531,9cyclictest8050irq/56-dwc_otg_11:20:021
32591995318,32cyclictest23-21ksoftirqd/111:50:111
3259199531,5cyclictest22143-21rm12:36:241
3259199531,3cyclictest0-21swapper/110:08:411
3259099535,24cyclictest0-21swapper/008:35:140
3259099533,5cyclictest19703-21ssh11:55:120
3259099533,36cyclictest31558-21tr10:50:240
3259099531,4cyclictest11604-21cut11:45:160
3259399522,35cyclictest0-21swapper/308:35:023
3259399521,4cyclictest0-21swapper/308:57:013
3259399521,39cyclictest0-21swapper/307:51:263
3259299522,14cyclictest8050irq/56-dwc_otg_09:20:202
3259199522,3cyclictest8050irq/56-dwc_otg_10:02:031
3259199521,6cyclictest0-21swapper/109:10:131
3259199521,5cyclictest727-21runrttasks07:34:521
32591995213,5cyclictest28413-1kworker/u9:210:50:251
3259199521,15cyclictest0-21swapper/109:49:281
3259199521,10cyclictest0-21swapper/109:16:021
32590995226,7cyclictest31509-21/usr/sbin/munin11:30:270
3259099522,38cyclictest0-21swapper/012:10:250
3259099522,34cyclictest499-21fschecks_count08:40:210
3259099522,28cyclictest23628-21tune2fs11:20:180
3259099522,25cyclictest2190-21ssh10:15:200
32590995215,22cyclictest11467-21if_wlan007:40:210
32590995213,6cyclictest11047-21latency_hist09:10:020
3259099521,35cyclictest10302-21grep07:35:300
3259399511,40cyclictest53292sleep308:53:563
3259399511,40cyclictest53292sleep308:53:563
3259399511,40cyclictest0-21swapper/307:37:333
3259399511,3cyclictest0-21swapper/308:37:483
32593995111,37cyclictest5357-21cron07:25:013
3259199519,7cyclictest9650irq/80-mmc008:55:011
3259199519,7cyclictest9650irq/80-mmc008:55:011
3259199515,3cyclictest0-21swapper/110:45:301
3259199512,4cyclictest0-21swapper/107:30:021
3259199511,12cyclictest0-21swapper/108:29:381
3259099517,29cyclictest0-21swapper/011:50:110
3259099514,24cyclictest0-21swapper/009:10:140
32590995127,6cyclictest26053-21fschecks_time10:05:200
3259099511,30cyclictest16185-21cron07:55:020
3259099511,29cyclictest20648-21wc08:05:220
3259399505,35cyclictest8050irq/56-dwc_otg_07:56:533
3259199508,8cyclictest8050irq/56-dwc_otg_11:58:401
3259199502,9cyclictest8050irq/56-dwc_otg_11:42:311
3259199501,8cyclictest0-21swapper/108:15:571
3259199501,6cyclictest16882-21munin-run09:55:021
3259199501,5cyclictest4020-21diskstats07:20:181
3259099504,10cyclictest0-21swapper/009:55:140
3259099501,29cyclictest28027-21/usr/sbin/munin12:05:220
3259099501,26cyclictest31846-21ssh10:11:300
3259099501,25cyclictest19002-21wc08:00:280
3259099501,25cyclictest19002-21wc08:00:280
3259399491,41cyclictest0-21swapper/307:32:303
3259399491,37cyclictest0-21swapper/308:07:253
3259199493,8cyclictest7850irq/56-dwc_otg08:45:141
3259199491,8cyclictest11808-21seq10:26:041
3259199491,8cyclictest11808-21seq10:26:041
3259099492,28cyclictest27411-21/usr/sbin/munin11:25:220
3259099491,29cyclictest13023-21aten_rbpower_en09:50:150
3259399482,43cyclictest0-21swapper/308:20:163
3259199487,38cyclictest13038-21ls07:45:171
3259199483,5cyclictest8050irq/56-dwc_otg_10:31:431
3259199482,10cyclictest7950irq/56-dwc_otg_07:59:211
3259099486,9cyclictest0-21swapper/008:25:300
32590994814,16cyclictest18812-21sh12:32:520
3259399471,43cyclictest3659-21latency_hist08:50:023
3259299477,25cyclictest0-21swapper/207:30:162
3259299474,26cyclictest0-21swapper/211:50:112
3259199476,31cyclictest0-21swapper/108:00:171
3259199476,31cyclictest0-21swapper/108:00:171
3259099474,12cyclictest7950irq/56-dwc_otg_09:20:130
32590994717,20cyclictest0-21swapper/007:20:590
32590994614,22cyclictest0-21swapper/010:32:230
3259199451,4cyclictest0-21swapper/108:56:421
32590994527,16cyclictest17033-21missed_timers07:55:230
32590994518,18cyclictest0-21swapper/010:26:450
32590994518,18cyclictest0-21swapper/010:26:450
3259099451,25cyclictest19175-21ntp_kernel_err10:35:250
320132453,35sleep00-21swapper/007:06:090
3259299447,27cyclictest0-21swapper/208:05:152
3259299447,24cyclictest0-21swapper/211:15:072
3259299447,24cyclictest0-21swapper/211:15:072
3259099447,4cyclictest125-21systemd-journal12:38:480
3259099446,24cyclictest0-21swapper/007:30:160
3259099441,24cyclictest7882-21/usr/sbin/munin11:00:300
3259299438,6cyclictest29-21rcuc/209:10:182
3259299437,24cyclictest0-21swapper/209:00:142
3259299433,20cyclictest31-21ksoftirqd/211:05:202
32592994322,10cyclictest8050irq/56-dwc_otg_10:10:012
3259199432,4cyclictest8050irq/56-dwc_otg_07:51:091
3259199431,4cyclictest0-21swapper/107:10:191
3259099431,28cyclictest8967-21fschecks_count09:45:190
3259299427,21cyclictest18455-21ssh10:35:162
3259199421,5cyclictest0-21swapper/108:34:461
3259299419,15cyclictest7950irq/56-dwc_otg_11:40:152
3259299417,22cyclictest0-21swapper/212:05:072
32592994123,10cyclictest31597-21sh10:50:262
3259099414,16cyclictest0-21swapper/009:00:290
3259299408,7cyclictest0-21swapper/211:29:042
3259299394,22cyclictest0-21swapper/208:25:152
32592993918,5cyclictest2911050wget08:30:152
32592993910,21cyclictest0-21swapper/208:35:152
3259099392,20cyclictest0-21swapper/007:18:500
3259299386,23cyclictest0-21swapper/211:10:072
32592993816,7cyclictest29-21rcuc/207:10:252
3259299381,18cyclictest27740-21latency_hist12:05:022
21042380,2sleep10-21swapper/107:15:171
3259299377,22cyclictest0-21swapper/210:20:162
3259299372,12cyclictest0-21swapper/210:25:172
3259299372,12cyclictest0-21swapper/210:25:172
32592993710,11cyclictest29-21rcuc/209:30:132
32592993710,11cyclictest29-21rcuc/209:30:132
3259299368,5cyclictest29-21rcuc/209:56:242
3259299363,5cyclictest5712-21awk10:20:022
32592993617,5cyclictest4518-21wget09:40:162
3259299361,6cyclictest32103-21ls11:30:212
32592993616,7cyclictest31-21ksoftirqd/212:35:182
4372350,2sleep20-21swapper/208:40:192
3259299356,19cyclictest7850irq/56-dwc_otg08:20:302
3259299352,4cyclictest7950irq/56-dwc_otg_12:20:142
3259299352,19cyclictest7850irq/56-dwc_otg12:10:272
3259299351,24cyclictest14928-21cut07:50:182
3259299351,22cyclictest16956-21latency07:55:222
70212340,3sleep232589-21cyclictest11:40:022
3259299341,5cyclictest16575-21perl12:30:192
32592993414,5cyclictest13237-21fschecks_count09:50:182
3259299341,20cyclictest8050irq/56-dwc_otg_09:30:022
3259199341,13cyclictest0-21swapper/108:07:481
3259299336,4cyclictest7950irq/56-dwc_otg_07:25:302
3259299336,4cyclictest0-21swapper/211:45:212
3259299335,5cyclictest7850irq/56-dwc_otg07:45:302
3259299335,12cyclictest8050irq/56-dwc_otg_07:20:022
3259299333,23cyclictest0-21swapper/210:30:122
3259299332,17cyclictest8050irq/56-dwc_otg_09:05:212
3259299331,9cyclictest23771-21grep10:40:312
32592993314,7cyclictest15881-21fschecks_time09:15:192
32592993313,9cyclictest19948-21latency_hist08:05:032
32592993313,9cyclictest19948-21latency_hist08:05:032
3259299322,18cyclictest8050irq/56-dwc_otg_08:15:262
32592993216,10cyclictest14503-21ssh12:27:302
3259299321,15cyclictest8050irq/56-dwc_otg_11:55:292
3259299321,13cyclictest7950irq/56-dwc_otg_08:10:022
32592993210,11cyclictest0-21swapper/210:14:232
319732325,21sleep20-21swapper/207:05:382
3259299318,4cyclictest7152-21kernelversion11:00:222
3259299314,8cyclictest7850irq/56-dwc_otg09:35:192
3259299314,6cyclictest7850irq/56-dwc_otg08:50:162
3259299314,6cyclictest7850irq/56-dwc_otg08:50:162
32592993112,6cyclictest22331-21/usr/sbin/munin10:00:272
3259299303,4cyclictest7850irq/56-dwc_otg11:20:262
3259299301,7cyclictest2047-21/usr/sbin/munin08:45:302
3259299301,7cyclictest10780-21modprobe07:40:012
32592993013,6cyclictest301ktimersoftd/207:42:022
3259299292,12cyclictest4483-21ntp_kernel_pll_07:20:252
3259299282,7cyclictest0-21swapper/208:55:242
307552275,12sleep10-21swapper/107:05:091
319952263,15sleep30-21swapper/307:05:563
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional