You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-28 - 16:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Sat Mar 28, 2026 12:43:39)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3166827260,6sleep131670-21cat07:10:121
1898521030,5sleep33177299cyclictest10:35:283
3176899843,44cyclictest0-21swapper/007:10:130
284692820,4sleep3381ktimersoftd/310:08:593
194272800,2sleep10-21swapper/111:14:071
191162800,5sleep23177199cyclictest09:20:152
191162800,5sleep23177199cyclictest09:20:152
3176899773,45cyclictest0-21swapper/009:40:230
3176899772,51cyclictest0-21swapper/012:05:230
3176899745,51cyclictest0-21swapper/007:55:250
31322740,2sleep30-21swapper/309:39:283
3177199703,50cyclictest7950irq/56-dwc_otg_10:10:302
3177199701,43cyclictest2125-21ntp_states10:15:342
3177199691,53cyclictest18554-21rm09:56:352
3177099698,18cyclictest7750irq/56-dwc_otg08:15:241
3177199681,43cyclictest24582-21ssh10:41:152
31768996841,5cyclictest7850irq/56-dwc_otg_10:10:250
3176899682,19cyclictest0-21swapper/011:45:240
31768996721,42cyclictest9-21ksoftirqd/012:15:230
3177199661,47cyclictest19425-21ntp_states10:35:342
31768996622,31cyclictest9-21ksoftirqd/010:55:220
31768996622,31cyclictest9-21ksoftirqd/010:55:220
3177199651,50cyclictest0-21swapper/209:39:322
3177199651,38cyclictest766-21apt-get11:30:242
31768996516,10cyclictest0-21swapper/008:20:240
3177199641,46cyclictest9381-21ssh11:01:292
3177199641,45cyclictest12665-21/usr/sbin/munin09:50:262
3177199641,45cyclictest12665-21/usr/sbin/munin09:50:262
3176899648,29cyclictest908-21runrttasks08:10:270
3177199633,54cyclictest0-21swapper/207:10:132
3177199631,48cyclictest12713-21ssh12:21:092
3177199631,48cyclictest12713-21ssh12:21:092
3177199631,42cyclictest7340-21ssh10:21:202
3177199631,42cyclictest7340-21ssh10:21:202
3177199631,42cyclictest31335-21rm11:27:312
3177199631,42cyclictest10747-21ssh09:47:512
3177199631,39cyclictest26366-21ssh09:28:232
3176899633,34cyclictest0-21swapper/012:10:230
3177199621,43cyclictest26350-21sh12:38:092
3177199621,42cyclictest24534-21sh10:04:332
3177099626,3cyclictest0-21swapper/110:55:221
3177099626,3cyclictest0-21swapper/110:55:221
3176899626,22cyclictest0-21swapper/010:40:250
3176899622,36cyclictest28433-21grep10:45:390
3176899622,34cyclictest20186-21latency_hist11:15:110
3177199611,46cyclictest16827-21ssh10:32:472
3177199611,46cyclictest16827-21ssh10:32:472
3177199611,42cyclictest6145-21ssh09:42:172
3177199611,42cyclictest17835-21ssh11:50:202
3177199611,41cyclictest11319-21ssh11:41:492
3177199611,40cyclictest5322-21rm12:13:092
3177199611,38cyclictest7950irq/56-dwc_otg_08:35:352
3177199611,37cyclictest7850irq/56-dwc_otg_10:29:172
3176899614,49cyclictest9-21ksoftirqd/008:30:240
3176899614,49cyclictest9-21ksoftirqd/008:30:240
3177199601,50cyclictest21904-21ssh11:16:012
3177199601,49cyclictest512-21ssh12:07:142
3177199601,46cyclictest0-21swapper/210:49:482
3177199601,42cyclictest11675-21chrt07:44:272
3177199591,42cyclictest29140-21ntp_states08:30:352
3177199591,42cyclictest29140-21ntp_states08:30:352
3177199591,41cyclictest15753-21cron11:10:102
3177199591,39cyclictest15986-21rm11:47:302
31768995940,4cyclictest0-21swapper/011:00:240
3177199581,43cyclictest0-21swapper/208:09:522
3177199581,39cyclictest9711-21ssh12:18:212
3177099582,10cyclictest7850irq/56-dwc_otg_09:30:221
3177199571,42cyclictest21747-21ssh12:32:202
3177199571,40cyclictest1540-21ssh10:52:432
3177199571,38cyclictest17462-21ssh12:27:042
3177199571,38cyclictest13049-21rm09:12:322
3177099576,8cyclictest7750irq/56-dwc_otg11:30:231
3176899571,53cyclictest468-21cut11:30:110
31768995715,26cyclictest24728-21aten_rbpower_en11:20:250
3176899571,39cyclictest13256-21cron11:45:110
3177199561,47cyclictest8517-21ssh11:39:222
3177199561,42cyclictest18817-21ssh11:13:042
3177199561,36cyclictest29577-21ssh09:31:312
3177199561,35cyclictest0-21swapper/210:58:532
3177199561,35cyclictest0-21swapper/210:58:532
3176899564,49cyclictest17877-21kworker/0:210:35:240
31768995624,16cyclictest0-21swapper/010:20:340
31768995624,16cyclictest0-21swapper/010:20:340
31768995616,19cyclictest21881-21/usr/sbin/munin10:00:360
3176899561,30cyclictest14213-21ssh10:30:170
3176899561,30cyclictest14213-21ssh10:30:170
3177199551,44cyclictest23918-21ssh11:56:232
3177199551,39cyclictest14819-21ssh09:15:132
3177199551,39cyclictest0-21swapper/207:29:182
3177199551,37cyclictest29075-21ssh10:10:022
3177199551,36cyclictest18745-21chrt08:02:342
3177199551,36cyclictest18745-21chrt08:02:342
3176899553,25cyclictest0-21swapper/007:25:230
31768995528,6cyclictest3418-21cut08:50:250
3177299542,12cyclictest12407-21ssh09:50:203
3177299542,12cyclictest12407-21ssh09:50:203
3177299541,4cyclictest0-21swapper/308:35:243
3177199541,50cyclictest23655-21/usr/sbin/munin08:15:352
3177199541,41cyclictest28353-21ssh12:01:402
3177199541,36cyclictest691-21ls08:40:382
3177199541,36cyclictest691-21ls08:40:382
3177199541,36cyclictest21481-21sort08:10:312
3176899549,23cyclictest8199-21aten_rbpower_cu09:45:250
31768995440,7cyclictest0-21swapper/008:55:320
31768995426,5cyclictest23321-21sh12:35:100
31768995426,5cyclictest1283-21munin-plugin-st08:45:110
31768995426,5cyclictest1283-21munin-plugin-st08:45:110
3177199531,47cyclictest4031-21ls07:20:392
3177199531,47cyclictest4031-21ls07:20:392
3177199531,36cyclictest24836-21apt-get08:20:232
3177199531,36cyclictest15587-21/usr/sbin/munin07:55:222
3177199531,35cyclictest26844-21ssh11:22:082
3176899536,25cyclictest0-21swapper/007:40:240
31768995326,3cyclictest0-21swapper/007:45:190
3176899531,27cyclictest9000-21modprobe11:40:100
165472530,1sleep10-21swapper/111:48:291
3177199521,47cyclictest2224-21ntp_states08:45:342
3177199521,37cyclictest0-21swapper/207:31:482
3177199521,36cyclictest0-21swapper/208:51:272
3176899529,17cyclictest0-21swapper/010:25:320
31768995227,4cyclictest9-21ksoftirqd/010:15:240
31768995226,4cyclictest7950irq/56-dwc_otg_12:20:250
31768995226,4cyclictest7950irq/56-dwc_otg_12:20:250
31768995226,3cyclictest0-21swapper/009:20:300
31768995226,3cyclictest0-21swapper/009:20:300
31768995225,4cyclictest32369-21fschecks_time09:35:270
3176899522,31cyclictest27780-21sh09:30:240
3176899521,29cyclictest9499-21sed07:35:380
31768995212,18cyclictest7950irq/56-dwc_otg_08:15:240
3177199512,33cyclictest7950irq/56-dwc_otg_08:25:322
3177199512,33cyclictest7950irq/56-dwc_otg_08:25:322
3177199511,46cyclictest13401-21sshd07:48:272
3177199511,36cyclictest9896-21runrttasks07:39:402
3176899514,25cyclictest0-21swapper/007:20:230
3176899514,25cyclictest0-21swapper/007:20:230
31768995133,4cyclictest7850irq/56-dwc_otg_09:55:290
31768995126,5cyclictest25537-21fschecks_count10:05:270
3176899511,31cyclictest0-21swapper/007:50:230
3176899511,27cyclictest20861-21cron08:10:100
3177199507,40cyclictest31767-21cyclictest09:00:262
3177199501,37cyclictest0-21swapper/207:54:422
31768995017,22cyclictest0-21swapper/008:48:190
3176899501,24cyclictest27760-21ssh12:00:380
3177199491,45cyclictest9161-21/usr/sbin/munin09:05:302
3177199491,38cyclictest0-21swapper/207:16:542
31768994928,2cyclictest0-21swapper/011:50:360
3176899492,32cyclictest0-21swapper/012:25:250
31768994921,5cyclictest7227-21fschecks_time09:00:290
31768994920,5cyclictest91750snmpd11:31:220
3176899484,39cyclictest11-21rcu_preempt09:25:230
3176899481,25cyclictest26872-21fschecks_count08:25:250
3176899481,25cyclictest26872-21fschecks_count08:25:250
312032482,38sleep00-21swapper/007:06:280
3177199471,26cyclictest0-21swapper/208:59:032
3177099478,25cyclictest0-21swapper/110:35:241
3177099477,24cyclictest0-21swapper/107:25:251
3176899479,21cyclictest3049-21latency_hist10:55:110
31768994724,6cyclictest7850irq/56-dwc_otg_09:50:310
31768994724,6cyclictest7850irq/56-dwc_otg_09:50:310
3176899472,24cyclictest23783-21aten_rbpower_vo12:35:260
31768994714,17cyclictest22114-21sh11:55:160
256232470,8sleep3381ktimersoftd/310:43:053
3177099467,25cyclictest0-21swapper/107:15:241
3177099467,24cyclictest0-21swapper/108:55:251
3177099461,27cyclictest7750irq/56-dwc_otg09:35:211
3176899466,25cyclictest0-21swapper/007:15:240
3176899465,22cyclictest0-21swapper/011:05:380
3176899462,23cyclictest15031-21basename09:15:260
3176899461,5cyclictest31842-21http08:40:110
3176899461,24cyclictest10366-21ssh09:10:110
3177299453,6cyclictest0-21swapper/312:15:253
3177099457,23cyclictest0-21swapper/108:20:161
3177099452,8cyclictest7750irq/56-dwc_otg09:40:221
31768994521,6cyclictest11966-21runrttasks09:10:400
3177299447,23cyclictest0-21swapper/310:45:243
3177099447,4cyclictest0-21swapper/109:10:241
3177099444,2cyclictest0-21swapper/107:10:231
3176899447,18cyclictest0-21swapper/011:15:310
31768994419,4cyclictest0-21swapper/008:00:340
31768994419,4cyclictest0-21swapper/008:00:340
3176899441,24cyclictest0-21swapper/007:30:350
40472431,38sleep34087-21ntp_states10:55:343
40472431,38sleep34087-21ntp_states10:55:343
3177299437,22cyclictest0-21swapper/307:50:253
3177099437,22cyclictest0-21swapper/107:40:241
3177099432,8cyclictest9348-21ssh11:40:231
108272430,5sleep13177099cyclictest10:25:361
3177099429,3cyclictest7750irq/56-dwc_otg09:45:231
3177099422,5cyclictest7750irq/56-dwc_otg10:00:231
3177099422,5cyclictest0-21swapper/110:30:231
3177099422,5cyclictest0-21swapper/110:30:231
31772994122,7cyclictest20629-21idleruntime-cro10:00:103
31770994120,11cyclictest23-21ksoftirqd/110:20:251
31770994120,11cyclictest23-21ksoftirqd/110:20:251
3177299404,3cyclictest7950irq/56-dwc_otg_07:55:243
3177099407,24cyclictest0-21swapper/107:55:251
3177099406,22cyclictest0-21swapper/108:05:161
3177299397,25cyclictest0-21swapper/307:45:223
3177099392,8cyclictest16696-21ssh09:55:221
279262390,5sleep33177299cyclictest12:00:553
27142391,4sleep30-21swapper/310:15:453
3177299386,6cyclictest0-21swapper/309:20:233
3177299386,6cyclictest0-21swapper/309:20:233
31772993817,5cyclictest30449-21sh11:25:593
31772993816,5cyclictest7950irq/56-dwc_otg_12:35:253
31772993816,5cyclictest16595-21if_wlan011:10:303
3177099384,20cyclictest0-21swapper/107:20:231
3177099384,20cyclictest0-21swapper/107:20:231
31770993818,6cyclictest5205-21aten_rbpower_vo11:35:251
31770993817,6cyclictest0-21swapper/109:20:221
31770993817,6cyclictest0-21swapper/109:20:221
31770993814,6cyclictest21-21rcuc/111:25:111
3177099381,24cyclictest30192-21ls08:35:111
3177099381,24cyclictest30192-21ls08:35:111
3177299372,19cyclictest0-21swapper/309:05:253
31772993710,5cyclictest7850irq/56-dwc_otg_09:10:213
3177099372,5cyclictest908-21runrttasks10:05:221
3177099371,27cyclictest14789-21ls07:50:391
3177299364,21cyclictest0-21swapper/311:00:273
31772993623,10cyclictest0-21swapper/309:40:293
31772993616,7cyclictest15388-21sh10:30:383
31772993616,7cyclictest15388-21sh10:30:383
3177299361,28cyclictest25081-21cut08:20:283
3177299361,24cyclictest29909-21ls10:10:293
3177299361,24cyclictest26806-21ls08:25:233
3177299361,24cyclictest26806-21ls08:25:233
31772993610,5cyclictest7750irq/56-dwc_otg11:40:103
3177099366,17cyclictest7750irq/56-dwc_otg12:15:301
31770993614,5cyclictest23-21ksoftirqd/108:35:391
31770993613,4cyclictest0-21swapper/108:25:191
31770993613,4cyclictest0-21swapper/108:25:191
78352350,7sleep121-21rcuc/109:00:371
3177299358,4cyclictest3101-21sshd07:20:263
3177299358,4cyclictest3101-21sshd07:20:263
3177299356,15cyclictest0-21swapper/309:25:213
3177299354,15cyclictest7950irq/56-dwc_otg_09:20:113
3177099359,5cyclictest3353-21ntp_states12:10:341
3177099358,6cyclictest23-21ksoftirqd/111:50:381
3177099358,12cyclictest7950irq/56-dwc_otg_10:50:401
3177099352,20cyclictest0-21swapper/111:10:111
31770993521,5cyclictest3416-21wget08:50:251
3177099351,5cyclictest19591-21wget12:30:251
3177099351,4cyclictest22185-21ssh11:55:211
31770993511,8cyclictest7850irq/56-dwc_otg_10:15:271
3177099351,15cyclictest0-21swapper/112:29:471
3177299347,14cyclictest0-21swapper/310:25:243
3177299345,6cyclictest21646-21grep11:15:443
3177299345,19cyclictest7750irq/56-dwc_otg08:13:053
31772993412,5cyclictest12276-21if_err_eth011:05:303
3177299341,24cyclictest0-21swapper/307:10:243
3177099347,15cyclictest0-21swapper/109:05:241
3177099346,4cyclictest7950irq/56-dwc_otg_09:25:201
3177099345,12cyclictest7750irq/56-dwc_otg10:45:111
3177099344,14cyclictest7850irq/56-dwc_otg_09:50:371
3177099344,14cyclictest7850irq/56-dwc_otg_09:50:371
100432340,2sleep10-21swapper/107:40:111
3177299339,4cyclictest0-21swapper/310:21:373
3177299339,4cyclictest0-21swapper/310:21:373
3177299338,13cyclictest0-21swapper/310:01:523
3177299334,15cyclictest7950irq/56-dwc_otg_08:40:233
3177299334,15cyclictest7950irq/56-dwc_otg_08:40:233
3177299331,24cyclictest31870-21ls10:50:293
3177299331,20cyclictest15773-21ls12:25:293
3177299331,17cyclictest1935-21timerwakeupswit11:30:383
3177299331,10cyclictest4893-21grep07:25:263
3177099337,6cyclictest7950irq/56-dwc_otg_12:35:361
3177099337,15cyclictest0-21swapper/112:05:241
3177099335,13cyclictest0-21swapper/108:40:231
3177099335,13cyclictest0-21swapper/108:40:231
3177099334,6cyclictest7850irq/56-dwc_otg_08:00:401
3177099334,6cyclictest7850irq/56-dwc_otg_08:00:401
3177099332,7cyclictest0-21swapper/111:00:221
31770993313,4cyclictest23-21ksoftirqd/110:45:581
3177099331,20cyclictest16083-21grep09:15:401
3177299328,14cyclictest0-21swapper/307:30:273
3177299327,12cyclictest7950irq/56-dwc_otg_07:45:113
3177299326,15cyclictest0-21swapper/311:20:253
3177299325,14cyclictest7950irq/56-dwc_otg_12:05:353
3177299324,14cyclictest7950irq/56-dwc_otg_08:15:283
31772993224,4cyclictest7950irq/56-dwc_otg_11:40:233
3177299321,7cyclictest7750irq/56-dwc_otg08:00:273
3177299321,7cyclictest7750irq/56-dwc_otg08:00:273
31772993214,11cyclictest19582-21grep08:05:283
3177099326,12cyclictest23-21ksoftirqd/108:15:101
3177099322,19cyclictest0-21swapper/111:28:221
31770993215,5cyclictest23-21ksoftirqd/112:04:401
31770993212,4cyclictest7750irq/56-dwc_otg12:20:411
31770993212,4cyclictest7750irq/56-dwc_otg12:20:411
312692322,22sleep20-21swapper/207:07:152
3177299317,15cyclictest0-21swapper/307:35:243
3177299316,4cyclictest0-21swapper/309:45:543
31772993114,4cyclictest7950irq/56-dwc_otg_11:55:233
3177299311,28cyclictest11402-21uniq12:20:303
3177299311,28cyclictest11402-21uniq12:20:303
3177099318,10cyclictest0-21swapper/111:17:081
3177099317,5cyclictest30552-21ls10:10:371
3177099315,5cyclictest7755-21/usr/sbin/munin07:30:411
31770993120,4cyclictest7950irq/56-dwc_otg_08:48:121
3177299303,12cyclictest7950irq/56-dwc_otg_09:30:403
3177299302,7cyclictest7850irq/56-dwc_otg_08:30:303
3177299302,7cyclictest7850irq/56-dwc_otg_08:30:303
3177299301,8cyclictest13941-21fschecks_time11:45:293
3177299301,5cyclictest21216-21taskset11:54:063
31770993019,5cyclictest12094-21aten_rbpower_vo07:45:231
311742303,21sleep30-21swapper/307:06:063
3177299296,4cyclictest7950irq/56-dwc_otg_12:30:403
3177299291,4cyclictest255350ssh12:10:213
3177299291,14cyclictest7950irq/56-dwc_otg_08:50:393
3177299291,13cyclictest7950irq/56-dwc_otg_08:55:263
3177299283,6cyclictest7950irq/56-dwc_otg_07:15:293
3177299281,3cyclictest7950irq/56-dwc_otg_08:45:223
3177299281,20cyclictest0-21swapper/309:02:143
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional