You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 23:03

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Wed Mar 25, 2026 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2058921020,2sleep00-21swapper/009:51:430
558599837,41cyclictest0-21swapper/210:05:152
170942820,1sleep00-21swapper/007:40:280
5583997951,7cyclictest7950irq/56-dwc_otg_11:50:200
103242790,4sleep210325-21unixbench_multi07:20:382
256272760,4sleep225630-21latency_hist08:05:102
190872750,5sleep1558499cyclictest11:08:211
190872750,5sleep1558499cyclictest11:08:211
190072730,5sleep0558399cyclictest11:08:120
190072730,5sleep0558399cyclictest11:08:120
273942710,2sleep20-21swapper/211:56:582
273942710,2sleep20-21swapper/211:56:582
223832670,2sleep10-21swapper/109:54:581
558599663,10cyclictest0-21swapper/210:40:202
142662660,2sleep20-21swapper/211:40:432
5585996533,10cyclictest25603-21apt-get09:20:232
5585996533,10cyclictest25603-21apt-get09:20:232
558399645,46cyclictest9468-21fschecks_count07:20:270
558399643,51cyclictest9550irq/80-mmc008:05:210
5583996345,4cyclictest1584-21unixbench_multi11:25:360
558399633,43cyclictest13105-21fschecks_count07:30:260
558399632,12cyclictest4056-21latency_hist08:35:090
5583996213,15cyclictest0-21swapper/007:55:210
558399618,30cyclictest7850irq/56-dwc_otg_10:05:150
558399611,49cyclictest30194-21fschecks_time09:25:260
5585996020,12cyclictest7020-21ssh10:15:132
5585996020,12cyclictest7020-21ssh10:15:132
558399605,37cyclictest101ktimersoftd/008:45:090
558399604,28cyclictest0-21swapper/010:45:190
5583996036,10cyclictest101ktimersoftd/007:15:260
5583996036,10cyclictest101ktimersoftd/007:15:260
5585995920,18cyclictest17483-21ls07:40:352
5583995915,23cyclictest27657-21latency_hist10:40:100
5583995911,45cyclictest101ktimersoftd/011:00:090
5585995818,15cyclictest3230-21ssh11:28:242
5585995815,15cyclictest28938-21ssh12:38:262
558499572,14cyclictest0-21swapper/111:30:201
558499572,14cyclictest0-21swapper/111:30:201
558399574,34cyclictest0-21swapper/011:20:210
558399574,32cyclictest0-21swapper/010:00:370
558399574,30cyclictest0-21swapper/007:25:210
5583995733,9cyclictest0-21swapper/011:00:350
5583995733,7cyclictest101ktimersoftd/009:25:100
5583995733,7cyclictest101ktimersoftd/009:25:100
5583995724,5cyclictest5820-21grep11:30:370
5583995724,5cyclictest5820-21grep11:30:370
5585995618,18cyclictest21236-21ssh09:52:542
5585995618,12cyclictest10132-21if_eth008:50:302
5585995616,28cyclictest4813-21ssh09:33:342
558499565,10cyclictest0-21swapper/108:45:211
558399566,46cyclictest101ktimersoftd/012:35:330
558399565,48cyclictest9-21ksoftirqd/010:15:100
5583995641,5cyclictest101ktimersoftd/007:45:360
5583995638,5cyclictest6355-21ps07:10:330
5583995637,8cyclictest0-21swapper/011:59:250
5583995637,8cyclictest0-21swapper/011:59:250
5583995634,9cyclictest9966-21fschecks_count08:50:280
558599552,7cyclictest0-21swapper/211:45:222
5585995514,14cyclictest11056-21taskset12:16:122
5585995511,10cyclictest7950irq/56-dwc_otg_09:40:202
5583995534,10cyclictest0-21swapper/012:10:380
5583995531,8cyclictest22237-21/usr/sbin/munin12:30:260
5583995524,13cyclictest7850irq/56-dwc_otg_12:25:340
558399552,26cyclictest18351-21apt09:50:090
558399552,26cyclictest18351-21apt09:50:090
558399551,44cyclictest10712-21unixbench_singl12:15:370
5585995418,20cyclictest17338-21ssh10:26:442
5583995421,10cyclictest101ktimersoftd/011:45:090
558399541,43cyclictest23088-21cut09:55:240
558599532,10cyclictest7950irq/56-dwc_otg_08:55:212
5585995318,19cyclictest1154-21ssh12:05:182
5584995326,6cyclictest908-21runrttasks07:40:221
558399536,25cyclictest0-21swapper/008:25:240
5583995332,7cyclictest101ktimersoftd/008:45:310
5583995320,13cyclictest0-21swapper/010:15:220
5583995320,13cyclictest0-21swapper/010:15:220
558599523,11cyclictest7950irq/56-dwc_otg_08:10:202
5585995212,25cyclictest23502-21date10:35:102
5583995238,3cyclictest0-21swapper/007:35:280
5583995233,9cyclictest29092-21grep10:40:370
5583995230,8cyclictest0-21swapper/009:30:520
5583995227,13cyclictest101ktimersoftd/010:35:090
5583995227,10cyclictest0-21swapper/012:02:350
558699517,18cyclictest0-21swapper/309:55:203
558599513,6cyclictest0-21swapper/208:25:232
5585995114,21cyclictest7850irq/56-dwc_otg_09:25:302
5585995112,14cyclictest29031-21cut11:20:262
558499516,28cyclictest0-21swapper/109:00:231
558399512,25cyclictest15240-21grep09:05:230
558399511,26cyclictest14002-21tr09:00:320
558399511,25cyclictest0-21swapper/012:05:240
5585995015,12cyclictest5598-21/usr/sbin/munin07:10:342
5585995015,12cyclictest21653-21munin-run12:30:092
5585995013,12cyclictest3701-21seq08:32:052
5585995013,11cyclictest15814-21ssh12:22:072
5585995012,16cyclictest10298-21ssh10:57:272
558599501,14cyclictest0-21swapper/212:00:192
5583995022,9cyclictest29439-21tr08:15:220
5583995017,14cyclictest17057-21apt-get11:45:200
5585994918,11cyclictest13194-21timerandwakeup11:00:362
5585994916,11cyclictest23632-21ssh09:17:212
5585994911,12cyclictest17007-21ssh09:47:452
5585994911,12cyclictest17007-21ssh09:47:452
558399495,6cyclictest9550irq/80-mmc010:30:100
558399493,5cyclictest20396-21ssh11:10:210
5583994933,8cyclictest0-21swapper/011:36:510
5583994922,8cyclictest24849-21uniq11:15:270
5583994922,8cyclictest20986-21basename09:15:090
558699487,25cyclictest29889-21ssh09:25:223
5585994820,12cyclictest24221-21ssh11:53:442
5585994817,18cyclictest4561-21/usr/sbin/munin10:50:322
5585994815,16cyclictest7850irq/56-dwc_otg_12:30:302
5585994815,11cyclictest15211-21awk07:35:302
5585994814,16cyclictest24173-21ssh09:55:412
5583994832,3cyclictest0-21swapper/010:23:590
558399482,9cyclictest21947-21ssh09:15:300
5583994823,3cyclictest0-21swapper/010:50:300
558399481,22cyclictest11632-21unixbench_singl09:40:370
9322470,6sleep1558499cyclictest10:07:181
558699476,9cyclictest18687-21chrt09:50:213
558599476,27cyclictest7950irq/56-dwc_otg_11:36:572
5585994714,18cyclictest23510-21taskset07:58:482
5585994713,18cyclictest8098-21ssh12:13:222
5585994713,18cyclictest19488-21ssh11:09:092
5585994713,18cyclictest19488-21ssh11:09:092
5585994713,12cyclictest30160-21ssh10:04:022
5585994712,12cyclictest0-21swapper/208:45:372
558499476,27cyclictest0-21swapper/112:30:201
558499472,5cyclictest0-21swapper/110:55:211
5584994724,7cyclictest5641-21wget07:10:231
558399471,38cyclictest13-21rcuc/007:53:330
558599469,11cyclictest6331-21ssh09:35:242
5585994619,13cyclictest22201-21ssh11:11:332
5585994612,23cyclictest30686-21chrt08:16:492
5585994612,13cyclictest11027-21/usr/sbin/munin07:25:282
5585994612,12cyclictest8067-21/usr/sbin/munin07:15:312
5585994612,12cyclictest8067-21/usr/sbin/munin07:15:312
5583994632,4cyclictest9411-21ssh09:39:070
5583994615,14cyclictest0-21swapper/008:35:330
558399461,24cyclictest31890-21tr08:20:320
50162462,36sleep00-21swapper/007:06:180
558699457,23cyclictest0-21swapper/309:40:223
558599457,25cyclictest0-21swapper/207:45:152
5585994513,15cyclictest14222-21ssh10:23:372
5585994512,12cyclictest23335-21latency_hist10:35:102
558499457,26cyclictest0-21swapper/107:55:221
558499457,24cyclictest0-21swapper/107:20:231
558499457,23cyclictest0-21swapper/108:20:241
558499452,10cyclictest7850irq/56-dwc_otg_11:50:201
558399459,19cyclictest16687-21diskmemload12:22:440
558599448,8cyclictest0-21swapper/210:12:522
558599447,25cyclictest0-21swapper/208:35:222
5585994415,19cyclictest14250-21taskset07:33:182
5585994412,15cyclictest2819-21ssh10:49:142
5585994412,11cyclictest7850irq/56-dwc_otg_11:17:162
5585994411,14cyclictest20705-21rm09:14:432
5585994411,11cyclictest5996-21/usr/sbin/munin08:40:222
558499445,34cyclictest19514-21ssh10:30:201
558499441,7cyclictest0-21swapper/109:45:221
558499441,7cyclictest0-21swapper/109:45:221
5583994426,8cyclictest0-21swapper/008:11:200
5583994420,4cyclictest24163-21fschecks_count08:00:250
558699437,26cyclictest0-21swapper/308:55:213
558699437,24cyclictest0-21swapper/311:35:153
5586994315,14cyclictest37-21rcuc/309:15:223
558599437,22cyclictest7950irq/56-dwc_otg_08:08:492
558699428,22cyclictest0-21swapper/311:25:213
5585994214,16cyclictest6204-21ssh11:31:152
5585994214,16cyclictest6204-21ssh11:31:152
558499427,25cyclictest0-21swapper/109:05:231
558499427,24cyclictest0-21swapper/107:45:151
558499426,13cyclictest7850irq/56-dwc_otg_10:15:321
558499426,13cyclictest7850irq/56-dwc_otg_10:15:321
558499424,26cyclictest61-21kswapd010:00:211
5584994217,5cyclictest11593-21fschecks_count10:20:281
558699417,21cyclictest0-21swapper/309:30:153
558699416,25cyclictest0-21swapper/308:50:153
558499417,25cyclictest0-21swapper/108:50:151
558499415,23cyclictest24545-21ssh11:15:231
558499411,3cyclictest7950irq/56-dwc_otg_11:55:231
558499411,3cyclictest7950irq/56-dwc_otg_11:55:231
558499408,20cyclictest61-21kswapd007:35:151
558499402,9cyclictest0-21swapper/109:10:211
5584994010,9cyclictest0-21swapper/108:25:231
558699397,24cyclictest0-21swapper/308:05:243
558699393,4cyclictest7950irq/56-dwc_otg_12:30:213
558699392,18cyclictest18715-1kworker/u9:010:30:183
558699391,26cyclictest7613-21ls07:15:253
558699391,26cyclictest7613-21ls07:15:253
558499397,7cyclictest0-21swapper/109:55:181
558499396,21cyclictest7750irq/56-dwc_otg11:26:381
558499395,25cyclictest7750irq/56-dwc_otg09:35:091
5584993930,5cyclictest221ktimersoftd/111:45:211
558499392,5cyclictest7950irq/56-dwc_otg_09:35:201
286692390,2sleep3558699cyclictest11:20:193
260082390,2sleep10-21swapper/112:35:131
558699381,24cyclictest7283-21ls07:15:103
558599382,23cyclictest0-21swapper/209:00:232
5585993812,12cyclictest16410-21sshd09:06:512
558499387,19cyclictest0-21swapper/110:42:551
558499385,24cyclictest25632-21ssh09:20:231
558499385,24cyclictest25632-21ssh09:20:231
558499384,22cyclictest7950irq/56-dwc_otg_08:10:101
5584993814,10cyclictest0-21swapper/108:15:361
5583993814,15cyclictest0-21swapper/008:56:340
5586993731,4cyclictest19311-21ls07:45:353
558499373,23cyclictest7750irq/56-dwc_otg08:10:291
558499372,9cyclictest1220-21aten_rbpower_cu12:05:221
558499372,16cyclictest0-21swapper/112:20:261
5584993711,4cyclictest0-21swapper/111:20:221
558699364,5cyclictest7750irq/56-dwc_otg10:00:233
558699364,18cyclictest0-21swapper/310:22:183
558699362,6cyclictest7950irq/56-dwc_otg_10:15:213
558699362,6cyclictest7950irq/56-dwc_otg_10:15:213
558699361,3cyclictest7950irq/56-dwc_otg_11:15:203
558699361,25cyclictest3957-21latency_hist08:35:103
558499367,16cyclictest0-21swapper/107:25:211
558499364,13cyclictest7750irq/56-dwc_otg10:15:101
5584993618,5cyclictest18072-21df12:25:251
558499361,7cyclictest21-21rcuc/110:40:091
558499361,5cyclictest7750irq/56-dwc_otg07:30:201
558499361,22cyclictest7750irq/56-dwc_otg08:35:301
558699359,5cyclictest7950irq/56-dwc_otg_10:50:163
558699355,23cyclictest27828-21ssh10:40:223
558699352,29cyclictest0-21swapper/311:10:203
558699352,26cyclictest8491-21latency_hist11:35:103
558699352,26cyclictest8491-21latency_hist11:35:103
558499356,20cyclictest7850irq/56-dwc_otg_11:38:401
558499354,23cyclictest0-21swapper/111:03:551
558499354,22cyclictest7850irq/56-dwc_otg_07:50:271
5584993518,4cyclictest21-21rcuc/109:25:381
558499351,7cyclictest4169-21ssh10:50:271
5584993515,4cyclictest0-21swapper/109:15:211
558499351,27cyclictest12534-21ls08:55:351
558699347,10cyclictest7950irq/56-dwc_otg_07:55:353
558699346,23cyclictest0-21swapper/309:20:233
558699346,23cyclictest0-21swapper/309:20:233
558699345,5cyclictest7950irq/56-dwc_otg_11:00:313
558699345,22cyclictest0-21swapper/310:58:213
558699344,6cyclictest7850irq/56-dwc_otg_11:55:423
558699344,6cyclictest7850irq/56-dwc_otg_11:55:423
5586993417,4cyclictest29651-21sh12:39:443
558699341,3cyclictest0-21swapper/311:45:233
5586993413,5cyclictest21061-21latency_hist09:15:103
558699341,25cyclictest13810-21latency09:00:293
558699341,23cyclictest13960-21cut12:20:263
558699341,17cyclictest10983-21ls07:25:093
558599344,12cyclictest7950irq/56-dwc_otg_08:20:392
5585993421,11cyclictest21181-21ls07:50:352
558499347,18cyclictest7850irq/56-dwc_otg_09:40:261
558499346,5cyclictest7850irq/56-dwc_otg_08:30:211
558499345,13cyclictest0-21swapper/111:40:261
558499341,18cyclictest0-21swapper/110:48:271
558699333,3cyclictest0-21swapper/312:15:203
558699332,13cyclictest7950irq/56-dwc_otg_07:30:093
5586993320,4cyclictest2845-21sh10:10:113
558699331,6cyclictest9500-21dpkg08:50:103
558699331,5cyclictest14620-21ssh09:45:233
558699331,5cyclictest14620-21ssh09:45:233
558499339,4cyclictest7750irq/56-dwc_otg11:10:351
558499334,18cyclictest13441-21ls12:20:101
558499333,4cyclictest23-21ksoftirqd/112:10:301
558499332,9cyclictest0-21swapper/112:00:211
51642333,23sleep20-21swapper/207:08:142
558699328,5cyclictest7950irq/56-dwc_otg_11:05:353
558699328,5cyclictest7950irq/56-dwc_otg_11:05:353
558699327,5cyclictest7950irq/56-dwc_otg_08:05:093
558699327,11cyclictest0-21swapper/310:35:253
558699325,23cyclictest16847-21ls09:10:103
558699325,20cyclictest5339-21ssh12:10:233
558699322,18cyclictest7850irq/56-dwc_otg_11:50:243
558699321,6cyclictest7750irq/56-dwc_otg12:25:273
558699321,4cyclictest6140-21aten_rbpower_cu09:35:233
5586993212,4cyclictest847-21latency_hist12:05:103
5586993211,7cyclictest0-21swapper/312:09:053
558499325,4cyclictest7750irq/56-dwc_otg10:25:311
558499321,5cyclictest7850irq/56-dwc_otg_08:45:091
558499321,20cyclictest0-21swapper/107:15:271
558499321,20cyclictest0-21swapper/107:15:271
558699316,5cyclictest7850irq/56-dwc_otg_08:25:333
558699314,8cyclictest31249-21aten_rbpower_po10:05:233
558699312,13cyclictest7850irq/56-dwc_otg_10:47:453
558699311,6cyclictest7950irq/56-dwc_otg_08:35:373
5586993114,5cyclictest15941-21sh10:25:293
558499311,24cyclictest24826-21timerandwakeup08:00:351
558699309,4cyclictest0-21swapper/311:40:113
558699306,13cyclictest0-21swapper/307:50:233
558699301,5cyclictest28170-21wc08:10:303
5586993014,9cyclictest37-21rcuc/308:20:333
558699291,7cyclictest14823-21aten_rbpower_po07:35:233
558699291,6cyclictest7850irq/56-dwc_otg_08:40:333
558699283,3cyclictest7950irq/56-dwc_otg_07:30:203
558699282,19cyclictest0-21swapper/307:41:463
558699281,16cyclictest0-21swapper/308:19:043
51522287,14sleep10-21swapper/107:08:051
50242264,15sleep30-21swapper/307:06:253
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional