You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-13 - 00:10

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Tue May 12, 2026 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1295121520,5sleep10-21swapper/116:26:251
2049021110,2sleep220491-21/usr/sbin/munin18:16:222
2521921060,1sleep20-21swapper/218:30:592
1582921020,4sleep015818-21apt-get18:05:570
13974999525,5cyclictest917-21snmpd20:01:100
203362920,3sleep20-21swapper/216:46:252
1397499891,65cyclictest0-21swapper/018:28:150
13974998737,4cyclictest6263-21grep20:41:280
144922850,4sleep231-21ksoftirqd/220:12:402
1397499853,51cyclictest0-21swapper/020:26:090
80562830,5sleep21398299cyclictest21:23:062
101972830,3sleep210189-21timerwakeupswit20:46:262
80002780,2sleep10-21swapper/118:48:091
236402780,2sleep20-21swapper/220:24:082
309942770,6sleep11397899cyclictest19:53:471
295082730,2sleep10-21swapper/118:36:011
206742730,2sleep120676-21rm20:59:541
198532720,2sleep0101ktimersoftd/019:40:580
13978997210,21cyclictest7750irq/56-dwc_otg20:06:091
224062710,6sleep11397899cyclictest16:51:281
16912700,2sleep30-21swapper/318:41:153
1397499705,17cyclictest7950irq/56-dwc_otg_21:11:110
13974997010,19cyclictest7850irq/56-dwc_otg_20:56:040
1397499672,45cyclictest0-21swapper/019:05:590
40322660,5sleep31398399cyclictest21:56:573
1397499661,42cyclictest0-21swapper/020:20:580
1397499652,45cyclictest0-21swapper/019:51:090
1397499647,39cyclictest14300-21fschecks_count16:31:160
170112630,3sleep30-21swapper/320:16:163
1397499627,38cyclictest0-21swapper/018:41:180
208322610,2sleep10-21swapper/121:38:471
1398399605,18cyclictest7950irq/56-dwc_otg_21:51:113
1398399605,18cyclictest7950irq/56-dwc_otg_21:51:113
1397499606,50cyclictest0-21swapper/016:51:130
13974996029,14cyclictest7850irq/56-dwc_otg_21:16:290
13974995929,16cyclictest26279-21modprobe21:45:580
1397499591,40cyclictest0-21swapper/021:01:230
13974995823,18cyclictest16152-21fschecks_time16:36:140
13974995822,16cyclictest0-21swapper/018:16:250
13974995815,4cyclictest24979-21diskmemload20:31:220
13974995815,4cyclictest24979-21diskmemload20:31:220
1397499581,41cyclictest14445-21aten_rbpower_po18:56:130
13974995733,4cyclictest8658-21latency_hist17:45:580
13974995730,5cyclictest29963-21grep18:36:150
13974995725,5cyclictest2521-21aten_rbpower_en21:56:120
1397499561,42cyclictest0-21swapper/018:54:310
1397499561,41cyclictest17738-21aten_rbpower_en16:41:120
13974995536,5cyclictest14461-21unixbench_multi20:51:260
13974995536,5cyclictest14461-21unixbench_multi20:51:260
1397499552,36cyclictest31477-21unixbench-2d17:16:240
1397499552,35cyclictest0-21swapper/018:46:250
13974995512,32cyclictest0-21swapper/021:32:080
13974995512,32cyclictest0-21swapper/021:32:080
1397499548,26cyclictest9422-21timerandwakeup20:06:240
13974995438,4cyclictest7750irq/56-dwc_otg19:31:240
13974995433,6cyclictest7850irq/56-dwc_otg_21:30:580
13974995425,5cyclictest28882-21fschecks_count17:11:160
1397499542,45cyclictest0-21swapper/019:06:240
1397499541,43cyclictest0-21swapper/020:36:360
1397499541,37cyclictest3352-21wget19:21:130
13974995328,7cyclictest7850irq/56-dwc_otg_18:21:130
13974995328,7cyclictest7850irq/56-dwc_otg_18:21:130
1397499531,38cyclictest0-21swapper/019:28:540
1397499531,34cyclictest24306-21ssh19:46:110
1397499531,34cyclictest24306-21ssh19:46:110
1397899525,10cyclictest7750irq/56-dwc_otg21:16:121
1397899523,39cyclictest21896-21cut18:21:161
1397899523,39cyclictest21896-21cut18:21:161
13974995236,6cyclictest25837-21ssh21:06:090
13974995235,5cyclictest7850irq/56-dwc_otg_17:51:100
13974995235,5cyclictest7850irq/56-dwc_otg_17:51:100
13974995222,17cyclictest1315-21grep19:56:270
13974995217,15cyclictest24627-21sh20:25:560
1397499521,40cyclictest0-21swapper/017:31:220
1397499521,38cyclictest0-21swapper/017:00:580
326552510,2sleep20-21swapper/220:35:312
326552510,2sleep20-21swapper/220:35:312
1397899505,6cyclictest7750irq/56-dwc_otg20:51:081
1397899505,6cyclictest7750irq/56-dwc_otg20:51:081
13974995029,5cyclictest19729-21fschecks_count16:46:170
13974995016,15cyclictest20445-21grep19:41:140
1397499501,41cyclictest0-21swapper/018:06:130
13974995013,5cyclictest0-21swapper/019:16:090
13974995013,5cyclictest0-21swapper/019:16:090
13974995013,23cyclictest0-21swapper/021:36:090
1397899494,7cyclictest792-21ssh20:36:091
1397499494,22cyclictest27236-21ssh19:11:110
1397499493,23cyclictest13966-21cyclictest20:46:230
13974994910,20cyclictest27001-21idleruntime21:46:150
1397899486,5cyclictest0-21swapper/119:46:101
1397899486,5cyclictest0-21swapper/119:46:101
1397899485,31cyclictest3324-21ssh19:21:131
1397899481,10cyclictest7750irq/56-dwc_otg18:51:111
13974994818,15cyclictest1865-21fschecks_count17:26:150
1397499481,38cyclictest0-21swapper/017:38:390
1397499481,35cyclictest0-21swapper/021:51:000
1397499481,35cyclictest0-21swapper/021:51:000
1397499481,27cyclictest947-21kworker/0:021:21:130
135302483,13sleep00-21swapper/016:29:170
317402470,8sleep11397899cyclictest19:16:151
317402470,8sleep11397899cyclictest19:16:151
1397899477,23cyclictest7750irq/56-dwc_otg19:56:101
13974994721,20cyclictest10504-21ls17:50:590
1397499471,36cyclictest0-21swapper/017:03:320
1398399468,24cyclictest0-21swapper/316:36:113
13983994629,6cyclictest20359-21ssh19:41:133
1397899467,29cyclictest0-21swapper/117:11:131
1397899467,19cyclictest0-21swapper/121:06:041
1397899466,32cyclictest13966-21cyclictest16:46:111
1397899462,10cyclictest0-21swapper/121:11:111
1398399457,23cyclictest0-21swapper/317:11:043
1398399457,23cyclictest0-21swapper/316:51:133
1398299456,25cyclictest0-21swapper/218:21:122
1398299456,25cyclictest0-21swapper/218:21:122
1397899453,4cyclictest7750irq/56-dwc_otg18:31:121
1397499452,32cyclictest0-21swapper/017:59:450
1397499451,6cyclictest26800-21tr17:06:130
1397499451,36cyclictest0-21swapper/020:11:050
1397499451,23cyclictest25489-21tr18:31:140
1397899447,23cyclictest0-21swapper/117:51:091
1397899447,23cyclictest0-21swapper/117:51:091
1398399437,23cyclictest0-21swapper/321:46:103
1398299437,26cyclictest0-21swapper/221:31:112
1398299437,26cyclictest0-21swapper/221:31:112
1397899437,24cyclictest0-21swapper/118:11:121
1397899437,22cyclictest0-21swapper/118:16:111
1397899434,6cyclictest7750irq/56-dwc_otg20:11:111
1397499431,32cyclictest0-21swapper/018:12:400
1398399428,5cyclictest0-21swapper/320:41:213
1397899427,22cyclictest0-21swapper/117:26:111
1397499421,30cyclictest958-21users17:21:290
1398299417,24cyclictest0-21swapper/219:01:112
1398299411,3cyclictest7850irq/56-dwc_otg_18:06:112
1398399401,20cyclictest0-21swapper/320:36:243
1398299407,23cyclictest0-21swapper/220:16:122
1398299402,12cyclictest20447-21sort19:41:142
1397899405,3cyclictest7750irq/56-dwc_otg19:36:121
13978994015,8cyclictest21-21rcuc/120:47:341
1397899401,31cyclictest0-21swapper/119:11:541
54042391,34sleep35407-21ssh21:20:503
249602390,1sleep10-21swapper/120:26:081
1398399397,22cyclictest0-21swapper/319:01:113
1398399395,23cyclictest27144-21ssh19:11:043
1398399392,5cyclictest7950irq/56-dwc_otg_17:41:093
13983993913,5cyclictest0-21swapper/319:21:133
1398399391,27cyclictest18558-21cut21:36:153
13982993916,6cyclictest4458-21/usr/sbin/munin20:01:142
1397899399,3cyclictest7750irq/56-dwc_otg16:31:041
1397899395,21cyclictest5851-21ssh21:21:121
1398399387,25cyclictest0-21swapper/318:21:123
1398399387,25cyclictest0-21swapper/318:21:123
1398399381,4cyclictest7950irq/56-dwc_otg_17:36:093
13982993813,6cyclictest3146-21cron17:30:572
13978993815,6cyclictest23-21ksoftirqd/119:26:201
13978993814,4cyclictest0-21swapper/121:41:111
1398399375,19cyclictest29617-21ssh18:36:113
13983993714,5cyclictest19613-21grep16:46:143
1398399371,22cyclictest16012-21apt-key18:05:593
1398399371,21cyclictest32492-21cut17:21:173
13983993711,4cyclictest7850irq/56-dwc_otg_20:21:083
13982993721,4cyclictest24236-21/usr/sbin/munin19:46:132
13982993721,4cyclictest24236-21/usr/sbin/munin19:46:132
1397899377,6cyclictest7750irq/56-dwc_otg19:41:261
1397899371,22cyclictest15894-21ls18:05:581
1398399361,8cyclictest0-21swapper/321:06:103
13982993621,5cyclictest2318850wget16:56:132
1397899367,6cyclictest23-21ksoftirqd/117:46:121
1397899366,22cyclictest0-21swapper/117:56:101
13978993619,7cyclictest23-21ksoftirqd/117:41:171
13978993613,17cyclictest221ktimersoftd/116:56:101
1398399355,16cyclictest0-21swapper/317:06:103
1398399351,26cyclictest11048-21ls21:26:273
1398299355,13cyclictest7850irq/56-dwc_otg_20:41:252
1398299354,7cyclictest7850irq/56-dwc_otg_16:36:112
13982993516,5cyclictest18601-21fschecks_time21:36:162
1398299351,3cyclictest7950irq/56-dwc_otg_20:26:122
1397899357,6cyclictest23-21ksoftirqd/119:06:291
1397899353,16cyclictest7850irq/56-dwc_otg_21:55:581
1397899353,16cyclictest7850irq/56-dwc_otg_21:55:581
1397899352,4cyclictest7850irq/56-dwc_otg_17:31:091
13978993521,5cyclictest7850irq/56-dwc_otg_21:46:101
1397899352,13cyclictest7850irq/56-dwc_otg_17:01:151
13978993515,5cyclictest21-21rcuc/121:30:591
13978993510,4cyclictest0-21swapper/120:41:131
1398399347,14cyclictest0-21swapper/319:51:113
1398399346,12cyclictest0-21swapper/318:55:583
1398399345,20cyclictest29145-21ssh20:31:113
1398399345,20cyclictest29145-21ssh20:31:113
1398399344,19cyclictest7950irq/56-dwc_otg_17:51:143
1398399344,19cyclictest7950irq/56-dwc_otg_17:51:143
1398399342,27cyclictest7950irq/56-dwc_otg_20:26:093
13983993415,5cyclictest14411-21idleruntime16:31:163
1398399341,23cyclictest2470-21switchtime17:26:243
1398299348,7cyclictest0-21swapper/219:10:592
1398299346,6cyclictest7850irq/56-dwc_otg_21:51:152
1398299346,6cyclictest7850irq/56-dwc_otg_21:51:152
1398299345,7cyclictest7950irq/56-dwc_otg_16:51:282
1398299344,15cyclictest7950irq/56-dwc_otg_19:30:582
13982993422,7cyclictest5332-21cron18:45:582
13982993420,12cyclictest7950irq/56-dwc_otg_19:36:162
1398299341,4cyclictest7950irq/56-dwc_otg_18:51:142
13982993410,5cyclictest4055-21/usr/sbin/munin17:31:202
1397899349,14cyclictest7950irq/56-dwc_otg_21:31:151
1397899349,14cyclictest7950irq/56-dwc_otg_21:31:151
1397899348,12cyclictest7950irq/56-dwc_otg_20:01:271
1397899347,14cyclictest0-21swapper/118:06:041
1397899345,6cyclictest7850irq/56-dwc_otg_20:25:581
1397899344,13cyclictest7750irq/56-dwc_otg20:32:551
1397899344,13cyclictest7750irq/56-dwc_otg20:32:551
1397899343,7cyclictest7850irq/56-dwc_otg_21:01:271
13978993419,4cyclictest7950irq/56-dwc_otg_19:33:391
13978993418,4cyclictest21-21rcuc/121:56:231
13978993414,5cyclictest9550irq/80-mmc018:30:581
15152330,6sleep21398299cyclictest17:25:582
1398399337,14cyclictest0-21swapper/317:16:103
1398399336,15cyclictest0-21swapper/318:01:113
1398399332,7cyclictest0-21swapper/321:35:043
1398399332,7cyclictest0-21swapper/321:35:043
13983993314,4cyclictest15124-21ntp_states18:56:223
1398399331,21cyclictest25712-21cut18:31:173
1398399331,11cyclictest7850irq/56-dwc_otg_20:50:583
1398299336,14cyclictest0-21swapper/217:11:042
1398299334,5cyclictest7850irq/56-dwc_otg_18:46:112
1398299333,9cyclictest26183-21ssh21:06:162
1398299333,16cyclictest7950irq/56-dwc_otg_18:11:132
13982993324,6cyclictest17280-21latency_hist20:55:592
13982993324,6cyclictest17280-21latency_hist20:55:592
1398299331,23cyclictest9424-21missed_timers17:46:192
1397899337,15cyclictest0-21swapper/116:41:121
1397899335,7cyclictest0-21swapper/117:06:111
1397899334,4cyclictest7750irq/56-dwc_otg17:16:101
1398399329,10cyclictest0-21swapper/320:09:313
1398399328,4cyclictest11651-1kworker/3:1H16:56:113
1398399327,15cyclictest0-21swapper/319:16:123
1398399327,15cyclictest0-21swapper/319:16:123
1398399327,14cyclictest0-21swapper/320:57:463
1398399326,16cyclictest0-21swapper/319:29:063
13983993211,3cyclictest0-21swapper/318:46:273
1398399320,15cyclictest13134-21ntp_states17:56:193
1398299327,16cyclictest0-21swapper/219:51:112
1398299323,5cyclictest7950irq/56-dwc_otg_21:18:422
13982993219,6cyclictest5701-21sh19:23:202
1398299321,5cyclictest7950irq/56-dwc_otg_21:41:032
13982993215,4cyclictest7950irq/56-dwc_otg_20:40:582
1398299321,19cyclictest0-21swapper/218:00:582
13982993210,6cyclictest7950irq/56-dwc_otg_21:26:272
1397899329,4cyclictest0-21swapper/120:16:251
1397899327,12cyclictest7950irq/56-dwc_otg_17:21:201
1397899321,20cyclictest14565-21ls18:56:131
13978993210,4cyclictest0-21swapper/118:41:191
13978993210,12cyclictest0-21swapper/119:05:331
1398399319,8cyclictest13586-21ssh20:51:153
1398399319,8cyclictest13586-21ssh20:51:153
1398399319,4cyclictest0-21swapper/319:58:553
1398399313,12cyclictest7950irq/56-dwc_otg_19:39:263
1398399312,5cyclictest7850irq/56-dwc_otg_21:21:103
13983993118,10cyclictest15165-21ssh20:13:543
1398399311,3cyclictest7750irq/56-dwc_otg21:15:583
1398299318,12cyclictest0-21swapper/219:56:452
1398299317,10cyclictest7950irq/56-dwc_otg_21:04:312
1398299316,4cyclictest17613-21aten_rbpower_vo20:56:142
1398299316,4cyclictest11319-21ntp_offset17:51:182
1398299316,4cyclictest11319-21ntp_offset17:51:182
1398299315,6cyclictest7950irq/56-dwc_otg_18:05:572
1398299315,5cyclictest7850irq/56-dwc_otg_21:56:212
1398299314,14cyclictest7950irq/56-dwc_otg_18:40:572
1398299313,4cyclictest7850irq/56-dwc_otg_18:56:112
1398299313,25cyclictest27207-21ssh19:11:092
13982993117,11cyclictest0-21swapper/216:31:212
13982993116,4cyclictest31834-21fschecks_time19:16:162
13982993116,4cyclictest31834-21fschecks_time19:16:162
1398299311,5cyclictest8140-21grep17:41:242
1398399307,10cyclictest7950irq/56-dwc_otg_19:46:253
1398399307,10cyclictest7950irq/56-dwc_otg_19:46:253
1398399307,10cyclictest0-21swapper/319:10:273
1398399305,5cyclictest7750irq/56-dwc_otg21:01:133
1398399305,3cyclictest7750irq/56-dwc_otg19:35:473
1398399303,13cyclictest4201-21/usr/sbin/munin20:01:133
1398299308,5cyclictest7950irq/56-dwc_otg_16:41:212
1398299307,5cyclictest0-21swapper/221:46:232
1398299304,3cyclictest0-21swapper/220:06:082
1398299303,5cyclictest7750irq/56-dwc_otg21:11:262
1398299301,23cyclictest0-21swapper/217:36:122
1397899304,13cyclictest7950irq/56-dwc_otg_17:38:551
134212305,16sleep213424-21resolvconf16:27:572
1398399297,4cyclictest7950irq/56-dwc_otg_18:16:243
1398399292,8cyclictest0-21swapper/318:26:243
1398399291,14cyclictest7750irq/56-dwc_otg17:46:203
1398299298,3cyclictest0-21swapper/219:35:412
1398299291,12cyclictest7750irq/56-dwc_otg17:01:262
1397899295,5cyclictest7750irq/56-dwc_otg16:40:581
1398399285,10cyclictest0-21swapper/321:41:433
1398399281,13cyclictest7950irq/56-dwc_otg_17:31:173
1398299285,5cyclictest7950irq/56-dwc_otg_17:06:172
1398299285,11cyclictest0-21swapper/218:30:572
1398399273,3cyclictest7950irq/56-dwc_otg_18:11:293
13983992714,4cyclictest17916-21entropy16:41:153
120942274,13sleep30-21swapper/316:26:043
1398399261,5cyclictest7850irq/56-dwc_otg_17:02:353
1398299264,4cyclictest7750irq/56-dwc_otg17:16:212
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional