You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-14 - 03:28

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Thu May 14, 2026 00:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1352521030,7sleep21056699cyclictest04:36:262
151722980,2sleep30-21swapper/308:20:263
69662890,2sleep00-21swapper/005:50:590
251912850,3sleep11056599cyclictest09:11:171
10564998413,29cyclictest2077-21ssh06:46:110
1056499831,79cyclictest9-21ksoftirqd/009:12:210
45542800,2sleep30-21swapper/306:48:303
1056599786,19cyclictest0-21swapper/108:06:121
1056599786,19cyclictest0-21swapper/108:06:121
1056499786,56cyclictest0-21swapper/009:21:110
10564997862,5cyclictest9-21ksoftirqd/005:56:140
204062770,6sleep21056699cyclictest07:07:092
204062770,6sleep21056699cyclictest07:07:092
73182760,3sleep07300-21switchtime06:51:250
1056499751,39cyclictest29453-21ssh09:56:100
1056499751,39cyclictest29453-21ssh09:56:100
1056499742,51cyclictest0-21swapper/008:31:090
1056599733,29cyclictest0-21swapper/105:56:101
226082720,5sleep21056699cyclictest09:46:372
10564997113,26cyclictest9-21ksoftirqd/007:20:580
19592690,1sleep10-21swapper/108:42:511
1056499681,57cyclictest0-21swapper/005:41:040
300302660,1sleep00-21swapper/008:37:260
300302660,1sleep00-21swapper/008:37:260
80102650,3sleep27980-21ssh07:31:252
10564996348,6cyclictest101ktimersoftd/008:46:250
1056499631,51cyclictest31488-21latency_hist08:00:590
1056499631,28cyclictest0-21swapper/009:06:130
1056499631,28cyclictest0-21swapper/009:06:130
99692613,12sleep00-21swapper/004:26:440
10564996053,5cyclictest0-21swapper/008:14:400
10564996041,7cyclictest7850irq/56-dwc_otg_06:40:590
10564996030,15cyclictest14254-21latency_hist07:00:580
1056499596,27cyclictest25334-21ssh09:51:090
1056799582,3cyclictest7950irq/56-dwc_otg_09:51:093
1056699587,9cyclictest7750irq/56-dwc_otg08:31:102
1056499583,46cyclictest21324-21kworker/0:106:41:460
10564995830,5cyclictest13-21rcuc/006:26:150
1056499582,38cyclictest32264-21sed07:21:260
10564995819,17cyclictest25118-21aten_rbpower_en05:11:110
205002570,31sleep37950irq/56-dwc_otg_08:26:183
1056499571,31cyclictest0-21swapper/008:26:240
1056699564,21cyclictest7950irq/56-dwc_otg_07:41:092
1056499561,4cyclictest13349-21switchtime08:56:250
1056499561,39cyclictest3987-21diskmemload08:06:180
1056499561,39cyclictest3987-21diskmemload08:06:180
1056499561,36cyclictest0-21swapper/007:41:260
10564995613,23cyclictest0-21swapper/008:25:590
1056599559,9cyclictest0-21swapper/104:31:101
1056599559,9cyclictest0-21swapper/104:31:101
1056499559,23cyclictest12807-21timerwakeupswit08:16:270
10564995542,4cyclictest101ktimersoftd/008:01:160
1056499551,5cyclictest0-21swapper/004:31:180
1056499551,5cyclictest0-21swapper/004:31:180
1056499542,41cyclictest0-21swapper/005:06:100
10564995422,8cyclictest7950irq/56-dwc_otg_09:16:100
10564995410,25cyclictest9-21ksoftirqd/009:05:590
10564995327,10cyclictest7950irq/56-dwc_otg_09:36:220
1056499532,32cyclictest0-21swapper/008:52:570
1056499531,39cyclictest23615-21aten_rbpower_po07:51:120
1056499531,38cyclictest9347-21cut09:31:180
10566995210,9cyclictest7850irq/56-dwc_otg_09:01:092
10564995230,5cyclictest21252-21latency_hist05:00:590
10564995223,14cyclictest7850irq/56-dwc_otg_07:40:590
10564995218,18cyclictest0-21swapper/005:20:590
10564995216,21cyclictest13-21rcuc/005:51:040
1056499521,34cyclictest0-21swapper/006:21:220
1056499512,35cyclictest0-21swapper/007:31:180
1056499512,34cyclictest19441-21aten_rbpower_cu07:46:120
1056499511,37cyclictest15809-21grep07:01:260
1056499511,31cyclictest21724-21ssh09:46:190
10564995024,7cyclictest7850irq/56-dwc_otg_06:16:220
1056499502,25cyclictest917-21snmpd04:36:170
10564995020,6cyclictest7850irq/56-dwc_otg_04:46:260
10564995019,17cyclictest21735-21awk06:31:110
10564995018,16cyclictest12544-21latency_hist06:06:000
10564995018,16cyclictest12544-21latency_hist06:06:000
1056499501,46cyclictest0-21swapper/009:26:150
1056799492,9cyclictest7950irq/56-dwc_otg_08:31:113
1056599497,26cyclictest0-21swapper/104:51:121
1056599494,18cyclictest7750irq/56-dwc_otg07:31:151
10564994917,17cyclictest0-21swapper/009:44:450
1056499491,31cyclictest0-21swapper/007:06:190
1056499491,31cyclictest0-21swapper/007:06:190
1056699488,26cyclictest0-21swapper/204:46:122
10565994818,10cyclictest23-21ksoftirqd/107:36:091
10564994826,6cyclictest7950irq/56-dwc_otg_07:26:210
1056499481,44cyclictest32599-21fschecks_time05:31:150
1056499476,24cyclictest0-21swapper/007:11:170
1056499472,35cyclictest0-21swapper/006:06:200
10564994719,9cyclictest101ktimersoftd/005:36:140
1056799467,24cyclictest0-21swapper/305:21:123
1056799467,24cyclictest0-21swapper/305:21:123
10564994620,20cyclictest0-21swapper/005:02:560
1056499461,32cyclictest30507-21aten_rbpower_cu05:26:110
1056799457,24cyclictest0-21swapper/306:01:123
1056799457,24cyclictest0-21swapper/306:01:123
1056699457,26cyclictest0-21swapper/206:16:112
1056699457,24cyclictest0-21swapper/206:11:112
1056699451,7cyclictest7750irq/56-dwc_otg09:56:092
1056699451,7cyclictest7750irq/56-dwc_otg09:56:092
1056499457,35cyclictest14577-21ls06:11:120
1056499451,35cyclictest0-21swapper/008:42:290
1056499451,31cyclictest0-21swapper/004:51:110
1056799447,24cyclictest0-21swapper/305:41:113
1056699446,25cyclictest0-21swapper/205:36:132
1056699444,25cyclictest0-21swapper/205:41:112
10566994417,8cyclictest7950irq/56-dwc_otg_07:56:102
10564994412,22cyclictest0-21swapper/005:21:120
10564994412,22cyclictest0-21swapper/005:21:120
288062430,3sleep20-21swapper/208:36:192
288062430,3sleep20-21swapper/208:36:192
1056699437,24cyclictest0-21swapper/205:51:102
10565994324,5cyclictest7750irq/56-dwc_otg06:45:581
10565994322,6cyclictest27286-21grep06:36:291
10564994322,6cyclictest7850irq/56-dwc_otg_04:41:180
1056699428,5cyclictest31-21ksoftirqd/205:31:122
1056699424,15cyclictest4679-21ssh09:26:092
10566994221,6cyclictest7850irq/56-dwc_otg_08:21:262
1056799417,24cyclictest39-21ksoftirqd/307:30:243
1056799417,22cyclictest0-21swapper/304:46:123
10567994125,5cyclictest32536-21rm07:21:413
1056699417,24cyclictest0-21swapper/208:16:132
10566994118,7cyclictest7850irq/56-dwc_otg_08:46:132
1056599411,6cyclictest0-21swapper/109:01:101
1056599411,3cyclictest0-21swapper/105:01:111
327572400,2sleep30-21swapper/305:31:173
1056799407,23cyclictest0-21swapper/305:51:103
10567994026,5cyclictest14053-21ssh09:36:253
1056799401,25cyclictest0-21swapper/305:36:123
1056699407,24cyclictest0-21swapper/208:41:132
1056699407,23cyclictest0-21swapper/204:56:122
1056599407,27cyclictest0-21swapper/106:26:111
1056599402,7cyclictest11939-21unixbench_singl06:01:271
1056599402,7cyclictest11939-21unixbench_singl06:01:271
1056599401,10cyclictest20607-21grep07:46:271
1056799397,23cyclictest0-21swapper/305:04:173
10567993918,5cyclictest39-21ksoftirqd/305:56:103
1056699398,14cyclictest23498-21ssh06:32:142
1056699394,23cyclictest0-21swapper/206:21:122
1056699394,14cyclictest7950irq/56-dwc_otg_07:16:232
10566993914,8cyclictest7950irq/56-dwc_otg_05:09:362
10566993913,6cyclictest7750irq/56-dwc_otg09:08:592
10566993913,6cyclictest7750irq/56-dwc_otg09:08:592
1056699391,15cyclictest0-21swapper/208:55:052
1056699391,13cyclictest7850irq/56-dwc_otg_09:12:102
1056599398,16cyclictest7950irq/56-dwc_otg_09:51:191
1056599397,23cyclictest0-21swapper/108:31:121
1056599392,9cyclictest15867-21sed07:01:261
1056599392,9cyclictest12730-21fschecks_time08:56:161
10565993914,5cyclictest17133-21aten_rbpower_en09:41:121
1056799387,20cyclictest0-21swapper/304:56:123
1056699381,22cyclictest301ktimersoftd/209:51:252
1056699381,22cyclictest28881-21ls05:21:142
1056699381,22cyclictest28881-21ls05:21:142
10566993811,8cyclictest7950irq/56-dwc_otg_06:43:172
1056599387,15cyclictest7850irq/56-dwc_otg_06:50:591
1056599382,5cyclictest16246-21grep06:16:111
10565993815,6cyclictest0-21swapper/107:06:171
10565993815,6cyclictest0-21swapper/107:06:171
1056599381,29cyclictest7950irq/56-dwc_otg_08:11:151
1056799379,10cyclictest31729-21/usr/sbin/munin08:01:193
1056799377,25cyclictest0-21swapper/307:31:113
10567993713,6cyclictest7750irq/56-dwc_otg09:31:253
1056799371,22cyclictest3783-21ls08:45:593
1056699378,14cyclictest0-21swapper/209:33:522
1056699372,30cyclictest28869-21ssh09:16:112
10566993714,8cyclictest7950irq/56-dwc_otg_08:01:252
1056599378,14cyclictest7850irq/56-dwc_otg_05:26:151
1056599372,8cyclictest5442-21wget05:46:121
1056599372,25cyclictest0-21swapper/109:31:081
1056599372,10cyclictest16257-21cut04:46:151
10565993716,9cyclictest21384-21diskmemload07:53:451
294592361,4sleep10-21swapper/109:56:121
294592361,4sleep10-21swapper/109:56:121
151302360,3sleep215132-21timerandwakeup04:41:252
111512361,4sleep211161-21ssh08:15:392
1056799367,22cyclictest0-21swapper/306:16:113
1056799367,10cyclictest37-21rcuc/308:41:003
1056799367,10cyclictest37-21rcuc/308:41:003
1056799366,20cyclictest15277-21ssh07:41:113
1056799365,16cyclictest37-21rcuc/307:05:053
10567993619,15cyclictest2425050ssh09:10:443
10567993619,15cyclictest2425050ssh09:10:443
10567993613,5cyclictest7950irq/56-dwc_otg_09:56:093
10567993613,5cyclictest7950irq/56-dwc_otg_09:56:093
1056699367,20cyclictest0-21swapper/209:21:112
1056699365,9cyclictest7850irq/56-dwc_otg_08:56:132
1056699363,14cyclictest7850irq/56-dwc_otg_06:59:572
1056699362,6cyclictest7950irq/56-dwc_otg_06:36:172
1056699362,4cyclictest7950irq/56-dwc_otg_07:46:112
10566993616,4cyclictest7950irq/56-dwc_otg_06:51:202
10566993610,8cyclictest7950irq/56-dwc_otg_08:27:062
1056599363,9cyclictest2778-21grep05:36:291
1056599361,8cyclictest8287-21aten_rbpower_en08:51:111
1056599361,8cyclictest0-21swapper/108:16:131
1056599361,7cyclictest1422-21latency_hist05:35:581
10565993616,5cyclictest908-21runrttasks06:56:051
10565993616,5cyclictest23195-21fschecks_time07:11:151
10565993616,4cyclictest28214-21aten_rbpower_cu08:36:121
10565993616,4cyclictest28214-21aten_rbpower_cu08:36:121
10565993612,6cyclictest13094-21latency06:06:201
1056799357,6cyclictest39-21ksoftirqd/308:46:223
1056799356,17cyclictest37-21rcuc/307:16:193
1056799355,24cyclictest39-21ksoftirqd/308:12:573
10567993514,4cyclictest37-21rcuc/308:21:233
1056799351,25cyclictest24961-21ssh09:50:553
10566993516,6cyclictest7750irq/56-dwc_otg05:56:192
10566993513,8cyclictest7750irq/56-dwc_otg07:25:032
10566993510,9cyclictest301ktimersoftd/206:26:232
1056599358,7cyclictest7850irq/56-dwc_otg_04:41:271
1056599356,7cyclictest7850irq/56-dwc_otg_06:55:591
1056599355,6cyclictest7850irq/56-dwc_otg_05:41:261
1056599353,21cyclictest0-21swapper/104:59:331
10565993513,5cyclictest0-21swapper/107:16:171
1056599351,21cyclictest23746-21ssh06:32:411
7012341,5sleep10-21swapper/109:21:131
315992341,27sleep10-21swapper/108:01:001
1056799347,12cyclictest37-21rcuc/307:07:493
1056799347,12cyclictest37-21rcuc/307:07:493
1056799346,18cyclictest0-21swapper/306:11:113
1056799344,10cyclictest15092-21cat04:41:243
10567993417,4cyclictest4649-21/usr/sbin/munin09:26:203
10567993416,12cyclictest21384-21diskmemload07:00:013
1056799341,2cyclictest0-21swapper/307:11:113
1056799341,25cyclictest0-21swapper/306:37:083
1056799341,21cyclictest0-21swapper/304:51:203
10567993411,4cyclictest37-21rcuc/307:36:193
10567993411,11cyclictest381ktimersoftd/307:58:323
1056699345,5cyclictest7950irq/56-dwc_otg_07:01:122
1056699345,5cyclictest7750irq/56-dwc_otg09:41:222
1056699343,5cyclictest29-21rcuc/208:07:002
1056699343,5cyclictest29-21rcuc/208:07:002
10566993419,5cyclictest26180-21sh07:15:092
10566993414,8cyclictest6095-21ssh07:30:092
1056599348,5cyclictest7750irq/56-dwc_otg05:15:591
1056599344,5cyclictest16088-21sort08:21:161
1056599343,9cyclictest0-21swapper/107:22:451
1056599341,22cyclictest12565-21ls04:36:121
1056599341,19cyclictest3803-21ssh07:26:251
1056799338,9cyclictest37-21rcuc/306:42:543
1056799337,17cyclictest39-21ksoftirqd/306:31:513
1056799337,10cyclictest39-21ksoftirqd/309:14:353
1056799336,16cyclictest39-21ksoftirqd/309:03:433
1056799335,16cyclictest543-21ssh09:21:113
1056799334,9cyclictest7950irq/56-dwc_otg_06:26:183
1056799333,20cyclictest39-21ksoftirqd/304:31:273
1056799333,20cyclictest39-21ksoftirqd/304:31:273
1056799332,16cyclictest0-21swapper/306:53:433
10567993315,4cyclictest4331-21processes08:06:233
10567993315,4cyclictest4331-21processes08:06:233
1056799331,4cyclictest0-21swapper/309:16:403
10567993310,10cyclictest0-21swapper/308:58:083
1056699337,5cyclictest19382-21latency_hist04:55:582
1056699337,4cyclictest7850irq/56-dwc_otg_06:49:452
1056699336,5cyclictest7850irq/56-dwc_otg_07:36:272
1056699333,3cyclictest7750irq/56-dwc_otg06:01:112
1056699333,3cyclictest7750irq/56-dwc_otg06:01:112
1056699331,7cyclictest31-21ksoftirqd/209:38:462
1056599337,16cyclictest0-21swapper/109:36:111
1056599335,5cyclictest0-21swapper/109:26:111
1056599332,7cyclictest20714-21/usr/sbin/munin08:26:221
10565993327,4cyclictest24408-21sh09:10:591
10565993327,4cyclictest24408-21sh09:10:591
1056599332,17cyclictest0-21swapper/107:41:121
10565993317,5cyclictest2694850wget05:16:121
10565993310,4cyclictest7850irq/56-dwc_otg_08:46:081
97662326,15sleep20-21swapper/204:26:262
1056799327,22cyclictest0-21swapper/306:21:123
1056799321,19cyclictest12868-21cut06:06:173
1056799321,16cyclictest27488-21ls07:55:593
1056799321,14cyclictest37-21rcuc/305:11:503
1056799321,13cyclictest39-21ksoftirqd/308:54:533
1056699321,6cyclictest11742-21unixbench_singl04:31:272
1056699321,6cyclictest11742-21unixbench_singl04:31:272
1056699321,21cyclictest0-21swapper/207:51:112
1056599323,7cyclictest21504-21cut09:46:161
1056599321,6cyclictest23367-21aten_rbpower_po05:06:131
1056799317,21cyclictest0-21swapper/305:16:123
1056799313,22cyclictest0-21swapper/309:45:093
1056799311,8cyclictest0-21swapper/305:08:543
10567993113,7cyclictest37-21rcuc/307:49:503
1056599318,4cyclictest0-21swapper/107:59:471
1056599312,10cyclictest7750irq/56-dwc_otg06:21:101
1056599311,7cyclictest29347-21ntpq05:21:211
1056599311,7cyclictest29347-21ntpq05:21:211
1056599311,12cyclictest7850irq/56-dwc_otg_09:18:161
91282305,14sleep19212-21awk04:26:171
1056699302,14cyclictest7950irq/56-dwc_otg_05:26:172
1056699301,6cyclictest7850irq/56-dwc_otg_05:01:262
1056599301,8cyclictest7190-21chrt05:51:081
1056799297,11cyclictest31664-21/usr/sbin/munin05:26:273
1056799293,13cyclictest7950irq/56-dwc_otg_04:36:153
1056699297,15cyclictest0-21swapper/205:16:122
1056699296,6cyclictest6534-21unixbench_singl05:46:272
1056699291,13cyclictest7950irq/56-dwc_otg_06:06:132
1056599291,17cyclictest14730-21cut06:11:151
98842283,14sleep30-21swapper/304:26:273
1056799281,15cyclictest6490-21unixbench_multi05:46:273
1056699286,13cyclictest31-21ksoftirqd/205:14:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional