You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-06-15 - 13:55

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Mon Jun 15, 2026 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1310322340,2sleep00-21swapper/019:05:290
1940121350,8sleep31388199cyclictest20:55:033
13878991191,4cyclictest0-21swapper/023:32:040
609021130,2sleep20-21swapper/222:05:032
13878991003,81cyclictest11411-21fschecks_time22:50:200
1387899991,76cyclictest17142-21unixbench_singl23:35:300
1387899991,76cyclictest17142-21unixbench_singl23:35:300
1387899962,73cyclictest0-21swapper/021:40:250
1387899941,74cyclictest27347-21latency_hist23:10:030
1387899934,75cyclictest25276-21taskset21:48:480
1387899932,79cyclictest0-21swapper/022:13:570
1387899922,78cyclictest27607-21chrt22:30:250
1387899912,74cyclictest5024-21sh00:00:290
1387899901,78cyclictest0-21swapper/023:04:280
1387899901,72cyclictest0-21swapper/000:25:320
1387899892,72cyclictest0-21swapper/023:10:180
1387899892,70cyclictest0-21swapper/022:55:570
1387899892,70cyclictest0-21swapper/022:55:570
1387899891,74cyclictest2494-21fschecks_time22:00:190
1387899872,60cyclictest0-21swapper/000:20:020
1387899871,71cyclictest0-21swapper/019:44:190
1387899862,69cyclictest0-21swapper/021:15:220
1387899861,70cyclictest0-21swapper/020:48:050
1387899856,61cyclictest0-21swapper/020:10:150
1387899851,71cyclictest0-21swapper/021:20:540
1387899851,69cyclictest0-21swapper/020:50:290
1387899851,40cyclictest1808-21munin-run22:00:030
1387899851,40cyclictest1808-21munin-run22:00:030
1387899841,70cyclictest0-21swapper/023:57:360
1387899832,69cyclictest0-21swapper/021:26:300
1387899832,64cyclictest0-21swapper/022:44:460
13878998326,43cyclictest8050irq/56-dwc_otg_19:55:260
13878998326,43cyclictest8050irq/56-dwc_otg_19:55:260
1387899831,74cyclictest0-21swapper/023:46:220
1387899831,4cyclictest7579-21sed22:05:320
1387899821,68cyclictest20945-21fschecks_count00:20:210
1387899811,57cyclictest9005-21ntp_kernel_pll_00:05:270
1387899811,57cyclictest9005-21ntp_kernel_pll_00:05:270
1387899802,68cyclictest0-21swapper/021:34:510
1387899802,53cyclictest0-21swapper/000:39:280
158562790,5sleep31388199cyclictest21:36:543
1387899796,54cyclictest32683-21grep23:15:280
1387899792,5cyclictest19324-21apt-get19:25:110
13878997913,14cyclictest9-21ksoftirqd/023:40:140
13878997910,57cyclictest0-21swapper/021:11:020
1387899781,4cyclictest7880-21unixbench_multi20:20:300
13878997812,49cyclictest13899-21aten_rbpower_po21:35:170
1387899771,60cyclictest28353-21/usr/sbin/munin23:50:170
1387899762,60cyclictest0-21swapper/000:33:590
1387899762,60cyclictest0-21swapper/000:33:590
1387899762,59cyclictest0-21swapper/000:14:290
1387899762,58cyclictest0-21swapper/019:11:490
1387899761,63cyclictest0-21swapper/022:22:240
1387899752,55cyclictest0-21swapper/023:28:060
1387899752,52cyclictest0-21swapper/022:16:450
1387899751,4cyclictest9329-21chrt22:47:320
1387899742,61cyclictest0-21swapper/021:52:580
1387899741,57cyclictest0-21swapper/023:20:250
1387899741,39cyclictest17284-21apt19:20:020
1387899741,2cyclictest0-21swapper/022:25:140
1387899741,2cyclictest0-21swapper/022:25:140
309462730,4sleep2301ktimersoftd/221:16:072
235602730,1sleep00-21swapper/021:05:220
235602730,1sleep00-21swapper/021:05:220
225832730,2sleep30-21swapper/323:04:093
1387899731,5cyclictest408-21ssh22:36:220
1387899722,67cyclictest0-21swapper/019:50:170
325452710,3sleep232530-21missed_timers20:00:242
13878997124,22cyclictest7950irq/56-dwc_otg_20:45:030
13878997124,22cyclictest7950irq/56-dwc_otg_20:45:030
1387899702,54cyclictest0-21swapper/020:06:430
1387899702,47cyclictest0-21swapper/021:04:170
1387899701,3cyclictest0-21swapper/020:55:250
1387899701,2cyclictest0-21swapper/020:28:280
225932690,2sleep20-21swapper/221:45:222
1387899692,52cyclictest0-21swapper/020:31:380
1387899681,4cyclictest0-21swapper/019:36:260
1387899672,47cyclictest0-21swapper/020:39:480
1387899665,45cyclictest769-21runrttasks20:01:190
1387899661,3cyclictest0-21swapper/019:49:520
17642650,2sleep30-21swapper/322:00:003
17642650,2sleep30-21swapper/322:00:003
1387899651,4cyclictest225392sleep019:33:210
1387899645,25cyclictest0-21swapper/020:15:160
1388099624,14cyclictest0-21swapper/221:04:512
133432614,8sleep30-21swapper/319:06:433
1387899601,3cyclictest188692sleep019:22:260
237532580,5sleep21388099cyclictest19:35:292
1387999584,7cyclictest7850irq/56-dwc_otg22:10:161
325942570,2sleep10-21swapper/100:34:501
325942570,2sleep10-21swapper/100:34:501
129972550,2sleep20-21swapper/222:12:522
139582530,2sleep30-21swapper/322:53:273
1387999522,5cyclictest0-21swapper/123:40:151
125682520,2sleep30-21swapper/321:33:273
1388099513,4cyclictest7950irq/56-dwc_otg_23:40:152
283352500,2sleep20-21swapper/200:29:352
1388099503,3cyclictest0-21swapper/220:05:162
1387999483,5cyclictest8050irq/56-dwc_otg_20:35:151
13879994825,5cyclictest7850irq/56-dwc_otg22:50:141
13879994711,4cyclictest7850irq/56-dwc_otg19:10:131
1388199463,20cyclictest5906-21ls20:15:313
1387999468,5cyclictest727-21runrttasks19:40:211
1387999467,25cyclictest0-21swapper/119:55:161
1387999467,25cyclictest0-21swapper/119:55:161
1387999467,24cyclictest0-21swapper/123:50:161
1387999459,7cyclictest0-21swapper/122:35:121
1387999457,26cyclictest0-21swapper/120:55:161
1388099447,22cyclictest0-21swapper/221:05:162
1388099447,22cyclictest0-21swapper/221:05:162
13880994423,6cyclictest9326-21latency_hist21:30:032
1387999447,24cyclictest0-21swapper/120:30:151
1388099425,4cyclictest8050irq/56-dwc_otg_23:15:152
1387999427,23cyclictest0-21swapper/123:05:081
13879994220,5cyclictest2427850wget23:45:181
1388199417,22cyclictest0-21swapper/320:25:163
1388199411,5cyclictest0-21swapper/323:35:133
1388199411,5cyclictest0-21swapper/323:35:133
13881994114,12cyclictest8050irq/56-dwc_otg_21:10:133
1388099411,4cyclictest6299-21ssh22:05:172
13879994119,6cyclictest17495-21switchtime00:15:291
13879994118,6cyclictest15550-21latency_hist20:45:031
13879994118,6cyclictest15550-21latency_hist20:45:031
1387999411,22cyclictest7850irq/56-dwc_otg00:25:251
13879994111,17cyclictest0-21swapper/100:00:151
232922401,34sleep323336-21ssh22:25:223
232922401,34sleep323336-21ssh22:25:223
1388199406,21cyclictest1103-21ssh21:20:143
1387999403,4cyclictest7850irq/56-dwc_otg21:55:191
1387999403,4cyclictest7850irq/56-dwc_otg21:55:191
13879994022,9cyclictest7850irq/56-dwc_otg22:25:191
13879994022,9cyclictest7850irq/56-dwc_otg22:25:191
261262390,5sleep31388199cyclictest00:25:343
1388099396,22cyclictest0-21swapper/219:30:172
1388099391,4cyclictest8050irq/56-dwc_otg_23:10:142
13880993914,16cyclictest23566-21wget23:05:172
1387999395,25cyclictest17977-21ssh21:40:101
13879993915,6cyclictest125-21systemd-journal19:51:111
1387999391,16cyclictest961-21ssh00:35:211
1388199381,21cyclictest27450-21ssh23:10:043
1388099389,19cyclictest0-21swapper/223:25:442
1388099384,6cyclictest125-21systemd-journal00:00:022
1387999386,16cyclictest7950irq/56-dwc_otg_19:25:021
1387999382,4cyclictest7850irq/56-dwc_otg21:10:141
13879993819,6cyclictest3514-21sshd23:59:371
13879993813,4cyclictest0-21swapper/122:00:141
1388199376,4cyclictest0-21swapper/300:37:503
1388199376,21cyclictest22201-21sh21:45:173
1388199373,19cyclictest8050irq/56-dwc_otg_23:10:193
1388099376,6cyclictest8050irq/56-dwc_otg_22:50:022
1388099371,25cyclictest12785-21ls00:10:222
1387999371,3cyclictest8050irq/56-dwc_otg_22:45:311
158242360,6sleep31388199cyclictest19:15:173
1388199368,6cyclictest7950irq/56-dwc_otg_00:00:163
1388199361,9cyclictest0-21swapper/321:40:103
1388199361,4cyclictest26666-21ssh21:50:213
13881993614,5cyclictest13605-21unixbench_multi00:10:333
1388099365,18cyclictest8050irq/56-dwc_otg_20:35:172
1388099364,27cyclictest0-21swapper/221:10:132
1388099362,4cyclictest18778-21ntp_kernel_pll_21:40:252
13880993616,6cyclictest8050irq/56-dwc_otg_19:15:022
1387999365,18cyclictest8050irq/56-dwc_otg_23:33:211
1387999363,23cyclictest0-21swapper/123:10:121
1387999361,6cyclictest20006-21ssh22:20:311
13879993616,5cyclictest7614-21sh23:25:131
1387999361,25cyclictest19886-21missed_timers19:25:211
13879993611,14cyclictest125-21systemd-journal20:51:231
128822360,4sleep20-21swapper/219:05:272
1388199357,15cyclictest0-21swapper/323:45:173
1388199353,7cyclictest8050irq/56-dwc_otg_22:35:133
13881993522,10cyclictest8050irq/56-dwc_otg_00:16:413
13881993521,8cyclictest2758-21/usr/sbin/munin22:40:223
13881993515,4cyclictest24964-21fschecks_count19:40:183
1388199351,23cyclictest15594-21ls23:35:033
1388199351,20cyclictest15732-21ls20:45:043
1388199351,20cyclictest15732-21ls20:45:043
1388099359,5cyclictest14855-21chrt22:55:032
1388099357,4cyclictest0-21swapper/223:23:082
1387999355,21cyclictest5402-21ssh21:25:171
13879993522,8cyclictest8050irq/56-dwc_otg_23:35:131
13879993522,8cyclictest8050irq/56-dwc_otg_23:35:131
13879993515,7cyclictest26657-21fschecks_time21:50:211
1387999351,32cyclictest3113-21uniq22:40:191
1388199346,13cyclictest8050irq/56-dwc_otg_20:30:253
1388199345,3cyclictest0-21swapper/300:05:143
1388199345,3cyclictest0-21swapper/300:05:143
1388199344,25cyclictest8050irq/56-dwc_otg_19:10:133
1388199341,4cyclictest8050irq/56-dwc_otg_22:15:293
1388199341,27cyclictest0-21swapper/322:47:393
1388099347,15cyclictest0-21swapper/220:50:162
1388099347,14cyclictest0-21swapper/219:25:132
1388099345,8cyclictest29-21rcuc/200:30:212
1388099345,8cyclictest29-21rcuc/200:30:212
13880993423,8cyclictest28406-21aten_rbpower_vo23:50:172
13880993414,5cyclictest9869-21sh21:30:182
1388099341,3cyclictest0-21swapper/221:35:262
1388099341,24cyclictest17463-21ls19:20:032
1388099341,23cyclictest21028-21uniq00:20:222
1388099341,16cyclictest18315-21ssh00:16:252
1387999349,5cyclictest8050irq/56-dwc_otg_00:20:181
1387999346,13cyclictest7850irq/56-dwc_otg20:25:171
13879993412,4cyclictest0-21swapper/121:36:321
1387999341,21cyclictest3827-21ssh21:22:571
13879993411,12cyclictest0-21swapper/123:02:101
13881993314,12cyclictest727-21runrttasks21:05:083
13881993314,12cyclictest727-21runrttasks21:05:083
1388199331,22cyclictest28838-21latency19:50:223
1388099336,24cyclictest0-21swapper/219:45:162
1388099334,6cyclictest7850irq/56-dwc_otg20:15:232
1388099331,7cyclictest9084-21taskset00:05:292
1388099331,7cyclictest9084-21taskset00:05:292
13880993312,10cyclictest0-21swapper/221:55:102
13880993312,10cyclictest0-21swapper/221:55:102
1387999337,15cyclictest0-21swapper/122:55:271
1387999337,15cyclictest0-21swapper/122:55:271
1387999333,7cyclictest7850irq/56-dwc_otg20:15:021
1387999332,7cyclictest7950irq/56-dwc_otg_19:35:141
1387999332,18cyclictest0-21swapper/120:20:201
1387999332,10cyclictest0-21swapper/119:30:171
13879993313,4cyclictest15689-21timerwakeupswit22:15:311
1387999331,13cyclictest7850irq/56-dwc_otg22:32:181
13879993311,12cyclictest0-21swapper/123:22:111
157922320,4sleep229-21rcuc/222:55:232
157922320,4sleep229-21rcuc/222:55:232
1388199327,15cyclictest0-21swapper/321:25:173
1388199322,14cyclictest8050irq/56-dwc_otg_22:35:023
1388199321,8cyclictest75050snmpd19:56:123
1388199321,8cyclictest75050snmpd19:56:123
13881993217,4cyclictest39-21ksoftirqd/321:15:233
13881993215,10cyclictest2197-21ssh23:57:113
1388199321,20cyclictest0-21swapper/323:24:233
13881993210,7cyclictest22822-1kworker/u9:200:30:153
13881993210,7cyclictest22822-1kworker/u9:200:30:153
1388099328,4cyclictest0-21swapper/223:37:442
1388099328,4cyclictest0-21swapper/223:37:442
1388099328,12cyclictest0-21swapper/222:20:262
1388099327,13cyclictest8050irq/56-dwc_otg_19:50:252
1388099326,7cyclictest31-21ksoftirqd/219:40:262
1388099323,14cyclictest7950irq/56-dwc_otg_21:52:422
1388099322,22cyclictest10136-21latency_hist20:30:032
1388099322,19cyclictest0-21swapper/222:32:402
1388099322,17cyclictest8050irq/56-dwc_otg_22:37:432
1388099321,20cyclictest16108-21grep20:45:192
1387999329,4cyclictest0-21swapper/100:12:351
1387999325,6cyclictest7850irq/56-dwc_otg00:05:171
1387999325,6cyclictest7850irq/56-dwc_otg00:05:171
1387999325,6cyclictest21301-21apt-get21:00:141
1387999323,12cyclictest7950irq/56-dwc_otg_21:16:181
1387999322,15cyclictest8050irq/56-dwc_otg_19:45:251
1387999322,13cyclictest7850irq/56-dwc_otg21:30:131
1387999321,5cyclictest16074-21sort20:45:181
13879993212,4cyclictest0-21swapper/121:46:141
13879993210,6cyclictest7850irq/56-dwc_otg23:15:041
1388199319,4cyclictest0-21swapper/323:25:053
1388199318,5cyclictest7950irq/56-dwc_otg_00:20:183
1388199316,12cyclictest0-21swapper/322:20:053
13881993115,5cyclictest17509-21/usr/sbin/munin19:20:203
13881993112,5cyclictest24671-21diskmemload22:57:433
13881993112,5cyclictest24671-21diskmemload22:57:433
1388199311,17cyclictest8050irq/56-dwc_otg_19:35:223
13881993110,4cyclictest0-21swapper/323:17:173
13881993110,4cyclictest0-21swapper/322:05:033
1388099313,4cyclictest8050irq/56-dwc_otg_20:10:262
1388099311,6cyclictest7950irq/56-dwc_otg_19:20:172
1388099311,10cyclictest3592-21rm21:22:322
1387999315,4cyclictest8050irq/56-dwc_otg_19:15:311
1387999314,7cyclictest7850irq/56-dwc_otg20:20:031
13879993116,5cyclictest727-21runrttasks21:08:511
13879993116,5cyclictest727-21runrttasks21:08:511
1388199308,3cyclictest0-21swapper/322:10:293
1388199307,3cyclictest0-21swapper/323:40:263
1388199306,6cyclictest8050irq/56-dwc_otg_22:00:193
1388199306,6cyclictest7850irq/56-dwc_otg20:35:203
1388199303,5cyclictest8050irq/56-dwc_otg_20:20:293
1388199302,18cyclictest0-21swapper/320:58:313
1388199302,12cyclictest8050irq/56-dwc_otg_20:45:183
1388199301,5cyclictest1193-21latency_hist20:05:023
1388199301,3cyclictest8050irq/56-dwc_otg_19:45:153
13881993010,10cyclictest0-21swapper/323:51:473
1388099309,4cyclictest0-21swapper/223:30:252
1388099308,5cyclictest25215-21ls23:45:302
1388099308,3cyclictest7950irq/56-dwc_otg_20:55:272
1388099304,16cyclictest727-21runrttasks00:03:242
13880993016,4cyclictest301ktimersoftd/220:31:552
13880993012,9cyclictest2823-21aten_rbpower_cu22:40:162
1388099301,20cyclictest14456-21ssh22:15:052
1388099301,15cyclictest8050irq/56-dwc_otg_00:35:202
1387999305,12cyclictest7850irq/56-dwc_otg22:05:231
1387999301,7cyclictest7950irq/56-dwc_otg_20:05:271
135272303,18sleep10-21swapper/119:09:001
1388199297,19cyclictest0-21swapper/320:10:153
1388199293,12cyclictest8050irq/56-dwc_otg_20:10:023
13880992922,4cyclictest20190-21/usr/sbin/munin23:00:272
1388199288,10cyclictest8050irq/56-dwc_otg_19:30:263
1388199282,3cyclictest0-21swapper/319:25:243
1388099286,4cyclictest8050irq/56-dwc_otg_20:40:172
1388099286,4cyclictest8050irq/56-dwc_otg_20:40:172
1388099281,15cyclictest0-21swapper/220:25:032
1388099281,14cyclictest8050irq/56-dwc_otg_22:25:132
1388099281,14cyclictest8050irq/56-dwc_otg_22:25:132
1387999287,12cyclictest0-21swapper/120:00:291
1388099262,4cyclictest0-21swapper/219:55:162
1388099262,4cyclictest0-21swapper/219:55:162
1388199257,15cyclictest0-21swapper/321:00:213
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional