You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-04-19 - 16:49

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Sun Apr 19, 2026 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1017922770,1sleep10-21swapper/116:26:021
3032521660,5sleep31173299cyclictest21:13:503
11729991203,83cyclictest0-21swapper/018:20:580
11729991203,83cyclictest0-21swapper/018:20:580
1879521000,2sleep30-21swapper/316:50:463
1172999981,81cyclictest14179-21/usr/sbin/munin18:06:070
1172999911,77cyclictest3792-21idleruntime-cro22:00:450
1172999911,68cyclictest0-21swapper/017:40:590
1172999911,68cyclictest0-21swapper/017:40:590
1172999901,79cyclictest0-21swapper/019:20:560
1172999891,66cyclictest31951-21fschecks_count17:26:040
1172999861,61cyclictest0-21swapper/020:20:580
1172999842,67cyclictest0-21swapper/021:00:580
11729998326,29cyclictest6600-21fschecks_count20:06:030
1172999831,77cyclictest19713-21rm21:40:460
1172999831,77cyclictest19713-21rm21:40:460
1172999831,74cyclictest0-21swapper/021:40:590
1172999831,71cyclictest29503-21ssh21:51:150
38182820,5sleep21173199cyclictest18:46:062
166862810,5sleep21173199cyclictest20:17:412
22252800,3sleep1221ktimersoftd/117:31:131
1172999801,31cyclictest17358-21unixbench_singl19:01:140
1172999791,67cyclictest3735-21grep21:21:040
1172999751,65cyclictest0-21swapper/018:52:500
11729997411,15cyclictest132850kworker/u9:221:18:230
11729997411,15cyclictest132850kworker/u9:221:18:230
1172999738,20cyclictest0-21swapper/019:17:290
11729997312,10cyclictest0-21swapper/017:30:560
95932720,4sleep231-21ksoftirqd/220:48:412
1172999728,21cyclictest28922-21runrttasks20:32:210
1172999722,59cyclictest0-21swapper/017:13:030
1172999722,12cyclictest0-21swapper/017:15:590
1172999721,32cyclictest1730-21ssh20:00:460
11729997117,5cyclictest25072sleep020:40:460
11729997031,30cyclictest7850irq/56-dwc_otg_18:30:580
1172999702,55cyclictest11434-21latency_hist21:30:460
1172999702,55cyclictest0-21swapper/017:05:460
1172999701,3cyclictest0-21swapper/020:20:460
1172999701,14cyclictest0-21swapper/019:36:090
1172999701,14cyclictest0-21swapper/019:36:090
326402690,5sleep332639-21/usr/sbin/munin21:16:123
326402690,5sleep332639-21/usr/sbin/munin21:16:123
11729996923,12cyclictest0-21swapper/016:45:570
11729996921,4cyclictest23771-21/usr/sbin/munin20:26:110
121842680,1sleep20-21swapper/218:01:012
11729996821,11cyclictest0-21swapper/019:46:360
1172999681,44cyclictest13012-21grep18:01:120
1172999681,21cyclictest0-21swapper/017:47:210
1172999681,16cyclictest3350-21wget18:46:000
1172999679,24cyclictest0-21swapper/020:45:580
1172999676,24cyclictest0-21swapper/021:48:120
1172999672,52cyclictest23843-21fschecks_time21:06:040
1172999672,51cyclictest0-21swapper/016:51:070
1172999671,56cyclictest31139-21cron18:40:460
11729996616,9cyclictest7850irq/56-dwc_otg_18:55:580
11729996616,3cyclictest0-21swapper/020:40:580
1172999661,24cyclictest0-21swapper/020:00:550
1172999661,18cyclictest0-21swapper/019:05:540
1172999661,17cyclictest18778-21timerandwakeup19:41:130
55812650,2sleep10-21swapper/120:05:041
141562650,3sleep014154-21ssh20:54:270
11729996517,4cyclictest7850irq/56-dwc_otg_17:21:130
1172999651,54cyclictest0-21swapper/018:25:570
1172999651,44cyclictest30060-21idleruntime-cro19:55:450
1173199643,10cyclictest7950irq/56-dwc_otg_19:00:562
11729996449,4cyclictest0-21swapper/016:55:510
1172999642,48cyclictest8531-21fschecks_count17:51:050
1172999641,4cyclictest11074-21memory20:11:070
11729996412,7cyclictest0-21swapper/019:29:200
1172999641,24cyclictest0-21swapper/018:44:370
1172999641,24cyclictest0-21swapper/018:44:370
11729996318,7cyclictest0-21swapper/017:36:440
1172999631,50cyclictest0-21swapper/019:31:030
1173199622,4cyclictest0-21swapper/219:26:142
1172999622,15cyclictest0-21swapper/021:15:190
1172999621,28cyclictest0-21swapper/017:07:510
1172999621,16cyclictest0-21swapper/016:36:520
1173199611,5cyclictest3939-21latency_hist22:00:462
1172999611,52cyclictest0-21swapper/018:16:010
1173199604,4cyclictest7850irq/56-dwc_otg_17:21:122
1173199601,5cyclictest25577-21systemd-journal18:55:002
1172999609,8cyclictest0-21swapper/019:12:320
1173299593,12cyclictest0-21swapper/320:51:003
1173199591,5cyclictest31949-21/usr/sbin/munin17:26:042
1172999591,32cyclictest0-21swapper/018:14:430
1173199586,3cyclictest0-21swapper/218:30:582
1173199585,4cyclictest0-21swapper/221:43:222
1173199583,4cyclictest15952-21wget21:36:022
1173199583,4cyclictest15952-21wget21:36:022
1173199581,3cyclictest0-21swapper/216:46:122
196382570,6sleep21173199cyclictest20:21:112
1173199572,9cyclictest3688-21fschecks_count21:21:032
1173199572,4cyclictest2167-21/usr/sbin/munin20:00:562
1173199571,5cyclictest18646-21taskset19:41:112
1173199571,4cyclictest14207-21/usr/sbin/munin18:06:082
11730995713,13cyclictest7750irq/56-dwc_otg20:45:591
1173099571,13cyclictest7750irq/56-dwc_otg17:50:591
11729995711,31cyclictest22596-21diskmemload21:31:400
1173199563,4cyclictest7850irq/56-dwc_otg_19:20:572
11731995616,5cyclictest22032-21aten_rbpower_cu19:46:012
1173099568,30cyclictest7750irq/56-dwc_otg18:20:581
1173099568,30cyclictest7750irq/56-dwc_otg18:20:581
1173099567,16cyclictest7750irq/56-dwc_otg16:40:561
1173099565,9cyclictest0-21swapper/120:25:571
1173199551,5cyclictest0-21swapper/219:55:112
11731995515,5cyclictest20618-21latency_hist16:55:462
11731995515,4cyclictest908-21runrttasks19:06:052
1172999552,42cyclictest16530-21head16:41:130
1173199547,4cyclictest7850irq/56-dwc_otg_20:41:012
1173199544,3cyclictest7950irq/56-dwc_otg_20:05:582
1173199541,3cyclictest0-21swapper/218:39:072
11731995413,7cyclictest12970-21ssh18:56:132
11731995411,7cyclictest31703-21ssh18:41:002
11731995411,7cyclictest31703-21ssh18:41:002
11731995410,7cyclictest25075-21if_eth019:11:042
1173099544,10cyclictest0-21swapper/116:35:561
1173199537,29cyclictest0-21swapper/216:55:512
1173199532,3cyclictest7950irq/56-dwc_otg_19:38:212
1173199532,3cyclictest7950irq/56-dwc_otg_19:38:212
1173199531,4cyclictest0-21swapper/219:56:002
1173199531,4cyclictest0-21swapper/217:15:592
1173199531,3cyclictest0-21swapper/220:57:402
11731995310,5cyclictest12887-21unixbench_multi21:31:142
1173199523,5cyclictest0-21swapper/220:26:112
1173199521,5cyclictest10701-21taskset21:29:372
1173199521,4cyclictest7950irq/56-dwc_otg_17:15:462
1173099526,12cyclictest7850irq/56-dwc_otg_21:10:571
1173099523,6cyclictest0-21swapper/120:50:591
1173299517,8cyclictest7950irq/56-dwc_otg_19:35:583
1173299517,8cyclictest7950irq/56-dwc_otg_19:35:583
1173199511,6cyclictest7750irq/56-dwc_otg20:51:022
1173199511,5cyclictest7950irq/56-dwc_otg_21:15:472
1173199511,4cyclictest19575-21ssh21:01:012
1173099516,11cyclictest3302-21ssh18:45:571
1173199504,8cyclictest7850irq/56-dwc_otg_17:30:562
1173199504,5cyclictest22596-21diskmemload21:09:072
1173199502,6cyclictest9450irq/48-DMA09:12:252
1173199501,4cyclictest7950irq/56-dwc_otg_18:00:462
1173199501,4cyclictest0-21swapper/221:53:092
1173099504,7cyclictest7750irq/56-dwc_otg20:15:571
11729995039,5cyclictest7850irq/56-dwc_otg_21:00:270
1173199491,5cyclictest0-21swapper/220:33:102
1173099496,29cyclictest28901-21sh19:16:001
1173099493,17cyclictest0-21swapper/121:56:011
1173199481,5cyclictest0-21swapper/220:11:032
1173099484,19cyclictest11307-21rm18:55:281
11730994826,4cyclictest10184-21aten_rbpower_po17:56:001
11730994818,6cyclictest7750irq/56-dwc_otg19:01:111
1172999489,20cyclictest0-21swapper/018:00:410
1173199472,10cyclictest29511-21memory19:16:082
1173099477,25cyclictest0-21swapper/118:25:511
1173199461,6cyclictest0-21swapper/221:50:462
1173099467,24cyclictest0-21swapper/117:00:591
1173099464,8cyclictest0-21swapper/117:45:591
11730994621,14cyclictest23-21ksoftirqd/121:45:581
1173299456,7cyclictest0-21swapper/320:20:563
1173299454,24cyclictest6949-21ssh20:45:583
11732994525,14cyclictest908-21runrttasks16:30:593
1173099458,23cyclictest0-21swapper/116:55:591
11730994524,8cyclictest7950irq/56-dwc_otg_19:55:271
11730994516,12cyclictest19623-21apt-get21:40:521
1173199447,24cyclictest0-21swapper/217:06:002
1173199445,25cyclictest0-21swapper/217:45:592
1173099449,4cyclictest28571-21chrt18:36:181
1173099447,24cyclictest0-21swapper/118:06:011
1173099447,23cyclictest0-21swapper/116:30:591
11730994422,7cyclictest7950irq/56-dwc_otg_18:30:481
11730994417,5cyclictest12104-21fschecks_count18:56:021
11730994412,23cyclictest0-21swapper/121:00:591
113632443,34sleep00-21swapper/016:29:380
1173299437,24cyclictest0-21swapper/319:26:003
1173299437,22cyclictest0-21swapper/318:25:593
1173199437,21cyclictest15898-21ls16:41:052
1173099438,10cyclictest0-21swapper/117:06:001
1173099437,25cyclictest0-21swapper/119:56:001
1173099437,22cyclictest0-21swapper/119:36:001
1173099437,22cyclictest0-21swapper/119:36:001
1173099434,10cyclictest20363-21ssh19:05:591
11730994320,14cyclictest0-21swapper/121:36:011
11730994320,14cyclictest0-21swapper/121:36:011
11730994316,15cyclictest7750irq/56-dwc_otg20:09:191
11730994315,15cyclictest7950irq/56-dwc_otg_19:32:061
11730994312,15cyclictest7950irq/56-dwc_otg_19:41:071
11730994311,20cyclictest14035-21rm20:15:251
1173199426,22cyclictest0-21swapper/218:25:522
1173099427,23cyclictest0-21swapper/120:41:011
1173099423,18cyclictest3339-21ssh19:23:341
11730994225,8cyclictest7750irq/56-dwc_otg17:35:571
11730994218,13cyclictest221ktimersoftd/121:09:471
225112410,4sleep122512-21latency_hist20:25:451
1173299416,25cyclictest0-21swapper/317:55:593
1173099416,23cyclictest0-21swapper/117:10:591
11730994125,4cyclictest7950irq/56-dwc_otg_19:12:461
11730994121,9cyclictest649-21rm18:41:461
11730994121,9cyclictest649-21rm18:41:461
11730994120,9cyclictest7950irq/56-dwc_otg_19:46:311
1173099411,6cyclictest15475-21munin-run21:35:461
11730994113,12cyclictest221ktimersoftd/120:40:471
109592410,3sleep337-21rcuc/319:32:003
1173299401,25cyclictest3525-21ls18:46:013
1173199407,5cyclictest0-21swapper/221:15:572
1173199407,5cyclictest0-21swapper/221:15:572
1173199401,10cyclictest0-21swapper/220:37:512
1173099406,19cyclictest6593-21rm19:26:491
11730994015,13cyclictest7950irq/56-dwc_otg_21:26:431
11730994015,13cyclictest7850irq/56-dwc_otg_17:26:041
11730994013,11cyclictest7950irq/56-dwc_otg_21:20:581
1173299397,18cyclictest7950irq/56-dwc_otg_19:19:313
1173099399,16cyclictest29065-21chrt20:32:381
1173099398,12cyclictest7950irq/56-dwc_otg_17:21:071
1173299388,15cyclictest7950irq/56-dwc_otg_21:51:143
1173299386,22cyclictest0-21swapper/318:10:593
11732993820,5cyclictest39-21ksoftirqd/320:06:143
1173199386,18cyclictest0-21swapper/216:35:592
1172999381,7cyclictest12322-21missed_timers16:31:060
1173299377,14cyclictest7850irq/56-dwc_otg_20:36:043
1173299376,5cyclictest7750irq/56-dwc_otg21:55:593
1173299375,5cyclictest7750irq/56-dwc_otg17:11:143
1173199371,24cyclictest12082-21cut16:31:032
1173099375,5cyclictest0-21swapper/118:10:571
11730993719,4cyclictest7950irq/56-dwc_otg_18:00:551
50112361,2sleep30-21swapper/320:42:513
257582350,2sleep30-21swapper/318:33:513
1173299354,25cyclictest7950irq/56-dwc_otg_17:45:573
11732993517,9cyclictest17640-21sh19:40:503
11732993512,4cyclictest0-21swapper/321:31:143
1173099358,8cyclictest23-21ksoftirqd/117:44:521
1173099358,8cyclictest23-21ksoftirqd/117:44:521
1173099355,17cyclictest1565-21rm21:18:011
1173099355,17cyclictest1565-21rm21:18:011
1173099352,6cyclictest28344-21ssh21:50:591
1173099351,24cyclictest17526-21ls16:46:021
1173299346,5cyclictest7750irq/56-dwc_otg19:21:113
1173299342,6cyclictest2550650ssh21:07:333
1173199341,7cyclictest7950irq/56-dwc_otg_17:44:412
1173199341,7cyclictest7950irq/56-dwc_otg_17:44:412
1173099347,7cyclictest23-21ksoftirqd/118:17:161
1173099345,5cyclictest7850irq/56-dwc_otg_20:56:081
1173299339,15cyclictest0-21swapper/318:16:033
1173299334,4cyclictest7750irq/56-dwc_otg19:56:043
1173299333,5cyclictest7750irq/56-dwc_otg20:01:103
1173299332,15cyclictest7850irq/56-dwc_otg_17:21:073
1173299331,7cyclictest0-21swapper/319:51:063
1173299331,5cyclictest7750irq/56-dwc_otg20:16:013
1173299331,4cyclictest0-21swapper/319:06:043
1173299331,3cyclictest0-21swapper/320:31:003
1173199331,6cyclictest9519-21tune2fs19:31:032
1173099335,7cyclictest7950irq/56-dwc_otg_17:20:461
1173299327,15cyclictest0-21swapper/318:55:593
1173299326,14cyclictest0-21swapper/317:40:593
1173299326,14cyclictest0-21swapper/317:40:593
1173299325,6cyclictest7750irq/56-dwc_otg21:41:143
1173299323,16cyclictest7950irq/56-dwc_otg_18:36:033
11732993223,6cyclictest381ktimersoftd/319:10:583
11732993217,5cyclictest19466-21aten_rbpower_vo21:01:003
1173299321,5cyclictest7950irq/56-dwc_otg_21:35:593
1173299321,5cyclictest7950irq/56-dwc_otg_21:35:593
1173299321,24cyclictest15862-21missed_timers20:56:083
1173299321,22cyclictest8662-21timerwakeupswit21:26:133
1173299321,22cyclictest15966-21missed_timers16:41:063
11731993224,5cyclictest19237-21wget18:21:002
11731993224,5cyclictest19237-21wget18:21:002
1173299316,5cyclictest7950irq/56-dwc_otg_18:21:143
1173299316,5cyclictest7950irq/56-dwc_otg_18:21:143
1173299313,8cyclictest15504-21kworker/u8:019:01:063
1173299312,4cyclictest22827-21sshd20:25:563
11732993110,4cyclictest0-21swapper/319:49:333
1173199311,5cyclictest17526-21df_inode18:16:012
1173199311,23cyclictest9249-21timerandwakeup17:51:142
11730993111,3cyclictest0-21swapper/116:55:461
1173299307,12cyclictest7950irq/56-dwc_otg_18:51:143
11732993015,4cyclictest39-21ksoftirqd/318:41:543
11732993015,4cyclictest39-21ksoftirqd/318:41:543
1173199306,14cyclictest0-21swapper/217:35:582
1173299292,6cyclictest0-21swapper/321:45:583
1173299292,19cyclictest0-21swapper/321:21:183
1173299292,18cyclictest0-21swapper/317:36:573
1173299292,11cyclictest29731-21cat17:20:463
11732992916,4cyclictest13889-21seq20:15:093
1173199292,6cyclictest24249-21munin-run17:05:452
1173299281,9cyclictest0-21swapper/317:35:083
1173299281,6cyclictest7750irq/56-dwc_otg17:06:013
1173299281,19cyclictest0-21swapper/317:53:273
1173299274,4cyclictest7750irq/56-dwc_otg16:40:463
1173299273,4cyclictest7750irq/56-dwc_otg16:56:033
1173299272,5cyclictest0-21swapper/317:01:113
111172274,15sleep20-21swapper/216:26:242
1173299266,18cyclictest815-21latency_hist17:30:463
1173299261,12cyclictest7850irq/56-dwc_otg_16:51:103
1173299251,7cyclictest11969-21aten_rbpower_en18:00:583
101182253,11sleep30-21swapper/316:26:013
1173299241,5cyclictest13722-21apt-get18:06:003
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional