You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-06 - 21:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot3s.osadl.org (updated Wed May 06, 2026 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13843210200,6sleep20-21swapper/216:26:132
1520899883,49cyclictest0-21swapper/021:01:080
204692840,3sleep00-21swapper/020:20:550
15208998322,38cyclictest0-21swapper/016:51:080
134092823,7sleep00-21swapper/016:26:080
1520899814,64cyclictest0-21swapper/017:56:080
1520899814,64cyclictest0-21swapper/017:56:080
15208998021,35cyclictest7850irq/56-dwc_otg_19:51:200
15208997925,41cyclictest20655-21latency_hist18:15:550
1520899776,54cyclictest0-21swapper/017:46:060
1520899771,71cyclictest23755-21latency19:06:140
1520899761,20cyclictest13050-21ssh20:51:050
1520899761,13cyclictest12681-21rm18:52:300
1520899757,50cyclictest0-21swapper/019:31:180
1520899751,55cyclictest14868-21ssh21:31:190
168422740,2sleep20-21swapper/220:55:522
15208997223,32cyclictest7850irq/56-dwc_otg_20:00:170
15208997214,49cyclictest15816-21missed_timers16:31:160
7242710,3sleep337-21rcuc/318:38:363
7242710,3sleep337-21rcuc/318:38:363
15211997141,8cyclictest10002-21apt-get17:46:063
15208997127,30cyclictest7750irq/56-dwc_otg20:45:560
15208997127,30cyclictest7750irq/56-dwc_otg20:45:560
1520899712,6cyclictest8953-21aten_rbpower_cu20:46:070
1520899712,20cyclictest0-21swapper/018:43:220
1520899711,20cyclictest24639-21sh20:25:570
281142690,5sleep226001-21diskmemload21:08:202
15211996927,26cyclictest22625-21/usr/sbin/munin18:21:223
1520899697,49cyclictest0-21swapper/018:46:160
1520899697,33cyclictest0-21swapper/018:21:080
1520899682,6cyclictest30296-21ssh21:50:540
1520899681,60cyclictest0-21swapper/017:10:550
1520899681,12cyclictest0-21swapper/020:34:420
1520899681,11cyclictest0-21swapper/018:38:100
1520899681,11cyclictest0-21swapper/018:38:100
282162670,5sleep31521199cyclictest18:32:203
15211996725,25cyclictest24900-21/usr/sbin/munin19:46:183
1520899678,5cyclictest12402-21/usr/sbin/munin17:51:160
15208996713,11cyclictest32185-21taskset17:16:160
15208996710,13cyclictest25142-21grep19:46:200
15211996624,20cyclictest8941-21ntpq20:06:193
15208996612,12cyclictest0-21swapper/018:35:270
1520899661,16cyclictest0-21swapper/018:16:140
15211996526,19cyclictest31370-21ssh19:54:513
1520899654,7cyclictest9450irq/48-DMA09:12:250
1520899654,15cyclictest25577-21systemd-journal17:04:440
1520899651,60cyclictest19149-21ssh19:40:170
1520899651,21cyclictest0-21swapper/019:15:020
15208996512,11cyclictest396-21chrt21:14:260
15208996512,11cyclictest396-21chrt21:14:260
15208996512,10cyclictest4936-21ssh21:20:050
1520899651,15cyclictest917-21snmpd21:45:410
43562640,1sleep30-21swapper/320:01:133
15211996429,11cyclictest8902-21apt-get20:46:063
15211996420,24cyclictest7950irq/56-dwc_otg_17:16:143
1520899642,59cyclictest0-21swapper/019:17:320
1520899642,59cyclictest0-21swapper/019:17:320
15208996418,7cyclictest7850irq/56-dwc_otg_21:56:340
15208996411,6cyclictest7750irq/56-dwc_otg21:51:200
1520899641,16cyclictest0-21swapper/021:06:130
15211996324,13cyclictest483650ssh21:19:543
15208996316,37cyclictest0-21swapper/020:06:220
1520899631,15cyclictest0-21swapper/017:11:140
1520899622,30cyclictest7356-21ssh19:26:040
1520899621,5cyclictest0-21swapper/019:45:560
1520899621,13cyclictest0-21swapper/019:24:410
15211996131,17cyclictest12001-21/usr/sbin/munin17:51:093
15211996125,23cyclictest787-21taskset21:53:433
15211996123,11cyclictest8289-21smartpower19:26:193
15211996122,19cyclictest19138-21ssh19:40:163
1520899619,10cyclictest6531-21ssh21:21:190
1520899611,15cyclictest0-21swapper/017:26:570
15211996030,17cyclictest27801-21users21:46:233
1521199602,15cyclictest0-21swapper/320:56:073
1520899603,18cyclictest0-21swapper/019:03:490
1520899601,17cyclictest12263-21/usr/sbin/munin20:11:230
1520899601,16cyclictest0-21swapper/020:57:180
15208996010,2cyclictest0-21swapper/018:56:230
15211995932,11cyclictest25782-21aten_rbpower_en21:06:093
15211995927,8cyclictest24400-21munin-run20:25:543
15211995925,14cyclictest7750irq/56-dwc_otg21:31:163
15211995920,25cyclictest14911-21rm19:35:113
1520899591,19cyclictest0-21swapper/021:36:550
15211995831,11cyclictest15224-21/usr/sbin/munin16:31:143
15211995828,8cyclictest29723-21aten_rbpower_en17:11:083
15211995827,12cyclictest6918-21rm20:42:543
15211995827,12cyclictest6918-21rm20:42:543
15211995824,14cyclictest8193-21ssh21:23:283
15208995819,4cyclictest4011-21ssh20:40:100
1520899581,14cyclictest0-21swapper/020:21:010
15211995732,8cyclictest23206-21ssh19:06:063
15211995727,18cyclictest2517-21ssh19:20:163
15211995727,18cyclictest2517-21ssh19:20:163
15211995726,11cyclictest2267-21taskset17:24:073
15211995726,11cyclictest2267-21taskset17:24:073
15211995724,12cyclictest23763-21latency_hist19:45:553
15211995724,12cyclictest16186-21sh20:54:413
15211995722,22cyclictest3663-21/usr/sbin/munin18:41:233
1520899571,16cyclictest6277-21sh20:03:070
1520899571,15cyclictest13084-21ssh21:29:490
15211995626,17cyclictest3961-21ssh20:40:053
15211995625,19cyclictest17443-21ssh18:58:383
15211995618,20cyclictest0-21swapper/321:11:533
15211995618,20cyclictest0-21swapper/321:11:533
15211995617,25cyclictest27624-21chrt17:05:543
1520899564,12cyclictest0-21swapper/016:57:410
1520899561,12cyclictest0-21swapper/016:47:050
261282550,2sleep126123-21ssh21:06:131
15211995522,19cyclictest21925-21taskset16:47:183
15211995515,22cyclictest0-21swapper/318:16:103
1520899558,7cyclictest0-21swapper/017:37:390
1520899554,18cyclictest60532chrt17:35:010
15208995512,8cyclictest0-21swapper/016:36:380
15208995512,8cyclictest0-21swapper/016:36:380
1520899551,14cyclictest0-21swapper/018:04:560
15211995429,13cyclictest19826-21ssh20:20:083
15211995420,11cyclictest21279-21ssh21:39:353
15211995418,24cyclictest908-21runrttasks16:52:383
1520899541,31cyclictest26158-21latency_hist18:30:550
1520899541,15cyclictest0-21swapper/016:42:100
243062530,2sleep124308-21fschecks_time19:46:101
15211995328,12cyclictest25266-21ssh21:05:423
15211995324,16cyclictest16692-21taskset18:02:293
15211995322,13cyclictest3878-21ssh21:56:453
15211995321,10cyclictest7750irq/56-dwc_otg20:28:453
15211995320,14cyclictest29327-21rm19:12:343
15211995319,11cyclictest7850irq/56-dwc_otg_17:10:553
15211995317,23cyclictest26001-21diskmemload20:14:293
15211995316,22cyclictest7950irq/56-dwc_otg_18:49:273
15211995228,11cyclictest1646-21ssh19:57:163
15211995223,11cyclictest4205-21taskset17:29:313
15211995219,12cyclictest32066-21ssh20:34:483
15211995218,9cyclictest7950irq/56-dwc_otg_21:42:523
1520999521,11cyclictest0-21swapper/121:51:051
15211995126,15cyclictest14005-21ssh18:54:563
15211995124,16cyclictest12304-21ssh21:28:243
15211995123,11cyclictest5469-21ssh19:23:053
15211995119,11cyclictest0-21swapper/317:37:393
15211995116,20cyclictest0-21swapper/319:01:443
241012500,2sleep20-21swapper/219:46:072
15211995018,16cyclictest7950irq/56-dwc_otg_16:44:453
1520899481,23cyclictest0-21swapper/017:21:360
1520899481,23cyclictest0-21swapper/017:21:360
1520899481,16cyclictest9732-21runrttasks17:45:440
15211994732,13cyclictest18709-21sed16:40:553
15211994732,13cyclictest18709-21sed16:40:553
15211994722,11cyclictest25577-21systemd-journal18:07:393
15211994715,18cyclictest0-21swapper/318:30:083
1520999473,12cyclictest0-21swapper/118:51:061
1521099464,6cyclictest7750irq/56-dwc_otg18:51:052
1521199457,3cyclictest0-21swapper/317:56:083
1521199457,3cyclictest0-21swapper/317:56:083
15211994429,12cyclictest8885-21ntp_offset17:41:173
15211994419,16cyclictest908-21runrttasks18:13:033
1520999445,8cyclictest5741-21ssh21:21:061
1520999444,7cyclictest7850irq/56-dwc_otg_17:11:051
15209994411,9cyclictest7950irq/56-dwc_otg_16:50:541
15211994328,12cyclictest24682-21if_eth016:56:123
1520999432,7cyclictest7750irq/56-dwc_otg20:11:051
15209994316,5cyclictest25834-21idleruntime-cro17:00:541
206412420,2sleep10-21swapper/121:38:261
15211994229,10cyclictest5945-21sshd17:33:393
1520999429,5cyclictest6176-21latency_hist22:00:551
53042410,2sleep121-21rcuc/119:22:461
317912410,5sleep231-21ksoftirqd/218:36:222
317912410,5sleep231-21ksoftirqd/218:36:222
1521099412,10cyclictest7850irq/56-dwc_otg_20:11:052
15210994118,4cyclictest17280-21aten_rbpower_en18:06:072
15209994121,9cyclictest7850irq/56-dwc_otg_19:26:101
15210994017,7cyclictest20768-21wget16:46:082
1520999407,15cyclictest7950irq/56-dwc_otg_21:11:081
1520999407,15cyclictest7950irq/56-dwc_otg_21:11:081
142592400,2sleep10-21swapper/121:31:091
15209993922,4cyclictest16984-21munin-plugin-st18:05:541
1521099382,4cyclictest7850irq/56-dwc_otg_17:11:062
15210993819,10cyclictest25086-21sh21:05:232
1520999382,13cyclictest1025-21ssh19:56:221
15209993819,6cyclictest17652-21fschecks_time20:56:131
15209993818,12cyclictest4730-21/usr/sbin/munin20:41:121
15209993818,12cyclictest4730-21/usr/sbin/munin20:41:121
15209993817,10cyclictest26054-21vmstat20:26:231
1520999381,6cyclictest17711-21unixbench_singl20:16:221
1521099377,24cyclictest0-21swapper/219:56:082
15210993720,5cyclictest1386-21latency_hist21:15:562
15210993720,5cyclictest1386-21latency_hist21:15:562
15210993718,5cyclictest29427-21sh21:49:192
15210993716,14cyclictest24320-21aten_rbpower_en16:56:072
1520999378,6cyclictest7750irq/56-dwc_otg16:31:231
1520999372,7cyclictest7950irq/56-dwc_otg_21:16:061
1520999372,25cyclictest16115-21rm20:54:331
15209993717,9cyclictest4556-21/usr/sbin/munin17:31:071
15209993715,14cyclictest14944-21ssh19:35:141
1521099362,6cyclictest0-21swapper/219:30:542
1521099361,6cyclictest7950irq/56-dwc_otg_20:56:062
15210993616,8cyclictest5642-21/usr/sbin/munin17:31:202
1521099361,23cyclictest17985-21timerandwakeup16:36:232
1521099361,23cyclictest17985-21timerandwakeup16:36:232
1520999369,4cyclictest7850irq/56-dwc_otg_18:50:051
15209993617,5cyclictest31424-21sh19:54:581
15209993616,5cyclictest10708-21sh21:26:191
15209993615,4cyclictest21-21rcuc/121:00:571
15209993614,6cyclictest7950irq/56-dwc_otg_17:46:141
15209993614,4cyclictest22534-21latency_hist18:20:551
305142350,3sleep231-21ksoftirqd/219:53:192
1521099352,7cyclictest31332-21latency_hist19:15:552
15210993519,7cyclictest7850irq/56-dwc_otg_20:31:102
1521099351,19cyclictest28706-21ls20:30:552
1520999358,14cyclictest0-21swapper/120:36:231
1520999356,18cyclictest23-21ksoftirqd/118:31:081
1520999355,6cyclictest2141-21latency_hist18:40:551
1520999355,6cyclictest2141-21latency_hist18:40:551
1520999355,5cyclictest7750irq/56-dwc_otg20:06:081
1520999352,19cyclictest27614-21ssh19:11:101
15209993513,6cyclictest0-21swapper/119:01:061
15209993513,4cyclictest29331-21sh21:49:081
15209993513,4cyclictest221ktimersoftd/117:56:141
15209993513,4cyclictest221ktimersoftd/117:56:141
1521099347,5cyclictest19659-21ssh19:40:552
1521099346,11cyclictest0-21swapper/218:56:212
1521099344,6cyclictest7850irq/56-dwc_otg_18:01:122
1521099343,11cyclictest7950irq/56-dwc_otg_21:31:132
15210993421,6cyclictest22267-21ssh21:40:572
1521099341,22cyclictest8342-21cpuspeed20:06:102
1520999343,17cyclictest7950irq/56-dwc_otg_16:36:131
1520999343,17cyclictest7950irq/56-dwc_otg_16:36:131
1520999342,8cyclictest22338-21/usr/sbin/munin21:41:171
15209993414,5cyclictest16780-21sh19:36:231
15209993413,5cyclictest178962ntp_kernel_pll_18:06:161
15209993412,4cyclictest0-21swapper/120:22:091
15209993412,4cyclictest0-21swapper/120:02:131
1520999341,16cyclictest27643-21latency_hist17:05:551
15209993410,5cyclictest15637-21sh18:56:171
147972344,21sleep30-21swapper/316:29:073
272102330,5sleep227212-21unixbench_singl17:01:222
1521099339,11cyclictest31850-21grep21:51:222
1521099337,6cyclictest7950irq/56-dwc_otg_16:31:232
1521099336,16cyclictest0-21swapper/218:48:172
1521099332,4cyclictest11149-21unixbench_singl21:26:232
15210993322,4cyclictest11508-21apt-get19:31:052
15210993320,4cyclictest4889-21cut20:41:092
15210993320,4cyclictest4889-21cut20:41:092
1521099331,7cyclictest6409-21aten_rbpower_cu17:36:082
15210993316,10cyclictest4969-21sh21:20:092
1521099331,15cyclictest7950irq/56-dwc_otg_17:41:232
1520999339,14cyclictest0-21swapper/118:41:411
1520999336,11cyclictest7950irq/56-dwc_otg_17:51:231
1520999335,6cyclictest23-21ksoftirqd/118:30:551
1520999331,5cyclictest0-21swapper/118:11:051
1521099327,15cyclictest29-21rcuc/219:21:112
1521099322,6cyclictest32625-21head17:16:222
15210993216,5cyclictest19951-21/usr/sbin/munin19:01:192
15210993216,5cyclictest1123-21kernelversion20:36:142
1521099321,17cyclictest7950irq/56-dwc_otg_17:46:122
1520999329,5cyclictest24097-21ssh19:06:201
1520999326,11cyclictest7950irq/56-dwc_otg_20:31:121
1520999324,9cyclictest21-21rcuc/119:41:211
1520999322,7cyclictest32287-21ntp_states19:16:171
1520999322,7cyclictest32287-21ntp_states19:16:171
15209993213,4cyclictest3597-21swap17:26:191
1521099317,5cyclictest7850irq/56-dwc_otg_17:51:232
1521099317,17cyclictest0-21swapper/221:37:412
1521099317,16cyclictest0-21swapper/220:50:052
1521099316,16cyclictest7950irq/56-dwc_otg_19:16:062
1521099316,16cyclictest7950irq/56-dwc_otg_19:16:062
15210993111,13cyclictest14130-21latency17:56:152
15210993111,13cyclictest14130-21latency17:56:152
15210993110,4cyclictest0-21swapper/216:51:222
1520999317,10cyclictest7950irq/56-dwc_otg_16:55:541
1520999316,5cyclictest8948-21proc_pri17:41:191
1520999312,6cyclictest0-21swapper/117:16:111
1520999311,15cyclictest0-21swapper/120:48:521
149372314,18sleep114966-21hostname16:30:551
1521099306,15cyclictest0-21swapper/220:21:072
1521099305,11cyclictest0-21swapper/221:20:562
1521099305,10cyclictest0-21swapper/218:42:452
1521099304,8cyclictest28165-21grep17:06:112
15210993024,4cyclictest2424-21idleruntime-cro17:25:552
15210993024,4cyclictest2424-21idleruntime-cro17:25:552
1521099302,13cyclictest0-21swapper/218:31:062
1521099301,7cyclictest624250ssh22:00:552
1521099301,5cyclictest0-21swapper/219:41:422
1521099301,4cyclictest0-21swapper/220:16:122
1520999307,9cyclictest7950irq/56-dwc_otg_18:21:191
1520999301,5cyclictest6697-21fschecks_time17:36:121
1521099292,6cyclictest22401-21apt18:20:542
15210992921,6cyclictest26008-21sh18:30:542
1521099291,24cyclictest4400-21ssh20:01:142
15210992910,9cyclictest23197-21apt-get19:06:082
1520999291,18cyclictest917-21snmpd17:09:451
1520999291,11cyclictest1130-21fschecks_count17:21:111
1520999291,11cyclictest1130-21fschecks_count17:21:111
1521099289,4cyclictest20412-21sh16:45:542
1521099282,5cyclictest7850irq/56-dwc_otg_17:26:012
1521099282,5cyclictest7750irq/56-dwc_otg18:11:052
1520999282,4cyclictest0-21swapper/116:41:221
1521099275,12cyclictest7950irq/56-dwc_otg_18:21:172
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional