You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-11 - 00:48

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Sun May 10, 2026 12:43:37)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5602880,3sleep0541-21ssh19:45:560
23397998527,48cyclictest0-21swapper/021:51:090
2339799851,29cyclictest21193-21grep17:51:240
2339799831,22cyclictest25598-21latency_hist20:15:570
2339799805,20cyclictest9229-21ssh19:56:110
23397998028,40cyclictest7950irq/56-dwc_otg_18:55:560
23397997921,5cyclictest3892-21cut17:06:110
2339799791,23cyclictest19527-21ntpq21:26:180
23405997829,23cyclictest21239-21ssh20:49:481
23397997826,5cyclictest7850irq/56-dwc_otg_21:01:110
23397997817,12cyclictest5603-21/usr/sbin/munin17:11:250
23397997726,5cyclictest7850irq/56-dwc_otg_20:55:340
2339799762,9cyclictest0-21swapper/021:21:060
23405997429,20cyclictest16753-21ssh20:44:091
23405997429,20cyclictest16753-21ssh20:44:091
2339799732,22cyclictest0-21swapper/019:07:070
2339799732,22cyclictest0-21swapper/019:07:070
23397997316,13cyclictest1764-21awk17:00:560
23405997125,27cyclictest19288-21ssh21:26:151
23397997125,4cyclictest7850irq/56-dwc_otg_19:35:570
23397997125,4cyclictest7850irq/56-dwc_otg_19:35:570
23397997117,5cyclictest7950irq/56-dwc_otg_18:16:160
23397997113,11cyclictest0-21swapper/020:36:130
2339799711,22cyclictest0-21swapper/019:36:150
2341399702,28cyclictest7850irq/56-dwc_otg_21:21:082
23405997029,23cyclictest4524-21/usr/sbin/munin17:06:201
23397997034,3cyclictest0-21swapper/020:56:080
23397997034,3cyclictest0-21swapper/020:56:080
2339799702,20cyclictest0-21swapper/021:45:570
23405996928,25cyclictest23928-21ssh20:13:141
23397996914,5cyclictest8645-21sshd19:55:350
2339799691,22cyclictest0-21swapper/019:15:570
23405996833,18cyclictest31455-21rm20:21:531
2339799681,7cyclictest2340-21cut21:06:110
23397996811,5cyclictest25577-21systemd-journal18:35:570
2339799675,6cyclictest7850irq/56-dwc_otg_20:35:570
2339799673,18cyclictest0-21swapper/017:46:140
2339799671,19cyclictest9263-21ntpq19:16:200
23405996630,18cyclictest23247-21ssh19:33:161
23405996630,18cyclictest23247-21ssh19:33:161
23405996627,20cyclictest12292-21ssh18:41:271
23405996627,20cyclictest12292-21ssh18:41:271
2339799666,4cyclictest28262-21grep21:36:260
2339799666,3cyclictest0-21swapper/017:01:130
23397996630,4cyclictest7850irq/56-dwc_otg_22:00:560
23397996624,4cyclictest0-21swapper/020:06:080
23397996624,4cyclictest0-21swapper/020:06:080
2339799661,20cyclictest7099-21ssh18:36:160
2340599657,28cyclictest0-21swapper/118:06:101
23405996536,9cyclictest7620-21ls21:51:181
23405996530,16cyclictest2018-21aten_rbpower_cu17:01:091
23405996529,14cyclictest6888-21latency_hist21:50:581
23405996525,14cyclictest20168-21grep17:51:111
2339799652,17cyclictest0-21swapper/020:16:170
2339799651,21cyclictest0-21swapper/021:12:450
2339799651,20cyclictest0-21swapper/018:56:170
23397996511,14cyclictest0-21swapper/019:50:060
23405996428,12cyclictest1968-21ssh19:07:351
23405996428,12cyclictest1968-21ssh19:07:351
23405996427,24cyclictest29170-21acpi18:16:091
23397996414,4cyclictest7750irq/56-dwc_otg20:21:140
2339799641,15cyclictest0-21swapper/021:48:230
96342630,1sleep20-21swapper/221:14:562
23405996328,13cyclictest16399-21latency_hist17:40:571
23405996323,13cyclictest21766-21ssh18:52:531
23405996322,21cyclictest732-21ssh21:03:471
23397996343,4cyclictest0-21swapper/016:36:020
2339799634,14cyclictest0-21swapper/019:26:230
23397996312,4cyclictest14933-21fschecks_time17:36:150
23405996233,20cyclictest8747-21ssh19:55:461
23405996233,12cyclictest28925-21ssh20:58:341
23405996233,12cyclictest28925-21ssh20:58:341
23405996232,21cyclictest8759-21df_inode19:16:131
23405996227,32cyclictest224650ssh21:45:261
23405996226,20cyclictest4147-21ssh19:49:561
2339799626,16cyclictest61-21kswapd020:26:080
2339799622,18cyclictest0-21swapper/018:44:500
2339799622,18cyclictest0-21swapper/018:44:500
2339799621,16cyclictest0-21swapper/020:49:320
23397996211,5cyclictest7850irq/56-dwc_otg_17:45:560
23405996129,19cyclictest2579250ssh20:55:231
23405996128,22cyclictest19404-21ssh20:07:331
23405996128,22cyclictest19404-21ssh20:07:331
23405996127,12cyclictest5831-21diskstats17:11:161
23405996127,11cyclictest17047-21ssh21:23:221
23405996118,20cyclictest7750irq/56-dwc_otg19:15:561
23397996122,5cyclictest0-21swapper/019:01:090
23397996122,4cyclictest7850irq/56-dwc_otg_16:44:480
2339799611,19cyclictest0-21swapper/021:32:130
2339799611,17cyclictest0-21swapper/018:50:130
23419996011,26cyclictest381ktimersoftd/321:56:073
23405996030,10cyclictest0-21swapper/119:36:071
23405996029,21cyclictest6640-21wget18:36:101
23405996023,14cyclictest12411-21grep21:56:241
2339799603,19cyclictest0-21swapper/021:20:490
2339799601,14cyclictest0-21swapper/020:45:560
2339799601,14cyclictest0-21swapper/020:45:560
23405995932,14cyclictest32273-21sh19:44:371
23405995932,12cyclictest17790-21sshd17:43:221
2339799591,18cyclictest0-21swapper/019:24:320
23405995828,19cyclictest4827-21ssh20:30:011
23405995826,20cyclictest1960-21diskmemload19:04:481
23405995825,13cyclictest14390-21latency_hist21:20:571
23405995825,12cyclictest14604-21unixbench_singl20:01:261
23405995821,19cyclictest2313850ssh21:31:101
23405995819,29cyclictest7850irq/56-dwc_otg_20:18:181
23405995727,19cyclictest908-21runrttasks21:09:461
23405995727,11cyclictest30523-21chrt16:50:221
23405995724,20cyclictest12781-21ssh20:39:261
23405995723,12cyclictest10341-21ls19:56:251
23405995721,13cyclictest0-21swapper/116:42:441
23397995729,8cyclictest7526-21wget17:16:110
23405995629,11cyclictest9325-21ssh20:35:421
23405995627,18cyclictest27034-21apt-get18:11:001
23405995626,13cyclictest32365-21taskset16:55:521
23405995626,12cyclictest14224-21ssh19:21:521
2339799563,33cyclictest22728-21switchtime17:56:240
23397995628,6cyclictest31063-21wget18:21:120
2339799561,20cyclictest0-21swapper/018:30:160
23405995531,8cyclictest9884-21ssh21:15:231
23405995531,12cyclictest4507-21ssh18:32:581
23405995525,19cyclictest18742-21sh18:50:041
23405995519,16cyclictest28989-21ssh21:37:321
2339799558,20cyclictest0-21swapper/016:46:530
2339799551,21cyclictest0-21swapper/017:22:000
2339799551,21cyclictest0-21swapper/017:22:000
23405995437,14cyclictest23469-21apt-get16:31:031
23405995437,14cyclictest23469-21apt-get16:31:031
23405995432,7cyclictest23895-21ssh18:56:101
23405995422,20cyclictest7750irq/56-dwc_otg19:29:181
2340599533,8cyclictest0-21swapper/117:16:071
23405995326,12cyclictest1635-21apt-config18:27:151
23405995323,16cyclictest12505-21runrttasks17:30:101
23405995320,20cyclictest10649-21seq17:24:311
23405995320,20cyclictest10649-21seq17:24:311
2339799531,11cyclictest0-21swapper/020:05:570
23405995229,14cyclictest25164-21sshd18:05:211
23405995224,11cyclictest19628-21taskset17:48:511
23405995221,15cyclictest14293-21sshd17:35:041
2341999512,38cyclictest7950irq/56-dwc_otg_20:16:103
2341399511,11cyclictest22894-21ssh21:31:052
2340599515,39cyclictest7750irq/56-dwc_otg17:56:091
2339799515,3cyclictest0-21swapper/018:06:100
2339799511,29cyclictest23309-21apt-key16:30:580
23405995025,10cyclictest32604-21chrt16:55:581
23405995018,14cyclictest27023-21cat16:40:561
2341999498,26cyclictest0-21swapper/317:16:103
23405994821,10cyclictest32401-21taskset18:24:251
2339799481,21cyclictest0-21swapper/018:01:220
2341999473,4cyclictest0-21swapper/316:46:083
229612473,12sleep00-21swapper/016:29:200
2341999467,24cyclictest0-21swapper/318:26:113
2341399466,25cyclictest0-21swapper/216:41:102
2339799461,27cyclictest27945-21wc18:11:170
2339799452,21cyclictest0-21swapper/016:35:580
2339799452,21cyclictest0-21swapper/016:35:580
2339799451,41cyclictest0-21swapper/017:26:450
2341399441,8cyclictest0-21swapper/220:46:092
23405994428,12cyclictest23412-21cut16:30:581
2341999435,22cyclictest14664-21ssh21:21:083
23413994319,4cyclictest29-21rcuc/220:01:202
23419994221,5cyclictest3085350wget16:51:103
23413994220,13cyclictest31-21ksoftirqd/221:51:132
23397994213,25cyclictest0-21swapper/017:31:110
217082420,2sleep30-21swapper/317:55:573
23413994122,4cyclictest7444-21wget17:16:102
23413994117,6cyclictest2393-21aten_rbpower_po18:31:092
2341399411,23cyclictest31-21ksoftirqd/217:05:572
2339799411,21cyclictest0-21swapper/016:51:200
252952400,1sleep20-21swapper/219:36:122
2341999408,26cyclictest0-21swapper/319:31:103
2341999408,26cyclictest0-21swapper/319:31:103
2341999408,22cyclictest0-21swapper/318:11:093
23413994025,5cyclictest2371850wget18:01:102
2341399401,5cyclictest19596-21ssh18:51:082
2341999397,4cyclictest0-21swapper/318:56:083
23419993919,9cyclictest7950irq/56-dwc_otg_18:41:123
23419993919,9cyclictest7950irq/56-dwc_otg_18:41:123
23419993914,12cyclictest0-21swapper/321:36:263
23419993912,6cyclictest0-21swapper/321:41:213
2341399396,12cyclictest31-21ksoftirqd/220:41:232
2341399396,12cyclictest31-21ksoftirqd/220:41:232
2341399391,9cyclictest0-21swapper/218:11:032
23413993916,6cyclictest29-21rcuc/220:21:222
23413993914,7cyclictest31-21ksoftirqd/221:00:572
23413993914,7cyclictest31-21ksoftirqd/221:00:572
2341999382,19cyclictest0-21swapper/319:30:353
23419993819,5cyclictest2247-21/usr/sbin/munin21:06:253
23419993818,4cyclictest2549850wget18:06:103
23419993817,5cyclictest9745-21sh20:36:033
23419993815,5cyclictest17714-21sh20:06:123
23419993815,5cyclictest17714-21sh20:06:123
23413993816,6cyclictest9935-21wget20:36:112
49602370,2sleep2301ktimersoftd/221:08:562
2341999375,20cyclictest6421-21ssh21:11:103
23419993715,4cyclictest8796-21fschecks_count19:16:153
23419993712,17cyclictest0-21swapper/319:13:193
2341399374,7cyclictest7750irq/56-dwc_otg16:31:082
2341399374,7cyclictest7750irq/56-dwc_otg16:31:082
2341399372,15cyclictest29-21rcuc/219:55:392
23413993718,4cyclictest2232-21sh19:46:272
215492376,17sleep221602-21aten_rbpower_en16:26:102
215492376,17sleep221602-21aten_rbpower_en16:26:102
2341999369,4cyclictest7850irq/56-dwc_otg_19:41:173
2341999367,24cyclictest0-21swapper/316:56:113
2341999367,13cyclictest7850irq/56-dwc_otg_17:56:263
2341999366,8cyclictest7950irq/56-dwc_otg_20:27:373
23419993623,8cyclictest7950irq/56-dwc_otg_19:06:073
23419993623,8cyclictest7950irq/56-dwc_otg_19:06:073
23419993612,17cyclictest0-21swapper/318:21:263
2341399362,3cyclictest7750irq/56-dwc_otg17:46:132
2341399362,12cyclictest7750irq/56-dwc_otg17:51:132
23413993612,9cyclictest31-21ksoftirqd/221:26:132
2341999354,11cyclictest7950irq/56-dwc_otg_19:01:243
2341999351,22cyclictest22781-21ls21:30:583
2341399359,13cyclictest0-21swapper/220:00:452
2341399357,10cyclictest29-21rcuc/217:29:462
2341399354,6cyclictest29-21rcuc/221:56:222
2341399352,4cyclictest0-21swapper/221:16:072
23413993519,5cyclictest29-21rcuc/218:41:232
23413993519,5cyclictest29-21rcuc/218:41:232
23413993512,4cyclictest29-21rcuc/219:01:242
282182340,2sleep30-21swapper/319:39:473
2341999347,12cyclictest7950irq/56-dwc_otg_19:51:113
2341999346,5cyclictest7750irq/56-dwc_otg19:21:233
2341999345,20cyclictest22235-21ssh20:51:023
2341999345,18cyclictest13959-21ssh20:41:093
2341999345,18cyclictest13959-21ssh20:41:093
23419993412,11cyclictest4221-21ssh21:46:293
2341999341,20cyclictest0-21swapper/316:36:023
2341399349,4cyclictest7850irq/56-dwc_otg_20:51:032
2341399348,13cyclictest7950irq/56-dwc_otg_16:46:142
2341399346,6cyclictest7750irq/56-dwc_otg21:50:572
2341399346,15cyclictest7850irq/56-dwc_otg_20:16:182
2341399341,7cyclictest7750irq/56-dwc_otg16:51:242
23413993415,5cyclictest29-21rcuc/219:16:062
23413993413,9cyclictest301ktimersoftd/218:56:142
23413993413,4cyclictest5424-21taskset17:10:572
2341399341,19cyclictest301ktimersoftd/221:42:432
2341999338,6cyclictest9298-21aten_rbpower_po17:21:103
2341999338,6cyclictest9298-21aten_rbpower_po17:21:103
2341999338,5cyclictest7750irq/56-dwc_otg21:51:233
2341999334,4cyclictest7750irq/56-dwc_otg19:56:153
2341999334,14cyclictest30861-21fschecks_count21:01:143
23419993325,6cyclictest6467-21rm18:35:583
2341999332,20cyclictest0-21swapper/318:47:543
2341399337,5cyclictest30135-21ssh19:41:222
2341399335,9cyclictest301ktimersoftd/221:01:222
2341399333,11cyclictest31-21ksoftirqd/219:27:102
2341399332,13cyclictest29-21rcuc/220:10:222
2341399332,13cyclictest29-21rcuc/220:10:222
23413993314,10cyclictest301ktimersoftd/218:24:322
23413993310,4cyclictest1960-21diskmemload18:46:232
2341999326,7cyclictest7850irq/56-dwc_otg_17:06:243
2341999326,14cyclictest0-21swapper/316:41:103
2341999325,12cyclictest7850irq/56-dwc_otg_20:56:143
2341999325,12cyclictest7850irq/56-dwc_otg_20:56:143
23419993213,4cyclictest381ktimersoftd/320:05:573
2341999321,26cyclictest22585-21switchtime20:11:213
2341999321,21cyclictest19866-21grep18:51:143
23419993210,13cyclictest0-21swapper/320:46:003
2341399328,11cyclictest7950irq/56-dwc_otg_17:11:202
2341399327,5cyclictest7750irq/56-dwc_otg20:11:092
2341399323,6cyclictest0-21swapper/217:31:102
2341399323,4cyclictest7850irq/56-dwc_otg_20:29:592
23413993210,12cyclictest29-21rcuc/218:38:312
23413993210,10cyclictest0-21swapper/219:33:222
23413993210,10cyclictest0-21swapper/219:33:222
2341999312,21cyclictest0-21swapper/318:32:473
23419993110,6cyclictest7850irq/56-dwc_otg_19:50:573
2341399313,15cyclictest7850irq/56-dwc_otg_17:21:172
2341399313,15cyclictest7850irq/56-dwc_otg_17:21:172
2341399312,4cyclictest7850irq/56-dwc_otg_18:06:072
2341399312,16cyclictest0-21swapper/219:23:452
2341399311,8cyclictest5163-21ssh19:11:212
2341399311,8cyclictest0-21swapper/218:27:142
2341399311,4cyclictest7750irq/56-dwc_otg16:36:182
2341999308,4cyclictest0-21swapper/321:31:163
2341999302,4cyclictest7850irq/56-dwc_otg_17:11:113
23419993021,6cyclictest18531-21fschecks_time17:46:143
2341999301,3cyclictest0-21swapper/320:31:133
2341399305,13cyclictest0-21swapper/220:31:102
2341399304,7cyclictest30871-21idleruntime-cro21:40:572
2341399303,8cyclictest7850irq/56-dwc_otg_17:41:252
23413993023,5cyclictest31-21ksoftirqd/219:06:122
23413993023,5cyclictest31-21ksoftirqd/219:06:122
2341399302,14cyclictest7850irq/56-dwc_otg_17:56:112
2341399301,18cyclictest0-21swapper/217:00:112
2341999297,20cyclictest29524-21ls18:16:163
2341999292,15cyclictest0-21swapper/316:31:133
2341999292,15cyclictest0-21swapper/316:31:133
2341999291,3cyclictest7950irq/56-dwc_otg_20:21:103
23419992912,5cyclictest13112-21fschecks_time17:31:143
2341399296,11cyclictest29-21rcuc/217:40:192
2341399292,4cyclictest0-21swapper/218:20:572
230502294,14sleep10-21swapper/116:30:261
2341999286,11cyclictest7950irq/56-dwc_otg_17:45:573
2341999281,6cyclictest7750irq/56-dwc_otg21:16:243
2341999273,4cyclictest2067-21cut17:01:103
2341999272,17cyclictest0-21swapper/318:02:313
227652263,14sleep30-21swapper/316:26:523
227652263,14sleep30-21swapper/316:26:523
2341999256,4cyclictest7950irq/56-dwc_otg_17:26:213
2341999242,12cyclictest7950irq/56-dwc_otg_17:38:093
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional