You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-10-16 - 06:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Fri Oct 16, 2026 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
24790995733,16cyclictest0-21swapper/200:05:142
24801995442,9cyclictest33366irq/58-eth022:50:143
2479099532,39cyclictest26960-21ssh22:30:182
2479099530,43cyclictest0-21swapper/220:15:132
24790994822,20cyclictest30565-21ssh22:50:142
2479099452,35cyclictest3450-21ssh22:35:122
24757994534,8cyclictest27910-21apt-get20:20:130
24757994434,6cyclictest8591-21apt-get20:00:130
24757994425,13cyclictest21694-21apt-get00:15:130
2475799441,34cyclictest6330-21ssh21:25:130
2475799441,34cyclictest6330-21ssh21:25:120
2480199423,29cyclictest33366irq/58-eth022:45:123
24768994231,5cyclictest1768-21ssh23:10:121
24757994214,19cyclictest12-21ksoftirqd/023:35:140
24757994128,10cyclictest30554-21apt-get22:50:150
24757994125,11cyclictest3677-21apt-get19:55:120
24757994120,15cyclictest0-21swapper/000:25:120
24757994111,10cyclictest12539-21ntp_192.168.11422:20:280
24790994028,4cyclictest18009-21ssh23:55:142
24757994030,6cyclictest10115-21apt-get21:45:120
24757994012,10cyclictest27381-21if_eth100:00:180
24757993926,9cyclictest14064-21apt-get22:05:130
24757993923,10cyclictest0-21swapper/022:55:140
24757993914,11cyclictest28983-21users20:50:320
24757993911,11cyclictest5678-21ssh00:23:340
2480199382,21cyclictest33366irq/58-eth000:05:123
2476899385,20cyclictest32402-21ssh23:45:101
2476899382,9cyclictest21-21ksoftirqd/120:30:111
2475799389,10cyclictest13840-21ssh23:34:480
24757993814,12cyclictest7997-21ssh21:42:550
24757993812,9cyclictest6269-21ssh22:17:260
24757993812,11cyclictest21239-21rm21:31:420
24757993812,10cyclictest11126-21diskmemload22:39:590
24757993810,8cyclictest0-21swapper/021:10:310
2480199373,18cyclictest33366irq/58-eth000:20:143
2475799379,10cyclictest22753-21users22:25:310
24757993725,9cyclictest0-21swapper/020:35:120
24757993714,10cyclictest23050-21/usr/sbin/munin20:45:230
24757993713,13cyclictest1409-21tr20:25:180
2475799371,25cyclictest32415-21sh23:45:120
24757993712,10cyclictest3469-21ssh23:28:090
24757993711,11cyclictest22930-21ssh00:32:400
2475799369,8cyclictest28473-21ntp_states19:10:340
2475799369,8cyclictest28473-21ntp_states19:10:330
2475799369,10cyclictest25577-21rm21:15:400
24757993628,4cyclictest8953-21apt-get23:50:120
24757993623,8cyclictest21487-21apt-get22:45:120
24757993622,10cyclictest0-21swapper/020:55:100
24757993617,15cyclictest0-21swapper/000:10:160
24757993613,8cyclictest1047-21ssh00:39:300
24757993612,8cyclictest27223-21ssh21:54:170
24757993611,10cyclictest29549-21rm21:35:490
24757993610,8cyclictest0-21swapper/022:02:340
24757993610,11cyclictest30522-21rm23:43:120
2480199351,20cyclictest33366irq/58-eth022:25:123
2479099354,20cyclictest22346irq/40-dwc2_hso20:55:102
2479099351,31cyclictest8957-21ssh23:50:122
2475799358,11cyclictest28888-21ssh21:55:190
2475799358,11cyclictest22872-21date20:15:020
2475799358,11cyclictest2236-21cut21:00:200
2475799358,10cyclictest4832-21ssh21:23:440
24757993512,10cyclictest5890-21threads19:20:270
24757993511,8cyclictest9993-21ssh00:07:190
24757993511,10cyclictest31745-21irqcore19:50:180
24757993510,9cyclictest27371-21tr19:45:170
2480199344,19cyclictest33366irq/58-eth023:50:143
2480199342,18cyclictest33366irq/58-eth021:40:153
2479099343,19cyclictest22346irq/40-dwc2_hso00:35:132
2475799349,8cyclictest6932-21cut21:05:170
2475799348,8cyclictest22328-21ssh23:02:000
2475799347,11cyclictest16935-21ssh23:17:010
24757993427,3cyclictest21638-21apt-get19:40:110
24757993418,13cyclictest0-21swapper/022:30:170
24757993413,8cyclictest19414-21ssh22:42:590
24757993413,8cyclictest19414-21ssh22:42:580
24757993413,7cyclictest11126-21diskmemload23:21:080
24757993411,8cyclictest23084-21ssh22:10:110
24757993410,8cyclictest21954-21ssh23:55:290
24757993410,8cyclictest16891-21ntp_states20:05:240
2480199332,15cyclictest33366irq/58-eth022:35:123
2479099331,25cyclictest16394-21sh00:30:192
24757993312,7cyclictest11126-21diskmemload23:12:140
24757993310,10cyclictest5407-21if_eth020:30:170
2479099323,19cyclictest22346irq/40-dwc2_hso20:50:142
2479099322,17cyclictest22346irq/40-dwc2_hso23:10:112
2475799329,10cyclictest611-21latency19:15:360
2475799323,7cyclictest131rcu_preempt19:30:100
24757993210,12cyclictest11126-21diskmemload23:05:130
2480199312,16cyclictest33366irq/58-eth020:45:113
2479099313,18cyclictest22346irq/40-dwc2_hso21:00:152
2479099313,18cyclictest22346irq/40-dwc2_hso20:25:132
2479099311,18cyclictest12721-21ssh00:10:162
2476899312,22cyclictest30776-21ssh00:20:151
24757993114,14cyclictest0-21swapper/019:25:130
2479099303,19cyclictest22346irq/40-dwc2_hso20:10:122
24768993015,10cyclictest0-21swapper/100:25:141
24768993014,11cyclictest0-21swapper/120:20:131
24757993016,10cyclictest14886-21df_inode20:40:150
2479099292,20cyclictest0-21swapper/221:20:172
2479099292,18cyclictest22346irq/40-dwc2_hso23:00:142
2479099292,11cyclictest32228-21ssh22:15:142
2476899292,19cyclictest31153-21ssh00:36:411
2476899291,21cyclictest11540-21ssh21:10:131
2479099282,21cyclictest0-21swapper/220:40:142
2479099282,19cyclictest0-21swapper/220:45:122
24790992818,6cyclictest27-21ksoftirqd/200:15:122
24790992816,8cyclictest27-21ksoftirqd/220:00:122
2476899281,20cyclictest21719-21ssh00:15:141
2480199272,17cyclictest33366irq/58-eth023:55:133
24801992715,8cyclictest33366irq/58-eth000:35:133
2479099277,7cyclictest17910-21sh22:25:142
2479099272,20cyclictest0-21swapper/200:23:432
2479099271,14cyclictest12447-21ssh22:40:102
2479099271,14cyclictest12447-21ssh22:40:092
2476899279,4cyclictest21-21ksoftirqd/121:55:451
24768992712,9cyclictest21-21ksoftirqd/122:25:591
2476899271,12cyclictest0-21swapper/100:30:271
24768992710,9cyclictest21-21ksoftirqd/122:36:251
2475799278,10cyclictest21493-21latency_hist19:40:010
2475799278,10cyclictest21493-21latency_hist19:40:010
2480199262,17cyclictest0-21swapper/321:20:163
2480199262,14cyclictest33366irq/58-eth022:20:153
2479099264,15cyclictest22346irq/40-dwc2_hso20:35:132
2479099263,17cyclictest22346irq/40-dwc2_hso20:20:132
2479099263,14cyclictest22346irq/40-dwc2_hso21:35:112
2479099262,8cyclictest0-21swapper/219:25:122
2479099262,18cyclictest7118-21ssh22:55:142
2479099261,18cyclictest32431-21ssh23:45:132
2476899263,4cyclictest131rcu_preempt19:40:091
2476899262,3cyclictest131rcu_preempt23:05:291
24768992613,4cyclictest0-21swapper/122:14:451
24768992610,7cyclictest21-21ksoftirqd/122:47:261
2479099254,13cyclictest22346irq/40-dwc2_hso19:20:142
24790992513,4cyclictest7225-21/usr/sbin/munin00:25:192
2479099251,12cyclictest25219-21rm23:05:142
24790992510,7cyclictest22717-21cron22:10:022
2476899259,7cyclictest21-21ksoftirqd/123:03:531
2476899258,7cyclictest21-21ksoftirqd/123:39:351
2476899252,5cyclictest21-21ksoftirqd/120:15:111
24768992512,4cyclictest21-21ksoftirqd/100:02:391
24768992511,8cyclictest21-21ksoftirqd/119:50:021
2475799253,9cyclictest24619-21cut20:15:180
2480199243,14cyclictest0-21swapper/300:15:143
2479099244,16cyclictest22346irq/40-dwc2_hso21:15:132
2479099241,9cyclictest14014-21latency_hist23:35:012
2479099241,5cyclictest31521-21ls19:50:162
2479099241,20cyclictest4539-21sendmail-msp22:00:012
2479099241,19cyclictest22346irq/40-dwc2_hso21:40:152
2479099241,14cyclictest24640-21tr22:45:222
2479099241,14cyclictest10788-21ssh23:15:132
2476899249,8cyclictest21-21ksoftirqd/123:19:341
2476899249,5cyclictest21-21ksoftirqd/121:17:131
2476899249,12cyclictest131rcu_preempt19:20:111
2476899248,7cyclictest21-21ksoftirqd/121:29:041
2476899248,7cyclictest21-21ksoftirqd/121:29:031
2476899245,8cyclictest21-21ksoftirqd/121:05:221
2476899244,3cyclictest131rcu_preempt23:30:341
24768992412,4cyclictest21-21ksoftirqd/121:40:281
2476899241,19cyclictest29681-21apt-get21:20:141
2476899241,16cyclictest4928-21ssh22:00:131
2476899241,12cyclictest10962-21kthreadcore21:45:191
24768992410,7cyclictest21-21ksoftirqd/120:40:201
24768992410,5cyclictest21-21ksoftirqd/122:18:561
24768992410,4cyclictest21-21ksoftirqd/100:12:001
24768992410,2cyclictest131rcu_preempt23:23:161
2480199235,11cyclictest33366irq/58-eth023:10:113
2480199233,13cyclictest0-21swapper/300:32:413
2480199233,11cyclictest0-21swapper/322:30:163
2479099239,6cyclictest11123-21cron21:10:002
24790992310,11cyclictest166-21systemd-network21:30:302
2476899237,9cyclictest21-21ksoftirqd/122:54:221
2476899237,7cyclictest21-21ksoftirqd/121:31:091
2476899237,7cyclictest21-21ksoftirqd/100:05:571
24768992311,7cyclictest0-21swapper/123:43:161
2476899231,16cyclictest8966-21sh23:50:141
24768992311,4cyclictest21-21ksoftirqd/121:00:021
2476899231,13cyclictest14067-21ssh22:05:131
24768992310,7cyclictest21-21ksoftirqd/120:50:021
24768992310,4cyclictest21-21ksoftirqd/122:41:381
24768992310,4cyclictest21-21ksoftirqd/122:41:371
24768992310,4cyclictest21-21ksoftirqd/122:22:091
2480199223,13cyclictest33366irq/58-eth022:40:113
2480199223,13cyclictest33366irq/58-eth022:40:103
2479099229,10cyclictest25410-21kthreadcore23:40:212
2479099227,12cyclictest26973-21apt00:00:102
2479099226,13cyclictest20017-21grep23:20:142
2479099224,15cyclictest14210-21kthreadcore19:30:192
2479099222,10cyclictest9073-21fschecks_time22:20:172
2479099221,18cyclictest15276-21cut23:35:202
2479099221,17cyclictest25120-21fschecks_count19:10:162
2479099221,17cyclictest25120-21fschecks_count19:10:152
2479099221,17cyclictest14702-21ssh21:10:242
24790992211,4cyclictest15007-21latency_hist21:30:022
24790992211,4cyclictest15007-21latency_hist21:30:012
2479099221,11cyclictest5279-21if_eth122:00:162
2479099221,10cyclictest32012-21latency_hist22:15:012
2479099221,10cyclictest20279-21kthreadcore21:50:202
2476899229,7cyclictest21-21ksoftirqd/121:35:261
2476899229,4cyclictest21-21ksoftirqd/119:55:311
2476899228,4cyclictest21-21ksoftirqd/123:55:401
2476899228,4cyclictest21-21ksoftirqd/122:55:591
2476899226,4cyclictest21-21ksoftirqd/121:54:031
2476899226,3cyclictest131rcu_preempt19:15:381
2476899223,6cyclictest21-21ksoftirqd/119:25:171
2476899221,19cyclictest28823-21ssh23:25:111
2476899220,6cyclictest30977-21/usr/sbin/munin22:30:321
24757992210,9cyclictest24765-21sed19:10:020
24757992210,9cyclictest24765-21sed19:10:010
2480199215,11cyclictest0-21swapper/300:29:423
2480199215,10cyclictest33366irq/58-eth021:15:153
2480199213,10cyclictest33366irq/58-eth023:40:133
2480199212,8cyclictest0-21swapper/319:40:103
2479099214,12cyclictest22346irq/40-dwc2_hso21:45:122
2479099213,15cyclictest22346irq/40-dwc2_hso19:55:132
2479099212,17cyclictest28660-21kthreadcore19:45:192
2479099211,9cyclictest5333-21cut20:30:152
2479099211,16cyclictest13729-21ps20:05:182
2479099211,11cyclictest22783-21kthreadcore19:40:172
2479099211,10cyclictest621-21latency19:15:362
2479099211,10cyclictest29590-21kthreadcore23:25:192
2476899214,3cyclictest131rcu_preempt19:15:011
2476899213,5cyclictest21-21ksoftirqd/120:00:021
2476899213,15cyclictest439-21fschecks_time20:25:161
2476899212,11cyclictest21-21ksoftirqd/120:35:231
24801992010,8cyclictest0-21swapper/322:02:443
2479099201,9cyclictest24844-21cat19:10:022
2479099201,9cyclictest24844-21cat19:10:012
2479099201,10cyclictest20157-21missed_timers19:35:232
2479099201,10cyclictest20157-21missed_timers19:35:222
2476899208,4cyclictest21-21ksoftirqd/119:30:151
2476899203,8cyclictest131rcu_preempt19:35:221
2476899203,8cyclictest131rcu_preempt19:35:211
2476899202,15cyclictest18375-21fschecks_count20:10:161
2480199198,6cyclictest0-21swapper/321:56:173
2480199197,9cyclictest0-21swapper/321:47:073
2480199194,11cyclictest0-21swapper/300:10:153
2480199192,11cyclictest0-21swapper/320:55:113
2480199192,10cyclictest0-21swapper/319:25:063
2480199191,13cyclictest0-21swapper/322:05:133
2480199191,13cyclictest0-21swapper/321:05:133
2480199191,12cyclictest0-21swapper/323:30:223
24801991910,6cyclictest33366irq/58-eth022:55:133
2476899198,4cyclictest21-21ksoftirqd/120:50:281
2476899191,5cyclictest54246usb-storage19:10:051
2476899191,3cyclictest131rcu_preempt19:50:171
2476899190,10cyclictest1809-21tune2fs21:00:161
2480199189,4cyclictest0-21swapper/323:46:013
2480199182,11cyclictest0-21swapper/320:35:133
2480199182,10cyclictest0-21swapper/323:00:153
2480199182,10cyclictest0-21swapper/320:50:143
2480199182,10cyclictest0-21swapper/319:55:133
2480199181,13cyclictest0-21swapper/300:00:303
24801991810,6cyclictest0-21swapper/321:36:153
24768991812,4cyclictest0-21swapper/120:05:141
2480199179,6cyclictest33366irq/58-eth023:15:133
2480199179,6cyclictest0-21swapper/322:17:593
2480199179,6cyclictest0-21swapper/321:32:033
2480199179,6cyclictest0-21swapper/321:14:093
2480199179,3cyclictest0-21swapper/323:36:503
2480199179,3cyclictest0-21swapper/323:27:443
2480199179,3cyclictest0-21swapper/323:07:553
2480199179,3cyclictest0-21swapper/321:50:283
2480199178,6cyclictest33366irq/58-eth023:20:113
2480199178,6cyclictest0-21swapper/322:10:133
2480199172,9cyclictest0-21swapper/320:40:143
2480199172,10cyclictest0-21swapper/320:20:143
2480199172,10cyclictest0-21swapper/320:10:143
2480199172,10cyclictest0-21swapper/320:00:053
2480199172,10cyclictest0-21swapper/319:30:113
24801991711,4cyclictest0-21swapper/321:25:223
24801991711,4cyclictest0-21swapper/321:25:213
2480199171,13cyclictest0-21swapper/320:20:013
2480199163,9cyclictest0-21swapper/319:45:103
2480199162,10cyclictest0-21swapper/319:20:133
2480199162,10cyclictest0-21swapper/319:10:143
2480199162,10cyclictest0-21swapper/319:10:133
2480199161,9cyclictest0-21swapper/319:15:173
2480199161,12cyclictest0-21swapper/320:25:133
2480199161,10cyclictest0-21swapper/321:00:183
2476899141,10cyclictest24740-21cyclictest19:10:021
2476899141,10cyclictest24740-21cyclictest19:10:011
2480199138,3cyclictest0-21swapper/319:50:233
2480199135,6cyclictest0-21swapper/320:05:183
2480199128,3cyclictest0-21swapper/320:30:183
2480199111,7cyclictest0-21swapper/319:35:183
2480199111,7cyclictest0-21swapper/319:35:173
2480199101,6cyclictest0-21swapper/319:10:023
2480199101,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional