You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-25 - 02:37

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Wed Mar 25, 2026 00:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
262199714,37cyclictest6666-21ssh21:20:122
259999682,20cyclictest12-21ksoftirqd/023:45:120
259999645,9cyclictest12-21ksoftirqd/021:25:120
2629995944,10cyclictest28366irq/58-eth023:05:133
262199553,30cyclictest12191-21ssh22:35:142
262199515,31cyclictest20617-21diskmemload22:05:142
259999512,42cyclictest1251-21ssh21:35:130
259999503,37cyclictest7363-21ssh00:20:140
262999485,26cyclictest28366irq/58-eth023:10:123
259999481,36cyclictest30096-21ssh21:15:130
259999455,12cyclictest121ksoftirqd/023:10:120
262999424,17cyclictest28366irq/58-eth000:10:143
262199412,31cyclictest10032-21ssh00:02:242
2621994122,12cyclictest261rcuc/223:00:132
2621994016,17cyclictest27-21ksoftirqd/223:55:152
259999402,4cyclictest131rcu_preempt00:15:150
262199372,18cyclictest8328-21sed21:55:232
262199372,18cyclictest29695-21kthreadcore22:25:232
2613993729,3cyclictest29273-21apt-get20:45:141
2613993729,3cyclictest29273-21apt-get20:45:131
259999371,28cyclictest1506-21ssh23:05:130
262999363,16cyclictest0-21swapper/322:16:223
262199361,19cyclictest15752-21ssh21:25:142
261399362,30cyclictest26532-21ssh22:25:161
262999342,28cyclictest28366irq/58-eth021:15:113
262199342,26cyclictest17034-21ssh23:49:232
2613993423,7cyclictest28070-21apt-get21:50:121
262999332,7cyclictest0-21swapper/322:05:143
262199332,26cyclictest22003-21ssh22:04:232
2621993311,16cyclictest27-21ksoftirqd/219:40:122
2613993316,8cyclictest17522-21apt-get22:20:111
259999339,7cyclictest12-21ksoftirqd/019:25:130
262199325,17cyclictest22346irq/40-dwc2_hso00:35:132
262199323,23cyclictest10562-21ssh23:10:132
262199322,25cyclictest14134-21ssh22:16:222
262199322,20cyclictest16475-21ssh00:25:152
2621993217,9cyclictest0-21swapper/222:10:142
2613993222,7cyclictest19654-21apt-get20:35:131
261399321,24cyclictest2061-21ssh00:35:141
261399321,20cyclictest0-21swapper/120:30:131
259999323,23cyclictest3721-21ssh00:00:120
262199316,8cyclictest27-21ksoftirqd/223:15:122
262199311,18cyclictest23474-21ssh23:35:162
2613993123,2cyclictest30079-21apt-get21:15:111
261399312,21cyclictest25456-21ssh00:30:121
261399311,24cyclictest27101-21ssh23:55:131
259999311,23cyclictest28901-21ssh23:20:130
262199304,19cyclictest30167-21ssh22:45:132
262199303,19cyclictest22346irq/40-dwc2_hso20:15:102
262199303,16cyclictest22346irq/40-dwc2_hso19:15:142
259999307,18cyclictest151rcuc/023:25:140
259999303,19cyclictest13445-21sshd20:25:240
262999298,11cyclictest28366irq/58-eth000:35:143
262999294,14cyclictest28366irq/58-eth022:30:123
262999294,14cyclictest28366irq/58-eth022:30:123
262199291,19cyclictest22346irq/40-dwc2_hso23:25:162
261399298,5cyclictest32434-21apt-get23:40:141
261399295,11cyclictest22572-21irqrtprio19:30:171
261399294,11cyclictest8307-21ssh23:07:491
261399294,10cyclictest11359-21wc19:15:271
2613992922,3cyclictest2815-21apt-get19:10:131
261399292,22cyclictest4105-21ssh22:47:231
261399292,11cyclictest5264-21switchtime20:50:271
2613992920,4cyclictest30758-21apt-get00:15:131
2613992914,11cyclictest24871-21apt-get23:00:131
259999292,19cyclictest839-21munin-node00:25:140
259999292,17cyclictest28082-21ssh21:50:130
262999281,12cyclictest0-21swapper/322:50:153
262199289,12cyclictest0-21swapper/221:00:132
262199284,8cyclictest21023-21ssh21:10:132
262199283,16cyclictest22346irq/40-dwc2_hso20:35:132
262199282,21cyclictest24729-21ssh22:23:232
261399285,7cyclictest20961-21kthreadcore23:15:181
261399284,8cyclictest14004-21ssh23:29:591
261399282,15cyclictest22629-21ssh21:45:251
2613992815,9cyclictest1232-21apt-get21:35:141
261399281,10cyclictest29964-21latency_hist22:45:021
261399281,10cyclictest29964-21latency_hist22:45:011
259999282,20cyclictest28879-21ssh22:06:500
262999278,4cyclictest28366irq/58-eth000:30:013
262999278,4cyclictest0-21swapper/322:56:433
262999273,11cyclictest28366irq/58-eth021:55:133
262999272,3cyclictest0-21swapper/321:40:143
2629992720,4cyclictest28366irq/58-eth023:30:133
262999271,5cyclictest0-21swapper/300:15:303
262199274,16cyclictest22346irq/40-dwc2_hso20:50:132
262199271,12cyclictest1816966ssh22:55:212
262199271,10cyclictest5445-21wc20:20:162
261399279,11cyclictest24717-21apt-get21:30:141
261399275,7cyclictest16788-21cat21:25:201
261399275,4cyclictest32666-21irqstats22:10:191
261399272,9cyclictest19555-21ssh22:01:301
261399272,19cyclictest13949-21ssh22:53:221
261399272,19cyclictest11195-21ssh21:57:221
261399272,11cyclictest20405-21find00:25:301
261399272,10cyclictest14399-21cut19:20:201
261399272,10cyclictest14399-21cut19:20:201
2613992719,4cyclictest341-21apt-get20:15:101
2613992718,5cyclictest3694-21apt-get00:00:111
261399271,10cyclictest12076-21grep23:10:201
261399270,12cyclictest7457-21df00:20:151
261399270,12cyclictest24009-21ssh22:59:331
261399270,11cyclictest24818-21processes21:10:291
261399270,10cyclictest1549-21ssh23:20:491
259999273,5cyclictest131rcu_preempt23:50:130
259999270,2cyclictest4748-21ssh21:55:140
262999264,3cyclictest0-21swapper/321:10:143
262999262,14cyclictest0-21swapper/322:25:153
262999261,6cyclictest0-21swapper/323:55:233
262999261,6cyclictest0-21swapper/321:35:023
262999261,5cyclictest0-21swapper/323:50:013
262999261,5cyclictest0-21swapper/322:45:213
262199266,15cyclictest22346irq/40-dwc2_hso21:15:132
262199266,10cyclictest21871-21cat19:30:012
262199263,15cyclictest22346irq/40-dwc2_hso21:50:122
262199263,15cyclictest22346irq/40-dwc2_hso19:30:132
262199262,15cyclictest0-21swapper/219:55:132
262199262,15cyclictest0-21swapper/219:55:122
2621992618,5cyclictest21540-21ssh00:10:022
262199261,15cyclictest27851-21ls21:50:022
2621992611,11cyclictest4645-21kthreadcore22:30:212
2621992611,11cyclictest4645-21kthreadcore22:30:202
261399266,4cyclictest25206-21sed00:10:241
261399264,6cyclictest12512-21kthreadcore21:40:211
261399264,10cyclictest30269-21kthreadcore20:10:231
261399263,8cyclictest25240-21ssh23:53:171
261399263,18cyclictest14451-21ssh23:30:151
261399262,8cyclictest17044-21ntp_192.168.11419:55:241
261399262,8cyclictest17044-21ntp_192.168.11419:55:231
2599992616,5cyclictest151rcuc/019:40:120
262999258,4cyclictest0-21swapper/322:21:433
262999256,4cyclictest28366irq/58-eth023:25:333
262999256,15cyclictest28366irq/58-eth021:50:133
262999254,13cyclictest0-21swapper/321:20:123
262999253,16cyclictest0-21swapper/322:01:503
262999252,19cyclictest28366irq/58-eth000:00:103
262999251,7cyclictest0-21swapper/323:50:163
262999251,7cyclictest0-21swapper/319:10:293
262999251,6cyclictest0-21swapper/321:25:143
262999251,5cyclictest0-21swapper/322:10:233
262999251,3cyclictest0-21swapper/323:35:223
262199253,14cyclictest22346irq/40-dwc2_hso20:05:132
262199252,12cyclictest1530-21ssh23:05:152
2621992513,5cyclictest2264-21ssh23:21:372
261399259,11cyclictest21-21ksoftirqd/122:16:231
261399255,6cyclictest27283-21cut19:35:191
261399255,6cyclictest14950-21sed21:00:291
261399255,6cyclictest11217-21ssh22:34:261
261399255,6cyclictest11217-21ssh22:34:251
261399253,10cyclictest1066-21kthreadcore19:40:211
261399251,10cyclictest15125-21kthreadcore22:35:221
261399250,9cyclictest17993-21cut19:25:191
261399250,11cyclictest16332-21ntpq00:05:271
259999253,12cyclictest151rcuc/019:45:120
259999252,18cyclictest630-21tcf-agent00:30:140
259999252,18cyclictest24898-21ssh23:00:130
259999251,3cyclictest131rcu_preempt22:00:120
2599992513,9cyclictest12-21ksoftirqd/000:05:120
2599992513,3cyclictest16657-21kthreadcore19:55:200
2599992513,3cyclictest16657-21kthreadcore19:55:200
262999245,13cyclictest0-21swapper/300:05:143
262999243,17cyclictest28366irq/58-eth019:45:123
262999243,14cyclictest28366irq/58-eth023:20:133
262999241,7cyclictest0-21swapper/321:35:163
262999241,6cyclictest0-21swapper/321:45:173
262999241,6cyclictest0-21swapper/320:30:013
262999241,6cyclictest0-21swapper/320:10:023
262999241,6cyclictest0-21swapper/319:30:013
262999241,10cyclictest0-21swapper/322:35:313
262199242,17cyclictest22346irq/40-dwc2_hso21:30:142
262199242,15cyclictest9991-21kworker/2:020:30:142
262199241,4cyclictest10114-21ssh21:40:132
2621992412,9cyclictest29156-21ssh00:13:342
2621992410,5cyclictest7322-21/usr/sbin/munin00:20:262
261399245,5cyclictest12614-21ssh23:45:251
261399244,6cyclictest23571-21irqcore20:05:171
261399242,8cyclictest20617-21diskmemload22:07:231
2613992420,2cyclictest23448-21apt-get23:35:131
2613992412,8cyclictest10010-21apt-get20:25:131
261399241,17cyclictest6658-21ssh21:20:121
261399240,10cyclictest26857-21kthreadcore20:40:201
259999245,2cyclictest131rcu_preempt22:15:110
259999243,15cyclictest30152-21ssh22:45:130
259999241,6cyclictest131rcu_preempt20:45:150
259999241,6cyclictest131rcu_preempt20:45:140
2599992412,9cyclictest18088-21diskmemload22:20:170
262999233,16cyclictest28366irq/58-eth021:00:133
262999231,8cyclictest0-21swapper/323:40:233
262999231,7cyclictest0-21swapper/322:40:303
262999231,7cyclictest0-21swapper/322:40:293
262999231,7cyclictest0-21swapper/319:35:263
262999231,6cyclictest0-21swapper/323:15:193
262999231,6cyclictest0-21swapper/323:00:193
262999231,6cyclictest0-21swapper/321:05:173
262999231,6cyclictest0-21swapper/320:35:233
262999231,6cyclictest0-21swapper/320:16:313
262999231,6cyclictest0-21swapper/320:00:013
262999231,6cyclictest0-21swapper/320:00:003
262999231,6cyclictest0-21swapper/319:50:293
262999231,6cyclictest0-21swapper/319:40:213
262999231,6cyclictest0-21swapper/319:20:203
262999231,6cyclictest0-21swapper/319:20:193
262999231,6cyclictest0-21swapper/300:30:183
262999231,6cyclictest0-21swapper/300:20:203
262999231,5cyclictest0-21swapper/319:15:273
262199238,12cyclictest9489-21cut22:50:232
262199233,4cyclictest0-21swapper/219:45:132
262199233,14cyclictest22346irq/40-dwc2_hso19:50:132
262199232,16cyclictest22346irq/40-dwc2_hso00:15:142
262199231,9cyclictest2070966rm21:10:022
262199231,3cyclictest131rcu_preempt23:30:152
261399235,6cyclictest6166-21kthreadcore19:45:201
261399235,6cyclictest5342-21cut20:20:161
261399235,5cyclictest21-21ksoftirqd/120:55:171
259999238,5cyclictest12-21ksoftirqd/000:12:560
259999237,4cyclictest12-21ksoftirqd/023:30:180
259999236,14cyclictest12-21ksoftirqd/021:45:130
259999232,17cyclictest32445-21ssh23:40:120
262999221,7cyclictest0-21swapper/320:50:273
262999221,7cyclictest0-21swapper/320:10:223
262999221,6cyclictest0-21swapper/320:45:193
262999221,6cyclictest0-21swapper/320:45:193
262999221,6cyclictest0-21swapper/320:40:213
262999221,6cyclictest0-21swapper/320:30:203
262999221,6cyclictest0-21swapper/320:00:203
262999221,6cyclictest0-21swapper/319:30:173
262999221,5cyclictest0-21swapper/320:25:013
262199225,14cyclictest0-21swapper/221:35:122
262199221,11cyclictest4908-21users23:40:322
261399224,5cyclictest21-21ksoftirqd/119:50:271
261399224,4cyclictest18939-21irqcore20:00:181
259999227,9cyclictest0-21swapper/021:42:370
2599992214,5cyclictest4069-21kthreadcore00:35:210
2599992213,3cyclictest0-21swapper/021:10:270
2599992210,4cyclictest12-21ksoftirqd/019:20:160
2599992210,4cyclictest12-21ksoftirqd/019:20:160
262199217,11cyclictest12413-21apt-get19:20:132
262199217,11cyclictest12413-21apt-get19:20:122
262199216,7cyclictest18790-21tail20:00:192
262199212,10cyclictest24738-21missed_timers22:40:252
262199212,10cyclictest24738-21missed_timers22:40:242
2621992113,5cyclictest25543-21ssh00:30:142
262199211,16cyclictest31577-21latency_hist19:40:012
262199211,10cyclictest26781-21kthreadcore20:40:202
262199211,10cyclictest18066-21cpu23:50:132
259999219,3cyclictest12-21ksoftirqd/023:59:320
259999218,4cyclictest12-21ksoftirqd/022:10:220
259999218,3cyclictest12-21ksoftirqd/021:30:320
259999216,4cyclictest0-21swapper/022:50:150
259999216,3cyclictest131rcu_preempt19:30:280
2599992110,3cyclictest12-21ksoftirqd/023:36:520
2599992110,3cyclictest12-21ksoftirqd/023:15:110
2599992110,3cyclictest12-21ksoftirqd/019:50:200
262999201,6cyclictest0-21swapper/320:55:173
262199203,8cyclictest29677-21irqcore20:45:182
262199203,8cyclictest29677-21irqcore20:45:182
262199202,14cyclictest30436-21kthreadcore20:10:232
262199201,9cyclictest6789-21grep20:55:162
262199201,16cyclictest13290-21wc20:25:212
259999209,3cyclictest12-21ksoftirqd/022:43:180
259999209,3cyclictest12-21ksoftirqd/022:43:180
259999208,4cyclictest12-21ksoftirqd/020:30:150
259999208,3cyclictest12-21ksoftirqd/022:30:130
259999208,3cyclictest12-21ksoftirqd/022:30:120
259999207,10cyclictest2691-21ls22:30:010
259999206,11cyclictest131rcu_preempt20:10:160
2599992011,7cyclictest0-21swapper/020:40:260
2599992011,3cyclictest131rcu_preempt20:50:160
262199191,9cyclictest6048-21wc19:10:222
261399191,9cyclictest17881-21kthreadcore21:05:221
259999199,2cyclictest12-21ksoftirqd/022:55:180
259999199,2cyclictest12-21ksoftirqd/019:35:210
259999198,3cyclictest131rcu_preempt21:00:170
259999198,2cyclictest12-21ksoftirqd/019:10:260
259999197,3cyclictest12-21ksoftirqd/021:20:110
259999197,3cyclictest12-21ksoftirqd/021:05:180
259999192,2cyclictest131rcu_preempt20:55:140
2599991913,4cyclictest0-21swapper/022:35:160
2599991913,4cyclictest0-21swapper/020:15:100
259999190,14cyclictest23317-21timerandwakeup20:35:290
262199181,8cyclictest2648-21latency_hist19:10:022
259999188,2cyclictest12-21ksoftirqd/020:00:160
259999181,14cyclictest8095-21ps19:15:170
259999180,2cyclictest131rcu_preempt20:05:160
259999174,4cyclictest12-21ksoftirqd/020:20:140
259999153,9cyclictest2597-21cyclictest19:10:010
261399145,7cyclictest2618-21latency_hist19:10:011
262999121,8cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional