You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-05-08 - 11:45

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Fri May 08, 2026 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2117999708,8cyclictest131rcu_preempt23:35:120
2117999621,7cyclictest131rcu_preempt21:30:130
2117999522,42cyclictest17011-21ssh22:45:130
2117999512,36cyclictest7156-21ssh21:10:100
2119999504,8cyclictest27-21ksoftirqd/221:30:132
21187995029,15cyclictest23015-21ssh00:00:131
2118799502,41cyclictest466-21ssh22:00:131
2117999502,28cyclictest15393-21ssh23:20:130
2121099495,26cyclictest35966irq/58-eth021:55:133
2121099473,26cyclictest35966irq/58-eth022:50:143
21210994533,8cyclictest35966irq/58-eth022:25:113
2119999448,21cyclictest12553-21ssh00:30:112
2118799441,31cyclictest8812-21ssh00:10:131
2121099432,25cyclictest0-21swapper/323:50:113
21187994315,17cyclictest4308-21ssh22:20:131
2117999431,33cyclictest20751-21ssh23:05:130
2118799422,33cyclictest13337-21ssh22:25:111
2121099412,24cyclictest35966irq/58-eth020:35:113
21210994120,17cyclictest0-21swapper/323:45:123
2121099402,21cyclictest35966irq/58-eth000:20:163
2118799403,26cyclictest24454-21sh23:25:121
21187994029,5cyclictest10164-21apt-get23:35:131
21210993922,12cyclictest35966irq/58-eth023:10:133
21187993911,19cyclictest0-21swapper/120:45:121
21187993910,10cyclictest27268-21latency_hist19:50:011
2121099384,20cyclictest35966irq/58-eth022:05:133
2121099383,19cyclictest35966irq/58-eth022:00:133
21187993816,18cyclictest0-21swapper/123:00:131
21187993811,11cyclictest8944-21ssh22:57:081
21187993810,9cyclictest21770-21diskstats00:35:171
2117999382,26cyclictest6793-21diskmemload21:55:130
2121099373,23cyclictest35966irq/58-eth022:45:123
2118799379,9cyclictest12326-21ssh21:47:341
2118799379,11cyclictest32757-21ntp_192.168.11421:40:261
2118799377,11cyclictest8233-21tr23:50:201
21187993720,14cyclictest7917-21apt-get22:40:111
21187993710,11cyclictest23936-21ssh22:11:081
2117999371,24cyclictest2717-21ssh22:55:130
2118799368,11cyclictest22303-21ssh23:59:411
2118799357,18cyclictest0-21swapper/122:35:141
21187993520,12cyclictest15366-21apt-get23:20:131
21187993510,7cyclictest2673-21ssh22:18:351
2117999356,7cyclictest131rcu_preempt22:00:120
2117999351,21cyclictest21656-21ssh00:35:150
2121099341,22cyclictest35966irq/58-eth021:35:133
21199993425,6cyclictest261rcuc/219:40:122
2118799349,9cyclictest15701-21rm22:07:071
2118799349,8cyclictest20890-21diskstats23:05:161
2118799349,8cyclictest10318-21ssh23:15:301
2118799349,15cyclictest22200-21ssh22:45:541
2118799349,12cyclictest32500-21irqcore00:05:171
2118799349,11cyclictest25064-21sh00:18:221
2118799349,11cyclictest25064-21sh00:18:221
21187993421,10cyclictest29764-21apt-get23:10:131
21187993411,9cyclictest18658-21ssh21:33:541
21187993410,10cyclictest30966-21ssh21:21:331
21187993410,10cyclictest14139-21grep20:10:131
21187993410,10cyclictest12526-21ssh21:11:081
2119999332,15cyclictest26934-21ssh21:37:572
2118799339,10cyclictest3905-21/usr/sbin/munin00:25:181
2118799338,10cyclictest16167-21cut21:50:201
2118799337,8cyclictest29728-21ntp_192.168.11422:50:261
2118799337,7cyclictest0-21swapper/122:33:011
21187993328,2cyclictest23851-21apt-get21:55:131
2118799332,24cyclictest22922-21ssh23:40:261
21187993310,15cyclictest32698-21cat20:25:281
21187993310,10cyclictest26420-21rm21:37:211
2121099326,16cyclictest0-21swapper/323:15:133
2121099324,15cyclictest35966irq/58-eth000:10:123
2121099321,21cyclictest35966irq/58-eth020:15:143
2121099321,21cyclictest35966irq/58-eth020:15:143
2119999321,8cyclictest27-21ksoftirqd/200:25:122
21199993211,4cyclictest27-21ksoftirqd/221:27:232
2118799329,8cyclictest9056-21ssh23:34:091
2118799329,10cyclictest19636-21missed_timers21:15:231
2118799328,8cyclictest0-21swapper/120:00:121
2118799328,10cyclictest31050-21ls19:20:151
2118799327,10cyclictest11086-21grep20:05:191
21187993211,8cyclictest13267-21ssh00:30:181
21179993210,4cyclictest131rcu_preempt23:45:130
2121099311,18cyclictest35966irq/58-eth000:30:113
21199993113,12cyclictest27-21ksoftirqd/223:53:312
2118799318,9cyclictest5921-21/usr/sbin/munin21:05:281
2118799317,8cyclictest27150-21ntp_192.168.11420:20:231
2118799317,10cyclictest9611-21usage_Anybus20:35:281
2118799313,17cyclictest12585-21kthreadcore20:40:201
21187993116,6cyclictest0-21swapper/119:10:131
2117999311,24cyclictest16127-21ssh21:15:120
2117999310,24cyclictest26991-21ssh00:20:150
2121099307,15cyclictest0-21swapper/300:05:123
2121099304,16cyclictest35966irq/58-eth022:35:133
2119999302,9cyclictest0-21swapper/223:55:112
2119999302,20cyclictest29783-21ssh23:10:142
2119999300,22cyclictest14809-21ssh21:50:122
2118799309,7cyclictest0-21swapper/121:25:071
2118799307,8cyclictest28682-21cat20:55:271
2118799307,10cyclictest24198-21usage_Anybus20:50:301
21187993019,7cyclictest0-21swapper/119:35:111
21187993011,7cyclictest959-21ssh23:45:511
2117999303,18cyclictest12-21ksoftirqd/020:05:120
21179993012,9cyclictest12-21ksoftirqd/023:50:110
2119999293,16cyclictest27346irq/40-dwc2_hso20:00:122
2119999292,7cyclictest27-21ksoftirqd/220:10:112
2119999291,14cyclictest9565-21ssh22:05:132
21187992914,12cyclictest0-21swapper/119:55:121
2117999297,13cyclictest12-21ksoftirqd/021:20:120
21179992913,10cyclictest151rcuc/021:45:140
2121099283,16cyclictest0-21swapper/323:20:113
2121099283,16cyclictest0-21swapper/321:22:533
2121099282,16cyclictest35966irq/58-eth000:35:153
21210992811,11cyclictest0-21swapper/323:59:113
2119999284,14cyclictest27346irq/40-dwc2_hso21:45:132
2118799288,8cyclictest27181-21uniq00:20:181
2118799287,16cyclictest0-21swapper/120:30:121
21187992821,3cyclictest27352-21apt-get19:50:131
21187992821,3cyclictest27352-21apt-get19:50:131
2117999283,19cyclictest12-21ksoftirqd/019:50:130
2117999283,19cyclictest12-21ksoftirqd/019:50:130
2117999282,20cyclictest30365-21ssh00:03:290
2117999282,19cyclictest13921-21ssh23:55:110
2121099279,13cyclictest35966irq/58-eth022:15:163
2121099272,21cyclictest35966irq/58-eth022:10:143
2119999275,11cyclictest27-21ksoftirqd/222:10:112
2119999273,16cyclictest27346irq/40-dwc2_hso23:20:122
2119999273,10cyclictest27000-21ssh00:20:162
21199992716,8cyclictest261rcuc/223:40:122
2117999273,16cyclictest12-21ksoftirqd/020:10:120
2117999272,20cyclictest20448-21ssh22:28:120
2117999272,19cyclictest26085-21ssh22:50:140
2117999272,19cyclictest17768-21ssh00:15:140
2117999272,19cyclictest17768-21ssh00:15:140
2121099263,15cyclictest0-21swapper/321:34:063
21210992613,7cyclictest35966irq/58-eth021:40:153
2119999268,12cyclictest27-21ksoftirqd/200:12:342
2119999267,16cyclictest131rcu_preempt22:20:132
2119999263,17cyclictest0-21swapper/223:05:132
2119999263,11cyclictest27-21ksoftirqd/200:00:212
21199992614,9cyclictest0-21swapper/222:45:132
21199992610,10cyclictest131rcu_preempt21:56:232
2118799265,8cyclictest0-21swapper/121:00:141
2121099253,15cyclictest0-21swapper/322:23:083
2119999257,10cyclictest131rcu_preempt22:02:012
2119999255,17cyclictest24546irq/40-fe98000019:55:132
2119999255,14cyclictest27-21ksoftirqd/222:58:262
2119999252,18cyclictest636-21ssh21:24:062
2119999251,9cyclictest6033-21ssh22:38:162
21199992517,5cyclictest28578-21ssh23:45:172
21187992511,10cyclictest22386-21ntpq20:15:261
21187992511,10cyclictest22386-21ntpq20:15:261
2117999253,15cyclictest11774-21ssh23:00:130
21179992511,4cyclictest12-21ksoftirqd/019:40:220
2121099249,10cyclictest0-21swapper/300:04:113
21210992410,11cyclictest0-21swapper/323:30:303
2119999247,11cyclictest27-21ksoftirqd/223:39:412
2119999246,7cyclictest27-21ksoftirqd/223:15:142
2119999244,14cyclictest27-21ksoftirqd/200:15:452
2119999244,14cyclictest27-21ksoftirqd/200:15:452
2119999243,15cyclictest27346irq/40-dwc2_hso19:15:152
2119999242,10cyclictest27640-21ssh22:15:152
2119999241,6cyclictest7921-21ssh22:40:112
21199992412,9cyclictest3150-21ssh21:42:252
21187992413,8cyclictest17762-21sshd19:40:071
21187992413,8cyclictest11603-21ntpq19:30:251
21187992412,9cyclictest26114-21apt-get19:15:141
2117999243,15cyclictest14835-21ssh21:50:140
2117999242,17cyclictest8791-21apt-get00:10:120
2117999241,2cyclictest131rcu_preempt21:35:200
21179992411,4cyclictest0-21swapper/019:55:280
2117999240,2cyclictest0-21swapper/021:40:130
2119999239,4cyclictest27-21ksoftirqd/219:50:162
2119999239,4cyclictest27-21ksoftirqd/219:50:162
2119999239,3cyclictest27-21ksoftirqd/200:35:172
2119999234,2cyclictest131rcu_preempt23:00:202
2119999234,13cyclictest27-21ksoftirqd/220:40:202
2119999232,6cyclictest0-21swapper/222:28:122
2119999232,13cyclictest27-21ksoftirqd/221:15:212
2119999230,18cyclictest21512-21grep19:10:162
21187992313,8cyclictest7228-21grep19:25:291
2117999233,3cyclictest131rcu_preempt00:25:200
2117999233,12cyclictest29768-21ssh23:10:130
21179992312,3cyclictest12-21ksoftirqd/022:18:510
2117999231,19cyclictest32147-21ssh00:05:130
2117999231,18cyclictest10733-21cut20:40:150
2121099221,8cyclictest0-21swapper/321:45:233
2119999226,12cyclictest27346irq/40-dwc2_hso19:20:132
2119999225,4cyclictest131rcu_preempt21:10:292
2119999224,13cyclictest27346irq/40-dwc2_hso20:45:122
2119999222,8cyclictest131rcu_preempt20:30:122
2119999222,10cyclictest12881-21latency_hist19:35:022
2119999221,6cyclictest27-21ksoftirqd/223:30:302
21199992212,8cyclictest0-21swapper/223:25:032
2119999221,17cyclictest0-21swapper/222:50:142
2117999229,3cyclictest12-21ksoftirqd/023:15:290
2117999223,2cyclictest131rcu_preempt19:45:290
2117999222,15cyclictest6793-21diskmemload21:25:140
2121099217,9cyclictest35966irq/58-eth021:50:143
2119999215,10cyclictest131rcu_preempt20:05:212
2119999212,4cyclictest27-21ksoftirqd/219:45:232
2119999212,17cyclictest0-21swapper/220:55:122
21199992111,8cyclictest0-21swapper/222:31:592
21199992111,8cyclictest0-21swapper/220:20:112
2117999218,4cyclictest131rcu_preempt22:39:470
2117999218,3cyclictest12-21ksoftirqd/022:14:080
2117999217,2cyclictest12-21ksoftirqd/000:30:250
2117999210,18cyclictest23941-21cut20:20:150
2121099202,13cyclictest35966irq/58-eth000:15:123
2121099202,13cyclictest35966irq/58-eth000:15:123
21210992010,7cyclictest35966irq/58-eth021:15:123
2119999206,11cyclictest7937-21cron19:30:012
2119999203,8cyclictest22184-21wc20:15:232
2119999203,8cyclictest22184-21wc20:15:232
2119999203,14cyclictest0-21swapper/220:50:122
2119999202,15cyclictest27346irq/40-dwc2_hso20:35:112
2119999201,9cyclictest32729-21kthreadcore21:00:212
2119999201,9cyclictest2406-21fschecks_time21:05:162
2119999201,15cyclictest31655-21apt-get00:05:022
2119999201,10cyclictest32096-21grep20:25:262
2119999201,10cyclictest13066-21fschecks_count19:35:152
2117999203,3cyclictest131rcu_preempt23:42:550
2117999203,3cyclictest131rcu_preempt19:35:010
2117999202,2cyclictest131rcu_preempt23:30:060
21179992013,5cyclictest0-21swapper/020:40:030
21179992012,6cyclictest1-21systemd19:12:250
21179992012,5cyclictest13147-21ntp_192.168.11422:05:250
2117999201,14cyclictest923-21apt-get20:30:120
2117999201,11cyclictest131rcu_preempt23:27:360
21179992010,3cyclictest12-21ksoftirqd/022:20:160
2121099199,6cyclictest0-21swapper/323:37:173
2121099193,11cyclictest0-21swapper/321:26:063
2121099193,11cyclictest0-21swapper/300:28:353
2121099192,10cyclictest0-21swapper/320:00:123
21210991913,4cyclictest0-21swapper/323:00:283
21210991912,5cyclictest0-21swapper/322:55:583
21210991912,4cyclictest0-21swapper/322:40:123
2121099191,13cyclictest0-21swapper/320:30:283
21210991910,6cyclictest0-21swapper/323:40:203
2119999197,9cyclictest21202-21sort19:10:022
2118799198,8cyclictest21167-21cyclictest19:10:021
2117999193,2cyclictest131rcu_preempt21:00:220
21179991913,3cyclictest12-21ksoftirqd/020:55:240
2117999191,2cyclictest131rcu_preempt22:32:370
2117999191,13cyclictest7933-21ssh22:40:120
2121099182,10cyclictest0-21swapper/320:40:043
2121099182,10cyclictest0-21swapper/320:20:133
2121099182,10cyclictest0-21swapper/319:45:163
2121099181,14cyclictest0-21swapper/321:10:193
2121099181,13cyclictest0-21swapper/319:50:213
2121099181,13cyclictest0-21swapper/319:50:213
2121099181,10cyclictest0-21swapper/323:30:013
2117999188,2cyclictest12-21ksoftirqd/019:35:150
2117999185,3cyclictest131rcu_preempt19:25:140
2117999183,11cyclictest0-21swapper/020:00:110
21179991810,3cyclictest131rcu_preempt20:15:140
21179991810,3cyclictest131rcu_preempt20:15:130
2117999180,7cyclictest15396-21cat20:45:110
2121099179,6cyclictest0-21swapper/322:31:553
2121099172,8cyclictest0-21swapper/320:50:123
2121099172,10cyclictest0-21swapper/319:55:143
2121099172,10cyclictest0-21swapper/319:40:133
2121099171,13cyclictest0-21swapper/321:00:013
2117999179,6cyclictest31854-21load20:25:210
2117999178,3cyclictest0-21swapper/021:05:180
2117999177,2cyclictest12-21ksoftirqd/020:50:210
2117999172,12cyclictest3231-21cat19:25:010
2117999171,2cyclictest131rcu_preempt19:15:270
2121099168,6cyclictest0-21swapper/323:05:583
2121099162,9cyclictest0-21swapper/320:45:133
2121099161,12cyclictest0-21swapper/319:40:013
2121099161,12cyclictest0-21swapper/319:15:303
2121099157,3cyclictest0-21swapper/320:05:143
2121099152,9cyclictest0-21swapper/321:00:133
2121099152,9cyclictest0-21swapper/320:10:133
2121099151,7cyclictest0-21swapper/319:20:133
2121099149,3cyclictest0-21swapper/321:05:183
2121099141,11cyclictest0-21swapper/319:15:023
2121099141,10cyclictest0-21swapper/320:25:153
2117999141,6cyclictest131rcu_preempt19:10:020
2121099122,6cyclictest0-21swapper/319:25:143
2121099111,7cyclictest0-21swapper/319:30:163
2121099111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional