You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2026-03-12 - 04:15

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Thu Mar 12, 2026 00:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
415199542,25cyclictest8908-21ssh22:50:132
415199521,31cyclictest20011-21ssh23:50:122
4159995115,28cyclictest28366irq/58-eth000:15:143
415199452,28cyclictest6947-21sh21:55:142
415999426,22cyclictest28366irq/58-eth022:45:133
415199424,23cyclictest23377-21ssh22:40:142
415199422,24cyclictest1545-21ssh22:10:122
4128994226,3cyclictest12-21ksoftirqd/022:30:120
415999413,23cyclictest28366irq/58-eth000:35:133
415199415,28cyclictest0-21swapper/220:35:132
415199415,28cyclictest0-21swapper/220:35:122
412899411,3cyclictest131rcu_preempt19:25:120
4159993629,4cyclictest28366irq/58-eth021:35:113
4159993629,4cyclictest28366irq/58-eth021:35:103
415999352,22cyclictest28366irq/58-eth023:10:113
415199352,26cyclictest0-21swapper/200:25:162
415999342,20cyclictest28366irq/58-eth022:50:123
415999341,18cyclictest28366irq/58-eth020:05:123
415999333,18cyclictest28366irq/58-eth021:45:133
415199337,13cyclictest0-21swapper/200:10:422
4159993216,11cyclictest28366irq/58-eth023:35:143
415199326,15cyclictest0-21swapper/223:01:042
415199325,15cyclictest22346irq/40-dwc2_hso20:00:132
415199323,18cyclictest30323-21ssh21:50:142
4151993217,10cyclictest1875-21apt-get20:15:162
4151993215,13cyclictest5713-21apt-get00:00:112
414399323,26cyclictest10995-21ssh23:45:141
415999313,13cyclictest28366irq/58-eth021:50:143
4159993118,9cyclictest28366irq/58-eth000:25:163
415199316,18cyclictest0-21swapper/222:45:122
415999304,16cyclictest28366irq/58-eth022:00:133
415199309,13cyclictest0-21swapper/222:17:012
415199308,16cyclictest0-21swapper/221:10:112
415199308,10cyclictest0-21swapper/219:20:102
415199306,12cyclictest0-21swapper/223:15:092
4151993016,9cyclictest5321-21apt-get22:30:122
415999293,16cyclictest28366irq/58-eth023:00:113
415999292,18cyclictest28366irq/58-eth023:20:113
415999291,17cyclictest28366irq/58-eth000:00:123
415199297,17cyclictest0-21swapper/219:55:112
415199293,16cyclictest15947-21ssh22:00:122
415199293,15cyclictest22346irq/40-dwc2_hso20:50:132
415199292,21cyclictest0-21swapper/220:20:132
4151992921,5cyclictest19523-21ssh23:30:232
4151992921,5cyclictest19523-21ssh23:30:232
414399291,15cyclictest0-21swapper/119:35:101
415999285,14cyclictest28366irq/58-eth022:05:123
415999281,16cyclictest28366irq/58-eth022:15:123
415199288,15cyclictest0-21swapper/200:30:112
415199288,13cyclictest0-21swapper/222:28:252
415199288,13cyclictest0-21swapper/222:07:152
415199287,4cyclictest0-21swapper/221:32:142
415199287,12cyclictest0-21swapper/223:14:312
415199286,14cyclictest0-21swapper/221:28:192
415199286,14cyclictest0-21swapper/221:28:182
415199283,15cyclictest22346irq/40-dwc2_hso21:05:132
415199282,18cyclictest22346irq/40-dwc2_hso21:20:122
415199281,20cyclictest0-21swapper/220:40:122
412899282,20cyclictest11795-21ssh00:38:530
415999272,16cyclictest28366irq/58-eth023:40:123
415999271,15cyclictest28366irq/58-eth000:30:123
415199275,13cyclictest0-21swapper/200:18:542
415199272,19cyclictest0-21swapper/219:50:132
415199272,16cyclictest22346irq/40-dwc2_hso00:05:112
415199271,21cyclictest0-21swapper/223:25:132
415199267,7cyclictest0-21swapper/221:43:142
4151992611,12cyclictest18195-21cron00:25:012
415199260,17cyclictest2957-21ssh21:35:112
415199260,17cyclictest2957-21ssh21:35:102
415199256,3cyclictest0-21swapper/221:16:582
415199255,12cyclictest22346irq/40-dwc2_hso20:30:102
415199253,13cyclictest22346irq/40-dwc2_hso20:05:132
4151992518,4cyclictest28286-21apt-get19:35:102
415199250,8cyclictest25386-21/usr/sbin/munin23:35:172
414399252,18cyclictest11739-21sh00:38:491
4143992511,5cyclictest21-21ksoftirqd/119:50:131
412899254,15cyclictest5725-21ssh00:00:110
412899252,19cyclictest19730-21ssh22:20:150
412899252,17cyclictest14352-21ssh22:35:130
415999248,9cyclictest28366irq/58-eth022:10:133
415199247,4cyclictest0-21swapper/222:21:262
415199243,7cyclictest8758-21usage_EthcTask00:35:322
415199241,13cyclictest3573-21ssh23:05:102
414399242,14cyclictest5322-21ssh22:30:111
414399240,11cyclictest3744-21ssh22:10:201
415999236,10cyclictest28366irq/58-eth023:45:153
415999232,7cyclictest28366irq/58-eth020:00:133
415999232,12cyclictest28366irq/58-eth023:55:123
415199237,4cyclictest0-21swapper/223:56:592
415199237,4cyclictest0-21swapper/223:46:122
415199236,4cyclictest0-21swapper/223:20:432
4151992310,10cyclictest0-21swapper/219:45:192
4151992310,10cyclictest0-21swapper/219:45:192
414399238,10cyclictest21-21ksoftirqd/123:50:121
414399235,14cyclictest171-21systemd-network23:40:131
414399231,17cyclictest1357-21ls22:10:021
414399231,16cyclictest27563-21ssh00:30:131
412899231,18cyclictest20200-21cut23:50:160
415999223,14cyclictest28366irq/58-eth023:05:133
415999221,4cyclictest0-21swapper/323:25:123
415199225,12cyclictest11142-21latency20:55:202
415199224,14cyclictest27223-21sed19:30:272
415199223,6cyclictest131rcu_preempt20:45:252
415199222,6cyclictest0-21swapper/223:40:142
415199222,11cyclictest4276-21sshd19:10:122
415199222,11cyclictest17847-21missed_timers22:35:222
415199221,12cyclictest9112-21sshd19:15:132
4151992210,9cyclictest261rcuc/222:55:132
4143992212,7cyclictest9054-21cron00:20:001
414399221,17cyclictest14046-21cut19:20:131
414399221,11cyclictest0-21swapper/122:50:141
412899223,15cyclictest3592-21ssh23:05:130
4128992214,4cyclictest0-21swapper/019:10:120
412899221,17cyclictest10987-21ssh23:25:260
412899221,16cyclictest21653-21ssh23:15:140
412899220,18cyclictest11619-21uniq20:25:160
415999213,8cyclictest0-21swapper/323:15:153
415199217,4cyclictest0-21swapper/220:10:132
415199215,8cyclictest19207-21grep19:25:192
415199214,5cyclictest0-21swapper/219:40:112
415199213,4cyclictest27-21ksoftirqd/221:45:322
415199211,16cyclictest14124-21kthreadcore20:25:212
414399214,13cyclictest21-21ksoftirqd/119:30:111
414399211,5cyclictest29943-21tr23:55:181
414399211,5cyclictest15582-21sendmail-msp22:00:021
4143992111,8cyclictest3194-21fschecks_time21:35:151
4143992111,8cyclictest3194-21fschecks_time21:35:151
414399211,16cyclictest19327-21awk22:20:011
414399211,15cyclictest30690-21ssh23:20:131
414399211,11cyclictest16343-21ssh23:30:141
414399211,11cyclictest16343-21ssh23:30:131
412899218,10cyclictest27937-21ps00:30:160
412899217,4cyclictest12-21ksoftirqd/000:21:470
4128992115,4cyclictest0-21swapper/021:55:140
4128992112,7cyclictest12-21ksoftirqd/020:05:120
412899211,17cyclictest29725-21ls20:10:170
412899211,10cyclictest26010-21latency_hist21:30:010
412899211,10cyclictest26010-21latency_hist21:30:010
415999202,15cyclictest28366irq/58-eth019:45:133
415999202,15cyclictest28366irq/58-eth019:45:133
4159992012,5cyclictest28366irq/58-eth000:10:133
415199201,15cyclictest16121-21missed_timers21:00:212
414399201,9cyclictest10809-21grep23:25:231
414399201,15cyclictest3583-21ssh23:05:131
414399201,15cyclictest21639-21ntp_states22:55:251
414399201,15cyclictest12899-21irqstats21:40:191
412899206,11cyclictest4677-21cat00:15:310
412899203,7cyclictest27927-21kthreadcore23:00:190
412899202,6cyclictest29000-21ssh23:35:230
412899202,2cyclictest131rcu_preempt20:55:260
412899200,18cyclictest28917-21grep22:25:160
415999199,8cyclictest0-21swapper/321:26:063
415999199,8cyclictest0-21swapper/321:26:063
415999199,6cyclictest0-21swapper/321:15:493
415999192,11cyclictest0-21swapper/320:45:143
4159991912,5cyclictest28366irq/58-eth021:10:123
415999191,13cyclictest0-21swapper/319:20:263
415999191,12cyclictest0-21swapper/320:10:243
415999191,11cyclictest0-21swapper/320:35:273
415999191,11cyclictest0-21swapper/320:35:263
4159991910,6cyclictest0-21swapper/321:20:203
414399199,7cyclictest16282-21apt-get20:30:111
414399198,8cyclictest22984-21awk22:40:011
414399196,3cyclictest22160-21diskmemload00:05:251
414399193,4cyclictest21-21ksoftirqd/122:43:151
414399193,11cyclictest0-21swapper/122:20:141
414399191,4cyclictest0-21swapper/121:25:221
414399191,4cyclictest0-21swapper/121:25:211
4143991912,4cyclictest30711-21open_files23:00:271
4143991911,6cyclictest24473-21cron20:05:011
414399191,15cyclictest3098-21apt-get20:50:121
414399191,14cyclictest24794-21sh22:05:011
414399191,12cyclictest30343-21ssh21:50:141
414399191,12cyclictest22160-21diskmemload21:11:101
414399191,10cyclictest18257-21latency_hist00:25:021
4143991910,6cyclictest23117-21usage_EthcTask00:25:321
414399190,5cyclictest131rcu_preempt00:00:111
412899197,9cyclictest0-21swapper/020:15:160
412899193,3cyclictest131rcu_preempt21:55:020
412899192,14cyclictest12186-21apt-get21:40:130
412899191,5cyclictest2234-21timerandwakeup23:20:280
412899191,3cyclictest26218-21awk21:10:250
4128991912,3cyclictest0-21swapper/022:15:200
412899191,10cyclictest14548-21missed_timers23:45:260
415999189,6cyclictest28366irq/58-eth022:55:113
415999189,6cyclictest0-21swapper/321:42:453
415999188,7cyclictest0-21swapper/321:30:253
415999188,6cyclictest28366irq/58-eth023:30:133
415999188,6cyclictest28366irq/58-eth023:30:133
415999183,11cyclictest0-21swapper/323:50:123
415999182,10cyclictest0-21swapper/320:20:143
415999182,10cyclictest0-21swapper/319:10:133
415999181,13cyclictest0-21swapper/319:30:273
415999181,11cyclictest0-21swapper/322:40:153
4159991810,6cyclictest0-21swapper/322:34:443
414399189,4cyclictest0-21swapper/119:45:151
414399189,4cyclictest0-21swapper/119:45:141
414399182,8cyclictest131rcu_preempt21:17:331
414399182,4cyclictest21-21ksoftirqd/122:25:441
414399182,10cyclictest27473-21ssh00:10:251
414399181,9cyclictest0-21swapper/123:37:561
4143991811,4cyclictest0-21swapper/119:40:241
414399181,13cyclictest6628-21latency_hist20:20:011
414399181,12cyclictest8276-21ssh21:20:131
4143991810,4cyclictest0-21swapper/122:45:131
414399180,10cyclictest22160-21diskmemload21:30:211
412899188,7cyclictest6171-21cut21:35:210
412899188,7cyclictest6171-21cut21:35:210
412899183,11cyclictest14825-21apt-get00:05:110
412899181,9cyclictest10089-21kthreadcore22:50:200
412899181,9cyclictest0-21swapper/021:30:210
4128991812,4cyclictest8241-21/usr/sbin/munin21:20:220
4128991812,3cyclictest24063-21cut22:40:190
4128991812,3cyclictest12-21ksoftirqd/000:25:160
412899181,13cyclictest2243-21ssh22:10:180
4128991810,6cyclictest29039-21ssh23:55:130
4128991810,5cyclictest25954-21cat21:45:300
415999179,6cyclictest0-21swapper/300:20:113
415999178,7cyclictest0-21swapper/300:06:303
415999176,5cyclictest0-21swapper/322:35:113
415999172,10cyclictest0-21swapper/321:55:143
415999172,10cyclictest0-21swapper/320:30:123
415999171,13cyclictest0-21swapper/322:25:163
415999171,10cyclictest0-21swapper/320:45:003
415199174,9cyclictest4156-21latency_hist19:10:022
414399179,6cyclictest18420-21memory19:55:211
414399179,5cyclictest2351-21usage_Anybus20:45:301
414399178,7cyclictest30503-21sed23:20:021
414399172,4cyclictest131rcu_preempt20:25:141
414399171,9cyclictest9293-21cut19:15:161
414399171,8cyclictest0-21swapper/123:12:231
414399171,4cyclictest0-21swapper/120:20:201
4143991710,5cyclictest0-21swapper/120:05:291
414399170,4cyclictest53146usb-storage21:00:051
412899178,6cyclictest27267-21kthreadcore00:10:210
412899178,6cyclictest2567-21cut22:45:220
412899175,9cyclictest31049-21tr20:45:180
412899173,11cyclictest423-21latency_hist19:40:010
412899172,11cyclictest15976-21ssh22:00:140
412899171,7cyclictest12334-21kthreadcore19:50:210
412899171,6cyclictest20601-21kthreadcore20:00:200
412899171,5cyclictest9381-21sed19:45:300
412899171,5cyclictest9381-21sed19:45:290
412899171,14cyclictest24103-21kthreadcore20:35:210
412899171,14cyclictest24103-21kthreadcore20:35:210
412899171,13cyclictest31597-21ssh21:15:140
4128991711,3cyclictest0-21swapper/023:40:250
4128991710,4cyclictest6995-21grep20:50:270
412899170,10cyclictest9266-21cut19:15:150
415999165,7cyclictest0-21swapper/322:20:143
415999162,11cyclictest0-21swapper/320:55:203
415999162,10cyclictest0-21swapper/320:15:163
415999161,9cyclictest0-21swapper/321:05:123
414399168,6cyclictest12459-21cron21:00:001
414399167,7cyclictest7859-21cat19:10:251
414399167,7cyclictest29343-21ntpq20:40:231
414399165,8cyclictest21336-21grep21:05:281
414399163,11cyclictest21598-21irqrtprio20:35:171
414399163,11cyclictest21598-21irqrtprio20:35:171
414399161,13cyclictest18711-21sshd19:25:151
414399160,7cyclictest131rcu_preempt21:45:121
412899168,6cyclictest17200-21cut23:30:200
412899168,6cyclictest17200-21cut23:30:190
412899166,7cyclictest13369-21grep23:10:180
412899164,10cyclictest29777-21grep20:40:270
412899163,11cyclictest16783-21kthreadcore19:20:200
412899163,11cyclictest13571-1kworker/u9:122:05:110
412899163,10cyclictest4563-21grep19:40:270
412899162,11cyclictest18062-21kthreadcore20:30:180
412899162,10cyclictest14028-21kthreadcore21:00:180
412899161,11cyclictest26807-21missed_timers19:30:220
412899161,11cyclictest21478-21grep21:05:300
412899160,11cyclictest17859-21ssh22:55:100
415999153,6cyclictest0-21swapper/321:00:123
415999151,9cyclictest28366irq/58-eth019:55:123
415999151,9cyclictest0-21swapper/320:25:133
414399157,6cyclictest476-21ntpq20:10:261
412899152,10cyclictest7093-21irqcore20:20:180
412899151,7cyclictest4166-21latency_hist19:10:010
412899151,11cyclictest15109-21apt-get19:55:120
415999149,3cyclictest0-21swapper/319:35:233
414399141,10cyclictest4175-21cat19:10:021
415999132,9cyclictest0-21swapper/319:15:143
415999122,7cyclictest0-21swapper/319:50:123
415999122,6cyclictest0-21swapper/319:40:123
415999122,6cyclictest0-21swapper/319:25:153
415999121,7cyclictest0-21swapper/320:50:153
41599991,6cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional