You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-11 - 08:18

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Thu Dec 11, 2025 00:45:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2718099731,4cyclictest131rcu_preempt22:30:120
2719399645,46cyclictest14644-21ssh21:30:151
2720299552,44cyclictest29156-21sh00:15:132
2718099553,6cyclictest131rcu_preempt19:15:110
2719399494,35cyclictest27690-21ssh23:35:141
2721099463,30cyclictest29566irq/58-eth022:15:123
2719399462,6cyclictest131rcu_preempt22:55:151
27202994412,14cyclictest27-21ksoftirqd/223:30:142
2718099441,33cyclictest9818-21ssh22:45:140
27210994322,14cyclictest29566irq/58-eth023:20:143
27202994318,19cyclictest27-21ksoftirqd/222:51:242
2719399432,37cyclictest30384-21ssh21:20:151
27210994218,18cyclictest321rcuc/322:30:123
2720299422,32cyclictest14629-21cut00:05:232
2721099413,26cyclictest29566irq/58-eth023:45:123
2721099412,26cyclictest29566irq/58-eth000:30:143
2721099412,23cyclictest29566irq/58-eth019:10:143
2718099411,30cyclictest26742-21sh23:15:150
2721099404,25cyclictest29566irq/58-eth000:10:143
27210994016,14cyclictest29566irq/58-eth000:35:143
2721099393,24cyclictest29566irq/58-eth020:30:133
2720299376,12cyclictest24664-21ssh22:15:122
27202993720,11cyclictest261rcuc/219:50:122
27193993724,9cyclictest21-21ksoftirqd/120:45:131
2719399371,32cyclictest29158-21sh00:15:141
2720299367,23cyclictest261rcuc/223:55:112
2720299359,9cyclictest2630-21ssh23:20:142
27202993521,10cyclictest131rcu_preempt21:05:122
2718099352,29cyclictest1434-21ssh22:40:120
2721099341,26cyclictest29566irq/58-eth022:55:283
2720299341,24cyclictest16440-21ssh22:10:162
2719399343,27cyclictest13688-21ssh21:10:141
2719399342,24cyclictest5113-21ntp_states22:40:241
2721099332,18cyclictest29566irq/58-eth022:35:143
27202993211,4cyclictest131rcu_preempt23:10:152
2719399323,26cyclictest11998-21ssh23:45:131
2718099324,16cyclictest12-21ksoftirqd/020:40:120
2718099322,22cyclictest21628-21missed_timers22:50:230
2718099321,23cyclictest23816-21ssh21:55:120
2721099315,18cyclictest0-21swapper/323:30:123
2720299313,13cyclictest27-21ksoftirqd/220:00:152
2718099314,19cyclictest4493-21ssh00:00:140
2720299303,22cyclictest0-21swapper/200:27:242
2720299303,20cyclictest31275-21ssh21:40:152
2719399307,16cyclictest21-21ksoftirqd/120:35:131
2719399302,22cyclictest11112-21ssh00:23:501
2719399302,21cyclictest622-21ssh22:20:141
2718099301,4cyclictest131rcu_preempt23:25:150
2720299295,16cyclictest26346irq/40-dwc2_hso20:35:142
2720299292,21cyclictest5470-21ssh22:01:382
2720299292,21cyclictest5470-21ssh22:01:382
2720299291,10cyclictest20910-21ssh00:10:132
2718099299,16cyclictest15511-21awk00:25:240
27210992821,4cyclictest29566irq/58-eth021:25:143
2721099281,18cyclictest29566irq/58-eth022:50:153
2720299281,15cyclictest8987-21ssh22:25:152
2718099282,22cyclictest27538-21ssh00:35:150
2718099282,20cyclictest11143-21ssh00:23:570
2718099282,18cyclictest22915-21ssh21:35:120
2718099282,17cyclictest20919-21ssh00:10:140
2721099276,13cyclictest29566irq/58-eth022:45:143
2721099273,16cyclictest0-21swapper/300:07:243
2721099273,15cyclictest0-21swapper/322:41:463
2721099272,15cyclictest0-21swapper/321:55:123
2721099271,19cyclictest0-21swapper/320:40:123
2721099271,17cyclictest29566irq/58-eth021:30:153
2721099271,11cyclictest0-21swapper/320:55:143
27210992711,10cyclictest29566irq/58-eth021:10:143
2720299279,15cyclictest27-21ksoftirqd/222:30:122
2720299273,18cyclictest26346irq/40-dwc2_hso23:25:152
2720299271,19cyclictest27769-21ls23:35:142
2720299271,18cyclictest26346irq/40-dwc2_hso19:20:152
2719399273,18cyclictest21-21ksoftirqd/123:05:141
2719399272,22cyclictest14630-21tr00:05:241
2719399271,17cyclictest16425-21ssh22:10:151
27193992711,10cyclictest21-21ksoftirqd/120:20:111
2721099262,18cyclictest0-21swapper/323:04:243
2719399263,18cyclictest19747-21ssh00:30:131
2718099263,11cyclictest0-21swapper/022:55:150
2721099259,11cyclictest0-21swapper/323:35:243
2721099253,17cyclictest0-21swapper/323:07:243
2721099253,17cyclictest0-21swapper/321:20:163
2720299253,16cyclictest26346irq/40-dwc2_hso21:00:142
2720299252,20cyclictest0-21swapper/220:40:132
2720299252,18cyclictest26346irq/40-dwc2_hso22:20:142
2720299251,7cyclictest11697-21ls00:25:012
2720299251,20cyclictest26758-21cut20:50:172
2720299251,16cyclictest26346irq/40-dwc2_hso21:45:112
2719399259,11cyclictest21-21ksoftirqd/121:58:241
2721099248,9cyclictest321rcuc/321:45:123
27210992412,9cyclictest0-21swapper/322:10:313
2720299243,15cyclictest26346irq/40-dwc2_hso23:15:142
2720299242,17cyclictest0-21swapper/222:05:242
2720299242,15cyclictest26346irq/40-dwc2_hso21:35:142
2720299242,15cyclictest26346irq/40-dwc2_hso20:25:162
27202992413,5cyclictest13297-21diskmemload22:35:142
2720299241,16cyclictest26346irq/40-dwc2_hso23:50:162
2720299241,16cyclictest26346irq/40-dwc2_hso20:05:122
2720299240,15cyclictest13713-21ls22:45:292
2719399242,17cyclictest21-21ksoftirqd/119:55:131
2718099243,15cyclictest11978-21ssh23:45:120
2718099242,18cyclictest14232-21apt-get19:30:140
27210992311,9cyclictest0-21swapper/321:50:233
2720299236,14cyclictest29517-21diskmemload00:35:222
2720299232,16cyclictest26346irq/40-dwc2_hso20:45:142
2720299232,16cyclictest26346irq/40-dwc2_hso20:15:132
2720299232,16cyclictest0-21swapper/223:49:252
2720299232,16cyclictest0-21swapper/221:57:242
2720299231,9cyclictest26279-21ssh00:33:242
27202992313,8cyclictest0-21swapper/221:50:192
27202992312,8cyclictest1760-21if_eth022:40:162
2720299230,5cyclictest19114-21ls19:35:132
2719399231,17cyclictest26750-21ssh23:15:151
2718099239,5cyclictest12-21ksoftirqd/022:20:130
2718099233,2cyclictest131rcu_preempt23:10:130
2718099231,15cyclictest3678-21ssh23:40:160
2718099230,19cyclictest30400-21ssh21:20:160
2721099223,14cyclictest0-21swapper/323:10:143
2721099223,13cyclictest0-21swapper/322:05:153
2721099223,13cyclictest0-21swapper/300:23:513
2720299227,7cyclictest16599-21kthreadcore21:30:252
2720299226,2cyclictest131rcu_preempt23:02:492
2720299225,9cyclictest2385-21grep20:25:022
2720299225,9cyclictest17692-21grep21:10:292
27202992216,4cyclictest26534-21ssh22:55:192
2720299221,5cyclictest4475-21find00:00:122
2720299221,17cyclictest10556-21cron20:00:002
2720299221,12cyclictest11843-21kthreadcore23:05:222
2719399223,17cyclictest23442-21munin-plugin-st21:55:011
2719399222,4cyclictest131rcu_preempt23:40:151
27193992216,4cyclictest21-21ksoftirqd/122:40:021
2719399221,18cyclictest14342-21uniq19:30:171
2718099222,18cyclictest1081-21ssh23:36:550
2721099213,13cyclictest0-21swapper/300:25:243
2721099212,13cyclictest0-21swapper/300:15:153
27210992111,8cyclictest0-21swapper/321:40:253
2721099211,16cyclictest321rcuc/319:50:123
2720299217,7cyclictest27478-21ntp_states19:40:272
2720299217,11cyclictest24110-21kthreadcore21:15:212
2720299214,11cyclictest26346irq/40-dwc2_hso19:25:152
2720299211,12cyclictest30350-21/usr/sbin/munin21:20:182
2720299211,12cyclictest28396-21kthreadcore19:10:192
2719399211,15cyclictest4055-21cut21:00:171
2719399211,13cyclictest4619-21cut00:00:161
2718099218,4cyclictest131rcu_preempt00:15:190
27180992114,5cyclictest12-21ksoftirqd/021:20:010
27210992012,6cyclictest0-21swapper/323:50:523
27210992010,7cyclictest29566irq/58-eth022:25:143
2720299206,6cyclictest27-21ksoftirqd/220:10:282
2720299202,15cyclictest9832-21grep21:25:242
2720299201,9cyclictest4595-21kthreadcore23:40:212
2720299201,16cyclictest14556-21irqcore19:30:192
2720299201,10cyclictest32604-21grep19:45:302
2720299201,10cyclictest3062-21latency19:15:212
2720299201,10cyclictest2588-21wc20:55:302
2719399206,3cyclictest131rcu_preempt00:35:041
2719399203,14cyclictest24334-21head20:10:281
2719399202,15cyclictest28588-21ssh23:55:121
2719399202,14cyclictest13398-21ssh22:45:251
2719399202,12cyclictest30344-21ssh22:16:411
2719399202,12cyclictest23721-21ssh22:51:471
2719399201,16cyclictest1417-21ls19:50:141
2719399201,10cyclictest15538-21latency_hist20:05:011
2718099208,9cyclictest19895-21grep00:30:160
2718099207,8cyclictest12-21ksoftirqd/022:00:150
2718099207,8cyclictest12-21ksoftirqd/022:00:150
2718099201,8cyclictest10185-21/usr/sbin/munin23:05:230
27180992011,6cyclictest20548-21cron00:10:010
2718099201,14cyclictest31652-21uniq20:55:180
2718099201,11cyclictest20200-21grep22:10:280
2721099192,13cyclictest0-21swapper/300:00:023
2721099192,10cyclictest0-21swapper/322:20:143
27210991910,6cyclictest0-21swapper/321:17:563
27210991910,6cyclictest0-21swapper/300:00:253
27210991910,5cyclictest0-21swapper/320:50:153
2720299191,5cyclictest0-21swapper/220:30:142
2719399198,8cyclictest23644-21cron19:40:011
2719399197,9cyclictest25689-21ntp_kernel_err21:15:261
2719399192,13cyclictest8176-21ssh22:05:151
2719399192,12cyclictest9371-21ssh23:04:251
2719399192,12cyclictest15508-21netstat00:25:241
27193991911,6cyclictest31263-21cron20:55:021
2719399191,11cyclictest18456-21ssh23:10:151
2719399190,14cyclictest2829-21ls20:25:191
2718099199,7cyclictest14416-21sed21:30:000
2718099192,13cyclictest20305-21ssh23:50:160
27180991912,4cyclictest8375-21grep22:05:170
27180991912,4cyclictest0-21swapper/021:51:500
27180991912,3cyclictest0-21swapper/021:30:160
2718099191,15cyclictest4073-21fschecks_time21:00:170
2718099191,10cyclictest28564-21/usr/sbin/munin23:55:290
2718099191,10cyclictest15884-21ssh21:10:210
2718099190,16cyclictest23701-21ls19:40:000
2721099189,6cyclictest0-21swapper/323:16:083
2721099189,6cyclictest0-21swapper/321:35:473
2721099189,3cyclictest0-21swapper/320:15:133
2721099183,9cyclictest0-21swapper/322:00:153
2721099183,9cyclictest0-21swapper/322:00:143
2721099182,8cyclictest0-21swapper/319:15:123
2721099182,13cyclictest0-21swapper/323:40:153
2721099182,10cyclictest0-21swapper/320:45:143
2721099182,10cyclictest0-21swapper/320:10:143
2721099182,10cyclictest0-21swapper/319:35:133
2721099181,12cyclictest0-21swapper/319:20:293
2721099181,11cyclictest29566irq/58-eth019:55:153
2720299181,8cyclictest27268-21latency_hist19:10:012
2719399187,8cyclictest17830-21sed22:30:181
2719399185,3cyclictest131rcu_preempt23:30:181
2719399183,10cyclictest0-21swapper/122:01:391
2719399183,10cyclictest0-21swapper/122:01:391
2719399182,11cyclictest12434-21ssh21:27:241
2719399181,9cyclictest31964-21cron19:15:001
2719399181,8cyclictest17806-21kthreadcore20:40:171
27193991812,4cyclictest0-21swapper/123:25:131
2719399181,14cyclictest7114-21apt-get21:45:101
2719399181,13cyclictest20300-21apt-get23:50:161
2718099189,6cyclictest937-21cron19:50:010
2718099182,5cyclictest12-21ksoftirqd/023:00:270
2718099182,11cyclictest31550-21ssh22:18:240
2718099182,10cyclictest0-21swapper/021:05:130
2718099181,14cyclictest9060-21sh22:25:150
2718099181,14cyclictest31256-21apt-get21:40:130
2718099181,13cyclictest12224-21apt-get20:35:130
2718099181,10cyclictest10685-21ls20:00:020
2721099172,9cyclictest0-21swapper/321:00:153
2721099172,9cyclictest0-21swapper/319:45:173
2721099172,9cyclictest0-21swapper/319:40:163
2721099172,10cyclictest0-21swapper/320:35:143
27210991711,4cyclictest0-21swapper/323:25:153
2721099171,12cyclictest0-21swapper/321:05:143
2721099171,11cyclictest0-21swapper/319:25:303
2719399179,6cyclictest31514-21fw_packets21:40:181
2719399176,5cyclictest837-21cut19:15:191
2719399173,11cyclictest5261-21cut19:20:191
2719399172,12cyclictest13045-21switchtime19:25:291
2719399172,10cyclictest0-21swapper/119:45:171
2719399171,6cyclictest6159-21memory23:20:251
27193991711,4cyclictest27257-21usage_Anybus21:35:321
2719399170,9cyclictest8954-21/usr/sbin/munin22:25:231
2719399170,14cyclictest31570-21ls20:55:171
2718099179,6cyclictest1-21systemd20:21:150
2718099174,10cyclictest2787-21fschecks_time20:25:180
2718099173,11cyclictest7643-21kthreadcore19:20:230
2718099173,10cyclictest0-21swapper/020:05:130
2718099171,13cyclictest30428-21timerwakeupswit20:50:300
2718099171,13cyclictest28630-21kthreadcore19:10:200
2718099171,12cyclictest2643-21ssh23:20:150
2718099171,11cyclictest22857-21ssh23:30:240
27180991710,5cyclictest135-21systemd-journal20:12:030
27180991710,4cyclictest7221-21sh21:45:120
2721099162,9cyclictest0-21swapper/319:30:153
2719399169,4cyclictest20880-21/usr/sbin/munin00:10:151
2719399166,7cyclictest9444-21cut21:05:181
2719399162,9cyclictest0-21swapper/120:30:131
2719399162,10cyclictest0-21swapper/120:15:151
2719399161,12cyclictest23980-21apt-get19:40:131
2718099167,6cyclictest9630-21wc19:25:170
2718099162,10cyclictest0-21swapper/019:50:120
2718099162,10cyclictest0-21swapper/019:40:040
2718099161,8cyclictest22508-21kernelversion20:45:180
2718099161,12cyclictest14293-21ntp_states20:00:280
2718099161,11cyclictest7882-21sed20:30:190
2718099161,11cyclictest25563-21ssh22:35:140
2721099152,10cyclictest0-21swapper/320:25:163
2719399151,11cyclictest15824-21fschecks_time20:05:161
2718099153,10cyclictest27177-21cyclictest19:10:010
2718099151,11cyclictest25401-21grep20:15:160
2721099142,8cyclictest0-21swapper/320:05:143
2721099141,10cyclictest0-21swapper/320:00:223
2719399141,10cyclictest27266-21latency_hist19:10:011
2721099122,7cyclictest0-21swapper/320:25:013
2721099111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional