You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-10 - 19:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot4s.osadl.org (updated Wed Dec 10, 2025 12:45:06)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
148999690,60cyclictest0-21swapper/211:05:122
1489996132,23cyclictest0-21swapper/211:00:122
1497995835,14cyclictest29566irq/58-eth011:40:123
147199573,37cyclictest5922-21ssh09:45:150
147199572,43cyclictest9709-21ssh10:10:140
148999503,42cyclictest8881-21apt-get11:40:122
148999492,37cyclictest0-21swapper/211:25:142
1471994830,8cyclictest12-21ksoftirqd/007:40:140
148999471,41cyclictest15538-21apt-get09:05:132
148999463,21cyclictest13339-21ssh09:50:142
148999463,21cyclictest13339-21ssh09:50:142
148999461,40cyclictest31154-21apt-get10:25:132
148999461,25cyclictest5483-21memory09:20:222
148999451,39cyclictest23948-21apt-get10:20:122
148999431,37cyclictest11302-21apt-get07:20:152
148999425,33cyclictest26346irq/40-dwc2_hso12:10:122
148999422,34cyclictest16327-21apt-get11:45:142
148999420,23cyclictest30221-21sh11:53:252
148999412,28cyclictest2072-21ssh10:50:132
148999411,33cyclictest19058-21apt-get08:35:142
148999411,32cyclictest0-21swapper/209:10:122
148999411,26cyclictest0-21swapper/212:30:242
148999410,35cyclictest23620-21apt-get09:35:122
148999403,35cyclictest26346irq/40-dwc2_hso11:55:142
148999401,33cyclictest5747-21apt-get10:30:132
1497993925,9cyclictest29566irq/58-eth010:30:133
148999391,35cyclictest21035-21df07:30:172
148999390,33cyclictest26346irq/40-dwc2_hso10:35:142
148999381,35cyclictest932-21snmpd12:39:132
148999381,34cyclictest0-21swapper/212:15:132
148999381,32cyclictest13721-21apt-get12:05:152
148999372,31cyclictest26413-21grep08:05:282
148999371,32cyclictest24718-21/usr/sbin/munin09:55:322
148999371,27cyclictest17529-21latency_hist08:00:012
148999370,32cyclictest0-21swapper/211:10:112
148999370,28cyclictest1049-21sshd08:50:142
148999370,28cyclictest1049-21sshd08:50:132
148999362,30cyclictest0-21swapper/211:20:142
148999362,29cyclictest26346irq/40-dwc2_hso10:55:132
148999361,33cyclictest5912-21ssh09:45:142
148999361,32cyclictest8081-21/usr/sbin/munin07:50:162
148999361,31cyclictest9942-21ntp_states07:15:272
148999361,31cyclictest32273-21df08:15:172
148999361,31cyclictest20315-21diskstats10:40:172
148999361,31cyclictest1931-21memory09:40:242
148999361,31cyclictest12972-21/usr/sbin/munin08:25:272
148999361,29cyclictest0-21swapper/208:30:122
148999361,25cyclictest831466rm11:39:342
148999361,25cyclictest3103-21/usr/sbin/munin07:45:192
148999360,32cyclictest0-21swapper/211:15:152
148999360,26cyclictest0-21swapper/210:05:152
148099361,29cyclictest13039-21ssh10:35:131
147199368,18cyclictest8888-21ssh11:40:120
147199360,28cyclictest18750-21ssh11:25:130
148999353,29cyclictest13519-21cat10:10:312
148999351,31cyclictest2022-21sed07:40:292
148999351,31cyclictest0-21swapper/210:00:132
148999351,30cyclictest9390-21ssh12:22:462
148999350,32cyclictest5061-21/usr/sbin/munin07:10:242
148999350,31cyclictest16277-21apt-get09:30:122
148999350,2cyclictest16782-21ssh10:15:172
148999350,26cyclictest31268-21ntp_kernel_pll_10:45:282
1497993427,4cyclictest29566irq/58-eth009:30:123
148999340,31cyclictest5878-21acpi08:55:152
148999340,30cyclictest29232-21cut08:45:212
148999340,28cyclictest0-21swapper/207:35:152
147199341,3cyclictest131rcu_preempt10:30:130
148999331,29cyclictest30789-21ntp_192.168.11408:10:242
148999331,29cyclictest24325-21cut08:40:212
148999331,29cyclictest20133-21diskmemload09:29:552
148999331,27cyclictest0-21swapper/212:00:142
148999331,24cyclictest14607-21/usr/sbin/munin12:25:232
148999331,24cyclictest14607-21/usr/sbin/munin12:25:232
149799323,15cyclictest29566irq/58-eth011:45:143
148999322,27cyclictest6827-21kthreadcore08:20:212
148999322,27cyclictest28413-21cut11:30:202
148999322,26cyclictest0-21swapper/208:00:132
148999321,28cyclictest28004-21irqcore09:15:192
148999321,28cyclictest15298-21sshd09:04:132
148999320,24cyclictest16367-21sed07:25:172
1471993216,10cyclictest0-21swapper/008:35:130
1471993010,14cyclictest12-21ksoftirqd/008:20:140
1497992915,10cyclictest0-21swapper/309:10:143
148999291,25cyclictest26346irq/40-dwc2_hso07:10:012
149799283,17cyclictest0-21swapper/312:39:473
148099282,20cyclictest27049-21irqcore12:35:191
1471992816,9cyclictest27529-21ssh09:35:300
147199281,19cyclictest27050-21irqcore12:35:190
148099273,21cyclictest6483-21ssh12:00:141
1497992612,8cyclictest29566irq/58-eth009:45:153
148099261,22cyclictest0-21swapper/110:05:171
149799254,11cyclictest29566irq/58-eth012:20:123
149799253,16cyclictest0-21swapper/308:10:123
148099255,17cyclictest11359-21ssh11:20:131
148099254,18cyclictest21-21ksoftirqd/109:40:141
147199255,10cyclictest0-21swapper/007:20:150
147199250,14cyclictest2077-21sshd10:50:130
1497992414,7cyclictest29566irq/58-eth011:50:163
148099241,19cyclictest28844-21ssh11:10:121
148099241,18cyclictest24620-21ssh12:32:191
148099241,17cyclictest13748-21ssh12:05:151
147199247,12cyclictest12-21ksoftirqd/010:40:140
149799233,15cyclictest29566irq/58-eth012:10:123
1497992311,9cyclictest0-21swapper/311:35:183
148099237,8cyclictest5111-21irqrtprio08:20:181
1480992314,4cyclictest131rcu_preempt10:50:151
148099231,19cyclictest3970-21find11:15:131
148099231,18cyclictest24366-21ls09:10:281
147199231,3cyclictest131rcu_preempt12:10:120
149799229,10cyclictest29566irq/58-eth011:05:133
149799224,11cyclictest0-21swapper/310:35:133
149799222,16cyclictest29566irq/58-eth011:10:133
148099223,17cyclictest26267-21ssh11:30:121
148099221,17cyclictest27598-21ssh09:15:151
148099221,17cyclictest1407-21sshd11:35:141
147199226,3cyclictest131rcu_preempt10:00:010
147199225,12cyclictest1463-21cyclictest09:05:120
147199222,16cyclictest842-21munin-node09:50:150
147199222,16cyclictest842-21munin-node09:50:150
147199221,17cyclictest2077-21cpuspeed09:20:130
147199221,15cyclictest11100-21ssh12:25:140
147199221,15cyclictest11100-21ssh12:25:140
1497992112,5cyclictest0-21swapper/311:20:143
1497992112,5cyclictest0-21swapper/310:30:023
1497992110,7cyclictest29566irq/58-eth012:00:153
148099214,9cyclictest0-21swapper/111:45:141
148099212,15cyclictest22140-21ssh11:05:131
148099211,5cyclictest53046usb-storage07:50:041
148099211,5cyclictest28018-21usage_Anybus09:35:311
1480992113,5cyclictest21739-21cat10:40:231
148099211,15cyclictest20718-21ssh09:55:171
148099211,14cyclictest21168-21ssh12:10:131
148099211,13cyclictest14774-21grep07:55:211
147199212,6cyclictest24032-21df_inode11:50:170
147199211,8cyclictest16422-21fschecks_count09:30:160
1471992114,5cyclictest5657-21cron08:55:010
1471992114,5cyclictest5657-21cron08:55:010
147199211,16cyclictest31345-21ssh11:55:150
149799204,12cyclictest0-21swapper/312:05:153
149799202,13cyclictest0-21swapper/310:45:163
149799202,11cyclictest0-21swapper/307:15:143
149799202,11cyclictest0-21swapper/307:10:153
1497992011,7cyclictest0-21swapper/309:20:193
1497992011,6cyclictest0-21swapper/310:20:213
149799201,14cyclictest29566irq/58-eth011:30:123
1497992010,6cyclictest0-21swapper/310:05:273
148099209,8cyclictest12276-21kthreadcore10:10:221
148099207,7cyclictest0-21swapper/109:05:131
148099202,3cyclictest131rcu_preempt11:55:141
148099202,15cyclictest17733-21apt-get08:00:121
148099201,16cyclictest6915-21kthreadcore09:45:201
1480992011,6cyclictest6224-21sshd07:13:461
148099201,16cyclictest15004-21ssh11:00:131
1480992011,6cyclictest10665-21latency_hist09:00:011
148099201,14cyclictest27124-21latency_hist08:10:011
148099200,17cyclictest5816-21ls10:30:151
148099200,17cyclictest25584-21ls07:35:021
148099200,16cyclictest8338-21ps07:15:221
147199207,2cyclictest131rcu_preempt09:15:010
147199206,4cyclictest131rcu_preempt08:25:190
147199202,15cyclictest8956-21ssh09:25:120
1471992016,2cyclictest530-21usb-storage07:45:040
1471992011,7cyclictest31810-21sed10:00:300
147199201,16cyclictest18790-21ssh12:30:140
147199201,10cyclictest15241-21cat11:20:290
1471992010,7cyclictest20716-21cron07:30:010
149799199,7cyclictest0-21swapper/310:11:583
149799198,7cyclictest0-21swapper/310:15:153
149799193,11cyclictest0-21swapper/312:32:193
149799192,11cyclictest0-21swapper/309:35:143
149799192,10cyclictest0-21swapper/308:05:143
1497991911,5cyclictest0-21swapper/309:55:323
149799191,13cyclictest0-21swapper/309:50:143
149799191,13cyclictest0-21swapper/309:50:143
149799191,12cyclictest0-21swapper/309:05:023
149799191,11cyclictest0-21swapper/310:50:153
148099197,3cyclictest131rcu_preempt09:50:211
148099197,3cyclictest131rcu_preempt09:50:211
148099195,9cyclictest0-21swapper/111:40:121
148099191,9cyclictest13410-21tr12:25:221
148099191,9cyclictest13410-21tr12:25:211
1480991912,4cyclictest0-21swapper/110:20:291
148099191,15cyclictest32180-21find08:15:141
148099191,14cyclictest32444-21users10:45:341
148099191,12cyclictest23932-21ssh11:50:151
148099190,16cyclictest22307-21ls08:35:231
147199197,9cyclictest15645-21sh10:35:220
147199192,14cyclictest13747-21ssh12:05:150
147199191,14cyclictest16773-21ssh10:15:160
147199191,10cyclictest29379-21kthreadcore12:15:210
1471991910,6cyclictest23621-21cron08:40:020
147199190,14cyclictest27343-21ls10:20:220
149799182,8cyclictest0-21swapper/307:35:143
149799182,10cyclictest0-21swapper/307:50:053
1497991811,4cyclictest0-21swapper/312:15:153
149799181,11cyclictest0-21swapper/311:15:173
149799181,10cyclictest0-21swapper/307:55:043
1497991810,5cyclictest0-21swapper/310:40:293
148099184,8cyclictest0-21swapper/107:35:141
148099183,13cyclictest17111-21kthreadcore07:25:201
148099183,12cyclictest7366-21cat12:20:281
148099182,12cyclictest29061-21kthreadcore08:10:201
1480991811,5cyclictest0-21swapper/108:40:291
148099181,14cyclictest3293-21cut09:20:191
148099181,13cyclictest30546-21ps08:45:231
1480991810,5cyclictest30232-21sh10:00:211
148099180,7cyclictest19826-21kthreadcore11:25:201
147199189,6cyclictest9082-21uname10:55:210
147199189,6cyclictest29374-21kthreadcore11:30:220
147199188,8cyclictest22571-21apt-get08:05:130
147199185,10cyclictest6165-21kthreadcore12:20:210
147199182,13cyclictest23708-21latency_hist11:50:020
1471991813,3cyclictest0-21swapper/008:55:280
1471991812,4cyclictest31345-21processes09:15:290
1471991811,5cyclictest6493-21ssh10:05:320
147199181,14cyclictest2294-21cut11:35:190
147199181,13cyclictest31679-21ssh10:45:310
147199181,12cyclictest2035-21sh09:40:250
1471991810,6cyclictest4935-21ssh10:28:560
1471991810,6cyclictest1519-21ssh11:11:010
1471991810,6cyclictest12766-21ssh12:03:420
1471991810,6cyclictest0-21swapper/007:55:010
149799179,6cyclictest0-21swapper/311:57:253
149799179,6cyclictest0-21swapper/311:26:193
149799179,5cyclictest29566irq/58-eth010:00:123
149799172,9cyclictest0-21swapper/309:40:143
149799172,9cyclictest0-21swapper/308:30:143
149799172,9cyclictest0-21swapper/307:30:173
149799172,12cyclictest0-21swapper/311:00:223
149799172,10cyclictest0-21swapper/309:05:143
149799171,13cyclictest0-21swapper/309:15:233
149799171,12cyclictest0-21swapper/308:55:193
149799171,11cyclictest0-21swapper/307:25:173
149799171,10cyclictest0-21swapper/308:50:163
149799171,10cyclictest0-21swapper/308:50:163
1497991710,4cyclictest0-21swapper/310:55:363
148099174,10cyclictest11735-21kthreadcore09:25:201
148099173,11cyclictest10811-21kthreadcore08:25:201
148099172,12cyclictest14474-21idleruntime08:30:171
148099172,12cyclictest11474-21fschecks_time07:20:191
148099172,12cyclictest10266-21kthreadcore10:55:221
148099172,11cyclictest21-21ksoftirqd/109:00:151
148099172,10cyclictest0-21swapper/110:25:141
148099171,4cyclictest131rcu_preempt09:30:161
1480991711,4cyclictest0-21swapper/112:15:141
148099171,13cyclictest3736-21kthreadcore08:50:221
148099171,13cyclictest3736-21kthreadcore08:50:221
148099171,12cyclictest5361-21tr07:45:231
148099171,12cyclictest17141-21ssh10:15:211
147199175,4cyclictest131rcu_preempt09:00:190
147199172,10cyclictest0-21swapper/007:55:150
147199171,8cyclictest27514-21fschecks_time08:10:160
147199171,12cyclictest22238-21df_inode11:05:150
147199171,12cyclictest14987-21apt-get11:00:120
147199171,12cyclictest11170-21sort11:20:020
149799168,6cyclictest0-21swapper/312:25:113
149799168,6cyclictest0-21swapper/312:25:113
149799162,9cyclictest0-21swapper/308:40:173
148099161,12cyclictest32343-21cut07:40:211
147199169,5cyclictest10262-21sed07:15:300
147199165,8cyclictest14228-21sort08:30:120
147199161,13cyclictest32189-21apt-get08:15:150
149799159,4cyclictest0-21swapper/307:20:203
149799151,11cyclictest0-21swapper/309:25:203
149799151,11cyclictest0-21swapper/308:35:173
148099151,11cyclictest1508-21ls07:10:011
147199158,5cyclictest1463-21cyclictest07:10:010
147199157,6cyclictest1745-21uniq07:10:170
147199152,9cyclictest0-21swapper/008:45:170
147199152,9cyclictest0-21swapper/008:00:140
147199151,12cyclictest20951-21apt-get07:30:170
147199151,10cyclictest26009-21grep07:35:180
149799149,3cyclictest0-21swapper/308:19:193
149799149,3cyclictest0-21swapper/307:40:273
149799142,7cyclictest0-21swapper/307:50:023
149799141,9cyclictest0-21swapper/308:45:183
149799138,3cyclictest0-21swapper/308:00:253
149799135,6cyclictest0-21swapper/308:25:163
149799121,9cyclictest0-21swapper/308:20:183
149799101,6cyclictest0-21swapper/307:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional