You are here: Home / Technical Services / OSADL QA Farm Real-time / 
2025-12-27 - 04:08

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sat Dec 27, 2025 00:45:15)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
869399575,47cyclictest8837-21ssh23:45:140
870199563,41cyclictest15107-21ssh22:35:141
869399541,42cyclictest3493-21ssh22:10:130
869399504,39cyclictest27117-21ssh22:05:150
869399465,34cyclictest14893-21ssh00:25:130
871399453,29cyclictest28366irq/58-eth019:55:123
869399442,31cyclictest32418-21ssh23:40:130
870199432,8cyclictest131rcu_preempt00:20:121
869399434,4cyclictest0-21swapper/020:40:120
871399425,27cyclictest28366irq/58-eth021:50:113
871399422,24cyclictest28366irq/58-eth000:30:133
870899421,20cyclictest15106-21ssh22:35:142
869399421,35cyclictest6323-21ssh22:30:140
871399413,24cyclictest28366irq/58-eth022:30:153
8701994119,14cyclictest21-21ksoftirqd/100:35:111
871399407,16cyclictest28366irq/58-eth022:55:133
871399404,23cyclictest28366irq/58-eth021:55:133
870199403,22cyclictest16316-1kworker/u9:021:30:131
869399392,12cyclictest12-21ksoftirqd/000:00:120
869399392,12cyclictest12-21ksoftirqd/000:00:120
869399385,25cyclictest26990-21diskmemload22:45:170
869399385,25cyclictest26990-21diskmemload22:45:160
870899373,25cyclictest14734-21ssh23:30:142
870899372,29cyclictest31057-21timerandwakeup22:05:302
870199371,25cyclictest2930-21ssh23:05:101
871399362,23cyclictest28366irq/58-eth022:25:153
870199364,22cyclictest21112-21ssh22:20:161
8701993619,12cyclictest0-21swapper/119:10:141
869399360,27cyclictest0-21swapper/021:50:100
870199353,24cyclictest12264-21ssh22:15:141
8693993515,12cyclictest0-21swapper/000:30:130
871399342,16cyclictest28366irq/58-eth021:20:103
869399341,26cyclictest11906-21ssh00:05:150
869399338,15cyclictest151rcuc/021:10:150
870199325,15cyclictest21-21ksoftirqd/119:45:141
869399322,22cyclictest15162-21ssh21:56:350
870899312,24cyclictest0-21swapper/222:01:292
870899312,18cyclictest12273-21ssh22:15:152
870199312,22cyclictest31058-21timerandwakeup22:05:291
869399310,8cyclictest12-21ksoftirqd/020:10:110
8693993019,6cyclictest21268-21sh21:25:130
8693993019,6cyclictest21268-21sh21:25:130
870899292,20cyclictest0-21swapper/219:45:162
870899292,18cyclictest22346irq/40-dwc2_hso00:35:112
870899292,18cyclictest20942-21grep20:30:012
870899291,19cyclictest5883-21ssh00:20:112
870899291,15cyclictest12488-21ssh21:20:112
870199297,12cyclictest21-21ksoftirqd/100:00:131
870199297,12cyclictest21-21ksoftirqd/100:00:121
870199291,22cyclictest5807-21ssh23:25:141
869399293,20cyclictest29589-21ssh00:15:150
871399280,17cyclictest28366irq/58-eth022:05:153
869399281,7cyclictest12-21ksoftirqd/000:20:120
869399281,21cyclictest17648-21ssh23:50:150
870899273,17cyclictest22346irq/40-dwc2_hso20:55:132
870899271,17cyclictest11637-21ls00:05:022
870899271,17cyclictest11637-21ls00:05:012
870199272,18cyclictest24269-21ssh21:45:141
8713992615,7cyclictest28366irq/58-eth023:40:123
8713992611,10cyclictest28366irq/58-eth021:30:143
870899261,14cyclictest8860-21ssh22:50:012
869399263,17cyclictest17817-21ssh22:55:140
869399263,16cyclictest6527-21ssh21:35:160
869399262,19cyclictest3764-21ssh21:15:130
871399254,12cyclictest0-21swapper/300:25:143
870899254,2cyclictest26990-21diskmemload00:25:452
870899254,16cyclictest22346irq/40-dwc2_hso20:05:112
870899254,16cyclictest22346irq/40-dwc2_hso20:05:112
870899253,14cyclictest32757-21kthreadcore20:40:202
870899252,20cyclictest12151-21latency19:10:222
870899251,7cyclictest1018666ssh21:55:192
870899251,21cyclictest22848-21kthreadcore19:55:222
870899251,16cyclictest23803-21kthreadcore20:30:222
870899251,15cyclictest5408-21ls23:25:012
870899251,15cyclictest3299-21kthreadcore19:35:202
870899251,15cyclictest3299-21kthreadcore19:35:202
870199258,4cyclictest131rcu_preempt22:25:291
870199253,18cyclictest4309-21ssh21:50:231
870199252,16cyclictest839-21munin-node21:40:131
870199252,14cyclictest1820-21latency_hist20:10:011
870199252,14cyclictest1820-21latency_hist20:10:011
8693992511,6cyclictest131rcu_preempt23:12:250
871399247,10cyclictest321rcuc/320:45:133
871399243,14cyclictest0-21swapper/322:11:303
870899242,18cyclictest9045-21timerandwakeup19:40:272
870899242,17cyclictest0-21swapper/221:40:282
870899242,13cyclictest19800-21kthreadcore23:50:222
8708992413,9cyclictest13019-21ssh22:32:492
870899241,21cyclictest25401-21sed21:25:282
870899241,21cyclictest25401-21sed21:25:282
870899240,15cyclictest10123-21kthreadcore20:50:212
870199248,3cyclictest14189-21sed19:15:171
870199241,17cyclictest26990-21diskmemload00:10:141
869399242,19cyclictest1771-21ssh21:30:290
8693992415,5cyclictest0-21swapper/020:25:150
870899236,14cyclictest17729-21sshd19:15:282
870899233,2cyclictest131rcu_preempt22:12:342
870899232,6cyclictest0-21swapper/221:38:292
870899232,3cyclictest26990-21diskmemload22:24:142
870899232,17cyclictest0-21swapper/221:54:292
870899232,15cyclictest0-21swapper/223:05:132
8708992318,3cyclictest924-21snmpd22:29:542
8708992312,9cyclictest30511-21ssh23:38:092
8708992312,9cyclictest15871-21ssh22:53:042
870899231,19cyclictest26784-21ssh23:00:172
870899231,19cyclictest26640-21kthreadcore20:35:192
870899231,14cyclictest25007-21kthreadcore19:25:192
870899231,14cyclictest15562-21mailstats23:10:292
870899231,13cyclictest2759-21latency_hist00:00:012
870899231,13cyclictest2759-21latency_hist00:00:012
870899231,10cyclictest12032-21/usr/sbin/munin00:05:172
8708992310,10cyclictest3988-21sshd22:45:292
870199238,3cyclictest17166systemd-network23:36:491
870199232,7cyclictest6440-21diskstats22:30:171
8693992316,4cyclictest5780-21ssh23:25:120
8693992315,5cyclictest0-21swapper/021:45:130
8693992312,3cyclictest0-21swapper/023:18:080
869399231,19cyclictest26830-21cut23:00:170
8693992311,3cyclictest0-21swapper/022:27:000
871399225,11cyclictest28366irq/58-eth023:50:163
8713992215,5cyclictest0-21swapper/323:15:163
8713992212,6cyclictest28366irq/58-eth023:00:153
8713992212,6cyclictest28366irq/58-eth000:35:113
8713992211,4cyclictest0-21swapper/300:05:153
870899229,4cyclictest27-21ksoftirqd/200:15:272
870899225,9cyclictest1-21systemd21:11:272
870899223,11cyclictest681-21kernelversion23:40:192
870899222,17cyclictest7156-21latency21:15:222
870899221,19cyclictest3006-21kthreadcore20:10:182
870899221,16cyclictest30017-21diskmemload21:30:132
870899221,13cyclictest28873-21kthreadcore19:30:192
870899221,13cyclictest28873-21kthreadcore19:30:182
870199226,3cyclictest131rcu_preempt19:40:171
870199221,18cyclictest16044-21ls20:55:231
870199221,17cyclictest26805-21cut23:00:181
869399229,5cyclictest0-21swapper/020:45:130
869399223,15cyclictest0-21swapper/023:35:110
869399221,3cyclictest131rcu_preempt19:30:130
869399221,3cyclictest131rcu_preempt19:30:130
8693992211,3cyclictest0-21swapper/022:22:500
871399213,12cyclictest0-21swapper/323:35:113
871399212,3cyclictest0-21swapper/323:45:143
8713992114,4cyclictest0-21swapper/322:40:243
871399211,15cyclictest0-21swapper/322:20:193
870899214,14cyclictest12215-21kthreadcore20:20:182
870899214,13cyclictest22346irq/40-dwc2_hso00:10:162
870899212,16cyclictest27501-21grep00:30:282
870899212,16cyclictest17720-21fschecks_count21:00:162
870899212,14cyclictest27-21ksoftirqd/223:25:142
870899212,12cyclictest18195-21tr22:55:182
870899211,8cyclictest24550-21sort21:45:182
870899211,12cyclictest26983-21cron21:10:012
8708992110,9cyclictest24928-21cut22:40:192
870199216,3cyclictest131rcu_preempt21:25:211
870199216,3cyclictest131rcu_preempt21:25:201
870199215,3cyclictest131rcu_preempt22:50:211
870199213,5cyclictest32243-21users21:10:341
870199212,16cyclictest3374-21kthreadcore19:35:201
870199212,16cyclictest3374-21kthreadcore19:35:201
870199212,13cyclictest13759-21ssh21:38:281
870199211,7cyclictest7411-21ntp_states21:15:261
8701992114,5cyclictest0-21swapper/100:18:381
869399219,3cyclictest12-21ksoftirqd/022:17:000
869399219,3cyclictest12-21ksoftirqd/000:10:140
869399218,3cyclictest131rcu_preempt19:50:120
869399216,8cyclictest12-21ksoftirqd/021:40:380
869399215,4cyclictest131rcu_preempt23:55:140
869399215,4cyclictest131rcu_preempt23:55:130
8693992115,4cyclictest0-21swapper/022:00:130
869399211,17cyclictest25944-21find20:35:110
869399211,15cyclictest12622-21uniq21:20:140
869399210,11cyclictest12-21ksoftirqd/023:32:030
8713992013,5cyclictest0-21swapper/300:17:503
8713992010,8cyclictest0-21swapper/321:35:073
871399200,9cyclictest28366irq/58-eth020:35:123
870899202,9cyclictest22425-21kthreadcore23:15:232
870899202,7cyclictest131rcu_preempt23:45:152
870899202,15cyclictest25738-21kthreadcore20:00:202
870899201,16cyclictest5449-21kthreadcore20:45:222
870899201,15cyclictest7936-21kthreadcore20:15:202
870899201,12cyclictest15365-21ps19:50:172
870199209,3cyclictest21-21ksoftirqd/123:42:111
870199204,14cyclictest24831-21timerandwakeup20:30:291
870199203,14cyclictest5729-21grep20:10:281
870199203,14cyclictest1380-21kthreadcore20:40:221
870199202,16cyclictest32187-21latency_hist00:35:011
870199202,16cyclictest16495-21fschecks_time20:25:161
870199202,15cyclictest8095-21kthreadcore20:15:201
870199202,15cyclictest30028-21grep23:55:241
870199202,15cyclictest30028-21grep23:55:241
870199202,15cyclictest18708-21kthreadcore23:50:211
8701992012,5cyclictest853-21irqrtprio22:45:211
8701992012,5cyclictest853-21irqrtprio22:45:211
8701992011,7cyclictest16756-21cat00:05:361
8701992011,6cyclictest21526-21ntp_states22:55:261
870199201,14cyclictest12479-21sh21:20:111
869399209,3cyclictest12-21ksoftirqd/022:41:320
869399208,9cyclictest17528-21head19:15:290
869399202,13cyclictest12-21ksoftirqd/020:15:130
8693992013,4cyclictest0-21swapper/022:39:040
869399201,15cyclictest20016-21cut19:55:160
8693992010,7cyclictest29751-21cut23:20:170
869399200,13cyclictest5504-21cut19:40:160
871399199,6cyclictest0-21swapper/323:07:223
871399197,6cyclictest0-21swapper/323:25:213
871399193,11cyclictest0-21swapper/321:41:293
8713991913,4cyclictest0-21swapper/322:51:093
8713991913,4cyclictest0-21swapper/322:45:173
8713991913,4cyclictest0-21swapper/322:45:173
8713991912,5cyclictest0-21swapper/321:46:013
8713991912,5cyclictest0-21swapper/300:21:523
871399191,13cyclictest0-21swapper/321:15:293
871399191,12cyclictest0-21swapper/320:55:173
871399191,11cyclictest0-21swapper/319:45:023
871399191,10cyclictest0-21swapper/319:55:033
8713991910,6cyclictest0-21swapper/322:00:303
870899192,9cyclictest18472-21/usr/sbin/munin19:20:172
870899191,15cyclictest8798-21latency_hist19:10:012
870199199,2cyclictest21-21ksoftirqd/123:47:421
870199193,3cyclictest26990-21diskmemload23:15:501
870199192,5cyclictest630-21tcf-agent23:33:051
870199192,15cyclictest32166-21tr23:20:211
870199192,15cyclictest23371-21wc19:55:271
870199192,14cyclictest21153-21grep21:00:271
870199191,8cyclictest26745-21sed22:05:031
870199191,8cyclictest15979-21grep00:25:201
8701991911,6cyclictest9455-21apt21:55:111
8701991911,6cyclictest23056-21sshd19:21:301
869399195,8cyclictest12-21ksoftirqd/023:05:070
8693991913,4cyclictest0-21swapper/019:10:140
8693991913,3cyclictest12-21ksoftirqd/020:50:300
869399191,11cyclictest0-21swapper/022:54:010
871399189,7cyclictest0-21swapper/323:14:123
871399182,8cyclictest0-21swapper/320:30:133
871399182,11cyclictest0-21swapper/319:30:133
871399182,11cyclictest0-21swapper/319:30:133
8713991812,4cyclictest0-21swapper/323:55:153
8713991812,4cyclictest0-21swapper/323:55:153
8713991812,4cyclictest0-21swapper/323:20:183
8713991812,4cyclictest0-21swapper/321:25:153
8713991812,4cyclictest0-21swapper/321:25:153
8713991811,5cyclictest0-21swapper/323:30:193
8713991810,6cyclictest0-21swapper/321:10:553
8713991810,6cyclictest0-21swapper/300:10:153
870199188,7cyclictest23958-21apt-get22:40:131
870199183,12cyclictest6974-21unixbench_singl20:45:281
870199183,12cyclictest25183-21grep20:00:181
870199182,14cyclictest14677-21kthreadcore20:20:221
870199182,13cyclictest11863-21unixbench_singl20:50:291
870199181,8cyclictest24048-21kthreadcore19:25:181
8701991810,5cyclictest4887-21kthreadcore22:10:191
8701991810,5cyclictest0-21swapper/119:50:211
869399187,8cyclictest24812-21tune2fs20:00:170
869399186,3cyclictest131rcu_preempt19:20:280
869399184,11cyclictest503-21cut00:35:170
8693991812,4cyclictest0-21swapper/019:35:290
8693991812,4cyclictest0-21swapper/019:35:290
869399181,13cyclictest28002-21latency_hist19:30:010
871399179,6cyclictest28366irq/58-eth022:15:133
871399177,6cyclictest321rcuc/319:25:133
8713991711,4cyclictest0-21swapper/322:35:163
8713991711,4cyclictest0-21swapper/320:25:293
8713991711,4cyclictest0-21swapper/320:10:253
8713991711,4cyclictest0-21swapper/300:00:223
8713991711,4cyclictest0-21swapper/300:00:223
871399171,10cyclictest0-21swapper/320:15:293
871399171,10cyclictest0-21swapper/320:05:263
871399171,10cyclictest0-21swapper/320:05:263
870199176,2cyclictest131rcu_preempt20:35:151
870199171,12cyclictest26992-21latency_hist21:10:021
869399175,9cyclictest10286-21cut19:45:170
869399174,10cyclictest19500-21cut21:00:210
869399173,9cyclictest0-21swapper/020:20:110
869399173,4cyclictest0-21swapper/021:05:140
869399172,12cyclictest12902-21grep20:55:170
869399171,3cyclictest131rcu_preempt20:10:020
869399171,3cyclictest131rcu_preempt20:10:010
870199162,12cyclictest29965-21kthreadcore19:30:201
870199162,12cyclictest29965-21kthreadcore19:30:191
870199162,12cyclictest12234-21ssh23:10:191
869399161,11cyclictest21322-21tune2fs20:30:170
871399151,8cyclictest0-21swapper/320:40:123
870199152,11cyclictest8722-21latency_hist19:10:011
869399151,10cyclictest8720-21latency_hist19:10:020
871399147,5cyclictest321rcuc/320:00:133
871399141,7cyclictest0-21swapper/320:20:143
871399141,7cyclictest0-21swapper/319:45:163
871399141,6cyclictest0-21swapper/321:05:233
871399141,11cyclictest0-21swapper/319:35:273
871399141,11cyclictest0-21swapper/319:35:263
871399138,3cyclictest0-21swapper/321:04:393
871399122,6cyclictest0-21swapper/320:50:053
871399122,6cyclictest0-21swapper/319:15:133
871399121,8cyclictest0-21swapper/319:10:263
871399121,7cyclictest0-21swapper/319:20:283
871399111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional