You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-23 - 22:33

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Sun Nov 23, 2025 12:45:14)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2318199531,42cyclictest12510-21ssh09:55:140
2318199531,42cyclictest12510-21ssh09:55:130
2319199513,6cyclictest131rcu_preempt09:25:141
2318199501,38cyclictest31334-21ssh11:50:160
2319199487,9cyclictest131rcu_preempt11:00:131
2319199483,9cyclictest131rcu_preempt11:45:121
2319199474,7cyclictest131rcu_preempt10:05:131
2319199472,9cyclictest131rcu_preempt10:20:141
2319199471,8cyclictest131rcu_preempt10:25:131
2319199453,35cyclictest8665-21ssh11:15:141
2319199451,8cyclictest10915-21ssh10:35:141
2319199449,5cyclictest131rcu_preempt09:05:131
23191994311,21cyclictest21-21ksoftirqd/108:55:131
2321299422,25cyclictest29566irq/58-eth010:35:153
2320099402,21cyclictest14279-21ssh12:00:142
2320099402,19cyclictest20638-21diskstats10:00:152
2319199392,26cyclictest16412-21ssh11:20:131
23212993822,8cyclictest0-21swapper/308:10:153
23200993824,2cyclictest18875-21apt-get10:40:142
2319199382,24cyclictest22213-21ssh12:05:141
2319199382,24cyclictest22213-21ssh12:05:141
23200993724,8cyclictest2194-21apt-get10:50:162
23200993724,8cyclictest2194-21apt-get10:50:152
2321299363,16cyclictest29566irq/58-eth010:20:153
2321299362,18cyclictest29566irq/58-eth009:55:143
2321299362,18cyclictest29566irq/58-eth009:55:133
23212993611,18cyclictest0-21swapper/312:15:153
2320099361,27cyclictest12507-21diskmemload09:55:132
2320099361,27cyclictest12507-21diskmemload09:55:132
2320099361,20cyclictest14486-21ssh12:20:152
2319199364,28cyclictest31136-21ssh12:30:161
2319199364,27cyclictest17287-21ssh09:15:111
2319199362,29cyclictest22771-21ssh12:25:111
2319199361,10cyclictest28307-21sed11:25:281
2319199361,10cyclictest12766-21kthreadcore10:55:231
2321299351,17cyclictest29566irq/58-eth011:40:143
2319199352,29cyclictest20510-21ssh10:00:121
2319199351,30cyclictest18886-21ssh10:40:131
2319199351,10cyclictest22664-21latency_hist12:25:011
2319199344,9cyclictest21889-21latency_hist12:05:011
2319199342,29cyclictest6270-21ssh12:15:131
2319199341,10cyclictest19728-21sed11:40:301
2319199341,10cyclictest16010-21cat09:55:221
2319199341,10cyclictest16010-21cat09:55:221
2318199340,27cyclictest2983-21ssh10:30:120
23200993323,7cyclictest17289-21apt-get09:15:132
2319199334,26cyclictest29869-21ssh09:45:151
2319199331,9cyclictest15311-21ssh11:40:011
2319199331,9cyclictest15311-21ssh11:40:011
2318199334,19cyclictest12-21ksoftirqd/008:45:140
2319199328,3cyclictest0-21swapper/111:57:281
2319199327,5cyclictest29345-21kthreadcore10:45:201
2319199322,10cyclictest17870-21kthreadcore08:10:201
2319199321,11cyclictest17018-21grep09:15:021
2319199321,10cyclictest24450-21munin-run07:45:011
2320099316,12cyclictest0-21swapper/211:29:112
2320099315,21cyclictest0-21swapper/209:50:172
2320099310,15cyclictest10128-21sed12:15:292
2319199318,4cyclictest1058-21idleruntime11:10:171
2319199318,4cyclictest1058-21idleruntime11:10:171
2319199317,6cyclictest24644-21awk08:15:261
2319199313,9cyclictest3151-21fschecks_count10:30:161
2319199312,11cyclictest31507-21cut11:50:171
2319199312,10cyclictest23800-21irqcore07:10:181
2319199311,9cyclictest15019-21ssh09:35:121
2319199311,10cyclictest3783-21hwlatdetect10:10:191
2320099307,12cyclictest0-21swapper/212:05:062
2320099307,12cyclictest0-21swapper/212:05:062
2320099302,21cyclictest14907-21ssh09:35:052
2319199304,19cyclictest6924-21ssh12:35:141
2319199303,7cyclictest25128-21ssh09:20:141
2319199301,9cyclictest29978-21ps07:50:181
2319199301,9cyclictest29978-21ps07:50:181
2319199301,10cyclictest9073-21tr09:30:201
2319199301,10cyclictest32138-21ssh11:30:131
2319199301,10cyclictest13351-21kthreadcore07:30:221
2318199306,4cyclictest131rcu_preempt11:35:140
2318199306,4cyclictest131rcu_preempt11:35:130
2321299293,17cyclictest29566irq/58-eth012:20:133
2321299292,15cyclictest29566irq/58-eth010:00:113
2320099298,8cyclictest0-21swapper/211:41:412
2320099297,13cyclictest0-21swapper/210:13:572
2320099292,11cyclictest25596-21ssh11:05:052
23200992919,7cyclictest2978-21apt-get10:30:122
2320099291,8cyclictest932-21snmpd11:15:132
2319199295,5cyclictest29103-21kthreadcore07:15:211
2319199294,17cyclictest4216-21kworker/u8:012:10:141
2319199291,9cyclictest9728-21date10:55:011
2319199291,9cyclictest9728-21date10:55:011
2319199291,10cyclictest11823-21xargs08:35:281
2319199291,10cyclictest11823-21xargs08:35:271
2319199290,10cyclictest14831-21cut08:40:211
2320099283,21cyclictest0-21swapper/212:38:142
2320099283,20cyclictest0-21swapper/210:06:102
2320099281,21cyclictest26346irq/40-dwc2_hso09:30:132
2319199282,9cyclictest5199-21kthreadcore08:30:201
2319199282,9cyclictest29505-21timerandwakeup11:05:431
2319199281,9cyclictest24857-21fschecks_time07:45:171
2319199281,9cyclictest11809-21df10:15:171
2319199281,10cyclictest5787-21cut07:25:161
2319199281,10cyclictest5787-21cut07:25:161
2320099274,12cyclictest0-21swapper/210:25:412
23200992722,2cyclictest22303-21apt-get09:40:132
2320099272,19cyclictest0-21swapper/207:30:132
2320099272,16cyclictest26346irq/40-dwc2_hso11:50:132
23200992717,7cyclictest29476-21apt-get07:50:122
23200992717,7cyclictest29476-21apt-get07:50:122
23200992717,4cyclictest26819-21apt-get10:45:122
23200992711,11cyclictest0-21swapper/210:15:152
2319199271,10cyclictest25950-21ntp_states09:40:261
2318199277,7cyclictest12-21ksoftirqd/012:18:020
2318199272,19cyclictest30143-21ssh11:27:130
2321299263,11cyclictest29566irq/58-eth012:35:143
2320099266,4cyclictest0-21swapper/212:30:562
2320099262,15cyclictest26346irq/40-dwc2_hso10:55:152
23200992620,3cyclictest26080-21apt-get08:20:142
23200992612,5cyclictest10906-21apt-get10:35:132
2320099261,15cyclictest26346irq/40-dwc2_hso07:40:142
2319199266,6cyclictest23757-21kthreadcore08:50:191
2318199267,7cyclictest12-21ksoftirqd/010:27:140
2318199266,8cyclictest12-21ksoftirqd/010:45:000
2318199263,12cyclictest12-21ksoftirqd/012:32:260
2318199262,21cyclictest6634-21ssh11:55:130
2321299253,14cyclictest0-21swapper/309:35:043
23212992510,9cyclictest29566irq/58-eth011:55:133
2320099254,13cyclictest0-21swapper/209:05:142
2320099253,4cyclictest27-21ksoftirqd/211:20:042
2320099252,9cyclictest27-21ksoftirqd/211:10:282
2320099252,9cyclictest27-21ksoftirqd/211:10:282
2320099251,9cyclictest30396-21ssh12:10:152
2318199258,7cyclictest12-21ksoftirqd/010:22:560
2318199257,7cyclictest12-21ksoftirqd/011:45:550
2318199253,4cyclictest12-21ksoftirqd/012:30:020
2318199252,18cyclictest22794-21ssh11:23:140
23181992512,7cyclictest0-21swapper/009:13:100
2318199251,15cyclictest12-21ksoftirqd/009:15:120
23181992510,7cyclictest12-21ksoftirqd/009:46:130
2321299243,15cyclictest0-21swapper/311:05:043
23212992412,9cyclictest0-21swapper/309:15:253
2320099242,16cyclictest0-21swapper/210:20:102
23200992416,4cyclictest29848-21apt-get09:45:162
23200992413,8cyclictest6617-21apt-get11:55:132
23200992412,10cyclictest24296-21kthreadcore11:45:202
2320099241,14cyclictest27-21ksoftirqd/211:01:042
2319199242,9cyclictest1545-21kthreadcore09:00:191
2318199249,6cyclictest12-21ksoftirqd/012:02:540
2318199249,4cyclictest12-21ksoftirqd/010:00:490
2318199248,7cyclictest12-21ksoftirqd/010:37:310
2318199248,7cyclictest12-21ksoftirqd/010:13:000
2318199248,10cyclictest0-21swapper/011:19:570
2318199247,7cyclictest12-21ksoftirqd/010:51:120
2318199247,7cyclictest12-21ksoftirqd/010:51:120
2318199247,4cyclictest12-21ksoftirqd/010:46:060
2318199246,8cyclictest12-21ksoftirqd/012:06:450
2318199246,8cyclictest12-21ksoftirqd/012:06:440
23181992412,9cyclictest21107-21sshd09:38:100
2321299239,8cyclictest29566irq/58-eth011:25:133
23212992313,7cyclictest29566irq/58-eth012:10:143
2321299231,16cyclictest0-21swapper/311:20:133
2320099232,10cyclictest0-21swapper/207:20:132
23200992312,9cyclictest30292-21ssh09:21:082
2320099230,10cyclictest19179-21cat07:35:302
23191992315,5cyclictest21-21ksoftirqd/108:20:141
2318199239,4cyclictest151rcuc/012:10:160
2318199238,8cyclictest12-21ksoftirqd/011:14:540
2318199238,8cyclictest12-21ksoftirqd/011:14:540
2318199238,6cyclictest12-21ksoftirqd/011:32:360
2318199237,7cyclictest12-21ksoftirqd/010:56:250
2318199236,8cyclictest12-21ksoftirqd/010:16:040
2318199236,7cyclictest12-21ksoftirqd/008:40:240
2318199236,4cyclictest12-21ksoftirqd/009:28:200
2318199235,7cyclictest12-21ksoftirqd/011:01:040
2318199235,3cyclictest131rcu_preempt12:20:070
2318199233,4cyclictest131rcu_preempt12:35:170
2318199232,15cyclictest899-21ssh10:06:110
2318199232,15cyclictest25279-21apt-get11:05:040
23181992310,6cyclictest12-21ksoftirqd/009:22:150
2321299223,12cyclictest0-21swapper/312:00:153
2320099229,3cyclictest0-21swapper/208:19:502
2320099227,13cyclictest261rcuc/211:31:472
2320099226,4cyclictest0-21swapper/212:28:092
2320099226,4cyclictest0-21swapper/209:25:122
2320099225,14cyclictest27824-21/usr/sbin/munin08:55:182
23200992214,4cyclictest28170-21apt-get07:15:122
2320099221,14cyclictest0-21swapper/211:38:332
2320099221,14cyclictest0-21swapper/211:38:322
2318199228,8cyclictest12-21ksoftirqd/008:05:260
2318199228,7cyclictest12-21ksoftirqd/007:35:160
2318199228,3cyclictest131rcu_preempt09:40:050
2318199227,3cyclictest12-21ksoftirqd/007:15:310
2318199226,4cyclictest12-21ksoftirqd/011:43:170
2318199224,8cyclictest131rcu_preempt08:25:240
2318199223,6cyclictest12-21ksoftirqd/007:25:150
2318199223,6cyclictest12-21ksoftirqd/007:25:150
2318199221,3cyclictest131rcu_preempt07:45:290
23181992210,4cyclictest12-21ksoftirqd/009:52:530
23212992110,7cyclictest29566irq/58-eth010:45:133
2320099217,11cyclictest16482-21df_inode08:10:152
2320099216,7cyclictest9235-21grep07:25:292
2320099216,7cyclictest9235-21grep07:25:282
23200992110,4cyclictest0-21swapper/208:00:262
2318199219,4cyclictest12-21ksoftirqd/008:30:330
2318199218,7cyclictest0-21swapper/008:55:260
2318199216,3cyclictest131rcu_preempt07:20:250
2318199214,8cyclictest12-21ksoftirqd/008:00:000
2318199214,5cyclictest12-21ksoftirqd/009:30:380
2318199213,9cyclictest12-21ksoftirqd/009:05:270
2321299205,9cyclictest29566irq/58-eth010:10:163
2321299205,7cyclictest29566irq/58-eth011:15:153
2321299202,11cyclictest0-21swapper/312:05:143
2321299202,11cyclictest0-21swapper/312:05:133
23212992011,7cyclictest0-21swapper/309:10:293
2320099206,7cyclictest23553-21wc07:10:182
2320099206,6cyclictest1581-21kthreadcore08:25:222
2320099206,12cyclictest3962-21irqstats08:30:182
2320099202,9cyclictest8113-21cpu08:35:132
2320099202,9cyclictest8113-21cpu08:35:132
2320099202,5cyclictest27-21ksoftirqd/208:50:262
2320099202,10cyclictest324-21irqcore09:00:172
2320099201,9cyclictest23270-21latency_hist07:10:012
23200992011,7cyclictest0-21swapper/208:40:122
2320099201,16cyclictest2884-21kthreadcore07:55:212
2320099201,15cyclictest10454-21kthreadcore09:10:192
2320099201,10cyclictest26859-21kthreadcore07:45:222
2320099201,10cyclictest20472-21kthreadcore08:45:232
2318199206,3cyclictest131rcu_preempt09:00:180
2318199205,3cyclictest131rcu_preempt08:00:250
2321299199,4cyclictest0-21swapper/310:05:133
2321299198,8cyclictest0-21swapper/309:25:533
2321299193,11cyclictest0-21swapper/311:35:263
2321299193,11cyclictest0-21swapper/311:35:253
2321299193,11cyclictest0-21swapper/310:19:113
2321299192,10cyclictest0-21swapper/308:40:143
2321299191,12cyclictest0-21swapper/309:50:223
2321299191,10cyclictest0-21swapper/309:20:133
2320099192,13cyclictest12186-21irqrtprio08:05:182
2319199196,8cyclictest21-21ksoftirqd/107:35:131
2319199191,11cyclictest5025-21tr09:50:221
2318199194,8cyclictest12-21ksoftirqd/008:50:170
2318199193,4cyclictest12-21ksoftirqd/008:15:330
2318199192,8cyclictest12-21ksoftirqd/007:15:010
2318199191,14cyclictest842-21munin-node07:30:130
2321299189,6cyclictest29566irq/58-eth011:45:133
2321299189,6cyclictest0-21swapper/310:32:063
2321299188,3cyclictest0-21swapper/309:45:513
2321299186,9cyclictest0-21swapper/311:10:073
2321299186,9cyclictest0-21swapper/311:10:063
2321299183,10cyclictest0-21swapper/311:30:093
2321299182,7cyclictest0-21swapper/311:00:153
2321299182,11cyclictest0-21swapper/310:25:113
2321299182,10cyclictest0-21swapper/312:25:123
2321299182,10cyclictest0-21swapper/308:55:143
2321299182,10cyclictest0-21swapper/308:30:143
2321299182,10cyclictest0-21swapper/307:20:123
23212991811,4cyclictest29566irq/58-eth011:50:243
23212991810,6cyclictest0-21swapper/312:31:543
2319199183,12cyclictest17823-21sort08:45:151
2318199188,4cyclictest12-21ksoftirqd/008:10:260
2318199186,3cyclictest131rcu_preempt08:20:270
2318199185,8cyclictest131rcu_preempt07:40:180
2318199185,3cyclictest131rcu_preempt07:10:010
2318199184,3cyclictest131rcu_preempt07:50:180
2318199184,3cyclictest131rcu_preempt07:50:170
2318199182,11cyclictest0-21swapper/008:35:130
2318199182,11cyclictest0-21swapper/008:35:120
2321299178,6cyclictest0-21swapper/309:32:533
2321299172,10cyclictest0-21swapper/309:00:143
2321299172,10cyclictest0-21swapper/308:50:143
2321299172,10cyclictest0-21swapper/308:45:153
2321299172,10cyclictest0-21swapper/307:35:133
2321299172,10cyclictest0-21swapper/307:30:133
2321299172,10cyclictest0-21swapper/307:25:143
2321299172,10cyclictest0-21swapper/307:25:133
23212991711,4cyclictest0-21swapper/310:58:303
23212991711,4cyclictest0-21swapper/310:50:393
23212991711,4cyclictest0-21swapper/310:50:393
23212991711,4cyclictest0-21swapper/309:40:193
2321299171,12cyclictest0-21swapper/308:20:013
2321299171,12cyclictest0-21swapper/307:45:193
2321299171,11cyclictest0-21swapper/310:40:163
2321299171,10cyclictest0-21swapper/308:35:163
2321299171,10cyclictest0-21swapper/308:35:163
2319199172,12cyclictest2226-21cut07:55:201
2319199171,12cyclictest0-21swapper/107:20:101
2319199170,14cyclictest7763-21kthreadcore08:00:181
2321299162,10cyclictest0-21swapper/307:50:153
2321299162,10cyclictest0-21swapper/307:50:143
2321299161,10cyclictest0-21swapper/308:20:163
2319199168,5cyclictest31400-21irqcore08:25:191
2319199168,5cyclictest12737-21kthreadcore08:05:201
2321299149,3cyclictest0-21swapper/308:25:163
2321299149,3cyclictest0-21swapper/307:55:193
2321299141,10cyclictest0-21swapper/309:10:013
2319199142,9cyclictest23206-21ls07:10:011
2321299138,3cyclictest0-21swapper/307:17:093
2321299130,9cyclictest0-21swapper/308:05:133
2321299128,2cyclictest33-21ksoftirqd/307:10:593
2321299123,7cyclictest0-21swapper/308:00:133
2321299123,7cyclictest0-21swapper/307:40:173
2321299101,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional