You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-22 - 01:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot4s.osadl.org (updated Fri Nov 21, 2025 12:45:12)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3080099671,47cyclictest19171-21ssh11:40:131
3080999591,11cyclictest0-21swapper/208:35:122
3080999433,22cyclictest2459-21ssh10:25:132
3080099401,29cyclictest15661-21ssh10:55:131
30815993925,11cyclictest29566irq/58-eth012:35:113
3081599375,25cyclictest29566irq/58-eth010:05:153
3081599372,17cyclictest29566irq/58-eth010:15:123
3080099371,22cyclictest29959-21ssh09:40:131
3079199371,25cyclictest28749-21ssh11:25:130
3081599362,18cyclictest29566irq/58-eth011:40:123
3081599351,17cyclictest29566irq/58-eth008:15:143
30809993417,8cyclictest3275-21rm12:30:182
3081599336,19cyclictest29566irq/58-eth009:25:123
30791993323,7cyclictest19163-21apt-get11:40:130
3081599325,19cyclictest29566irq/58-eth010:50:143
3081599324,18cyclictest29566irq/58-eth012:15:133
3081599323,15cyclictest29566irq/58-eth009:45:123
30809993218,9cyclictest2043-21ssh10:24:572
30809993212,11cyclictest131rcu_preempt11:40:292
3079199324,5cyclictest20234-21/usr/sbin/munin10:55:310
3079199320,9cyclictest21056-21df_inode11:20:160
3080999318,8cyclictest27-21ksoftirqd/211:55:012
3080099311,19cyclictest11673-21ssh12:35:111
3081599304,12cyclictest29566irq/58-eth010:20:133
3080999308,16cyclictest27-21ksoftirqd/210:40:012
3080999308,15cyclictest27-21ksoftirqd/207:15:242
3080999306,17cyclictest27-21ksoftirqd/208:15:242
3080999304,14cyclictest131rcu_preempt09:40:342
3080999303,5cyclictest27-21ksoftirqd/212:25:112
3080999303,5cyclictest27-21ksoftirqd/212:25:112
3080999302,14cyclictest5421-21ssh11:10:112
30800993017,4cyclictest21-21ksoftirqd/110:45:021
30800993013,14cyclictest21-21ksoftirqd/107:50:141
3079199303,11cyclictest30807-21ssh10:42:100
3079199301,12cyclictest16580-21ntp_192.168.11411:15:230
3080999291,14cyclictest11915-21ssh10:10:122
3080999291,14cyclictest11915-21ssh10:10:122
3080999290,22cyclictest5092-21kthreadcore10:05:212
3080099299,4cyclictest131rcu_preempt11:10:161
3079199294,8cyclictest7090-21find07:15:280
3079199293,21cyclictest13119-21ssh09:50:130
3079199293,21cyclictest13119-21ssh09:50:130
3079199292,11cyclictest3853-21ssh11:07:580
3081599283,16cyclictest29566irq/58-eth007:35:133
3080999289,13cyclictest0-21swapper/207:35:172
3080999288,11cyclictest27-21ksoftirqd/211:45:222
3080999285,13cyclictest0-21swapper/209:59:392
3080999284,17cyclictest27-21ksoftirqd/209:05:252
3080999283,17cyclictest26346irq/40-dwc2_hso07:20:112
3080999280,15cyclictest131rcu_preempt08:55:132
30800992815,4cyclictest21-21ksoftirqd/111:15:201
30800992813,4cyclictest21-21ksoftirqd/110:05:211
30800992811,9cyclictest21-21ksoftirqd/111:20:161
30800992811,4cyclictest21-21ksoftirqd/109:20:181
3079199286,6cyclictest0-21swapper/012:35:220
3079199285,4cyclictest12-21ksoftirqd/008:15:230
3079199284,5cyclictest19387-21cut09:10:190
3079199283,9cyclictest1734-21ssh12:28:390
3079199283,9cyclictest1734-21ssh12:28:390
3079199283,8cyclictest10758-21ssh11:34:430
3079199283,10cyclictest5179-21ssh09:45:110
3079199282,8cyclictest27716-21ssh09:36:390
30791992814,10cyclictest8078-21apt-get10:50:150
3079199281,10cyclictest31598-21ssh11:46:090
3080999278,7cyclictest27-21ksoftirqd/212:15:052
3080999278,7cyclictest0-21swapper/211:57:432
3080999278,12cyclictest27-21ksoftirqd/209:45:432
3080999277,8cyclictest131rcu_preempt10:30:272
3080999276,14cyclictest27-21ksoftirqd/211:05:022
3080999276,13cyclictest27-21ksoftirqd/212:10:192
3080999275,16cyclictest27-21ksoftirqd/208:25:222
3080999274,14cyclictest27-21ksoftirqd/210:00:152
3080999273,17cyclictest27-21ksoftirqd/210:15:192
3080999272,18cyclictest27-21ksoftirqd/209:50:202
3080999272,18cyclictest27-21ksoftirqd/209:50:202
3080999272,17cyclictest26346irq/40-dwc2_hso09:30:122
3080999272,16cyclictest0-21swapper/208:30:302
3080999272,12cyclictest32286-21netstat11:25:232
30809992714,3cyclictest0-21swapper/207:30:122
30809992713,3cyclictest0-21swapper/211:31:152
3080999271,18cyclictest4934-21cut07:50:172
30809992711,3cyclictest131rcu_preempt09:15:242
30809992711,3cyclictest131rcu_preempt09:15:242
30809992710,11cyclictest0-21swapper/212:21:052
3080099279,9cyclictest21-21ksoftirqd/108:15:231
3080099276,11cyclictest21-21ksoftirqd/112:05:191
3080099272,18cyclictest1923-21rm10:24:461
30800992712,9cyclictest629-21Async14:00:481
30800992710,8cyclictest21-21ksoftirqd/111:45:261
3079199275,6cyclictest29743-21ssh09:16:050
3079199275,6cyclictest29743-21ssh09:16:050
3079199275,4cyclictest12-21ksoftirqd/008:25:180
3079199275,4cyclictest10926-21ntpq09:25:250
3079199272,10cyclictest4069-21apt-get10:05:160
3079199272,10cyclictest23811-21ssh12:01:280
3079199272,10cyclictest23738-21ssh10:38:050
3079199271,9cyclictest15464-21ntp_192.168.11410:10:220
3079199271,9cyclictest15464-21ntp_192.168.11410:10:220
3079199271,10cyclictest15436-21ssh11:56:210
3081599264,16cyclictest0-21swapper/309:40:133
3081599262,14cyclictest29566irq/58-eth010:55:133
3081599261,16cyclictest29566irq/58-eth009:35:133
3081599261,14cyclictest29566irq/58-eth007:45:113
3080999268,12cyclictest0-21swapper/210:54:292
3080999267,4cyclictest27-21ksoftirqd/210:40:322
3080999265,15cyclictest27-21ksoftirqd/211:23:412
3080999265,13cyclictest27-21ksoftirqd/210:45:222
3080999263,16cyclictest27-21ksoftirqd/209:20:442
3080999262,2cyclictest0-21swapper/210:55:282
3080999262,16cyclictest7262-21ssh09:25:122
30809992612,7cyclictest27-21ksoftirqd/212:35:112
30809992610,4cyclictest0-21swapper/211:36:392
3080099269,8cyclictest21-21ksoftirqd/112:04:141
3080099268,9cyclictest21-21ksoftirqd/111:37:391
3080099267,8cyclictest21-21ksoftirqd/109:25:241
3080099266,12cyclictest21-21ksoftirqd/112:20:211
30800992619,4cyclictest201rcuc/109:55:121
30800992610,8cyclictest21-21ksoftirqd/110:40:211
30800992610,7cyclictest21-21ksoftirqd/111:25:161
3079199265,7cyclictest10741-21sh12:15:150
3079199264,5cyclictest12-21ksoftirqd/010:00:260
3079199263,9cyclictest16045-21ssh11:35:320
3079199263,8cyclictest15401-21grep07:25:190
3079199263,14cyclictest24404-21ssh10:15:340
3079199262,8cyclictest29892-21ssh09:40:090
3079199261,9cyclictest16004-21ssh10:32:190
30791992619,4cyclictest8246-21apt-get07:20:110
30791992618,5cyclictest2973-21apt-get12:30:140
3079199261,10cyclictest9265-21sh11:54:310
3079199260,9cyclictest20994-21fschecks_time09:55:150
3079199260,12cyclictest5935-21ssh10:46:530
3079199260,10cyclictest14231-21cat08:00:020
3081599257,10cyclictest0-21swapper/310:25:113
3081599253,14cyclictest0-21swapper/309:50:133
3081599253,14cyclictest0-21swapper/309:50:133
3081599251,17cyclictest0-21swapper/311:25:133
3080999258,11cyclictest0-21swapper/209:11:312
3080999257,11cyclictest27-21ksoftirqd/207:25:262
3080999256,13cyclictest27-21ksoftirqd/212:05:252
3080999255,15cyclictest27-21ksoftirqd/209:00:202
3080999254,13cyclictest27-21ksoftirqd/209:35:262
3080999253,15cyclictest27-21ksoftirqd/207:40:172
3080999253,15cyclictest27-21ksoftirqd/207:40:172
3080999251,8cyclictest27-21ksoftirqd/211:08:122
30809992514,5cyclictest27-21ksoftirqd/212:00:382
3080099259,7cyclictest21-21ksoftirqd/111:04:461
3080099259,7cyclictest21-21ksoftirqd/110:31:371
3080099258,9cyclictest21-21ksoftirqd/109:19:301
3080099258,9cyclictest21-21ksoftirqd/109:19:301
3080099258,7cyclictest21-21ksoftirqd/109:10:171
3080099258,4cyclictest21-21ksoftirqd/112:12:431
3080099258,4cyclictest21-21ksoftirqd/110:15:271
3080099256,8cyclictest21-21ksoftirqd/108:00:141
3080099256,7cyclictest21-21ksoftirqd/111:55:421
3080099256,3cyclictest131rcu_preempt11:50:121
3080099255,10cyclictest21-21ksoftirqd/109:45:481
3080099252,9cyclictest21-21ksoftirqd/110:55:011
30800992513,3cyclictest21-21ksoftirqd/111:30:191
30800992510,3cyclictest131rcu_preempt12:27:081
30800992510,3cyclictest131rcu_preempt12:27:081
3079199257,4cyclictest12-21ksoftirqd/007:30:210
3079199255,4cyclictest30282-21ntp_192.168.11412:05:270
3079199254,5cyclictest27152-21apt-get10:20:130
3079199253,7cyclictest9796-21basename10:30:010
3079199253,7cyclictest0-21swapper/008:35:260
3079199253,4cyclictest2258-21apt-get12:10:140
3079199252,9cyclictest14262-21ntpq08:30:250
3079199251,9cyclictest21120-21kthreadcore12:20:220
30791992511,10cyclictest32335-21apt-get09:20:130
3079199250,11cyclictest8921-21missed_timers11:10:220
3081599245,12cyclictest29566irq/58-eth010:45:143
3081599245,11cyclictest0-21swapper/309:10:123
3080999249,8cyclictest27-21ksoftirqd/211:15:172
3080999246,13cyclictest27-21ksoftirqd/207:45:142
3080999242,14cyclictest26346irq/40-dwc2_hso08:10:122
3080999240,15cyclictest13195-21ls07:55:272
3080099249,3cyclictest131rcu_preempt08:30:191
3080099248,8cyclictest21-21ksoftirqd/109:30:251
3080099248,4cyclictest21-21ksoftirqd/111:05:131
3080099246,6cyclictest131rcu_preempt07:15:191
3080099245,3cyclictest131rcu_preempt07:20:251
3080099244,9cyclictest21-21ksoftirqd/109:50:221
3080099244,9cyclictest21-21ksoftirqd/109:50:221
3080099244,3cyclictest131rcu_preempt08:25:311
3080099243,4cyclictest21-21ksoftirqd/107:30:171
3080099243,3cyclictest131rcu_preempt10:00:231
3080099243,3cyclictest131rcu_preempt07:40:281
3080099243,3cyclictest131rcu_preempt07:40:281
3080099243,10cyclictest21-21ksoftirqd/110:38:071
3080099241,5cyclictest21-21ksoftirqd/110:10:111
3080099241,5cyclictest21-21ksoftirqd/110:10:111
30800992411,4cyclictest21-21ksoftirqd/108:10:171
3079199247,5cyclictest0-21swapper/008:10:130
3079199246,4cyclictest15755-21ntpq09:05:290
3079199245,5cyclictest14796-21/usr/sbin/munin09:30:150
3079199242,9cyclictest4835-21switchtime08:20:280
30791992412,9cyclictest23026-21apt-get11:00:150
3079199240,9cyclictest8543-21tail07:50:330
3079199240,9cyclictest6972-21cat08:55:300
3081599231,16cyclictest29566irq/58-eth011:50:123
3080999236,9cyclictest27-21ksoftirqd/207:10:202
30809992312,7cyclictest0-21swapper/208:50:122
3080099238,7cyclictest21-21ksoftirqd/110:25:131
3080099236,8cyclictest21-21ksoftirqd/109:35:201
3080099236,7cyclictest21-21ksoftirqd/108:35:261
3080099235,3cyclictest131rcu_preempt12:30:271
3080099235,3cyclictest131rcu_preempt07:55:221
3080099234,3cyclictest131rcu_preempt09:05:321
30800992311,4cyclictest131rcu_preempt08:50:281
3079199237,4cyclictest0-21swapper/008:00:170
3079199234,4cyclictest29697-21cat08:45:310
3079199233,7cyclictest1284-21mailstats08:50:270
3079199230,9cyclictest25091-21latency_hist08:45:020
3081599223,12cyclictest0-21swapper/311:15:133
3080999227,9cyclictest27-21ksoftirqd/207:10:022
3080999224,8cyclictest27-21ksoftirqd/208:40:232
3080999223,13cyclictest27-21ksoftirqd/208:10:012
3080999221,9cyclictest0-21swapper/208:20:142
3080999221,15cyclictest0-21swapper/208:45:112
3080099225,3cyclictest131rcu_preempt08:20:171
3080099225,3cyclictest131rcu_preempt08:05:131
3080099224,5cyclictest21-21ksoftirqd/107:35:181
3080099223,6cyclictest131rcu_preempt09:00:261
3080099223,3cyclictest131rcu_preempt08:55:161
3080099223,3cyclictest131rcu_preempt08:45:261
3079199229,9cyclictest19159-21apt-get08:05:100
3079199225,4cyclictest0-21swapper/007:40:180
3079199225,4cyclictest0-21swapper/007:40:180
3079199223,5cyclictest12-21ksoftirqd/007:10:160
3079199223,3cyclictest3357-21ntp_192.168.11407:45:220
3079199222,5cyclictest0-21swapper/007:35:290
3079199221,4cyclictest12-21ksoftirqd/009:00:120
3081599213,13cyclictest0-21swapper/312:20:153
3081599212,14cyclictest0-21swapper/311:05:153
3081599212,10cyclictest29566irq/58-eth011:10:113
30815992115,3cyclictest0-21swapper/310:00:233
3080999212,11cyclictest131rcu_preempt08:00:122
3080099213,9cyclictest21-21ksoftirqd/107:25:181
3080099211,4cyclictest21-21ksoftirqd/108:40:151
3081599204,10cyclictest0-21swapper/312:10:143
3081599203,12cyclictest0-21swapper/310:40:163
3081599202,10cyclictest29566irq/58-eth009:15:123
3081599202,10cyclictest29566irq/58-eth009:15:123
30815992010,6cyclictest29566irq/58-eth009:55:133
3080099206,4cyclictest21-21ksoftirqd/107:10:251
3080099201,5cyclictest21-21ksoftirqd/107:10:021
30800992013,5cyclictest0-21swapper/107:45:211
3081599199,5cyclictest0-21swapper/312:05:163
3081599193,11cyclictest0-21swapper/310:40:013
3081599193,10cyclictest0-21swapper/310:30:153
3081599192,12cyclictest29566irq/58-eth011:45:153
3081599192,10cyclictest29566irq/58-eth009:20:133
3081599191,13cyclictest0-21swapper/312:25:133
3081599191,13cyclictest0-21swapper/312:25:133
3081599191,13cyclictest0-21swapper/307:55:163
30815991910,6cyclictest29566irq/58-eth010:10:173
30815991910,6cyclictest29566irq/58-eth010:10:173
30815991910,6cyclictest0-21swapper/311:34:133
3081599189,6cyclictest0-21swapper/312:00:363
3081599184,7cyclictest29566irq/58-eth009:00:123
3081599183,7cyclictest0-21swapper/308:35:123
3081599182,10cyclictest0-21swapper/312:30:153
3081599182,10cyclictest0-21swapper/311:55:133
3081599182,10cyclictest0-21swapper/309:05:173
3081599182,10cyclictest0-21swapper/308:50:143
3081599182,10cyclictest0-21swapper/308:45:153
3081599182,10cyclictest0-21swapper/308:40:143
3081599182,10cyclictest0-21swapper/308:30:153
3081599182,10cyclictest0-21swapper/308:20:153
3081599182,10cyclictest0-21swapper/308:05:113
3081599182,10cyclictest0-21swapper/307:20:123
3081599182,10cyclictest0-21swapper/307:15:143
3081599181,13cyclictest0-21swapper/311:20:243
3081599181,13cyclictest0-21swapper/307:50:123
3081599181,12cyclictest0-21swapper/307:30:263
3081599179,6cyclictest0-21swapper/311:35:053
3081599172,9cyclictest0-21swapper/308:55:043
3081599172,9cyclictest0-21swapper/307:10:153
3081599172,11cyclictest0-21swapper/311:00:163
3081599172,10cyclictest0-21swapper/309:30:133
30815991712,2cyclictest0-21swapper/308:10:113
3081599162,9cyclictest0-21swapper/307:40:053
3081599162,9cyclictest0-21swapper/307:40:053
3081599162,10cyclictest0-21swapper/307:25:113
3079199151,10cyclictest30874-21ls07:10:020
3081599141,11cyclictest0-21swapper/308:00:223
3081599140,10cyclictest0-21swapper/308:25:143
3081599121,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional