You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-28 - 09:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Fri Nov 28, 2025 00:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3209599933,19cyclictest27-21ksoftirqd/222:00:132
3208699512,39cyclictest7659-21ssh22:40:141
3209599484,11cyclictest27-21ksoftirqd/222:10:132
3209599442,9cyclictest18355-21diskmemload00:25:152
3209599416,8cyclictest27-21ksoftirqd/219:25:122
3208699380,31cyclictest2887-21ssh21:20:141
3209599372,24cyclictest27-21ksoftirqd/223:50:162
3209599372,24cyclictest27-21ksoftirqd/223:50:152
32075993520,10cyclictest24761-21apt-get00:30:160
3207599331,27cyclictest0-21swapper/022:15:120
32107993215,8cyclictest29566irq/58-eth022:30:143
3209599325,18cyclictest19764-21ssh21:30:132
3208699324,20cyclictest21-21ksoftirqd/123:20:141
3209599317,13cyclictest27-21ksoftirqd/223:35:002
3209599311,7cyclictest0-21swapper/219:35:142
3209599311,20cyclictest4258-21ssh21:40:132
3208699313,19cyclictest10495-21ssh00:00:141
3207599319,19cyclictest0-21swapper/021:05:150
3207599314,10cyclictest24549-21ssh21:50:260
3207599314,10cyclictest18207-21ssh21:28:280
32075993123,5cyclictest974-21apt-get23:15:130
32075993123,4cyclictest4240-21apt-get21:40:130
3209599303,20cyclictest0-21swapper/200:35:142
3209599300,17cyclictest9658-21ssh00:20:152
3207599304,10cyclictest2634-21ssh21:38:160
32075993021,4cyclictest20737-21apt-get20:05:140
32075993020,8cyclictest12-21ksoftirqd/023:25:140
3209599292,21cyclictest14259-21ssh22:43:042
3209599292,10cyclictest27-21ksoftirqd/219:20:162
3208699292,4cyclictest131rcu_preempt21:25:161
3208699292,18cyclictest2428-21ssh23:55:131
3207599292,9cyclictest30460-21ssh00:10:300
3207599292,13cyclictest10974-21tr23:20:200
32075992919,7cyclictest26817-21apt-get23:50:160
32075992919,7cyclictest26817-21apt-get23:50:150
32075992914,12cyclictest23382-21apt-get22:30:140
3210799288,16cyclictest0-21swapper/323:30:143
32095992812,10cyclictest0-21swapper/222:46:532
32086992812,4cyclictest21-21ksoftirqd/122:45:511
3208699281,20cyclictest10490-21ssh23:40:161
3207599285,7cyclictest26263-21ssh23:49:450
3207599285,10cyclictest28958-21latency_hist19:40:030
3207599284,9cyclictest22404-21ntp_states21:10:280
3207599284,9cyclictest22404-21ntp_states21:10:280
3207599284,7cyclictest24035-21rm00:06:520
3207599284,6cyclictest22147-21grep21:30:230
3207599284,10cyclictest10816-21ssh21:24:560
3207599283,7cyclictest4412-21grep20:20:220
32075992822,3cyclictest5444-21apt-get22:00:140
3207599282,11cyclictest22572-21ssh23:07:090
3207599281,11cyclictest32125-21diskstats00:35:160
32075992811,13cyclictest0-21swapper/020:45:130
32107992713,6cyclictest0-21swapper/323:25:143
3209599278,4cyclictest131rcu_preempt22:05:162
3209599277,13cyclictest131rcu_preempt21:25:162
3209599272,19cyclictest0-21swapper/223:45:152
3209599272,19cyclictest0-21swapper/220:50:162
32095992717,8cyclictest0-21swapper/221:15:262
3209599271,17cyclictest0-21swapper/221:05:282
32095992711,4cyclictest27-21ksoftirqd/221:35:252
32095992710,14cyclictest1155-21cron19:45:012
3209599270,18cyclictest21167-21kthreadcore00:05:222
3209599270,15cyclictest26346irq/40-dwc2_hso21:45:132
3208699275,10cyclictest21-21ksoftirqd/119:35:011
3208699274,14cyclictest31743-21ssh22:35:141
3208699273,6cyclictest21-21ksoftirqd/122:30:131
3208699271,18cyclictest18682-21ssh23:45:141
3207599277,9cyclictest131rcu_preempt22:55:140
3207599274,9cyclictest4187-21df20:55:150
3207599274,5cyclictest12694-21kthreadcore23:40:230
3207599273,8cyclictest3724-21ssh21:58:080
3207599273,7cyclictest14942-21ssh00:21:250
3207599273,10cyclictest20937-21ssh22:45:510
3207599272,8cyclictest11788-21grep00:00:230
3207599272,3cyclictest0-21swapper/021:45:290
3207599272,20cyclictest2222-21ssh23:35:130
32075992721,2cyclictest31719-21apt-get20:50:150
3207599270,9cyclictest8485-21df23:00:140
3207599270,14cyclictest15049-21tr19:25:190
3207599270,11cyclictest28302-21grep21:15:200
3210799265,18cyclictest0-21swapper/300:05:133
3210799261,12cyclictest29566irq/58-eth020:35:133
3210799261,12cyclictest29566irq/58-eth020:35:123
3209599266,7cyclictest27-21ksoftirqd/223:35:312
3209599263,5cyclictest27-21ksoftirqd/200:00:162
3209599262,8cyclictest29403-21find21:55:122
3209599262,3cyclictest131rcu_preempt00:30:352
3209599262,19cyclictest0-21swapper/223:18:382
3209599262,17cyclictest27-21ksoftirqd/223:40:262
32095992615,5cyclictest27-21ksoftirqd/219:15:192
32095992613,7cyclictest0-21swapper/222:20:332
32095992613,7cyclictest0-21swapper/200:10:142
3208699265,10cyclictest21-21ksoftirqd/119:30:021
32086992616,5cyclictest0-21swapper/120:55:131
32086992613,10cyclictest3392-21ssh23:35:201
3208699261,17cyclictest0-21swapper/120:20:141
32086992610,4cyclictest21-21ksoftirqd/123:16:411
3207599267,5cyclictest0-21swapper/022:23:150
3207599265,5cyclictest26693-21munin-run20:45:010
3207599264,7cyclictest22491-21ssh22:10:170
3207599264,6cyclictest6026-21ntp_192.168.11423:55:250
3207599264,6cyclictest21343-21cat20:35:300
3207599264,6cyclictest21343-21cat20:35:290
3207599264,3cyclictest131rcu_preempt00:25:160
3207599263,6cyclictest9935-21cut22:40:210
3207599263,3cyclictest131rcu_preempt22:37:290
3207599261,9cyclictest17527-21ntp_192.168.11422:05:270
3207599261,9cyclictest12947-21ntp_192.168.11419:20:270
3207599261,16cyclictest31855-21ssh23:31:380
3207599261,10cyclictest16388-21cut20:00:180
3209599257,10cyclictest131rcu_preempt23:00:202
3209599256,6cyclictest131rcu_preempt22:15:202
3209599255,7cyclictest27-21ksoftirqd/222:53:502
3209599255,14cyclictest0-21swapper/222:36:142
3209599255,13cyclictest27-21ksoftirqd/222:25:442
3209599255,13cyclictest27-21ksoftirqd/222:25:442
32095992515,4cyclictest27-21ksoftirqd/219:45:172
3209599251,19cyclictest0-21swapper/221:00:192
3209599251,18cyclictest6435-21ls19:50:182
3209599251,17cyclictest27-21ksoftirqd/223:25:102
3209599251,16cyclictest0-21swapper/221:10:292
3209599251,16cyclictest0-21swapper/221:10:282
32095992511,11cyclictest16041-21sshd23:23:122
3208699259,9cyclictest21-21ksoftirqd/123:27:421
3208699259,4cyclictest21-21ksoftirqd/122:56:051
3208699258,9cyclictest21-21ksoftirqd/122:15:301
3208699255,4cyclictest131rcu_preempt21:45:161
3208699254,3cyclictest131rcu_preempt00:22:361
3208699253,3cyclictest131rcu_preempt21:35:321
32086992512,4cyclictest21-21ksoftirqd/100:13:311
32086992510,9cyclictest1862-21ssh00:15:161
32086992510,8cyclictest21-21ksoftirqd/123:10:541
32086992510,4cyclictest21-21ksoftirqd/123:00:171
32086992510,4cyclictest21-21ksoftirqd/121:40:321
3207599258,8cyclictest0-21swapper/019:50:150
3207599258,5cyclictest0-21swapper/019:55:160
3207599258,10cyclictest1850-21apt-get00:15:140
3207599255,4cyclictest12-21ksoftirqd/023:10:140
3207599254,7cyclictest20741-21ssh22:26:490
3207599254,7cyclictest20741-21ssh22:26:480
3210799247,14cyclictest0-21swapper/322:25:143
3210799247,14cyclictest0-21swapper/322:25:133
3210799246,13cyclictest0-21swapper/323:00:123
3210799244,11cyclictest0-21swapper/323:15:143
3210799242,12cyclictest0-21swapper/323:35:123
32107992414,7cyclictest0-21swapper/323:50:163
32107992414,7cyclictest0-21swapper/323:50:163
32107992413,8cyclictest0-21swapper/321:10:153
32107992413,8cyclictest0-21swapper/321:10:143
3209599247,11cyclictest27-21ksoftirqd/221:54:102
3209599241,9cyclictest0-21swapper/220:40:142
3209599241,6cyclictest0-21swapper/223:56:382
32095992412,4cyclictest27-21ksoftirqd/223:10:192
3209599241,16cyclictest0-21swapper/222:32:592
3209599241,14cyclictest27-21ksoftirqd/222:55:302
32095992410,7cyclictest27-21ksoftirqd/223:05:162
3208699249,9cyclictest21-21ksoftirqd/122:10:091
3208699249,4cyclictest21-21ksoftirqd/100:35:201
3208699248,5cyclictest131rcu_preempt22:28:431
3208699248,5cyclictest131rcu_preempt22:28:421
3208699248,4cyclictest21-21ksoftirqd/121:32:401
3208699245,4cyclictest131rcu_preempt22:20:191
3208699244,3cyclictest131rcu_preempt23:50:151
3208699244,3cyclictest131rcu_preempt23:50:141
3208699243,5cyclictest21-21ksoftirqd/121:10:171
3208699243,5cyclictest21-21ksoftirqd/121:10:161
3208699243,3cyclictest131rcu_preempt21:15:271
3208699242,3cyclictest131rcu_preempt20:05:161
32086992410,4cyclictest21-21ksoftirqd/121:55:141
3207599245,6cyclictest9245-21if_eth121:00:220
3207599245,4cyclictest12-21ksoftirqd/022:50:130
3207599244,5cyclictest12-21ksoftirqd/020:30:280
3207599242,4cyclictest12-21ksoftirqd/019:35:000
3207599241,9cyclictest25903-21cut20:10:190
32107992316,2cyclictest0-21swapper/320:45:143
3210799231,14cyclictest0-21swapper/321:45:123
3209599234,8cyclictest27-21ksoftirqd/200:15:252
3209599231,8cyclictest0-21swapper/220:22:372
3209599230,11cyclictest6503-21ssh21:20:252
3208699239,8cyclictest21-21ksoftirqd/122:50:571
3208699239,8cyclictest21-21ksoftirqd/120:15:121
3208699239,5cyclictest21-21ksoftirqd/100:07:531
3208699238,5cyclictest21-21ksoftirqd/120:30:221
3208699237,4cyclictest21-21ksoftirqd/100:34:281
3208699237,2cyclictest131rcu_preempt23:08:011
3208699235,7cyclictest28957-21latency_hist19:40:021
3208699233,6cyclictest21-21ksoftirqd/100:25:201
3208699233,4cyclictest21-21ksoftirqd/120:50:261
3208699232,5cyclictest0-21swapper/120:40:221
3208699230,7cyclictest8552-21unixbench_singl19:15:291
3207599239,3cyclictest0-21swapper/019:40:120
3207599234,4cyclictest0-21swapper/019:50:020
3207599231,18cyclictest9406-21ls19:20:010
3207599231,18cyclictest4479-21apt-config19:15:000
3207599230,8cyclictest30510-21fschecks_count20:15:170
3210799225,12cyclictest29566irq/58-eth019:15:143
3210799223,13cyclictest0-21swapper/323:40:153
3210799222,15cyclictest29566irq/58-eth022:20:143
3210799222,13cyclictest0-21swapper/321:55:163
32107992215,3cyclictest0-21swapper/322:55:143
3209599229,4cyclictest27-21ksoftirqd/219:10:202
3209599226,10cyclictest131rcu_preempt19:30:302
3209599223,4cyclictest131rcu_preempt20:55:142
3209599222,10cyclictest27-21ksoftirqd/219:55:202
3209599221,8cyclictest21907-21basename20:40:012
3209599221,8cyclictest21907-21basename20:40:002
32095992211,9cyclictest0-21swapper/220:45:162
3209599221,18cyclictest16051-21grep20:00:142
3208699229,7cyclictest21-21ksoftirqd/119:50:011
3208699228,8cyclictest21-21ksoftirqd/122:06:591
3208699228,3cyclictest131rcu_preempt21:50:251
3208699225,4cyclictest131rcu_preempt22:00:121
3208699225,3cyclictest131rcu_preempt21:00:321
3208699224,8cyclictest131rcu_preempt19:40:241
3208699221,9cyclictest17500-21timerandwakeup21:05:301
3208699221,6cyclictest21-21ksoftirqd/120:00:271
32086992214,5cyclictest0-21swapper/120:45:141
32086992212,4cyclictest21-21ksoftirqd/119:22:261
32086992210,4cyclictest21-21ksoftirqd/123:30:591
3207599225,5cyclictest12-21ksoftirqd/019:10:020
3210799213,14cyclictest0-21swapper/300:20:143
3210799213,13cyclictest0-21swapper/321:50:163
32107992111,6cyclictest0-21swapper/300:25:153
3209599217,11cyclictest27-21ksoftirqd/220:15:142
3208699218,4cyclictest21-21ksoftirqd/119:10:191
3208699214,3cyclictest131rcu_preempt20:35:191
3208699214,3cyclictest131rcu_preempt20:35:181
3208699213,3cyclictest131rcu_preempt20:25:121
3208699211,4cyclictest21-21ksoftirqd/119:55:241
3210799209,6cyclictest0-21swapper/322:10:133
3210799208,9cyclictest0-21swapper/321:35:233
3210799203,8cyclictest0-21swapper/322:05:173
3210799203,12cyclictest0-21swapper/322:01:383
3210799202,6cyclictest0-21swapper/322:35:163
32107992014,4cyclictest0-21swapper/320:00:123
32107992014,3cyclictest0-21swapper/321:20:133
32107992014,2cyclictest0-21swapper/323:20:133
32107992012,4cyclictest0-21swapper/300:35:143
3210799201,13cyclictest0-21swapper/322:40:123
3209599204,9cyclictest27-21ksoftirqd/220:35:002
3209599201,8cyclictest29376-21grep20:10:292
3209599201,13cyclictest0-21swapper/220:05:132
3208699209,7cyclictest21-21ksoftirqd/120:10:141
3208699201,3cyclictest131rcu_preempt19:50:201
3207599207,3cyclictest0-21swapper/020:25:140
3210799199,7cyclictest0-21swapper/321:42:223
3210799194,11cyclictest0-21swapper/321:15:263
3210799192,9cyclictest29566irq/58-eth023:05:133
3210799192,13cyclictest0-21swapper/300:10:223
32107991911,5cyclictest29566irq/58-eth000:15:163
3210799191,12cyclictest0-21swapper/323:55:123
3210799191,11cyclictest0-21swapper/319:55:163
3210799191,11cyclictest0-21swapper/300:00:193
32107991910,7cyclictest0-21swapper/323:10:263
32107991910,3cyclictest0-21swapper/319:25:133
3209599192,14cyclictest11333-21find20:25:292
3210799189,7cyclictest0-21swapper/323:45:083
3210799189,6cyclictest29566irq/58-eth022:50:133
3210799189,6cyclictest0-21swapper/322:46:223
3210799187,8cyclictest0-21swapper/319:30:263
3210799183,11cyclictest0-21swapper/321:30:063
3210799182,10cyclictest0-21swapper/319:20:163
3210799181,12cyclictest0-21swapper/320:50:243
3210799181,10cyclictest0-21swapper/320:55:283
3210799179,5cyclictest29566irq/58-eth021:25:163
3210799172,9cyclictest0-21swapper/321:05:163
3210799172,9cyclictest0-21swapper/320:40:143
3210799172,9cyclictest0-21swapper/320:25:143
3210799172,9cyclictest0-21swapper/320:15:133
3210799172,9cyclictest0-21swapper/319:50:173
3210799172,9cyclictest0-21swapper/319:45:123
3210799172,11cyclictest0-21swapper/322:15:203
32107991711,3cyclictest0-21swapper/300:30:163
32107991710,2cyclictest0-21swapper/320:05:133
3210799168,2cyclictest0-21swapper/319:10:133
3210799162,9cyclictest0-21swapper/319:40:063
3210799162,9cyclictest0-21swapper/319:35:153
32107991611,2cyclictest0-21swapper/321:00:163
3210799155,7cyclictest0-21swapper/320:10:133
3208699151,3cyclictest131rcu_preempt19:10:021
3210799148,4cyclictest0-21swapper/320:30:203
3210799133,7cyclictest0-21swapper/320:20:183
3209599121,8cyclictest32143-21cat19:10:012
3210799111,7cyclictest0-21swapper/319:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional