You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-04 - 15:42

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot4s.osadl.org (updated Thu Dec 04, 2025 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
187199621,4cyclictest0-21swapper/211:45:142
187199582,4cyclictest0-21swapper/212:35:142
1871995639,12cyclictest0-21swapper/211:05:152
186299501,35cyclictest30720-21ssh11:45:131
186299491,7cyclictest131rcu_preempt08:40:141
185499492,39cyclictest9963-21ssh10:50:130
187199481,12cyclictest0-21swapper/211:20:162
187199466,34cyclictest22728-21ssh11:40:132
187199464,5cyclictest0-21swapper/208:05:132
187199452,7cyclictest0-21swapper/210:25:132
188299435,24cyclictest29566irq/58-eth011:50:133
187199431,10cyclictest0-21swapper/208:45:142
187199423,29cyclictest21789-21ssh12:00:142
185499412,5cyclictest131rcu_preempt09:15:120
187199403,22cyclictest2072-21ssh10:45:132
187199403,22cyclictest2072-21ssh10:45:132
1871994021,12cyclictest530-21usb-storage10:40:132
1871994021,12cyclictest530-21usb-storage10:40:122
187199391,9cyclictest9998-21unixbench_singl08:55:312
187199391,8cyclictest9957-21apt-get10:50:122
186299391,28cyclictest3584-21ssh10:25:151
185499391,34cyclictest21499-21ssh12:20:140
185499391,34cyclictest21499-21ssh12:20:130
187199385,30cyclictest2344-21irqcore07:10:172
187199381,9cyclictest24717-21kthreadcore12:20:222
187199381,9cyclictest24717-21kthreadcore12:20:222
185499381,34cyclictest22704-21ssh09:55:140
187199371,8cyclictest9821-21cut07:15:232
187199371,8cyclictest2994-21wc11:25:292
187199371,8cyclictest22768-21kthreadcore09:10:222
185499373,7cyclictest131rcu_preempt12:05:130
185499372,29cyclictest11006-21ssh09:25:140
187199363,6cyclictest7719-21kthreadcore12:30:232
187199362,9cyclictest6193-21kthreadcore10:05:222
187199361,9cyclictest17665-21sshd11:55:272
187199361,9cyclictest1521-21hwlatdetect08:50:182
187199361,9cyclictest14571-21latency_hist11:35:022
187199361,9cyclictest13238-21hwlatdetect07:55:162
187199361,8cyclictest7416-21apt09:45:012
187199361,8cyclictest7416-21apt09:45:012
187199361,7cyclictest30153-21fschecks_time10:00:152
187199361,7cyclictest13839-21usage_EthcTask08:25:292
187199361,6cyclictest12749-21cat11:10:332
187199361,12cyclictest7532-21cut08:20:232
187199361,12cyclictest31746-21kthreadcore07:40:212
185499363,31cyclictest20398-21ssh10:15:130
187199353,29cyclictest18335-21kworker/u8:009:30:192
187199351,9cyclictest13096-21latency_hist12:15:012
187199351,8cyclictest27465-21/usr/sbin/munin08:10:252
187199351,8cyclictest21265-21kthreadcore08:35:202
187199351,13cyclictest19213-21kthreadcore10:55:222
186299352,29cyclictest5211-21ssh10:05:141
1862993515,11cyclictest0-21swapper/111:55:121
187199342,26cyclictest0-21swapper/207:20:142
187199342,12cyclictest26346irq/40-dwc2_hso09:50:132
187199341,9cyclictest8106-21latency_hist07:50:022
187199341,8cyclictest15470-21users09:00:322
187199341,7cyclictest25527-21apt11:00:132
187199341,7cyclictest19482-21irqrtprio10:35:202
187199341,12cyclictest15970-21sort09:05:192
187199341,12cyclictest15970-21sort09:05:182
187199341,11cyclictest8655-21ssh09:21:102
187199340,8cyclictest31574-21kthreadcore12:05:212
185499343,23cyclictest12-21ksoftirqd/011:50:120
185499342,23cyclictest11059-21ssh10:30:140
1854993417,10cyclictest12-21ksoftirqd/008:15:120
188299332,19cyclictest29566irq/58-eth012:30:153
187199334,4cyclictest25421-21grep09:35:202
187199331,9cyclictest20364-21/usr/sbin/munin10:15:272
187199331,8cyclictest23912-21kthreadcore07:30:212
187199331,6cyclictest27992-21head08:40:312
187199331,5cyclictest16292-21irqcore11:15:202
187199331,3cyclictest26209-21diskstats07:35:152
187199330,8cyclictest11715-21sed10:30:202
1862993323,5cyclictest28156-21sh10:20:171
1862993323,5cyclictest28156-21sh10:20:161
185499333,26cyclictest17690-21ssh10:55:170
1854993324,6cyclictest0-21swapper/011:15:150
185499331,4cyclictest131rcu_preempt09:20:130
188299325,13cyclictest29566irq/58-eth011:35:133
187199321,9cyclictest17155-21grep07:25:192
187199321,8cyclictest849-21kthreadcore08:15:192
187199321,8cyclictest30610-21kthreadcore10:20:242
187199321,8cyclictest30610-21kthreadcore10:20:232
187199321,7cyclictest7763-21/usr/sbin/munin09:45:252
187199321,5cyclictest16989-21grep10:10:302
187199321,25cyclictest28383-21ssh09:15:152
187199320,5cyclictest19919-21ssh11:36:092
1862993221,8cyclictest0-21swapper/112:25:141
187199318,3cyclictest18476-21irqstats08:00:182
187199318,3cyclictest18476-21irqstats08:00:172
187199311,9cyclictest22706-21/usr/sbin/munin09:55:142
187199311,9cyclictest0-21swapper/212:25:102
187199311,8cyclictest13246-21/usr/sbin/munin12:15:242
187199311,8cyclictest10001-21kthreadcore07:50:202
187199311,8cyclictest0-21swapper/209:25:142
186299311,21cyclictest23659-21ssh11:20:171
186299310,28cyclictest17999-21ssh09:30:151
187199301,6cyclictest11772-21ssh11:51:412
186299302,25cyclictest9974-21ssh10:50:151
185499302,22cyclictest22743-21ssh11:40:150
187199290,9cyclictest18981-21sshd08:30:432
186299291,21cyclictest5420-21ssh12:30:161
185499293,16cyclictest31467-21ssh11:25:130
1862992818,3cyclictest0-21swapper/110:35:141
185499285,15cyclictest12-21ksoftirqd/009:50:140
185499274,15cyclictest0-21swapper/010:10:140
185499272,18cyclictest13646-21ssh12:35:140
185499272,18cyclictest13285-21ssh12:15:140
1854992715,6cyclictest121ksoftirqd/008:50:130
187199261,8cyclictest1877-21latency_hist07:10:012
1862992610,12cyclictest0-21swapper/110:45:141
1862992610,12cyclictest0-21swapper/110:45:141
185499267,8cyclictest12-21ksoftirqd/008:10:130
185499263,18cyclictest151rcuc/011:30:140
1882992511,10cyclictest29566irq/58-eth012:20:153
1882992511,10cyclictest29566irq/58-eth012:20:143
186299251,18cyclictest6113-21apt-get08:55:151
185499252,19cyclictest29728-21ssh12:25:160
185499252,18cyclictest25538-21ssh11:00:140
185499242,18cyclictest1261-21sshd10:00:260
185499241,19cyclictest4647-21ls11:05:280
188299232,13cyclictest0-21swapper/307:25:133
186299232,3cyclictest131rcu_preempt10:55:151
1862992314,6cyclictest21785-21ssh12:00:131
185499234,13cyclictest48-21kcompactd008:25:120
185499233,11cyclictest0-21swapper/007:40:150
1854992313,7cyclictest26286-21cron10:40:010
188299222,13cyclictest29566irq/58-eth011:40:143
188299222,13cyclictest29566irq/58-eth011:30:133
186299225,3cyclictest131rcu_preempt11:50:281
186299224,15cyclictest167-21systemd-network07:45:121
1862992212,4cyclictest530-21usb-storage10:40:121
1862992212,4cyclictest530-21usb-storage10:40:121
186299221,17cyclictest19428-21ls09:50:291
186299221,16cyclictest1303-21apt-get08:50:131
185499221,18cyclictest26890-21uniq10:40:160
185499221,18cyclictest26890-21uniq10:40:150
185499221,17cyclictest31741-21rm10:20:270
185499221,17cyclictest31741-21rm10:20:270
188299219,9cyclictest29566irq/58-eth011:15:283
1882992111,7cyclictest29566irq/58-eth012:15:143
1882992111,6cyclictest29566irq/58-eth009:15:133
1862992111,6cyclictest16700-21kthreadcore11:35:221
186299211,15cyclictest32174-21ls08:45:221
186299211,10cyclictest3117-21cron07:45:001
186299210,16cyclictest6795-21ls07:15:011
185499213,12cyclictest7795-21ssh09:45:150
185499212,15cyclictest32735-21ssh09:40:130
185499212,15cyclictest32735-21ssh09:40:130
185499212,13cyclictest842-21munin-node09:10:150
185499212,13cyclictest21794-21ssh12:00:140
185499211,17cyclictest18005-21sh09:30:150
188299208,9cyclictest0-21swapper/312:27:423
1882992011,6cyclictest0-21swapper/309:46:133
188299201,11cyclictest29566irq/58-eth009:50:133
186299209,7cyclictest13873-21hwlatdetect12:35:171
186299201,16cyclictest752-21ssh11:05:141
186299201,16cyclictest0-21swapper/108:05:141
186299201,12cyclictest9733-21cut08:25:161
186299201,10cyclictest0-21swapper/110:16:251
185499209,6cyclictest1846-21cyclictest07:10:130
185499208,3cyclictest131rcu_preempt12:35:020
185499201,8cyclictest11087-21fschecks_count09:00:160
1854992012,5cyclictest12545-21awk10:10:010
1854992012,4cyclictest12-21ksoftirqd/011:55:130
1854992011,7cyclictest27241-21sshd08:08:230
185499201,15cyclictest18382-21ssh11:35:270
185499201,10cyclictest0-21swapper/010:48:400
185499201,10cyclictest0-21swapper/010:48:400
1854992010,7cyclictest18997-21cron08:35:010
188299199,8cyclictest0-21swapper/311:00:283
188299198,8cyclictest0-21swapper/310:18:033
188299192,12cyclictest0-21swapper/312:00:153
188299192,11cyclictest29566irq/58-eth010:45:143
188299192,11cyclictest29566irq/58-eth010:45:143
188299192,11cyclictest0-21swapper/309:25:153
188299192,11cyclictest0-21swapper/308:25:133
188299192,10cyclictest0-21swapper/312:10:053
188299192,10cyclictest0-21swapper/307:10:143
1882991911,6cyclictest0-21swapper/310:30:313
1882991911,6cyclictest0-21swapper/309:14:023
188299191,15cyclictest0-21swapper/311:45:203
1882991910,6cyclictest0-21swapper/310:55:163
1882991910,5cyclictest29566irq/58-eth010:20:163
1882991910,5cyclictest29566irq/58-eth010:20:163
186299199,6cyclictest10607-21mailstats11:30:301
186299193,12cyclictest11183-21fschecks_count10:30:161
186299192,6cyclictest26413-21ntp_states11:40:271
186299191,7cyclictest22669-21/usr/sbin/munin09:55:161
186299191,7cyclictest16136-21/usr/sbin/munin11:15:191
1862991912,4cyclictest29941-21/usr/sbin/munin10:00:271
1862991912,4cyclictest0-21swapper/112:10:021
186299191,13cyclictest32066-21apt-get08:15:021
1862991910,6cyclictest21-21ksoftirqd/109:10:141
186299190,13cyclictest344-21ls09:40:131
186299190,13cyclictest344-21ls09:40:131
186299190,12cyclictest13146-21cut07:55:151
185499198,9cyclictest25889-21basename07:35:010
185499198,7cyclictest5876-21cron11:50:010
185499192,13cyclictest28826-21ntp_states09:35:290
1854991911,6cyclictest0-21swapper/010:25:180
1854991910,6cyclictest0-21swapper/008:20:170
188299189,6cyclictest0-21swapper/312:36:103
188299189,6cyclictest0-21swapper/310:11:233
188299189,6cyclictest0-21swapper/309:56:063
188299189,5cyclictest29566irq/58-eth009:40:123
188299189,5cyclictest29566irq/58-eth009:40:123
188299188,7cyclictest0-21swapper/311:20:223
188299188,7cyclictest0-21swapper/310:05:293
188299187,9cyclictest0-21swapper/312:05:073
188299187,7cyclictest29566irq/58-eth010:25:133
188299182,10cyclictest0-21swapper/311:05:143
188299182,10cyclictest0-21swapper/309:20:143
188299182,10cyclictest0-21swapper/308:35:133
188299182,10cyclictest0-21swapper/307:30:143
188299182,10cyclictest0-21swapper/307:20:163
1882991812,4cyclictest29566irq/58-eth009:30:153
188299181,11cyclictest0-21swapper/307:45:123
1882991810,6cyclictest0-21swapper/310:54:533
186299187,8cyclictest7030-21grep07:15:171
186299182,10cyclictest0-21swapper/108:30:141
186299181,8cyclictest28706-21needreboot09:35:261
1862991813,3cyclictest0-21swapper/108:20:291
186299181,13cyclictest8258-21sh11:10:141
186299181,13cyclictest7898-21ssh09:45:171
186299181,12cyclictest13281-21ssh12:15:131
1862991810,5cyclictest0-21swapper/109:00:181
185499189,6cyclictest0-21swapper/008:40:250
185499187,8cyclictest6929-21df07:15:160
185499186,9cyclictest16069-21tail09:05:190
185499186,9cyclictest16069-21tail09:05:190
185499182,11cyclictest0-21swapper/008:00:130
185499182,11cyclictest0-21swapper/008:00:130
185499181,14cyclictest5821-21kthreadcore12:10:200
185499181,13cyclictest8259-21ssh11:10:140
188299179,6cyclictest0-21swapper/310:35:523
188299179,6cyclictest0-21swapper/309:35:383
188299179,3cyclictest0-21swapper/310:42:063
188299179,3cyclictest0-21swapper/310:42:053
188299178,6cyclictest0-21swapper/310:02:323
188299172,9cyclictest0-21swapper/308:50:153
188299172,12cyclictest0-21swapper/311:10:143
188299172,10cyclictest0-21swapper/311:55:123
188299172,10cyclictest0-21swapper/307:50:153
188299171,13cyclictest0-21swapper/311:25:213
186299178,6cyclictest0-21swapper/109:05:131
186299178,6cyclictest0-21swapper/109:05:131
186299174,4cyclictest18699-21kthreadcore07:25:211
186299174,10cyclictest31337-21cut09:15:221
186299173,11cyclictest32618-21kthreadcore12:05:241
186299172,12cyclictest24897-21kthreadcore12:20:221
186299172,12cyclictest24897-21kthreadcore12:20:221
186299172,10cyclictest0-21swapper/108:35:051
186299171,9cyclictest32472-21diskstats08:15:161
186299171,3cyclictest0-21swapper/110:13:341
1862991712,3cyclictest0-21swapper/108:00:121
1862991712,3cyclictest0-21swapper/108:00:121
1862991711,3cyclictest1-21systemd07:20:161
186299170,8cyclictest25934-21latency_hist07:35:011
185499171,14cyclictest25105-21kthreadcore11:20:230
1854991711,4cyclictest12079-21grep07:50:310
1854991710,4cyclictest22582-21cat08:35:210
185499170,14cyclictest8403-21tr08:55:230
185499170,14cyclictest26354-21ls07:35:160
188299162,9cyclictest0-21swapper/308:10:143
188299162,7cyclictest0-21swapper/307:35:133
188299161,10cyclictest0-21swapper/307:55:133
186299169,4cyclictest31585-21entropy11:25:161
186299164,9cyclictest11375-21irqcore09:25:191
186299162,11cyclictest9337-21kthreadcore07:50:191
186299162,11cyclictest7490-21timerwakeupswit09:20:291
186299161,12cyclictest26097-21/usr/sbin/munin07:35:231
186299161,12cyclictest25650-21diskstats11:00:171
185499169,5cyclictest3116-21cron07:45:010
185499161,12cyclictest16531-21sort07:25:150
185499161,11cyclictest11761-21diskstats07:20:170
185499160,13cyclictest13332-21ps07:55:170
185499151,11cyclictest28935-21apt-get08:45:130
188299142,8cyclictest29566irq/58-eth008:05:153
188299141,6cyclictest0-21swapper/308:30:203
186299141,10cyclictest1916-21cat07:10:011
188299138,3cyclictest0-21swapper/309:05:263
188299138,3cyclictest0-21swapper/309:05:253
188299132,9cyclictest0-21swapper/308:15:043
188299131,10cyclictest0-21swapper/308:40:213
185499131,10cyclictest1846-21cyclictest07:10:010
188299127,3cyclictest0-21swapper/309:00:233
188299123,7cyclictest0-21swapper/308:45:173
188299122,8cyclictest0-21swapper/307:40:253
188299122,7cyclictest0-21swapper/309:00:013
188299121,5cyclictest0-21swapper/308:00:193
188299121,5cyclictest0-21swapper/308:00:183
188299112,5cyclictest0-21swapper/307:15:153
188299111,7cyclictest0-21swapper/308:20:203
188299111,7cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional