You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-12-03 - 04:14

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #9, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack9slot4s (updated Wed Dec 03, 2025 00:59:03)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
134009912821221,37cyclictest13704-21/usr/sbin/munin19:25:140
134019911571137,16cyclictest3324-21strings20:20:241
134019911521133,16cyclictest0-21swapper/119:25:131
130822564517,31sleep10-21swapper/119:22:181
130082563516,32sleep00-21swapper/019:21:310
1340099148126,6cyclictest0-21swapper/020:20:240
274152880,4sleep041ktimersoftd/022:22:530
63252860,3sleep041ktimersoftd/023:08:130
270302850,5sleep1211ktimersoftd/100:03:151
279732840,4sleep1211ktimersoftd/100:37:371
46752770,8sleep00-21swapper/022:32:500
160632550,5sleep0111rcuc/022:45:080
220902490,7sleep01340099cyclictest22:17:090
1340099451,12cyclictest81rcu_preempt22:05:300
315282430,5sleep041ktimersoftd/021:20:160
267872431,25sleep126797-21ssh21:49:021
13401994210,14cyclictest0-21swapper/123:57:411
13400994221,6cyclictest0-21swapper/019:56:390
13400994124,6cyclictest7979-21ntp_states00:50:320
13400994123,12cyclictest23249-21ntp_states23:25:250
13400994117,8cyclictest3-21ksoftirqd/023:47:000
13400994117,8cyclictest3-21ksoftirqd/022:02:270
284222401,35sleep128432-21ssh22:24:211
13401994030,7cyclictest24855-21latency_hist00:35:011
13400994023,5cyclictest10946-21diskstats00:20:160
13400994023,13cyclictest13397-21wget21:35:120
13400994017,7cyclictest3-21ksoftirqd/000:15:190
13400994016,7cyclictest3-21ksoftirqd/021:53:030
13400994015,8cyclictest3-21ksoftirqd/022:42:300
13401993922,12cyclictest25569-21/usr/sbin/munin19:55:121
13400993923,5cyclictest19005-21sort19:35:290
13400993922,7cyclictest4347-21ssh21:25:210
13400993916,8cyclictest3-21ksoftirqd/020:57:210
13400993823,5cyclictest24377-21irqstats00:00:220
13400993823,11cyclictest7286-21ls22:35:250
13400993823,11cyclictest21741-21wget19:45:110
13400993822,5cyclictest592-21fschecks_time21:55:170
107692384,5sleep121242-21diskmemload23:12:401
292672371,32sleep129278-21ssh22:58:351
1340199378,13cyclictest40950irq/93-i91522:10:311
1340099377,19cyclictest0-21swapper/000:31:060
13400993722,5cyclictest9174-21irqstats21:30:220
13400993722,5cyclictest28055-21ntpdc23:30:250
13400993722,4cyclictest19528-21rm23:55:200
13400993722,11cyclictest20500-21cut21:05:270
13400993718,7cyclictest18333-21ssh00:27:360
13400993715,7cyclictest3-21ksoftirqd/020:35:430
1340199362,16cyclictest19351-21ssh23:21:071
13400993620,5cyclictest17768-21df23:20:130
13400993620,11cyclictest5051-21ssh00:14:210
13400993620,11cyclictest5051-21ssh00:14:200
13400993619,6cyclictest22063-21hddtemp_smartct21:10:180
1340099361,22cyclictest6204-21gdbus21:42:550
75192351,30sleep17590-21users20:30:301
236952353,5sleep021242-21diskmemload22:52:290
222612351,30sleep122271-21ssh22:17:251
192732351,4sleep119287-21ssh22:48:031
150172356,25sleep128960-1kworker/1:1H21:35:571
1340199358,23cyclictest0-21swapper/122:33:221
1340199353,13cyclictest8206-21sh22:02:591
1340099359,17cyclictest0-21swapper/023:44:350
13400993520,11cyclictest31176-21cat00:40:300
13400993519,5cyclictest32264-21diskstats23:35:160
13400993518,5cyclictest0-21swapper/020:40:220
13400993516,6cyclictest21552-21cron19:45:000
13400993515,6cyclictest29543-21df_inode20:05:150
13400993513,12cyclictest0-21swapper/020:28:430
13400993512,9cyclictest14619-21rm22:10:050
77302340,5sleep1201rcuc/100:50:281
1340199347,23cyclictest0-21swapper/100:20:051
13400993420,9cyclictest27652-21df20:00:140
13400993420,5cyclictest0-21swapper/019:30:180
13400993419,5cyclictest0-21swapper/020:50:240
13400993413,7cyclictest3-21ksoftirqd/020:12:080
13400993413,10cyclictest29218-21ssh00:05:220
13400993410,8cyclictest8366-21ssh23:10:170
13400993410,8cyclictest18299-21ssh23:54:360
13400993410,7cyclictest24702-21ssh21:45:570
317412331,4sleep131823-21memory20:10:241
237482331,5sleep10-21swapper/121:11:391
1340199331,6cyclictest11295-21ssh22:05:291
1340099332,18cyclictest0-21swapper/000:45:200
13400993317,5cyclictest31138-21hddtemp_smartct23:00:190
13400993316,6cyclictest29589-21ssh00:39:580
13400993316,6cyclictest18476-21ntp_states21:00:260
13400993315,6cyclictest27743-21ssh21:15:300
13400993314,4cyclictest0-21swapper/022:59:470
302822321,5sleep10-21swapper/123:33:021
181742321,4sleep118187-21sh23:54:261
13401993210,18cyclictest0-21swapper/121:27:291
13400993219,5cyclictest0-21swapper/023:15:120
13400993214,6cyclictest6924-21iostat_ios20:30:200
13400993211,7cyclictest3-21ksoftirqd/020:19:110
16832311,5sleep10-21swapper/121:55:411
1340199315,22cyclictest0-21swapper/100:06:051
1340199313,8cyclictest18934-21memory21:40:241
13401993120,7cyclictest21242-21diskmemload23:07:111
13400993118,5cyclictest0-21swapper/022:25:230
13400993115,8cyclictest0-21swapper/020:45:240
301392301,5sleep10-21swapper/121:18:551
1340199304,22cyclictest0-21swapper/122:39:321
1340199302,9cyclictest17933-21df_inode21:00:161
1340199301,15cyclictest28306-21sh21:50:191
13401993011,14cyclictest16154-21irqstats00:25:241
287742291,5sleep10-21swapper/120:01:491
233052291,5sleep10-21swapper/119:48:411
1340199299,16cyclictest0-21swapper/122:26:401
1340199295,20cyclictest0-21swapper/122:50:331
13401992919,6cyclictest19658-21ls19:39:591
13401992917,8cyclictest22-21ksoftirqd/122:40:151
13401992913,12cyclictest0-21swapper/100:11:351
13401992913,12cyclictest0-21swapper/100:11:341
1340199291,24cyclictest6076-21df00:15:131
1340199291,24cyclictest31154-21sed00:40:301
13401992910,15cyclictest104502chrt21:31:171
1340099297,5cyclictest0-21swapper/019:52:560
308832281,5sleep10-21swapper/120:09:111
151522281,5sleep10-21swapper/120:50:371
1340199289,15cyclictest0-21swapper/123:25:021
1340199289,15cyclictest0-21swapper/123:01:471
1340199289,15cyclictest0-21swapper/121:24:581
1340199288,16cyclictest0-21swapper/123:19:361
1340199288,16cyclictest0-21swapper/100:47:171
1340199283,18cyclictest8501-21diskstats20:35:171
1340199282,14cyclictest0-21swapper/123:41:371
1340199281,5cyclictest81rcu_preempt23:46:431
13401992814,10cyclictest22-21ksoftirqd/123:35:101
117822281,5sleep10-21swapper/120:44:421
59742271,5sleep10-21swapper/120:28:051
170462271,4sleep10-21swapper/120:55:481
13401992718,5cyclictest20904-21ls19:40:291
13401992717,6cyclictest17194-21ls19:30:291
133792271,5sleep10-21swapper/120:46:451
1340199262,11cyclictest0-21swapper/120:15:161
13401992517,5cyclictest22-21ksoftirqd/121:08:211
1340199232,17cyclictest24200-21iostat_ios19:50:231
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional