You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-01 - 08:13

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot6s.osadl.org (updated Tue Jul 01, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
22383211695,16sleep10-21swapper/119:06:231
22314211087,18sleep00-21swapper/019:05:300
22384210687,13sleep20-21swapper/219:06:242
22438210392,6sleep30-21swapper/319:07:053
160442770,0sleep00-21swapper/021:51:150
290932710,1sleep229-21rcuc/221:31:592
232412710,1sleep20-21swapper/220:30:162
155382680,0sleep20-21swapper/222:56:252
192132420,1sleep2301ktimersoftd/221:55:162
25532260,1sleep20-21swapper/200:23:592
2269399260,2cyclictest0-21swapper/020:35:120
308822250,1sleep3381ktimersoftd/319:30:003
2269399250,2cyclictest0-21swapper/020:55:120
2269599240,22cyclictest12-21rcu_preempt22:10:162
213202240,1sleep20-21swapper/200:10:102
2269399231,20cyclictest0-21swapper/023:00:180
2269399230,3cyclictest6510-21diskmemload21:28:390
2269399230,3cyclictest0-21swapper/022:30:220
2269399230,21cyclictest0-21swapper/023:25:270
2269399220,3cyclictest9403-21ssh22:17:470
2269399220,3cyclictest6510-21diskmemload22:51:060
2269399220,3cyclictest28599-21sh22:37:020
2269399220,3cyclictest18874-21ssh00:06:540
2269399220,3cyclictest10762-21munin-node0
2269399220,3cyclictest0-21swapper/023:35:260
2269399220,3cyclictest0-21swapper/022:55:260
2269399220,3cyclictest0-21swapper/022:42:430
2269399220,3cyclictest0-21swapper/021:56:380
2269399220,3cyclictest0-21swapper/021:42:300
2269399220,3cyclictest0-21swapper/021:23:180
2269399220,13cyclictest0-21swapper/000:23:090
22695992118,2cyclictest0-21swapper/221:50:132
22693992110,9cyclictest1388-21awk19:35:240
2269399210,3cyclictest4244-21ssh22:12:380
2269399210,3cyclictest13765-21ssh22:21:540
2269399210,2cyclictest5723-21ssh22:46:260
2269399210,2cyclictest14709-21ssh00:00:590
2269399210,2cyclictest0-21swapper/023:47:210
2269399210,2cyclictest0-21swapper/023:43:150
2269399210,2cyclictest0-21swapper/023:21:380
2269399210,2cyclictest0-21swapper/023:18:060
2269399210,2cyclictest0-21swapper/023:13:180
2269399210,2cyclictest0-21swapper/022:25:380
2269399210,2cyclictest0-21swapper/022:07:400
2269399210,2cyclictest0-21swapper/022:03:100
2269399210,2cyclictest0-21swapper/021:30:090
2269399210,2cyclictest0-21swapper/019:40:190
2269399210,2cyclictest0-21swapper/000:39:460
2269399210,2cyclictest0-21swapper/000:34:500
2269399210,2cyclictest0-21swapper/000:25:470
2269399210,2cyclictest0-21swapper/000:11:220
2269399210,12cyclictest0-21swapper/021:45:140
250812200,0sleep30-21swapper/321:28:203
22695992017,2cyclictest567-21kworker/u8:021:03:042
2269399200,3cyclictest5010-21snmpd19:45:180
2269399200,3cyclictest0-21swapper/020:54:100
2269399200,2cyclictest7726-21ssh23:55:020
2269399200,2cyclictest7726-21ssh23:55:020
2269399200,2cyclictest0-21swapper/023:50:140
2269399200,2cyclictest0-21swapper/023:32:590
2269399200,2cyclictest0-21swapper/023:05:140
2269399200,2cyclictest0-21swapper/021:35:420
2269399200,2cyclictest0-21swapper/021:15:020
2269399200,2cyclictest0-21swapper/021:10:300
2269399200,2cyclictest0-21swapper/020:30:180
2269399200,2cyclictest0-21swapper/020:26:460
2269399200,2cyclictest0-21swapper/019:20:300
2269399200,2cyclictest0-21swapper/000:15:100
22695991916,2cyclictest611-21kworker/u8:300:30:042
22695991916,2cyclictest26495-21kworker/u8:123:18:012
22695991916,2cyclictest22761-21kworker/u8:022:17:042
22695991916,2cyclictest16861-21kworker/u8:120:51:052
2269399190,3cyclictest31060-21aten2.4-expect19:30:140
2269399190,2cyclictest0-21swapper/021:05:190
2269399190,2cyclictest0-21swapper/021:05:180
2269399190,2cyclictest0-21swapper/021:00:220
2269399190,2cyclictest0-21swapper/020:45:180
2269399190,2cyclictest0-21swapper/020:40:190
2269399190,2cyclictest0-21swapper/020:20:220
2269399190,2cyclictest0-21swapper/020:10:180
2269399190,2cyclictest0-21swapper/020:05:140
2269399190,2cyclictest0-21swapper/019:50:230
2269399190,2cyclictest0-21swapper/019:25:140
10952190,2sleep11094-21http_loadtime20:55:161
22695991815,2cyclictest8793-21kworker/u8:223:26:042
22695991815,2cyclictest8229-21kworker/u8:119:56:012
22695991815,2cyclictest8229-21kworker/u8:119:53:042
22695991815,2cyclictest26431-21kworker/u8:220:49:042
22695991815,2cyclictest26431-21kworker/u8:220:42:042
22695991815,2cyclictest26137-21kworker/u8:122:48:042
22695991815,2cyclictest20432-21kworker/u8:219:24:042
22695991815,2cyclictest20221-21kworker/u8:223:13:042
22695991815,2cyclictest20075-21kworker/u8:222:38:052
22695991815,2cyclictest14040-21kworker/u8:023:02:042
22695991815,2cyclictest13731-21kworker/u8:122:26:052
22695991814,2cyclictest0-21swapper/221:15:132
22693991810,7cyclictest0-21swapper/020:15:220
2269399180,2cyclictest0-21swapper/019:55:180
2269399180,2cyclictest0-21swapper/019:18:270
2269399180,2cyclictest0-21swapper/019:10:140
22695991714,2cyclictest8793-21kworker/u8:223:38:012
22695991714,2cyclictest5146-21kworker/u8:321:44:042
22695991714,2cyclictest3170-21kworker/u8:023:22:042
22695991714,2cyclictest26572-21kworker/u8:119:32:042
22695991714,2cyclictest26137-21kworker/u8:122:54:062
22695991714,2cyclictest23742-21kworker/u8:123:47:042
22695991714,2cyclictest22761-21kworker/u8:022:00:052
22695991714,2cyclictest2263-21kworker/u8:022:44:042
22695991714,2cyclictest20432-21kworker/u8:220:06:042
22695991714,2cyclictest20432-21kworker/u8:220:02:052
22695991714,2cyclictest20432-21kworker/u8:219:26:052
22695991714,2cyclictest1767-21kworker/u8:223:56:012
22695991714,2cyclictest1767-21kworker/u8:223:56:012
22695991714,2cyclictest16890-21kworker/u8:221:22:042
22695991714,2cyclictest15608-21kworker/u8:023:44:042
22696991612,3cyclictest5146-21kworker/u8:321:42:053
22696991612,2cyclictest0-21swapper/300:20:223
22695991613,2cyclictest7402-21kworker/u8:100:19:042
22695991613,2cyclictest567-21kworker/u8:021:08:042
22695991613,2cyclictest567-21kworker/u8:021:08:042
22695991613,2cyclictest22761-21kworker/u8:022:09:042
22695991613,2cyclictest20432-21kworker/u8:219:38:052
22695991613,2cyclictest10443-21kworker/u8:020:29:052
22696991512,2cyclictest7402-21kworker/u8:100:01:043
22696991512,2cyclictest567-21kworker/u8:021:20:023
22696991512,2cyclictest0-21swapper/323:07:413
22696991512,2cyclictest0-21swapper/322:51:213
2269699150,13cyclictest0-21swapper/321:45:133
22695991512,2cyclictest6130-21kworker/u8:019:16:042
22695991512,2cyclictest567-21kworker/u8:020:56:012
22695991512,2cyclictest31526-21kworker/u8:221:38:052
22695991512,2cyclictest22761-21kworker/u8:022:31:042
22695991512,2cyclictest20432-21kworker/u8:219:49:042
22695991512,2cyclictest20432-21kworker/u8:219:43:042
22695991512,2cyclictest10443-21kworker/u8:020:19:042
2269499151,3cyclictest9680-21hddtemp_smartct21:45:211
2269499151,12cyclictest9632-21ssh22:18:081
2269499150,2cyclictest0-21swapper/122:50:201
2269499150,13cyclictest0-21swapper/122:56:251
22696991411,2cyclictest8692-21kworker/u8:100:35:043
22696991411,2cyclictest0-21swapper/323:56:543
22696991411,2cyclictest0-21swapper/323:56:543
22696991411,2cyclictest0-21swapper/323:30:353
22696991411,2cyclictest0-21swapper/323:25:273
22696991411,2cyclictest0-21swapper/323:20:193
22696991411,2cyclictest0-21swapper/322:59:223
22696991411,2cyclictest0-21swapper/322:39:523
22696991411,2cyclictest0-21swapper/321:50:423
22696991411,2cyclictest0-21swapper/321:15:163
2269699140,2cyclictest0-21swapper/323:45:213
2269699140,2cyclictest0-21swapper/322:09:223
2269699140,12cyclictest0-21swapper/323:00:293
2269699140,12cyclictest0-21swapper/322:29:063
2269699140,12cyclictest0-21swapper/322:15:173
22695991411,2cyclictest611-21kworker/u8:300:27:042
22695991411,2cyclictest22761-21kworker/u8:022:23:042
22695991411,2cyclictest15608-21kworker/u8:023:53:052
22695991411,2cyclictest15608-21kworker/u8:023:34:042
22695991411,2cyclictest14040-21kworker/u8:023:08:012
22695991411,2cyclictest10443-21kworker/u8:020:21:042
2269599140,2cyclictest0-21swapper/221:25:142
2269499140,3cyclictest19108-21sh23:33:371
2269499140,12cyclictest0-21swapper/122:05:051
22696991310,2cyclictest1657-21ssh22:42:583
22696991310,2cyclictest0-21swapper/323:50:173
22696991310,2cyclictest0-21swapper/323:40:233
22696991310,2cyclictest0-21swapper/323:39:553
22696991310,2cyclictest0-21swapper/323:18:183
22696991310,2cyclictest0-21swapper/322:30:393
22696991310,2cyclictest0-21swapper/322:23:463
22696991310,2cyclictest0-21swapper/321:37:243
22696991310,2cyclictest0-21swapper/321:12:533
22696991310,2cyclictest0-21swapper/320:40:193
22696991310,2cyclictest0-21swapper/300:26:433
22696991310,2cyclictest0-21swapper/300:11:573
22696991310,2cyclictest0-21swapper/300:09:493
2269699130,12cyclictest1261-21ssh22:10:173
2269699130,12cyclictest0-21swapper/319:10:253
2269699130,11cyclictest28077-21ssh23:10:113
22695991310,2cyclictest24883-21apt-get20:35:122
22695991310,2cyclictest0-21swapper/200:00:162
2269599130,2cyclictest0-21swapper/221:45:272
2269499131,11cyclictest22692-21cyclictest23:20:231
22694991310,2cyclictest332250irq/24-eth123:38:041
22694991310,2cyclictest16838-21ssh21:20:151
22694991310,2cyclictest0-21swapper/123:01:491
22694991310,2cyclictest0-21swapper/122:01:201
22694991310,2cyclictest0-21swapper/121:58:251
22694991310,2cyclictest0-21swapper/121:50:051
22694991310,2cyclictest0-21swapper/121:30:271
22694991310,2cyclictest0-21swapper/121:27:451
22694991310,2cyclictest0-21swapper/121:12:001
22694991310,2cyclictest0-21swapper/119:50:161
22694991310,2cyclictest0-21swapper/119:25:161
22694991310,2cyclictest0-21swapper/119:15:151
22694991310,2cyclictest0-21swapper/100:23:561
2269499130,2cyclictest0-21swapper/121:35:091
2269499130,2cyclictest0-21swapper/121:15:381
2269499130,2cyclictest0-21swapper/100:01:551
2269499130,12cyclictest0-21swapper/123:56:471
2269499130,12cyclictest0-21swapper/123:56:471
2269499130,12cyclictest0-21swapper/123:40:161
2269499130,12cyclictest0-21swapper/123:05:201
2269499130,12cyclictest0-21swapper/121:00:271
2269499130,12cyclictest0-21swapper/120:25:191
2269499130,12cyclictest0-21swapper/100:30:281
2269499130,12cyclictest0-21swapper/100:15:151
2269499130,11cyclictest6510-21diskmemload22:25:401
2269499130,11cyclictest0-21swapper/100:10:581
2269699129,2cyclictest24915-21apt-get19:15:103
2269699129,2cyclictest0-21swapper/322:01:173
2269699129,2cyclictest0-21swapper/319:50:173
2269699121,10cyclictest23161-1kworker/u9:100:18:153
2269699120,1cyclictest0-21swapper/321:31:323
2269699120,11cyclictest0-21swapper/322:47:473
2269699120,11cyclictest0-21swapper/321:55:273
2269699120,11cyclictest0-21swapper/320:45:193
2269699120,11cyclictest0-21swapper/319:40:193
2269699120,11cyclictest0-21swapper/300:31:073
2269599129,2cyclictest0-21swapper/221:10:142
2269599129,2cyclictest0-21swapper/219:10:202
2269599120,2cyclictest0-21swapper/200:37:262
2269499129,2cyclictest6510-21diskmemload23:28:041
2269499129,2cyclictest0-21swapper/122:48:591
2269499129,2cyclictest0-21swapper/122:33:221
2269499129,2cyclictest0-21swapper/120:15:481
2269499129,2cyclictest0-21swapper/120:00:141
2269499129,2cyclictest0-21swapper/100:37:191
2269499121,9cyclictest8735-21munin-node1
2269499121,9cyclictest4586-21apt-get21:05:141
2269499121,9cyclictest4586-21apt-get21:05:141
2269499121,9cyclictest12696-21cat22:20:291
2269499121,9cyclictest0-21swapper/120:40:121
2269499121,9cyclictest0-21swapper/120:05:111
2269499120,11cyclictest20432-21kworker/u8:219:41:041
2269499120,11cyclictest0-21swapper/123:53:481
2269499120,11cyclictest0-21swapper/100:09:191
2269699118,2cyclictest0-21swapper/319:20:293
2269599118,2cyclictest10443-21kworker/u8:020:12:052
2269599118,2cyclictest0-21swapper/200:06:032
2269499118,2cyclictest6510-21diskmemload22:35:021
2269499118,2cyclictest0-21swapper/122:10:071
2269499118,2cyclictest0-21swapper/121:40:051
2269499118,2cyclictest0-21swapper/120:50:221
2269499111,8cyclictest0-21swapper/119:45:141
2269499111,8cyclictest0-21swapper/119:10:131
2269499111,3cyclictest0-21swapper/123:15:191
2269499111,2cyclictest0-21swapper/123:10:131
2269499111,2cyclictest0-21swapper/120:45:121
2269499111,2cyclictest0-21swapper/119:35:241
2269499110,9cyclictest26572-21kworker/u8:119:33:041
2269499110,9cyclictest0-21swapper/120:30:141
2269499110,3cyclictest3549-21munin-node1
2269499110,2cyclictest0-21swapper/120:20:161
2269499110,10cyclictest10443-21kworker/u8:020:13:041
2269699107,2cyclictest0-21swapper/321:00:533
2269699107,2cyclictest0-21swapper/320:55:193
2269699107,2cyclictest0-21swapper/320:05:163
2269699106,2cyclictest0-21swapper/319:45:233
2269699100,9cyclictest26431-21kworker/u8:220:37:053
2269699100,9cyclictest20432-21kworker/u8:219:37:053
2269499107,2cyclictest0-21swapper/122:40:221
2269499101,8cyclictest31497-21run-parts23:45:251
2269499100,9cyclictest0-21swapper/120:35:171
2269499100,8cyclictest0-21swapper/119:20:201
226969990,8cyclictest0-21swapper/320:17:123
226969990,1cyclictest0-21swapper/320:22:373
226969985,2cyclictest0-21swapper/321:05:133
226969985,2cyclictest0-21swapper/321:05:133
226969985,2cyclictest0-21swapper/320:13:503
226969984,2cyclictest5129-21runrttasks20:02:203
226969983,3cyclictest22845-21apt20:30:143
226969973,3cyclictest31075-21basename20:50:123
226969973,3cyclictest13138-1kworker/u9:019:30:153
226969971,4cyclictest9103-21df19:55:143
226969970,1cyclictest0-21swapper/320:25:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional