You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 12:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack3slot6s.osadl.org (updated Fri May 02, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1172521460,5sleep11696899cyclictest21:53:131
2065521240,1sleep20-21swapper/223:40:152
16793211092,12sleep30-21swapper/319:08:073
16909210786,16sleep00-21swapper/019:09:360
16901210480,19sleep20-21swapper/219:09:312
16786210387,11sleep10-21swapper/119:08:011
17632870,0sleep00-21swapper/023:54:410
318262850,1sleep10-21swapper/123:51:011
297252800,0sleep00-21swapper/021:39:010
235272730,0sleep10-21swapper/122:37:421
16969992817,9cyclictest0-21swapper/219:35:122
130122250,2sleep113018-21aten2.4_r3power23:00:151
226692230,0sleep00-21swapper/023:10:070
1696999221,3cyclictest12923-21ssh23:00:142
1696999221,19cyclictest0-21swapper/222:30:122
156582220,1sleep30-21swapper/322:30:133
1696999211,2cyclictest0-21swapper/222:35:132
1696999210,19cyclictest0-21swapper/222:45:132
1696999210,19cyclictest0-21swapper/222:25:142
1696899211,18cyclictest0-21swapper/121:55:121
16967992017,2cyclictest0-21swapper/019:40:190
1696799200,2cyclictest0-21swapper/023:00:140
37882190,0sleep30-21swapper/323:23:293
305052190,1sleep30-21swapper/323:17:233
1696999182,2cyclictest0-21swapper/222:53:042
16968991815,2cyclictest7998-21kworker/u8:120:53:041
1696899180,16cyclictest0-21swapper/121:15:111
1697099171,3cyclictest12493-21ssh22:59:553
1697099171,14cyclictest0-21swapper/321:40:133
16969991714,2cyclictest21419-21kworker/u8:200:35:052
1696999171,13cyclictest0-21swapper/221:16:112
16968991714,2cyclictest783-21kworker/u8:021:47:051
16968991714,2cyclictest12409-21kworker/u8:122:31:031
1697099160,14cyclictest23263-21grep23:10:163
16968991613,2cyclictest2869-21kworker/u8:122:52:051
16968991613,2cyclictest22749-21kworker/u8:219:25:041
16968991613,2cyclictest15862-21kworker/u8:000:10:041
16968991613,2cyclictest12-21rcu_preempt21:10:161
16967991612,2cyclictest8263-21munin-node0
16970991510,3cyclictest10344-21sh23:30:153
1697099150,13cyclictest0-21swapper/321:45:273
1696999151,12cyclictest3745-21runrttasks23:30:052
1696999151,12cyclictest0-21swapper/222:55:282
1696999150,13cyclictest26669-21ssh00:19:492
1696999150,13cyclictest0-21swapper/223:55:162
1696999150,13cyclictest0-21swapper/223:08:042
1696999150,13cyclictest0-21swapper/222:40:052
1696999150,13cyclictest0-21swapper/219:20:132
16968991512,2cyclictest7998-21kworker/u8:120:16:041
16968991512,2cyclictest7959-21kworker/u8:223:12:051
16968991512,2cyclictest5767-21kworker/u8:022:27:041
16968991512,2cyclictest24177-21kworker/u8:122:41:041
16968991512,2cyclictest23247-21kworker/u8:022:11:051
16968991512,2cyclictest15179-21kworker/u8:123:06:031
16968991512,2cyclictest14799-21kworker/u8:221:29:051
16968991512,2cyclictest14799-21kworker/u8:221:24:041
16968991512,2cyclictest14578-21kworker/u8:119:43:041
1696899151,13cyclictest0-21swapper/123:34:021
1696899151,12cyclictest7516-21diskstats20:05:171
1696899151,12cyclictest21997-21cut21:30:241
1696899150,3cyclictest0-21swapper/122:23:311
16967991510,3cyclictest0-21swapper/021:49:080
1696799150,13cyclictest0-21swapper/021:05:170
1697099142,7cyclictest606-21ssh21:10:003
1697099140,12cyclictest0-21swapper/300:10:273
16969991411,2cyclictest7998-21kworker/u8:120:08:052
16969991411,2cyclictest26963-21kworker/u8:123:53:042
16969991410,2cyclictest25381-21chrt23:12:062
1696999140,9cyclictest0-21swapper/222:02:052
1696999140,2cyclictest0-21swapper/221:51:492
1696999140,12cyclictest8360-21tr00:00:192
1696999140,12cyclictest597-21diskmemload23:25:092
1696999140,12cyclictest0-21swapper/221:57:112
16968991411,2cyclictest32569-21kworker/u8:223:44:051
16968991411,2cyclictest31464-21kworker/u8:220:21:051
16968991411,2cyclictest31464-21kworker/u8:220:01:051
16968991411,2cyclictest31464-21kworker/u8:219:55:051
16968991411,2cyclictest31464-21kworker/u8:219:52:041
16968991411,2cyclictest26963-21kworker/u8:123:47:041
16968991411,2cyclictest25257-21kworker/u8:019:33:051
16968991411,2cyclictest17522-21kworker/u8:222:09:041
16968991411,2cyclictest16685-21kworker/u8:019:21:041
16968991411,2cyclictest14578-21kworker/u8:119:45:041
1696899140,9cyclictest0-21swapper/120:39:051
1696899140,12cyclictest0-21swapper/100:37:101
1696899140,10cyclictest0-21swapper/123:17:051
1696899140,10cyclictest0-21swapper/119:36:041
16967991411,2cyclictest597-21diskmemload21:33:480
16967991411,2cyclictest22957-21ssh23:42:280
16967991411,2cyclictest0-21swapper/022:20:160
16967991411,2cyclictest0-21swapper/020:50:190
16967991411,2cyclictest0-21swapper/000:37:370
16967991411,2cyclictest0-21swapper/000:31:180
16967991410,3cyclictest25830-21munin-node0
1696799140,2cyclictest0-21swapper/019:30:180
1696799140,13cyclictest0-21swapper/022:01:390
1696799140,12cyclictest597-21diskmemload00:25:190
1696799140,12cyclictest26963-21kworker/u8:123:57:040
1696799140,12cyclictest0-21swapper/022:30:230
1696799140,12cyclictest0-21swapper/021:44:290
1696799140,12cyclictest0-21swapper/021:26:080
1696799140,12cyclictest0-21swapper/020:45:210
1696799140,12cyclictest0-21swapper/000:10:040
132792140,2sleep00-21swapper/021:21:290
22962130,2sleep10-21swapper/100:27:061
1697099132,6cyclictest0-21swapper/319:50:013
1697099131,6cyclictest0-21swapper/320:05:003
1697099130,3cyclictest31342-21rm00:24:233
1697099130,11cyclictest0-21swapper/320:30:253
1696999138,3cyclictest597-21diskmemload23:15:152
1696999136,5cyclictest3745-21runrttasks20:28:042
16969991310,2cyclictest0-21swapper/221:45:122
16969991310,2cyclictest0-21swapper/220:20:202
16969991310,2cyclictest0-21swapper/200:28:492
1696999130,2cyclictest0-21swapper/221:32:492
1696999130,12cyclictest0-21swapper/222:10:122
1696999130,12cyclictest0-21swapper/200:05:382
1696999130,11cyclictest7958-21ssh22:21:292
1696999130,11cyclictest0-21swapper/221:25:322
16968991310,2cyclictest7998-21kworker/u8:120:55:051
16968991310,2cyclictest7998-21kworker/u8:120:32:041
16968991310,2cyclictest26710-21kworker/u8:021:07:041
16968991310,2cyclictest16685-21kworker/u8:019:18:041
16968991310,2cyclictest14799-21kworker/u8:221:40:021
16968991310,2cyclictest0-21swapper/122:47:531
16968991310,2cyclictest0-21swapper/122:03:521
16968991310,2cyclictest0-21swapper/121:35:231
16968991310,2cyclictest0-21swapper/100:09:041
1696899130,2cyclictest597-21diskmemload00:21:201
1696899130,2cyclictest0-21swapper/123:35:221
1696899130,2cyclictest0-21swapper/123:20:181
1696899130,2cyclictest0-21swapper/120:25:241
1696899130,2cyclictest0-21swapper/100:32:101
1696899130,12cyclictest0-21swapper/122:56:501
1696899130,12cyclictest0-21swapper/122:15:321
1696899130,11cyclictest0-21swapper/100:15:201
16967991310,2cyclictest597-21diskmemload21:16:210
16967991310,2cyclictest0-21swapper/023:35:120
16967991310,2cyclictest0-21swapper/023:20:520
16967991310,2cyclictest0-21swapper/022:55:230
16967991310,2cyclictest0-21swapper/022:45:440
16967991310,2cyclictest0-21swapper/022:35:240
16967991310,2cyclictest0-21swapper/022:28:000
16967991310,2cyclictest0-21swapper/021:55:250
16967991310,2cyclictest0-21swapper/020:15:240
1696799130,2cyclictest597-21diskmemload00:20:150
1696799130,2cyclictest0-21swapper/023:32:020
1696799130,2cyclictest0-21swapper/023:16:140
1696799130,2cyclictest0-21swapper/022:50:310
1696799130,2cyclictest0-21swapper/022:15:030
1696799130,2cyclictest0-21swapper/022:10:020
1696799130,2cyclictest0-21swapper/022:05:090
1696799130,2cyclictest0-21swapper/021:52:360
1696799130,12cyclictest0-21swapper/022:42:190
1696799130,12cyclictest0-21swapper/000:15:080
1696799130,11cyclictest26686-21kworker/u8:123:25:040
1696799130,11cyclictest0-21swapper/023:08:020
139812130,1sleep00-21swapper/000:05:320
67132120,2sleep20-21swapper/200:31:032
1697099129,2cyclictest23010-21latency_hist22:05:013
1697099129,2cyclictest0-21swapper/322:10:173
1697099127,3cyclictest5838-21perl23:25:163
1697099122,6cyclictest0-21swapper/321:29:593
1697099121,5cyclictest0-21swapper/322:25:013
1697099120,2cyclictest0-21swapper/323:50:213
1697099120,2cyclictest0-21swapper/300:30:263
1697099120,11cyclictest0-21swapper/322:54:063
1697099120,10cyclictest0-21swapper/323:38:273
1697099120,10cyclictest0-21swapper/322:46:003
1696999129,2cyclictest32569-21kworker/u8:223:35:052
1696999129,2cyclictest32569-21kworker/u8:223:21:032
1696999129,2cyclictest0-21swapper/223:46:222
1696999129,2cyclictest0-21swapper/221:10:312
1696999129,2cyclictest0-21swapper/220:45:312
1696999129,2cyclictest0-21swapper/220:15:152
1696999129,2cyclictest0-21swapper/219:30:262
1696999129,2cyclictest0-21swapper/200:21:402
1696999122,7cyclictest0-21swapper/220:33:592
1696999122,7cyclictest0-21swapper/219:57:592
1696999120,2cyclictest0-21swapper/221:40:402
1696999120,2cyclictest0-21swapper/221:20:102
1696999120,2cyclictest0-21swapper/220:40:172
1696999120,2cyclictest0-21swapper/219:40:212
1696999120,2cyclictest0-21swapper/200:10:262
1696999120,11cyclictest0-21swapper/221:35:282
1696999120,11cyclictest0-21swapper/219:15:202
1696899129,2cyclictest20811-21kworker/u8:220:42:041
1696899129,2cyclictest0-21swapper/123:25:141
1696899129,2cyclictest0-21swapper/100:03:121
1696899127,3cyclictest9136-21munin-node1
1696899120,11cyclictest0-21swapper/123:56:021
1696899120,11cyclictest0-21swapper/119:10:191
1696899120,10cyclictest0-21swapper/121:00:191
1696799120,2cyclictest0-21swapper/020:40:170
1696799120,11cyclictest0-21swapper/021:10:480
1696799120,11cyclictest0-21swapper/020:25:140
1696799120,11cyclictest0-21swapper/019:20:250
1696799120,11cyclictest0-21swapper/019:15:160
92992110,1sleep30-21swapper/300:00:543
178072110,2sleep317817-21ssh21:59:573
1697099118,2cyclictest0-21swapper/320:20:163
1697099118,2cyclictest0-21swapper/319:30:163
1697099117,2cyclictest0-21swapper/300:07:563
1697099116,3cyclictest28192-21sh23:47:573
1697099115,4cyclictest3594-21Xorg22:06:583
1697099112,5cyclictest0-21swapper/320:54:593
1697099112,5cyclictest0-21swapper/320:19:593
1697099112,5cyclictest0-21swapper/319:40:003
1697099112,5cyclictest0-21swapper/300:30:013
1697099112,5cyclictest0-21swapper/300:19:593
1697099111,6cyclictest0-21swapper/321:04:593
1697099111,5cyclictest0-21swapper/319:20:013
1697099110,9cyclictest14773-21ssh23:00:553
1697099110,9cyclictest0-21swapper/323:07:363
1697099110,2cyclictest0-21swapper/321:50:403
1697099110,10cyclictest0-21swapper/323:44:133
1696999118,2cyclictest7998-21kworker/u8:120:57:052
1696999118,2cyclictest32257-21kworker/u8:122:19:042
1696999118,2cyclictest26710-21kworker/u8:021:03:052
1696999118,2cyclictest14578-21kworker/u8:119:29:042
1696999116,3cyclictest20583-21grep20:35:272
1696999116,3cyclictest0-21swapper/219:50:182
1696999110,7cyclictest0-21swapper/222:06:042
1696999110,10cyclictest0-21swapper/220:00:222
1696999110,10cyclictest0-21swapper/219:10:152
1696799112,7cyclictest0-21swapper/019:11:590
1696799112,6cyclictest0-21swapper/020:23:590
1696799110,10cyclictest0-21swapper/019:25:010
272692100,1sleep00-21swapper/020:55:160
1697099107,2cyclictest597-21diskmemload22:41:363
1697099107,2cyclictest0-21swapper/321:10:053
1697099106,2cyclictest0-21swapper/321:38:513
1697099105,3cyclictest7369-21tr20:05:173
1697099105,3cyclictest3148-21avahi-daemon21:15:263
1697099105,3cyclictest20906-21fschecks_count22:35:153
1697099105,3cyclictest17695-21tune2fs19:10:163
1697099101,3cyclictest23017-21munin-run20:44:593
1697099100,9cyclictest0-21swapper/323:58:033
1697099100,9cyclictest0-21swapper/321:22:583
1697099100,2cyclictest597-21diskmemload00:36:013
1697099100,2cyclictest0-21swapper/322:26:173
1697099100,2cyclictest0-21swapper/322:16:263
1696999105,3cyclictest31466-21cpu19:45:182
1696999105,3cyclictest25442-21cpu20:50:162
1696899100,3cyclictest0-21swapper/120:45:301
1696799107,2cyclictest0-21swapper/020:35:140
1696799107,2cyclictest0-21swapper/019:50:270
1696799107,2cyclictest0-21swapper/019:45:230
1696799106,2cyclictest0-21swapper/020:10:180
1696799102,6cyclictest0-21swapper/019:55:260
1696799102,4cyclictest7075-21copy20:05:000
1696799100,7cyclictest0-21swapper/020:04:590
169709994,3cyclictest3353-21sort19:55:183
169709994,3cyclictest29596-21tr19:40:213
169709994,3cyclictest23997-21sed20:45:223
169709994,3cyclictest0-21swapper/321:30:293
169709994,3cyclictest0-21swapper/320:25:273
169709994,3cyclictest0-21swapper/319:50:143
169699994,3cyclictest31308-21tail21:05:132
169699990,6cyclictest0-21swapper/220:10:282
169679994,3cyclictest3745-21runrttasks21:02:360
169679990,8cyclictest25257-21kworker/u8:019:35:050
169709985,2cyclictest0-21swapper/319:20:133
169709984,2cyclictest0-21swapper/320:55:193
169709983,3cyclictest19258-21aten2.4-expect20:35:123
169709983,3cyclictest14578-21kworker/u8:119:25:133
169709983,3cyclictest10153-21pzem_power20:10:253
169679980,7cyclictest0-21swapper/020:30:070
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional