You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-11-05 - 10:27

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot6s.osadl.org (updated Wed Nov 05, 2025 00:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7411212096,19sleep00-21swapper/019:09:030
7389211996,17sleep10-21swapper/119:08:481
7283211695,16sleep20-21swapper/219:07:262
5603211494,12sleep30-21swapper/319:05:153
248802750,0sleep30-21swapper/321:11:333
41372740,1sleep20-21swapper/221:56:472
276302730,0sleep10-21swapper/100:34:261
261592730,0sleep00-21swapper/021:45:420
186352710,0sleep20-21swapper/223:50:472
129002670,0sleep10-21swapper/121:32:491
173362640,5sleep3752199cyclictest22:10:183
123922240,1sleep20-21swapper/220:40:202
751699221,19cyclictest0-21swapper/123:30:151
751699220,3cyclictest3631-21munin-node1
7515992216,4cyclictest6761-21ssh22:00:110
751699210,4cyclictest7673-21ssh21:27:151
751699210,3cyclictest2438-21ps20:15:231
751699210,3cyclictest20582-21taskset00:26:031
751699210,3cyclictest0-21swapper/123:52:551
751699210,3cyclictest0-21swapper/121:35:221
60682210,1sleep00-21swapper/023:05:140
751999200,15cyclictest0-21swapper/223:12:022
751999200,15cyclictest0-21swapper/220:13:042
751699201,3cyclictest0-21swapper/121:55:291
751699200,3cyclictest9549-21ntp_kernel_pll_22:35:191
751699200,3cyclictest8497-21sh00:14:541
751699200,3cyclictest7661-21fschecks_time23:40:151
751699200,3cyclictest28557-21sendmail_mailqu21:15:221
751699200,3cyclictest28436-21df_inode22:55:141
751699200,3cyclictest2557-21grep23:35:151
751699200,3cyclictest24457-21hddtemp_smartct19:50:151
751699200,3cyclictest18686-21munin-node1
751699200,3cyclictest14188-21munin-node1
751699200,3cyclictest0-21swapper/123:55:141
751699200,3cyclictest0-21swapper/123:01:431
751699200,3cyclictest0-21swapper/122:50:071
751699200,3cyclictest0-21swapper/122:20:261
751699200,3cyclictest0-21swapper/121:52:221
751699200,3cyclictest0-21swapper/121:41:101
751699200,3cyclictest0-21swapper/121:20:271
751699200,3cyclictest0-21swapper/121:13:311
751699200,3cyclictest0-21swapper/120:25:141
751699200,3cyclictest0-21swapper/100:01:471
175032200,0sleep00-21swapper/022:10:190
751699190,3cyclictest31350-21ssh00:05:031
751699190,3cyclictest25737-21memory23:25:191
751699190,3cyclictest23195-21unixbench_multi19:45:271
751699190,3cyclictest0-21swapper/122:40:181
751699190,3cyclictest0-21swapper/122:28:111
751699190,3cyclictest0-21swapper/120:03:261
751699190,3cyclictest0-21swapper/100:17:191
751699190,2cyclictest0-21swapper/123:21:341
751699190,17cyclictest0-21swapper/121:45:181
751699190,11cyclictest0-21swapper/122:00:571
751599190,2cyclictest9-21ksoftirqd/023:10:130
751599190,18cyclictest0-21swapper/019:30:130
180752190,1sleep00-21swapper/023:17:290
752199181,15cyclictest22218-21diskmemload23:50:283
7519991815,2cyclictest2714-21kworker/u8:123:02:052
751999180,16cyclictest0-21swapper/200:36:062
751999180,14cyclictest0-21swapper/221:07:052
751999180,14cyclictest0-21swapper/220:46:052
751999180,13cyclictest0-21swapper/220:50:012
751699182,14cyclictest16024-21grep23:15:181
751699181,3cyclictest22218-21diskmemload22:15:211
751699181,3cyclictest0-21swapper/119:25:151
751699181,11cyclictest22218-21diskmemload23:14:131
751699180,3cyclictest28880-21tune2fs00:35:141
751699180,3cyclictest0-21swapper/123:05:101
751699180,3cyclictest0-21swapper/122:49:191
751699180,3cyclictest0-21swapper/121:05:091
752199171,3cyclictest0-21swapper/321:23:193
7519991714,2cyclictest2714-21kworker/u8:122:42:042
7519991714,2cyclictest12280-21kworker/u8:123:27:052
751999171,13cyclictest0-21swapper/200:02:052
751999170,13cyclictest0-21swapper/223:09:042
751699177,8cyclictest15820-21munin-node1
751699171,3cyclictest22218-21diskmemload22:05:151
751699170,3cyclictest19859-21munin-node1
751699170,2cyclictest0-21swapper/120:46:111
751699170,15cyclictest0-21swapper/122:10:271
751699170,15cyclictest0-21swapper/119:35:201
7515991714,2cyclictest4324-21nscd22:55:120
7521991611,3cyclictest2001-21expr21:55:153
751999160,2cyclictest0-21swapper/220:57:052
751999160,2cyclictest0-21swapper/220:23:052
751999160,15cyclictest0-21swapper/222:00:052
751699161,6cyclictest0-21swapper/123:45:441
751699160,14cyclictest9766-21munin-node1
751699160,14cyclictest0-21swapper/120:20:291
751599162,12cyclictest0-21swapper/000:15:210
751599161,3cyclictest4680-21sh21:57:310
7515991611,3cyclictest7856-21sh19:10:140
751599161,13cyclictest0-21swapper/022:21:510
752199151,12cyclictest14237-21irqstats22:40:213
751999151,2cyclictest0-21swapper/221:34:052
7519991512,2cyclictest5846-21kworker/u8:122:12:052
7519991512,2cyclictest28082-21kworker/u8:023:38:022
7519991512,2cyclictest22709-21kworker/u8:000:34:042
7519991512,2cyclictest2209-21kworker/u8:300:19:052
7519991512,2cyclictest13815-21kworker/u8:100:26:062
751999151,12cyclictest15182-21apt-get23:15:122
7519991510,3cyclictest0-21swapper/222:05:272
751999150,2cyclictest0-21swapper/221:43:042
751999150,2cyclictest0-21swapper/220:17:052
751999150,2cyclictest0-21swapper/219:17:042
751999150,14cyclictest0-21swapper/219:57:052
751999150,14cyclictest0-21swapper/219:29:052
751699153,3cyclictest30185-21cpuspeed_turbos20:05:161
751699153,3cyclictest200052sleep121:02:301
751699152,3cyclictest10023-21cpuspeed_turbos20:35:131
751699151,3cyclictest0-21swapper/120:50:131
751699151,3cyclictest0-21swapper/120:40:211
751699151,3cyclictest0-21swapper/120:10:261
751699151,3cyclictest0-21swapper/119:55:181
751699151,3cyclictest0-21swapper/119:20:151
751699151,3cyclictest0-21swapper/119:20:151
751699151,3cyclictest0-21swapper/119:10:181
751699150,4cyclictest8609-21tr20:30:211
751599151,3cyclictest12107-21ssh21:31:340
7515991510,3cyclictest7833-21ssh21:27:300
7515991510,3cyclictest0-21swapper/023:30:160
7515991510,3cyclictest0-21swapper/021:20:180
751599150,3cyclictest22933-21cpuspeed_turbos21:10:160
751599150,13cyclictest0-21swapper/000:11:420
752199149,3cyclictest364-21awk20:10:313
752199149,3cyclictest29852-21aten2.4_r3power23:30:153
7521991411,2cyclictest0-21swapper/320:05:193
7521991411,2cyclictest0-21swapper/319:10:193
752199140,2cyclictest0-21swapper/323:00:123
752199140,2cyclictest0-21swapper/322:57:083
752199140,2cyclictest0-21swapper/322:51:373
752199140,13cyclictest0-21swapper/320:35:183
7519991411,2cyclictest27330-21kworker/u8:120:26:012
7519991411,2cyclictest18406-21kworker/u8:223:34:042
7519991411,2cyclictest12371-21kworker/u8:121:36:052
7519991411,2cyclictest0-21swapper/222:50:182
751999140,2cyclictest0-21swapper/223:41:042
751999140,2cyclictest0-21swapper/219:32:052
751999140,13cyclictest0-21swapper/223:49:052
751999140,13cyclictest0-21swapper/220:00:012
751999140,12cyclictest0-21swapper/223:57:082
751999140,12cyclictest0-21swapper/221:45:212
751599141,3cyclictest0-21swapper/022:45:450
751599141,12cyclictest0-21swapper/023:27:330
751599140,2cyclictest22218-21diskmemload23:52:130
751599140,12cyclictest22218-21diskmemload22:37:530
752199132,6cyclictest0-21swapper/319:25:013
752199132,6cyclictest0-21swapper/319:25:003
7521991310,2cyclictest22218-21diskmemload22:35:283
7521991310,2cyclictest0-21swapper/323:06:173
752199130,2cyclictest28651-21latency20:00:183
752199130,2cyclictest0-21swapper/322:08:583
752199130,2cyclictest0-21swapper/321:42:093
752199130,12cyclictest0-21swapper/323:46:483
752199130,12cyclictest0-21swapper/321:18:293
752199130,11cyclictest0-21swapper/323:39:283
751999131,10cyclictest15360-21ssh00:20:282
7519991310,2cyclictest22656-21kworker/u8:121:15:042
7519991310,2cyclictest0-21swapper/222:57:272
7519991310,2cyclictest0-21swapper/221:26:342
7519991310,2cyclictest0-21swapper/221:10:172
751999130,2cyclictest0-21swapper/223:24:272
751999130,2cyclictest0-21swapper/222:35:162
751999130,2cyclictest0-21swapper/222:21:162
751999130,2cyclictest0-21swapper/221:20:022
751999130,12cyclictest22218-21diskmemload00:07:252
751999130,12cyclictest0-21swapper/222:47:342
751999130,12cyclictest0-21swapper/222:30:252
751999130,12cyclictest0-21swapper/221:50:242
751999130,12cyclictest0-21swapper/200:12:562
751599138,3cyclictest0-21swapper/023:01:510
751599132,7cyclictest0-21swapper/023:49:590
751599132,7cyclictest0-21swapper/020:05:000
751599132,6cyclictest0-21swapper/020:30:000
751599132,6cyclictest0-21swapper/019:35:000
7515991310,2cyclictest9-21ksoftirqd/000:27:080
7515991310,2cyclictest0-21swapper/023:59:470
7515991310,2cyclictest0-21swapper/023:42:130
7515991310,2cyclictest0-21swapper/021:38:380
7515991310,2cyclictest0-21swapper/000:37:420
751599130,2cyclictest0-21swapper/022:28:310
751599130,2cyclictest0-21swapper/020:35:150
751599130,2cyclictest0-21swapper/019:20:270
751599130,2cyclictest0-21swapper/019:20:270
751599130,2cyclictest0-21swapper/000:08:110
751599130,12cyclictest11-21rcuc/021:52:470
751599130,12cyclictest0-21swapper/022:41:170
751599130,11cyclictest0-21swapper/000:01:410
241042130,2sleep30-21swapper/323:24:193
157752130,2sleep00-21swapper/000:21:070
752199129,2cyclictest22218-21diskmemload21:29:153
752199129,2cyclictest12776-21ssh23:11:483
752199129,2cyclictest0-21swapper/323:55:133
752199129,2cyclictest0-21swapper/322:01:373
752199129,2cyclictest0-21swapper/300:15:183
752199129,2cyclictest0-21swapper/300:05:203
752199121,2cyclictest22218-21diskmemload21:50:133
752199120,2cyclictest0-21swapper/323:15:123
752199120,10cyclictest0-21swapper/322:26:173
751999129,2cyclictest1216-21kworker/u8:119:10:052
751999129,2cyclictest0-21swapper/222:25:062
751999129,2cyclictest0-21swapper/220:30:232
751999122,7cyclictest0-21swapper/221:02:582
751999122,7cyclictest0-21swapper/220:07:592
751999120,11cyclictest0-21swapper/222:15:112
751999120,11cyclictest0-21swapper/220:35:152
751999120,11cyclictest0-21swapper/219:50:132
751999120,11cyclictest0-21swapper/219:40:172
751599129,2cyclictest9-21ksoftirqd/022:07:590
751599129,2cyclictest0-21swapper/022:51:100
751599129,2cyclictest0-21swapper/022:16:080
751599127,3cyclictest0-21swapper/021:05:000
751599122,7cyclictest0-21swapper/020:12:590
7515991210,1cyclictest0-21swapper/019:50:120
751599120,2cyclictest0-21swapper/022:31:300
751599120,11cyclictest11-21rcuc/020:05:250
751599120,11cyclictest0-21swapper/023:36:330
751599120,11cyclictest0-21swapper/023:20:270
751599120,11cyclictest0-21swapper/021:41:520
751599120,11cyclictest0-21swapper/019:45:190
751599120,11cyclictest0-21swapper/000:30:400
291762120,2sleep00-21swapper/021:15:440
752199118,2cyclictest0-21swapper/323:29:383
752199118,2cyclictest0-21swapper/322:20:223
752199118,2cyclictest0-21swapper/319:50:173
752199113,6cyclictest0-21swapper/300:10:253
752199112,5cyclictest0-21swapper/300:05:003
752199111,5cyclictest0-21swapper/319:35:013
752199110,9cyclictest0-21swapper/322:47:313
752199110,10cyclictest0-21swapper/321:47:183
752199110,10cyclictest0-21swapper/321:31:043
751999113,6cyclictest0-21swapper/219:20:252
751999113,6cyclictest0-21swapper/219:20:252
751599118,2cyclictest0-21swapper/020:20:300
751599112,6cyclictest0-21swapper/019:16:110
751599111,6cyclictest0-21swapper/021:05:180
318822110,2sleep30-21swapper/300:38:313
218712110,2sleep30-21swapper/319:45:123
168972110,2sleep30-21swapper/321:36:273
752199107,2cyclictest0-21swapper/300:24:213
752199106,2cyclictest15727-21latency_hist20:49:593
752199106,2cyclictest0-21swapper/319:55:293
752199101,5cyclictest0-21swapper/322:30:013
752199100,7cyclictest0-21swapper/319:40:243
752199100,6cyclictest0-21swapper/320:50:243
752199100,4cyclictest0-21swapper/320:56:023
752199100,4cyclictest0-21swapper/320:40:023
752199100,2cyclictest0-21swapper/300:25:283
751999107,2cyclictest13281-21kworker/u8:219:35:042
751999102,7cyclictest0-21swapper/219:47:012
751599102,6cyclictest0-21swapper/020:15:270
751599102,6cyclictest0-21swapper/019:56:190
751599102,5cyclictest5911-21gdm-simple-gree19:43:590
751599100,2cyclictest11-21rcuc/020:45:180
174092100,2sleep00-21swapper/020:52:560
75219996,2cyclictest0-21swapper/320:20:463
75219994,3cyclictest9184-21sshd20:30:493
75219994,3cyclictest5129-21runrttasks20:29:303
75219994,3cyclictest23748-21grep00:30:133
75219994,3cyclictest21430-21munin-node3
75219992,6cyclictest0-21swapper/319:35:043
75219990,7cyclictest5591-21kworker/u8:221:08:053
75219990,4cyclictest0-21swapper/321:00:043
75159994,3cyclictest9012-21timerwakeupswit20:30:290
75159994,3cyclictest19447-21chrt20:58:170
75159991,6cyclictest0-21swapper/019:26:110
75159991,3cyclictest0-21swapper/020:40:160
75219983,3cyclictest14822sleep320:15:093
75219983,3cyclictest13848-21aten2.4_r3power19:25:123
75219983,3cyclictest10421-21mii-tool19:15:163
75219980,6cyclictest7098-21ssh23:40:073
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional