You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-07-10 - 16:35

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack3slot6s.osadl.org (updated Thu Jul 10, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32101212299,17sleep10-21swapper/107:06:251
32088212096,6sleep30-21swapper/307:06:153
32029211994,18sleep00-21swapper/007:05:280
32191211394,13sleep20-21swapper/207:07:342
241502890,1sleep224157-21tail08:05:302
323792770,1sleep10-21swapper/108:26:241
10092560,0sleep30-21swapper/311:39:243
26222280,1sleep30-21swapper/312:13:483
138862220,0sleep10-21swapper/111:51:211
32408992113,6cyclictest7188-21ssh12:18:041
32409991916,2cyclictest7893-21kworker/u8:010:22:062
32409991916,2cyclictest24463-21kworker/u8:011:10:042
276452190,1sleep327651-21cpuspeed_turbos11:00:123
32409991815,2cyclictest29351-21kworker/u8:212:17:042
32409991815,2cyclictest12835-21kworker/u8:110:46:052
216122180,1sleep20-21swapper/209:47:292
74622170,0sleep00-21swapper/010:05:260
32409991714,2cyclictest25070-21kworker/u8:310:37:042
32409991714,2cyclictest12645-21kworker/u8:211:18:062
3240899170,15cyclictest15070-21munin-node1
3240899170,15cyclictest0-21swapper/110:55:041
258392170,0sleep10-21swapper/110:25:151
32409991613,2cyclictest9748-21kworker/u8:211:56:062
32409991613,2cyclictest6803-21kworker/u8:111:25:042
32409991613,2cyclictest29428-21kworker/u8:109:41:042
32409991613,2cyclictest29351-21kworker/u8:212:22:052
32408991611,3cyclictest3629-21users07:15:281
3240899160,1cyclictest0-21swapper/110:05:441
3240899160,14cyclictest0-21swapper/110:37:331
3241099151,3cyclictest0-21swapper/311:40:153
32410991512,2cyclictest0-21swapper/312:15:173
3241099151,12cyclictest26587-21apt-get08:15:053
3241099150,13cyclictest14036-21ssh11:19:383
3240999151,3cyclictest9933-21ssh11:47:502
3240999151,3cyclictest253712sleep209:50:452
3240999151,3cyclictest1579-21ssh10:33:432
3240999151,3cyclictest14679-21aten2.4_r3power11:20:122
32409991512,2cyclictest6803-21kworker/u8:111:35:042
32409991512,2cyclictest29428-21kworker/u8:109:23:052
32409991512,2cyclictest21506-21kworker/u8:110:05:052
32409991512,2cyclictest21257-21kworker/u8:110:28:052
32409991512,2cyclictest18632-21kworker/u8:009:13:052
32409991511,2cyclictest0-21swapper/210:52:012
3240999151,12cyclictest0-21swapper/210:00:192
3240999150,2cyclictest0-21swapper/210:56:372
3240999150,2cyclictest0-21swapper/209:56:372
3240999150,2cyclictest0-21swapper/209:56:372
3240999150,2cyclictest0-21swapper/208:10:272
3240999150,13cyclictest0-21swapper/209:07:052
32408991512,2cyclictest0-21swapper/111:30:341
32408991512,2cyclictest0-21swapper/109:36:541
32408991511,2cyclictest10065-21grep12:20:271
3240899150,2cyclictest16495-21diskmemload11:09:341
3240899150,2cyclictest16495-21diskmemload09:24:541
3240899150,13cyclictest31204-21latency_hist12:10:011
3240899150,13cyclictest18294-21latency_hist12:30:001
3240899150,13cyclictest16495-21diskmemload09:16:151
3240899150,13cyclictest0-21swapper/109:13:341
201742151,3sleep120259-21kernelversion11:25:161
144222150,0sleep30-21swapper/307:45:003
55302140,3sleep10-21swapper/109:30:281
43872140,2sleep00-21swapper/011:42:030
32410991411,2cyclictest0-21swapper/312:34:243
32410991411,2cyclictest0-21swapper/310:45:183
32410991411,2cyclictest0-21swapper/310:35:153
32410991411,2cyclictest0-21swapper/310:25:263
32410991411,2cyclictest0-21swapper/307:20:173
32410991410,2cyclictest13312-21runrttasks11:50:273
3241099140,3cyclictest27100-21ssh12:38:413
3241099140,12cyclictest6435-21ssh11:10:263
3241099140,12cyclictest0-21swapper/311:30:293
3241099140,12cyclictest0-21swapper/310:41:063
3241099140,12cyclictest0-21swapper/310:20:143
32409991411,2cyclictest8897-21kworker/u8:112:37:052
32409991411,2cyclictest3934-21kworker/u8:009:34:062
32409991411,2cyclictest29351-21kworker/u8:212:14:052
32409991411,2cyclictest23817-21kworker/u8:209:17:042
3240999140,13cyclictest16495-21diskmemload10:14:562
3240999140,13cyclictest0-21swapper/211:50:172
3240999140,12cyclictest16495-21diskmemload10:41:582
3240999140,12cyclictest0-21swapper/208:15:042
3240899149,4cyclictest0-21swapper/110:11:021
32408991411,2cyclictest0-21swapper/112:09:231
32408991411,2cyclictest0-21swapper/111:56:041
32408991411,2cyclictest0-21swapper/111:16:231
32408991411,2cyclictest0-21swapper/110:51:531
32408991411,2cyclictest0-21swapper/110:30:041
32408991411,2cyclictest0-21swapper/107:20:151
3240899140,2cyclictest0-21swapper/112:30:341
3240899140,2cyclictest0-21swapper/112:00:341
3240899140,2cyclictest0-21swapper/111:01:331
3240899140,2cyclictest0-21swapper/110:42:441
3240899140,13cyclictest0-21swapper/109:53:041
3240899140,12cyclictest26587-21apt-get08:15:051
3240899140,12cyclictest2490-21grep11:40:141
3240899140,12cyclictest16495-21diskmemload09:42:311
3240899140,12cyclictest16495-21diskmemload09:25:181
3240899140,12cyclictest16394-40update-apt-xapi07:48:391
3240899140,12cyclictest0-21swapper/111:49:551
3240899140,12cyclictest0-21swapper/110:45:451
3240899140,12cyclictest0-21swapper/110:15:181
3240899140,12cyclictest0-21swapper/110:00:011
3240899140,12cyclictest0-21swapper/109:45:561
32407991410,2cyclictest10679-21ssh09:35:430
3240799140,12cyclictest24284-21ssh11:30:010
3240799140,12cyclictest16495-21diskmemload10:51:590
3240799140,12cyclictest16495-21diskmemload09:50:340
3240799140,12cyclictest0-21swapper/012:35:160
3240799140,12cyclictest0-21swapper/012:11:520
3240799140,12cyclictest0-21swapper/011:09:410
3240799140,12cyclictest0-21swapper/010:49:040
3240799140,12cyclictest0-21swapper/010:37:410
3240799140,12cyclictest0-21swapper/008:20:170
3241099139,3cyclictest0-21swapper/311:20:133
32410991310,2cyclictest20144-21ssh11:58:253
32410991310,2cyclictest0-21swapper/311:07:303
32410991310,2cyclictest0-21swapper/310:50:493
32410991310,2cyclictest0-21swapper/308:25:183
32410991310,2cyclictest0-21swapper/308:20:163
32410991310,2cyclictest0-21swapper/307:10:173
3241099130,2cyclictest0-21swapper/312:21:383
3241099130,2cyclictest0-21swapper/312:02:213
3241099130,2cyclictest0-21swapper/310:31:313
3241099130,2cyclictest0-21swapper/309:35:213
3241099130,2cyclictest0-21swapper/309:33:593
3241099130,2cyclictest0-21swapper/309:15:173
3241099130,12cyclictest32406-21cyclictest09:11:393
3241099130,12cyclictest16495-21diskmemload12:05:573
3241099130,12cyclictest0-21swapper/311:45:113
3241099130,12cyclictest0-21swapper/311:25:153
3241099130,12cyclictest0-21swapper/310:59:193
3241099130,12cyclictest0-21swapper/310:14:443
3241099130,12cyclictest0-21swapper/310:05:153
3241099130,12cyclictest0-21swapper/310:00:183
3241099130,12cyclictest0-21swapper/309:26:473
3241099130,12cyclictest0-21swapper/308:05:193
3241099130,11cyclictest0-21swapper/309:51:313
3240999138,3cyclictest0-21swapper/207:45:002
32409991310,2cyclictest766-21df_abs08:30:132
32409991310,2cyclictest3934-21kworker/u8:009:39:052
32409991310,2cyclictest28315-21kworker/u8:007:12:022
32409991310,2cyclictest22780-21kworker/u8:112:06:052
32409991310,2cyclictest12835-21kworker/u8:111:00:012
32409991310,2cyclictest0-21swapper/211:43:132
32409991310,2cyclictest0-21swapper/209:25:412
32409991310,2cyclictest0-21swapper/208:20:262
32409991310,2cyclictest0-21swapper/208:00:132
32409991310,2cyclictest0-21swapper/207:48:422
32409991310,2cyclictest0-21swapper/207:20:122
3240999130,2cyclictest0-21swapper/211:06:112
3240999130,2cyclictest0-21swapper/210:15:152
3240999130,2cyclictest0-21swapper/208:50:252
3240999130,2cyclictest0-21swapper/208:40:242
3240999130,2cyclictest0-21swapper/208:25:152
3240999130,12cyclictest0-21swapper/212:34:202
32408991310,2cyclictest0-21swapper/111:10:181
3240899130,2cyclictest0-21swapper/110:03:101
3240899130,2cyclictest0-21swapper/108:30:161
3240899130,12cyclictest0-21swapper/109:55:541
3240899130,12cyclictest0-21swapper/108:20:241
3240899130,12cyclictest0-21swapper/107:35:151
3240899130,11cyclictest0-21swapper/107:10:221
3240799131,11cyclictest0-21swapper/009:45:370
32407991310,2cyclictest0-21swapper/012:05:130
32407991310,2cyclictest0-21swapper/012:00:280
32407991310,2cyclictest0-21swapper/011:57:420
32407991310,2cyclictest0-21swapper/011:46:320
32407991310,2cyclictest0-21swapper/011:16:270
32407991310,2cyclictest0-21swapper/011:12:080
32407991310,2cyclictest0-21swapper/010:44:160
32407991310,2cyclictest0-21swapper/010:30:550
32407991310,2cyclictest0-21swapper/010:20:180
32407991310,2cyclictest0-21swapper/010:00:330
32407991310,2cyclictest0-21swapper/009:56:110
32407991310,2cyclictest0-21swapper/009:56:100
32407991310,2cyclictest0-21swapper/009:13:070
3240799130,2cyclictest16495-21diskmemload12:19:030
3240799130,2cyclictest0-21swapper/011:35:540
3240799130,2cyclictest0-21swapper/011:27:070
3240799130,2cyclictest0-21swapper/011:22:060
3240799130,2cyclictest0-21swapper/011:00:270
3240799130,2cyclictest0-21swapper/010:10:220
3240799130,2cyclictest0-21swapper/009:26:210
3240799130,2cyclictest0-21swapper/009:20:210
3240799130,2cyclictest0-21swapper/007:48:130
3240799130,2cyclictest0-21swapper/007:15:170
3240799130,12cyclictest0-21swapper/012:31:410
3240799130,12cyclictest0-21swapper/012:22:300
3240799130,12cyclictest0-21swapper/011:50:030
3240799130,12cyclictest0-21swapper/010:57:400
3240799130,12cyclictest0-21swapper/009:40:470
3240799130,12cyclictest0-21swapper/009:05:200
3241099129,2cyclictest11747-21which08:55:253
3241099129,2cyclictest0-21swapper/310:15:183
3241099129,2cyclictest0-21swapper/309:57:193
3241099129,2cyclictest0-21swapper/309:57:193
3241099129,2cyclictest0-21swapper/309:40:263
3241099120,2cyclictest0-21swapper/312:25:563
3241099120,11cyclictest0-21swapper/309:46:253
3241099120,11cyclictest0-21swapper/309:21:163
3241099120,11cyclictest0-21swapper/309:05:153
3241099120,11cyclictest0-21swapper/308:00:143
3240999129,2cyclictest10656-21kworker/u8:108:57:052
3240999129,2cyclictest0-21swapper/211:30:112
3240999120,2cyclictest0-21swapper/212:27:142
3240999120,2cyclictest0-21swapper/212:01:532
3240999120,11cyclictest0-21swapper/208:45:162
3240999120,11cyclictest0-21swapper/207:15:262
3240899129,2cyclictest0-21swapper/112:36:061
3240899129,2cyclictest0-21swapper/108:50:241
3240899121,9cyclictest13760-21cat07:40:251
3240899120,11cyclictest18400-21kworker/u8:108:05:051
3240899120,11cyclictest0-21swapper/111:36:441
3240899120,11cyclictest0-21swapper/110:21:221
3240899120,11cyclictest0-21swapper/108:00:141
3240899120,10cyclictest0-21swapper/107:30:221
3240799129,2cyclictest0-21swapper/012:25:480
3240799129,2cyclictest0-21swapper/010:25:190
3240799129,2cyclictest0-21swapper/008:05:200
3240799120,2cyclictest0-21swapper/007:40:100
3240799120,11cyclictest16495-21diskmemload09:31:010
3240799120,11cyclictest0-21swapper/010:17:210
3240799120,11cyclictest0-21swapper/009:17:120
3240799120,11cyclictest0-21swapper/008:30:200
3240799120,11cyclictest0-21swapper/008:10:270
3240799120,11cyclictest0-21swapper/007:55:210
3241099118,2cyclictest0-21swapper/309:00:203
3241099110,10cyclictest0-21swapper/308:45:123
3240999118,2cyclictest0-21swapper/209:00:132
3240899111,8cyclictest6646-21apt-get08:45:101
3240899110,9cyclictest0-21swapper/109:05:111
3240899110,9cyclictest0-21swapper/108:40:111
3240799118,2cyclictest0-21swapper/009:00:130
3240799118,2cyclictest0-21swapper/008:35:130
3240799118,2cyclictest0-21swapper/007:20:130
3240799110,9cyclictest0-21swapper/008:15:030
3240799110,10cyclictest18400-21kworker/u8:108:47:050
3240799110,10cyclictest0-21swapper/007:10:160
3241099107,2cyclictest0-21swapper/307:48:203
3240999107,2cyclictest6438-21kworker/u8:007:51:042
3240999105,3cyclictest7258-21wc07:25:182
3240899107,2cyclictest0-21swapper/109:00:161
3240899107,2cyclictest0-21swapper/108:35:011
3240899107,2cyclictest0-21swapper/107:25:261
3240899100,8cyclictest0-21swapper/108:55:141
3240899100,3cyclictest16744-21munin-node1
3240799100,9cyclictest6438-21kworker/u8:008:02:050
3240799100,8cyclictest2177-21packagekitd07:25:090
201882100,2sleep30-21swapper/307:55:313
324109996,2cyclictest0-21swapper/308:40:133
324109996,2cyclictest0-21swapper/308:35:053
324109996,2cyclictest0-21swapper/307:27:593
324109994,3cyclictest5129-21runrttasks07:30:153
324099994,3cyclictest19016-21cpuspeed_turbos07:55:162
324099994,3cyclictest11517-21pzem_current07:35:192
324099993,2cyclictest3816-21unixbench_singl08:35:262
324099990,8cyclictest0-21swapper/207:30:372
324079994,3cyclictest8751-21head08:50:140
324079990,8cyclictest6438-21kworker/u8:007:34:050
324079990,7cyclictest31614-21latency08:25:200
26367290,2sleep30-21swapper/308:12:173
324109985,2cyclictest9513-21tr08:50:183
324109984,3cyclictest429-21aten2.4_r3power08:30:123
324109984,3cyclictest10530-21aten2.4_r3power07:35:123
324109980,7cyclictest0-21swapper/307:15:223
324109980,2cyclictest0-21swapper/307:53:593
324089984,3cyclictest4166-21rsyslogd07:55:161
324089983,3cyclictest25240-21grep08:10:161
324079985,2cyclictest0-21swapper/008:43:260
324079984,3cyclictest278-21jbd2/sda1-807:50:150
324079983,3cyclictest4365-21nscd08:55:150
324079983,3cyclictest10743-21sort07:35:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional