You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-02 - 23:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack3slot6s.osadl.org (updated Fri May 02, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2322821350,0sleep10-21swapper/110:24:351
296342115104,6sleep20-21swapper/207:07:332
29660211397,11sleep30-21swapper/307:07:543
29590210897,6sleep00-21swapper/007:06:590
29484210089,6sleep10-21swapper/107:05:361
168762760,0sleep10-21swapper/110:17:251
57642750,1sleep1221ktimersoftd/112:17:351
82212740,1sleep30-21swapper/310:09:173
105252710,0sleep10-21swapper/109:38:501
145752700,0sleep10-21swapper/111:54:141
75782280,0sleep10-21swapper/111:14:171
272092240,1sleep123-21ksoftirqd/109:22:421
133012230,1sleep10-21swapper/111:20:061
2985299200,2cyclictest0-21swapper/309:40:133
2985299200,18cyclictest0-21swapper/310:50:133
29851992017,2cyclictest4193-21kworker/u8:210:56:042
29851992016,2cyclictest31598-21kworker/u8:110:34:042
29851992015,3cyclictest16663-21ssh09:45:162
2985199200,2cyclictest0-21swapper/207:40:112
2985199200,18cyclictest0-21swapper/209:20:132
2985199200,18cyclictest0-21swapper/208:10:132
2985099200,18cyclictest0-21swapper/110:50:141
2984999200,2cyclictest0-21swapper/007:40:120
2984999200,18cyclictest0-21swapper/007:35:150
96652190,0sleep00-21swapper/009:37:330
29851991916,2cyclictest4193-21kworker/u8:211:16:042
29851991916,2cyclictest28665-21kworker/u8:012:19:042
2984999180,16cyclictest0-21swapper/008:10:130
29851991714,2cyclictest14549-21kworker/u8:112:28:052
29851991613,2cyclictest4193-21kworker/u8:210:48:042
29851991613,2cyclictest10370-21kworker/u8:109:44:042
29851991611,3cyclictest0-21swapper/212:37:462
2985099161,13cyclictest0-21swapper/112:30:391
2985099160,14cyclictest0-21swapper/110:56:171
2984999161,3cyclictest0-21swapper/012:20:010
2984999160,3cyclictest0-21swapper/011:48:480
224392160,1sleep011-21rcuc/011:29:240
2985299151,12cyclictest2277-21diskstats08:40:173
2985299150,3cyclictest6736-21irqstats08:50:153
2985299150,13cyclictest0-21swapper/309:30:203
29851991512,2cyclictest3886-21kworker/u8:011:47:042
29851991512,2cyclictest30816-21kworker/u8:311:07:032
29851991512,2cyclictest24316-21kworker/u8:208:31:032
29851991512,2cyclictest22660-21kworker/u8:111:00:052
29851991512,2cyclictest18330-21kworker/u8:011:32:032
29851991512,2cyclictest14054-21kworker/u8:109:10:052
2985199151,12cyclictest3320-21apt-get11:10:122
2985199151,12cyclictest27676-21ssh10:28:532
2985199151,12cyclictest0-21swapper/207:45:282
29851991510,3cyclictest16023-21ssh11:22:062
2985199150,13cyclictest13896-21diskmemload12:20:412
2985099152,7cyclictest0-21swapper/109:00:001
2985099152,2cyclictest0-21swapper/111:03:201
29850991510,3cyclictest0-21swapper/110:35:151
2985099150,3cyclictest0-21swapper/109:10:191
2985099150,13cyclictest21917-21ssh11:28:361
2984999151,12cyclictest25227-21ssh10:59:300
2984999151,12cyclictest2116-21rm10:02:470
29849991510,3cyclictest2266-21/usr/sbin/munin12:15:000
2984999150,3cyclictest0-21swapper/011:35:410
2984999150,13cyclictest11337-21ssh12:23:340
2984999150,13cyclictest0-21swapper/011:40:150
2984999150,13cyclictest0-21swapper/009:20:200
93082140,1sleep00-21swapper/008:55:270
2985299149,3cyclictest0-21swapper/312:25:193
29852991410,2cyclictest800-21cpuspeed_turbos11:40:153
2985299140,12cyclictest31904-21ps11:05:203
2985299140,12cyclictest25855-21ssh11:00:073
2985299140,12cyclictest10591-21fschecks_time09:00:173
2985299140,12cyclictest0-21swapper/311:15:173
2985199149,3cyclictest331-21ssh12:12:052
29851991411,2cyclictest7797-21kworker/u8:207:14:052
29851991411,2cyclictest5560-21kworker/u8:107:18:042
29851991411,2cyclictest4193-21kworker/u8:211:26:052
29851991411,2cyclictest31598-21kworker/u8:110:35:042
29851991411,2cyclictest19678-21ssh10:53:312
29851991411,2cyclictest0-21swapper/212:34:212
29851991411,2cyclictest0-21swapper/208:40:212
29851991410,2cyclictest0-21swapper/209:57:112
2985199140,2cyclictest0-21swapper/210:16:012
2985199140,2cyclictest0-21swapper/210:00:562
2985199140,13cyclictest0-21swapper/208:45:122
2985199140,13cyclictest0-21swapper/208:45:112
2985199140,12cyclictest3049-21nscd07:25:152
2985199140,12cyclictest28058-21sshd11:35:112
2985199140,12cyclictest23067-21ssh09:18:552
2985199140,12cyclictest0-21swapper/209:31:512
2985099140,13cyclictest0-21swapper/108:50:241
2985099140,12cyclictest0-21swapper/108:40:171
2984999141,3cyclictest17411-21ssh09:12:530
29849991411,2cyclictest17188-21ssh09:45:180
2984999140,3cyclictest22029-21date08:10:010
2984999140,3cyclictest0-21swapper/011:33:400
2984999140,2cyclictest0-21swapper/011:14:480
2984999140,12cyclictest27559-21ssh12:06:340
2984999140,12cyclictest0-21swapper/012:04:020
2984999140,12cyclictest0-21swapper/010:17:570
2984999140,12cyclictest0-21swapper/010:09:560
2984999140,12cyclictest0-21swapper/008:00:110
211312140,2sleep00-21swapper/012:33:390
55242130,5sleep121-21rcuc/111:45:041
29852991310,2cyclictest0-21swapper/310:35:193
29852991310,2cyclictest0-21swapper/310:25:163
2985299130,2cyclictest0-21swapper/311:28:143
2985299130,2cyclictest0-21swapper/307:15:053
2985299130,11cyclictest0-21swapper/310:12:273
2985299130,11cyclictest0-21swapper/309:52:383
29851991310,2cyclictest5928-21kworker/u8:207:54:042
29851991310,2cyclictest30493-21kworker/u8:311:52:042
29851991310,2cyclictest24316-21kworker/u8:208:28:052
29851991310,2cyclictest11676-21ssh10:12:042
29851991310,2cyclictest11537-21kworker/u8:209:09:042
29851991310,2cyclictest0-21swapper/210:40:122
29851991310,2cyclictest0-21swapper/210:20:412
29851991310,2cyclictest0-21swapper/210:06:132
29851991310,2cyclictest0-21swapper/209:50:202
29851991310,2cyclictest0-21swapper/209:25:122
29851991310,2cyclictest0-21swapper/208:05:112
2985199130,2cyclictest13896-21diskmemload12:08:022
2985199130,2cyclictest0-21swapper/212:00:032
2985199130,2cyclictest0-21swapper/208:35:142
2985199130,11cyclictest0-21swapper/211:40:562
2985099138,3cyclictest26903-21df_abs09:55:181
2985099138,3cyclictest0-21swapper/111:30:201
2985099138,3cyclictest0-21swapper/110:27:581
2985099138,3cyclictest0-21swapper/107:50:121
2985099131,10cyclictest0-21swapper/110:10:261
29850991310,2cyclictest0-21swapper/112:05:181
29850991310,2cyclictest0-21swapper/109:51:191
29850991310,2cyclictest0-21swapper/109:32:471
2985099130,12cyclictest0-21swapper/109:25:241
2985099130,11cyclictest0-21swapper/112:25:231
2985099130,11cyclictest0-21swapper/112:01:301
2985099130,11cyclictest0-21swapper/108:35:221
2985099130,11cyclictest0-21swapper/107:45:261
2984999138,3cyclictest32164-21ssh10:33:140
29849991310,2cyclictest29848-21cyclictest11:04:280
29849991310,2cyclictest0-21swapper/012:37:190
29849991310,2cyclictest0-21swapper/011:17:120
29849991310,2cyclictest0-21swapper/011:07:440
29849991310,2cyclictest0-21swapper/010:51:110
29849991310,2cyclictest0-21swapper/010:25:140
29849991310,2cyclictest0-21swapper/010:20:230
29849991310,2cyclictest0-21swapper/010:10:170
2984999130,3cyclictest13287-21ssh11:20:050
2984999130,2cyclictest21987-21ssh09:50:160
2984999130,2cyclictest0-21swapper/011:58:590
2984999130,2cyclictest0-21swapper/011:51:590
2984999130,2cyclictest0-21swapper/010:45:560
2984999130,12cyclictest0-21swapper/009:55:080
2984999130,12cyclictest0-21swapper/009:40:050
2984999130,12cyclictest0-21swapper/007:55:150
191122130,2sleep10-21swapper/109:46:521
316002120,2sleep10-21swapper/110:32:251
2985299129,2cyclictest0-21swapper/310:30:063
2985299129,2cyclictest0-21swapper/310:20:363
2985299129,2cyclictest0-21swapper/309:59:453
2985299129,2cyclictest0-21swapper/307:25:153
2985299120,2cyclictest0-21swapper/310:55:273
2985299120,2cyclictest0-21swapper/309:35:203
2985299120,2cyclictest0-21swapper/309:25:143
2985299120,11cyclictest0-21swapper/312:30:043
2985299120,11cyclictest0-21swapper/312:03:073
2985299120,11cyclictest0-21swapper/311:35:383
2985299120,11cyclictest0-21swapper/310:00:143
2985299120,11cyclictest0-21swapper/309:22:153
2985299120,11cyclictest0-21swapper/307:40:153
2985299120,11cyclictest0-21swapper/307:20:173
2985299120,10cyclictest0-21swapper/312:35:363
2985299120,10cyclictest0-21swapper/312:13:093
2985199129,2cyclictest5928-21kworker/u8:207:35:052
2985199129,2cyclictest18235-21kworker/u8:008:01:052
2985199129,2cyclictest15784-21kworker/u8:107:57:042
2985199122,7cyclictest0-21swapper/207:30:582
2985199120,2cyclictest0-21swapper/209:35:432
2985199120,2cyclictest0-21swapper/208:55:252
2985199120,11cyclictest0-21swapper/211:55:162
2985199120,11cyclictest0-21swapper/208:20:282
2985099129,2cyclictest0-21swapper/112:35:171
2985099129,2cyclictest0-21swapper/111:36:281
2985099129,2cyclictest0-21swapper/109:43:041
2985099128,2cyclictest0-21swapper/107:35:171
2985099127,3cyclictest0-21swapper/108:15:061
2985099124,3cyclictest20297-21head08:05:121
2985099122,7cyclictest4572-21gdm-simple-gree08:14:581
2985099122,7cyclictest13896-21diskmemload11:05:011
2985099122,7cyclictest0-21swapper/108:56:011
2985099121,9cyclictest6675-21df_abs10:40:171
2985099121,9cyclictest30067-21munin-node1
2985099121,9cyclictest0-21swapper/110:45:171
2985099121,9cyclictest0-21swapper/110:00:181
2985099121,9cyclictest0-21swapper/107:55:171
2985099121,9cyclictest0-21swapper/107:20:141
2985099121,3cyclictest5594-21grep10:05:271
2985099120,2cyclictest0-21swapper/107:30:261
2985099120,11cyclictest13896-21diskmemload11:42:061
2985099120,11cyclictest0-21swapper/112:22:151
2985099120,11cyclictest0-21swapper/111:16:181
2984999129,2cyclictest30297-21apt-get08:30:110
2984999129,2cyclictest0-21swapper/010:40:590
2984999129,2cyclictest0-21swapper/009:16:200
2984999129,2cyclictest0-21swapper/007:15:150
2984999122,6cyclictest0-21swapper/007:30:010
2984999122,5cyclictest0-21swapper/008:45:010
2984999121,6cyclictest0-21swapper/008:40:000
2984999120,11cyclictest0-21swapper/012:25:070
2984999120,11cyclictest0-21swapper/010:36:240
2984999120,11cyclictest0-21swapper/009:30:000
2984999120,11cyclictest0-21swapper/009:27:110
2984999120,11cyclictest0-21swapper/008:45:190
2984999120,11cyclictest0-21swapper/008:45:180
2984999120,11cyclictest0-21swapper/007:48:060
183412120,2sleep10-21swapper/111:57:311
2985299118,2cyclictest13896-21diskmemload11:59:203
2985299118,2cyclictest0-21swapper/311:33:553
2985299118,2cyclictest0-21swapper/310:16:273
2985299118,2cyclictest0-21swapper/309:45:363
2985299112,6cyclictest0-21swapper/307:58:013
2985299110,9cyclictest0-21swapper/312:05:553
2985299110,9cyclictest0-21swapper/309:10:223
2985299110,2cyclictest0-21swapper/311:21:063
2985299110,10cyclictest0-21swapper/311:50:373
2985299110,10cyclictest0-21swapper/311:45:463
2985299110,10cyclictest0-21swapper/311:12:013
2985299110,10cyclictest0-21swapper/309:17:163
2985199118,2cyclictest0-21swapper/208:15:052
2985199110,10cyclictest0-21swapper/209:04:102
2985099117,2cyclictest21-21rcuc/107:25:161
2985099117,2cyclictest0-21swapper/108:20:171
2985099111,8cyclictest3745-21runrttasks07:40:101
2985099111,6cyclictest0-21swapper/112:10:101
2985099111,6cyclictest0-21swapper/108:30:231
2985099111,3cyclictest28283-21munin-node1
2985099110,9cyclictest1461-21latency_hist07:20:001
2985099110,9cyclictest0-21swapper/109:15:121
2985099110,9cyclictest0-21swapper/109:05:161
2985099110,3cyclictest0-21swapper/108:45:131
2985099110,3cyclictest0-21swapper/108:45:121
2984999118,2cyclictest0-21swapper/008:25:200
2984999116,3cyclictest0-21swapper/009:00:160
2984999111,5cyclictest0-21swapper/007:35:000
131752110,2sleep313896-21diskmemload10:46:073
2985299107,2cyclictest0-21swapper/310:40:053
2985299105,3cyclictest13059-21wc09:05:203
2985299102,7cyclictest0-21swapper/308:48:593
2985299102,7cyclictest0-21swapper/308:48:583
2985299102,5cyclictest0-21swapper/308:08:583
2985299100,8cyclictest0-21swapper/312:20:223
2985299100,2cyclictest31854-21apt-get07:15:003
2985299100,2cyclictest0-21swapper/312:15:393
2985199107,2cyclictest24316-21kworker/u8:208:50:042
2984999107,2cyclictest0-21swapper/009:05:220
2984999107,2cyclictest0-21swapper/008:20:300
2984999105,3cyclictest7858-21grep08:55:000
298529996,2cyclictest0-21swapper/308:20:203
298529995,2cyclictest14723-21perl07:50:123
298529991,5cyclictest0-21swapper/308:31:583
298529991,4cyclictest0-21swapper/308:40:013
298529990,8cyclictest0-21swapper/308:15:203
298529990,6cyclictest0-21swapper/308:57:023
298519990,2cyclictest0-21swapper/207:20:242
298509990,8cyclictest0-21swapper/108:00:011
298499994,3cyclictest2128-21fschecks_count07:20:140
298499994,3cyclictest15953-21latency_hist07:55:010
298529984,2cyclictest18730-21fschecks_time08:00:163
298529983,3cyclictest5985-21aten2.4-expect07:30:133
298529983,3cyclictest3745-21runrttasks08:26:553
298529983,3cyclictest3745-21runrttasks08:13:353
298529983,3cyclictest12047-21aten2.4-expect07:45:163
298499981,4cyclictest0-21swapper/008:15:130
298499980,7cyclictest0-21swapper/007:10:150
298529973,2cyclictest8229-21cpuspeed_turbos07:35:173
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional