You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-09-18 - 16:02

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot6s.osadl.org (updated Thu Sep 18, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23401211895,18sleep30-21swapper/307:08:183
23258211393,15sleep00-21swapper/007:06:280
23220211192,13sleep20-21swapper/207:05:572
23497210486,13sleep10-21swapper/107:09:331
133812740,0sleep30-21swapper/312:00:493
39292710,0sleep20-21swapper/210:45:152
48212280,1sleep00-21swapper/012:25:270
2356599260,24cyclictest0-21swapper/211:00:112
2356499241,21cyclictest7619-21diskmemload11:55:131
60072230,0sleep10-21swapper/109:40:261
2356499222,18cyclictest332250irq/24-eth112:15:121
2356599211,18cyclictest0-21swapper/209:05:122
2356499211,2cyclictest0-21swapper/111:40:131
2356399210,2cyclictest0-21swapper/007:55:150
23566992017,2cyclictest24422-21kworker/u8:110:08:043
23566992017,2cyclictest18356-21kworker/u8:210:00:053
23566992016,2cyclictest4864-21kworker/u8:209:45:053
2356599200,2cyclictest0-21swapper/209:35:142
2356499200,2cyclictest0-21swapper/110:40:121
2356499200,2cyclictest0-21swapper/110:00:121
2356499200,2cyclictest0-21swapper/107:45:111
2356699190,2cyclictest0-21swapper/307:45:123
2356599190,17cyclictest0-21swapper/208:15:122
23563991915,2cyclictest5129-21runrttasks07:15:110
201242190,1sleep20-21swapper/209:21:512
108082190,1sleep121-21rcuc/109:12:281
23566991815,2cyclictest15585-21kworker/u8:008:53:053
23566991815,2cyclictest13151-21kworker/u8:211:48:053
80372170,0sleep20-21swapper/211:55:262
23566991714,2cyclictest24513-21kworker/u8:211:07:053
23566991714,2cyclictest12833-21kworker/u8:109:52:053
23566991714,2cyclictest12621-21kworker/u8:111:16:043
2356499171,14cyclictest1538-21apt-get10:10:101
2356499170,15cyclictest0-21swapper/109:32:411
23566991613,2cyclictest9047-21kworker/u8:109:14:063
23566991613,2cyclictest8383-21kworker/u8:212:34:043
23566991613,2cyclictest6040-21kworker/u8:010:52:053
23566991613,2cyclictest29905-21kworker/u8:011:37:053
23566991613,2cyclictest24422-21kworker/u8:110:20:023
23566991613,2cyclictest24422-21kworker/u8:110:17:053
23566991613,2cyclictest20694-21kworker/u8:009:59:043
23566991613,2cyclictest20694-21kworker/u8:009:41:043
23566991611,3cyclictest26103-21ssh12:15:123
2356699160,2cyclictest0-21swapper/308:55:123
2356499160,14cyclictest0-21swapper/109:25:201
58192151,3sleep05982-21aten2.4-expect09:05:140
23566991512,2cyclictest8250-21kworker/u8:111:23:053
23566991512,2cyclictest782-21kworker/u8:011:53:053
23566991512,2cyclictest5773-21kworker/u8:208:08:043
23566991512,2cyclictest27347-21kworker/u8:007:32:033
23566991512,2cyclictest25464-21kworker/u8:010:38:063
23566991512,2cyclictest20694-21kworker/u8:009:38:053
23566991512,2cyclictest19539-21kworker/u8:310:58:053
23566991512,2cyclictest19539-21kworker/u8:310:58:043
23566991512,2cyclictest15585-21kworker/u8:008:42:043
23566991512,2cyclictest11711-21kworker/u8:012:11:053
23566991510,3cyclictest0-21swapper/310:25:093
23566991510,3cyclictest0-21swapper/308:25:233
23566991510,3cyclictest0-21swapper/308:25:233
2356699150,13cyclictest0-21swapper/307:10:183
2356599151,12cyclictest19830-21ssh12:08:182
2356599151,12cyclictest13063-21cpuspeed_turbos09:15:172
23565991510,3cyclictest2369-21sed11:50:172
23565991510,3cyclictest0-21swapper/212:26:412
2356599150,2cyclictest0-21swapper/210:11:432
2356599150,13cyclictest0-21swapper/208:25:122
2356599150,13cyclictest0-21swapper/208:25:122
23564991510,3cyclictest4994-21sh12:25:291
2356499150,14cyclictest0-21swapper/112:06:591
2356499150,13cyclictest7619-21diskmemload10:45:191
2356499150,13cyclictest0-21swapper/111:11:301
2356399152,8cyclictest20924-21ssh12:10:000
2356399152,8cyclictest0-21swapper/011:15:010
2356399152,7cyclictest0-21swapper/008:40:010
2356399150,7cyclictest0-21swapper/011:22:040
2356399150,3cyclictest19405-21ssh09:55:120
157912150,3sleep315897-21awk11:30:203
2356699149,3cyclictest0-21swapper/310:45:183
23566991411,2cyclictest8250-21kworker/u8:111:27:053
23566991411,2cyclictest5926-21kworker/u8:009:16:053
23566991411,2cyclictest5926-21kworker/u8:009:09:043
23566991411,2cyclictest30821-21kworker/u8:210:10:023
23566991411,2cyclictest27347-21kworker/u8:007:22:043
23566991411,2cyclictest24513-21kworker/u8:211:10:053
23566991411,2cyclictest24422-21kworker/u8:110:31:043
23566991411,2cyclictest20694-21kworker/u8:009:25:043
23566991411,2cyclictest15585-21kworker/u8:008:48:053
23566991411,2cyclictest14916-21kworker/u8:209:20:053
23566991411,2cyclictest11711-21kworker/u8:012:09:043
23566991411,2cyclictest0-21swapper/312:29:273
2356699140,2cyclictest26519-21ssh11:41:063
23565991411,2cyclictest11450-21kworker/u8:108:00:062
23565991411,2cyclictest0-21swapper/211:24:332
23565991411,2cyclictest0-21swapper/210:57:122
23565991411,2cyclictest0-21swapper/210:57:112
23565991411,2cyclictest0-21swapper/210:15:232
23565991411,2cyclictest0-21swapper/210:04:232
2356599140,2cyclictest0-21swapper/212:16:232
2356599140,2cyclictest0-21swapper/207:55:152
2356599140,13cyclictest0-21swapper/209:33:122
2356599140,13cyclictest0-21swapper/208:35:132
2356599140,12cyclictest29459-21aten2.4-expect11:45:132
2356599140,12cyclictest0-21swapper/212:11:332
2356599140,12cyclictest0-21swapper/209:12:482
2356499149,3cyclictest32544-21rm12:20:281
23564991411,2cyclictest0-21swapper/112:03:091
23564991411,2cyclictest0-21swapper/111:09:291
23564991411,2cyclictest0-21swapper/110:33:501
23564991411,2cyclictest0-21swapper/109:57:191
23564991411,2cyclictest0-21swapper/109:53:401
23564991410,2cyclictest0-21swapper/111:22:521
2356499140,2cyclictest17888-21cat08:15:001
2356499140,13cyclictest0-21swapper/112:34:191
2356499140,13cyclictest0-21swapper/110:08:101
2356499140,13cyclictest0-21swapper/109:45:091
2356499140,12cyclictest7419-21rm10:15:181
2356499140,12cyclictest0-21swapper/112:35:201
2356499140,12cyclictest0-21swapper/110:58:271
2356499140,12cyclictest0-21swapper/110:58:271
2356499140,12cyclictest0-21swapper/108:45:141
2356399142,7cyclictest0-21swapper/010:35:010
2356399142,7cyclictest0-21swapper/008:15:010
2356399142,6cyclictest0-21swapper/010:55:010
23563991411,2cyclictest0-21swapper/011:50:140
2356399140,13cyclictest0-21swapper/011:55:280
2356399140,12cyclictest0-21swapper/012:31:320
2356399140,12cyclictest0-21swapper/010:05:210
267372130,3sleep126854-21memory10:35:181
23566991310,2cyclictest28926-21kworker/u8:012:38:053
23566991310,2cyclictest28926-21kworker/u8:012:20:063
23566991310,2cyclictest27347-21kworker/u8:007:19:053
23566991310,2cyclictest20380-21ssh11:01:393
23566991310,2cyclictest19539-21kworker/u8:310:43:053
23566991310,2cyclictest15585-21kworker/u8:008:32:053
23566991310,2cyclictest11450-21kworker/u8:108:03:053
23566991310,2cyclictest0-21swapper/311:55:193
2356599139,2cyclictest0-21swapper/207:15:232
2356599138,3cyclictest23280-21ssh11:05:132
2356599138,3cyclictest0-21swapper/210:34:252
23565991310,2cyclictest6880-21ssh09:41:122
23565991310,2cyclictest30538-21ssh11:12:232
23565991310,2cyclictest0-21swapper/212:21:032
23565991310,2cyclictest0-21swapper/212:01:362
23565991310,2cyclictest0-21swapper/211:40:172
23565991310,2cyclictest0-21swapper/211:35:132
23565991310,2cyclictest0-21swapper/209:25:532
23565991310,2cyclictest0-21swapper/209:00:102
2356599130,2cyclictest7619-21diskmemload11:16:052
2356599130,2cyclictest0-21swapper/210:50:432
2356599130,2cyclictest0-21swapper/210:09:142
2356599130,2cyclictest0-21swapper/209:51:322
2356599130,2cyclictest0-21swapper/209:47:532
2356599130,2cyclictest0-21swapper/208:10:142
2356599130,2cyclictest0-21swapper/207:50:192
2356599130,12cyclictest0-21swapper/212:31:062
2356599130,12cyclictest0-21swapper/211:25:532
2356599130,12cyclictest0-21swapper/210:44:532
2356599130,12cyclictest0-21swapper/210:38:082
2356599130,12cyclictest0-21swapper/210:28:382
2356599130,12cyclictest0-21swapper/209:55:032
2356599130,11cyclictest16988-21ssh11:31:212
2356499132,6cyclictest14644-21ssh11:30:001
23564991310,2cyclictest332250irq/24-eth110:28:401
23564991310,2cyclictest0-21swapper/112:14:191
23564991310,2cyclictest0-21swapper/111:28:501
23564991310,2cyclictest0-21swapper/110:21:491
23564991310,2cyclictest0-21swapper/109:05:221
23564991310,2cyclictest0-21swapper/108:40:011
23564991310,2cyclictest0-21swapper/107:10:121
2356499130,2cyclictest0-21swapper/111:53:491
2356499130,11cyclictest0-21swapper/111:45:151
2356499130,11cyclictest0-21swapper/109:39:201
2356499130,11cyclictest0-21swapper/109:23:111
2356399139,2cyclictest0-21swapper/009:42:430
2356399138,3cyclictest0-21swapper/008:25:100
2356399138,3cyclictest0-21swapper/008:25:090
2356399132,7cyclictest0-21swapper/009:05:010
2356399132,6cyclictest0-21swapper/007:25:000
23563991311,1cyclictest0-21swapper/012:15:160
23563991310,2cyclictest19539-21kworker/u8:310:59:050
23563991310,2cyclictest19539-21kworker/u8:310:59:040
23563991310,2cyclictest0-21swapper/011:27:370
23563991310,2cyclictest0-21swapper/009:52:190
23563991310,2cyclictest0-21swapper/009:38:280
23563991310,2cyclictest0-21swapper/009:33:090
2356399130,2cyclictest0-21swapper/012:00:250
2356399130,2cyclictest0-21swapper/010:40:150
2356399130,2cyclictest0-21swapper/008:40:190
2356399130,12cyclictest0-21swapper/011:38:590
2356399130,12cyclictest0-21swapper/010:45:120
2356399130,12cyclictest0-21swapper/010:20:590
2356399130,11cyclictest0-21swapper/011:45:120
2356399130,11cyclictest0-21swapper/011:41:170
130992130,2sleep00-21swapper/012:35:050
123692130,2sleep00-21swapper/009:14:560
2356699122,7cyclictest0-21swapper/309:02:593
2356699122,7cyclictest0-21swapper/308:24:013
2356699121,6cyclictest11561-21copy07:59:593
2356699120,2cyclictest0-21swapper/308:10:133
2356699120,11cyclictest0-21swapper/308:18:043
2356599129,2cyclictest0-21swapper/210:20:052
2356599129,2cyclictest0-21swapper/207:40:112
2356599120,2cyclictest0-21swapper/212:35:572
2356599120,2cyclictest0-21swapper/207:45:242
2356599120,2cyclictest0-21swapper/207:30:092
2356599120,11cyclictest0-21swapper/208:50:192
2356599120,11cyclictest0-21swapper/208:30:242
2356499129,2cyclictest0-21swapper/111:00:011
2356499129,2cyclictest0-21swapper/110:50:301
2356499121,9cyclictest13389-21perl09:15:181
2356499120,2cyclictest0-21swapper/111:37:421
2356499120,11cyclictest0-21swapper/111:15:141
2356499120,11cyclictest0-21swapper/108:50:251
2356499120,11cyclictest0-21swapper/108:00:211
2356399129,2cyclictest7619-21diskmemload10:35:040
2356399129,2cyclictest0-21swapper/011:17:290
2356399129,2cyclictest0-21swapper/011:08:390
2356399129,2cyclictest0-21swapper/010:29:540
2356399129,2cyclictest0-21swapper/009:20:070
2356399128,2cyclictest24557-21sh12:13:060
2356399122,7cyclictest0-21swapper/008:54:590
2356399121,10cyclictest0-21swapper/012:20:090
2356399120,2cyclictest0-21swapper/010:04:590
2356399120,2cyclictest0-21swapper/009:25:250
2356399120,2cyclictest0-21swapper/009:15:160
2356399120,2cyclictest0-21swapper/008:00:050
2356399120,11cyclictest12933-21ssh09:48:010
2356399120,11cyclictest0-21swapper/011:00:120
2356399120,11cyclictest0-21swapper/010:17:250
2356399120,11cyclictest0-21swapper/010:10:290
2356399120,11cyclictest0-21swapper/008:30:130
2356399120,11cyclictest0-21swapper/007:40:120
2356399120,10cyclictest0-21swapper/011:30:070
23142120,8sleep20-21swapper/207:35:182
2356699118,2cyclictest12-21rcu_preempt09:30:143
2356599118,2cyclictest0-21swapper/208:20:282
2356599111,7cyclictest0-21swapper/207:24:582
2356599111,5cyclictest0-21swapper/208:50:012
2356599110,1cyclictest0-21swapper/208:05:192
2356499118,2cyclictest0-21swapper/108:05:191
2356499111,8cyclictest1454-21apt-get07:35:111
2356499110,9cyclictest28120-21grep07:20:121
2356499110,9cyclictest0-21swapper/108:55:131
2356499110,9cyclictest0-21swapper/108:40:241
2356499110,9cyclictest0-21swapper/108:25:101
2356499110,9cyclictest0-21swapper/108:25:101
2356499110,9cyclictest0-21swapper/108:20:161
2356499110,9cyclictest0-21swapper/107:30:111
2356499110,9cyclictest0-21swapper/107:25:201
2356499110,2cyclictest0-21swapper/109:00:141
2356399118,2cyclictest0-21swapper/007:20:250
2356399118,2cyclictest0-21swapper/007:14:530
2356399110,1cyclictest0-21swapper/008:20:160
2356399110,10cyclictest0-21swapper/007:35:140
2356699105,3cyclictest30705-21ntpq07:25:213
2356699100,2cyclictest0-21swapper/307:53:043
2356599102,6cyclictest0-21swapper/207:10:232
2356599101,5cyclictest0-21swapper/207:30:002
2356599100,7cyclictest0-21swapper/208:55:592
2356599100,7cyclictest0-21swapper/208:43:582
2356499107,2cyclictest0-21swapper/107:50:131
2356499100,8cyclictest0-21swapper/107:55:151
2356499100,8cyclictest0-21swapper/107:40:111
2356499100,3cyclictest25785-21munin-node1
2356399107,2cyclictest0-21swapper/008:45:140
2356399105,3cyclictest31953-21aten2.4-expect07:30:100
2356399100,3cyclictest14608-21iostat_ios08:05:180
2356399100,1cyclictest0-21swapper/008:55:120
235669995,2cyclictest37-21rcuc/308:35:143
235669994,3cyclictest2825-21head07:35:263
235669994,3cyclictest0-21swapper/307:40:123
235649996,2cyclictest0-21swapper/108:15:101
235649990,8cyclictest0-21swapper/108:30:101
235639994,3cyclictest8560-21ntpq07:50:180
235639994,3cyclictest6925-21timerwakeupswit07:45:260
235639994,3cyclictest17528-21chrt08:12:250
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional