You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-03-19 - 19:38

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #3, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack3slot6s.osadl.org (updated Wed Mar 19, 2025 12:43:49)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
23425211997,16sleep30-21swapper/307:07:453
23397211997,16sleep00-21swapper/007:07:220
23327211897,16sleep20-21swapper/207:06:282
23569211799,13sleep10-21swapper/107:09:361
197852760,0sleep30-21swapper/309:58:583
183532750,0sleep00-21swapper/012:16:200
297402730,0sleep20-21swapper/212:30:132
217972720,1sleep00-21swapper/012:20:170
305682610,0sleep20-21swapper/209:35:162
210402480,1sleep021038-21ssh11:44:410
23629993214,16cyclictest0-21swapper/210:50:072
4172240,0sleep20-21swapper/211:21:462
2362999232,19cyclictest0-21swapper/208:55:152
2362999212,17cyclictest0-21swapper/210:40:052
23629992118,2cyclictest9979-21kworker/u8:108:05:042
23629992118,2cyclictest6884-21kworker/u8:107:22:052
2362999210,19cyclictest0-21swapper/210:30:072
2362899211,18cyclictest6209-21diskmemload10:40:091
23629992017,2cyclictest2868-21kworker/u8:211:28:052
23629992017,2cyclictest28092-21kworker/u8:008:47:052
2362999200,2cyclictest0-21swapper/209:50:082
2362999200,16cyclictest0-21swapper/208:16:032
2362799200,2cyclictest0-21swapper/009:50:080
23629991916,2cyclictest27979-21kworker/u8:207:33:042
2362799190,2cyclictest0-21swapper/010:40:090
2363099181,3cyclictest30068-21ssh11:20:003
23629991815,2cyclictest5778-21kworker/u8:012:04:042
2362999180,2cyclictest0-21swapper/207:16:042
2362999180,16cyclictest0-21swapper/210:12:032
2362999174,8cyclictest6122-21kworker/2:110:03:042
23629991714,2cyclictest9979-21kworker/u8:108:37:052
2362999170,2cyclictest0-21swapper/212:08:042
2362999170,2cyclictest0-21swapper/210:15:052
2362999170,12cyclictest0-21swapper/210:28:042
2362999170,11cyclictest0-21swapper/208:54:052
23627991712,3cyclictest0-21swapper/009:43:120
2363099161,3cyclictest0-21swapper/312:15:523
23629991613,2cyclictest6500-21kworker/u8:109:27:042
23629991613,2cyclictest20952-21kworker/u8:111:59:042
23629991613,2cyclictest20952-21kworker/u8:111:59:042
2362999160,2cyclictest0-21swapper/209:44:052
2362999160,2cyclictest0-21swapper/207:26:052
2362999160,11cyclictest0-21swapper/211:11:022
2362899162,13cyclictest23-21ksoftirqd/110:14:591
23628991613,2cyclictest23-21ksoftirqd/110:06:191
23628991612,2cyclictest0-21swapper/110:27:291
2362899160,14cyclictest6209-21diskmemload11:20:481
2362899160,14cyclictest0-21swapper/111:10:151
23627991611,3cyclictest25419-21rm11:15:100
19532160,2sleep339-21ksoftirqd/309:39:583
2363099152,7cyclictest0-21swapper/308:55:003
2363099151,3cyclictest8805-21ssh09:11:273
2363099151,2cyclictest3789-21ssh11:26:203
2363099151,12cyclictest26985-21cat10:40:223
2363099151,12cyclictest25248-21ssh12:24:123
2363099150,2cyclictest0-21swapper/310:36:503
2363099150,2cyclictest0-21swapper/309:49:003
2363099150,13cyclictest6209-21diskmemload12:11:003
2363099150,13cyclictest6209-21diskmemload10:27:403
2362999152,7cyclictest0-21swapper/208:30:002
23629991512,2cyclictest992-21kworker/u8:112:39:052
23629991512,2cyclictest30527-21kworker/u8:109:45:052
23629991512,2cyclictest2868-21kworker/u8:211:34:052
23629991512,2cyclictest20952-21kworker/u8:111:51:042
23629991512,2cyclictest20952-21kworker/u8:111:51:042
23629991512,2cyclictest20383-21kworker/u8:110:38:052
23629991512,2cyclictest13574-21kworker/u8:011:37:042
23629991511,2cyclictest16769-21ntpdc09:55:152
2362999150,2cyclictest0-21swapper/208:12:042
2362999150,13cyclictest0-21swapper/212:20:192
23628991512,2cyclictest23-21ksoftirqd/110:59:081
2362899151,12cyclictest23-21ksoftirqd/112:03:481
23628991510,3cyclictest0-21swapper/110:50:211
2362899150,2cyclictest0-21swapper/109:45:491
2362899150,13cyclictest1523-21cat10:50:001
2362899150,13cyclictest0-21swapper/112:26:281
2362799150,13cyclictest13850-21ssh12:12:010
2363099142,7cyclictest0-21swapper/308:05:583
23630991411,2cyclictest6209-21diskmemload11:40:203
23630991411,2cyclictest0-21swapper/312:00:303
23630991411,2cyclictest0-21swapper/311:49:203
23630991411,2cyclictest0-21swapper/310:56:403
23630991411,2cyclictest0-21swapper/310:12:303
23630991411,2cyclictest0-21swapper/309:54:103
2363099141,11cyclictest6209-21diskmemload11:35:373
2363099140,3cyclictest20080-21sh09:23:283
2363099140,2cyclictest0-21swapper/312:33:503
2363099140,2cyclictest0-21swapper/311:56:203
2363099140,2cyclictest0-21swapper/311:56:203
2363099140,2cyclictest0-21swapper/311:04:103
2363099140,2cyclictest0-21swapper/309:30:073
2363099140,13cyclictest0-21swapper/310:51:103
2363099140,13cyclictest0-21swapper/308:10:003
2363099140,12cyclictest0-21swapper/310:02:143
2363099140,12cyclictest0-21swapper/308:20:143
23629991411,2cyclictest6500-21kworker/u8:109:19:052
2362999140,9cyclictest0-21swapper/209:34:042
23628991411,2cyclictest21-21rcuc/109:42:031
23628991411,2cyclictest21-21rcuc/109:19:421
23628991411,2cyclictest0-21swapper/111:27:181
23628991411,2cyclictest0-21swapper/111:16:281
23628991411,2cyclictest0-21swapper/111:06:181
2362899141,12cyclictest0-21swapper/109:27:491
2362899141,11cyclictest0-21swapper/109:55:201
2362899140,2cyclictest0-21swapper/110:00:391
2362899140,2cyclictest0-21swapper/107:55:131
2362899140,13cyclictest0-21swapper/112:22:081
2362899140,13cyclictest0-21swapper/112:08:081
2362899140,12cyclictest625-21cut08:50:241
2362899140,12cyclictest0-21swapper/109:50:151
2362899140,12cyclictest0-21swapper/108:20:261
23627991411,2cyclictest0-21swapper/012:27:310
23627991411,2cyclictest0-21swapper/011:21:110
23627991411,2cyclictest0-21swapper/010:08:120
2362799140,2cyclictest0-21swapper/011:09:210
2362799140,13cyclictest0-21swapper/012:36:310
2362799140,13cyclictest0-21swapper/010:47:110
2362799140,13cyclictest0-21swapper/010:02:520
2362799140,12cyclictest6269-21copy09:10:000
2362799140,12cyclictest4086-21ssh11:26:460
2362799140,12cyclictest2470-21munin-node0
2362799140,12cyclictest10013-21rm09:13:140
2362799140,12cyclictest0-21swapper/011:35:560
2362799140,12cyclictest0-21swapper/011:01:010
2362799140,12cyclictest0-21swapper/010:16:010
2362799140,12cyclictest0-21swapper/009:47:360
2362799140,12cyclictest0-21swapper/009:36:240
227022140,2sleep20-21swapper/211:46:532
315802130,2sleep00-21swapper/010:11:270
2363099138,3cyclictest13118-21ssh09:15:293
2363099132,6cyclictest0-21swapper/309:30:013
23630991310,2cyclictest0-21swapper/311:50:093
23630991310,2cyclictest0-21swapper/311:50:093
23630991310,2cyclictest0-21swapper/308:45:253
2363099130,2cyclictest0-21swapper/312:27:003
2363099130,2cyclictest0-21swapper/311:11:003
2363099130,2cyclictest0-21swapper/310:47:563
2363099130,12cyclictest0-21swapper/312:36:113
2363099130,12cyclictest0-21swapper/311:30:113
2363099130,12cyclictest0-21swapper/311:20:403
2363099130,12cyclictest0-21swapper/310:20:173
2363099130,12cyclictest0-21swapper/310:15:503
2363099130,11cyclictest6209-21diskmemload12:08:003
2363099130,11cyclictest6209-21diskmemload10:05:263
2363099130,11cyclictest5053-21sh09:41:553
2363099130,11cyclictest0-21swapper/311:08:273
2362999132,6cyclictest0-21swapper/208:25:002
2362999132,6cyclictest0-21swapper/207:45:002
23629991310,2cyclictest6884-21kworker/u8:107:11:052
23629991310,2cyclictest6500-21kworker/u8:109:13:042
23629991310,2cyclictest28357-21kworker/u8:211:07:042
23629991310,2cyclictest28357-21kworker/u8:210:55:042
23629991310,2cyclictest26887-21kworker/u8:010:09:052
2362999130,2cyclictest30299-21copy10:45:042
2362899138,3cyclictest16160-21perl10:30:101
2362899132,9cyclictest221ktimersoftd/107:30:291
2362899132,7cyclictest0-21swapper/108:50:001
2362899132,2cyclictest221ktimersoftd/108:35:111
23628991310,2cyclictest23-21ksoftirqd/109:37:091
23628991310,2cyclictest221ktimersoftd/112:31:481
23628991310,2cyclictest221ktimersoftd/108:15:191
23628991310,2cyclictest0-21swapper/112:37:081
23628991310,2cyclictest0-21swapper/112:18:581
23628991310,2cyclictest0-21swapper/110:15:141
23628991310,2cyclictest0-21swapper/109:10:091
2362899130,2cyclictest0-21swapper/108:30:261
2362899130,12cyclictest0-21swapper/112:12:281
2362899130,12cyclictest0-21swapper/111:57:491
2362899130,12cyclictest0-21swapper/111:57:491
2362899130,12cyclictest0-21swapper/110:23:391
2362899130,11cyclictest0-21swapper/111:35:351
2362799139,2cyclictest0-21swapper/007:30:220
23627991310,2cyclictest0-21swapper/012:06:210
23627991310,2cyclictest0-21swapper/012:01:440
23627991310,2cyclictest0-21swapper/011:57:110
23627991310,2cyclictest0-21swapper/011:57:110
23627991310,2cyclictest0-21swapper/011:45:530
23627991310,2cyclictest0-21swapper/011:32:520
23627991310,2cyclictest0-21swapper/010:56:110
23627991310,2cyclictest0-21swapper/010:37:060
23627991310,2cyclictest0-21swapper/008:40:140
23627991310,2cyclictest0-21swapper/008:40:140
2362799130,2cyclictest0-21swapper/011:53:550
2362799130,2cyclictest0-21swapper/011:53:550
2362799130,2cyclictest0-21swapper/011:10:570
2362799130,2cyclictest0-21swapper/009:29:420
2362799130,2cyclictest0-21swapper/009:23:130
2362799130,12cyclictest0-21swapper/012:31:410
2362799130,12cyclictest0-21swapper/010:51:560
2362799130,12cyclictest0-21swapper/010:32:320
2362799130,12cyclictest0-21swapper/010:27:070
2362799130,12cyclictest0-21swapper/010:22:320
2362799130,12cyclictest0-21swapper/009:30:150
180952130,2sleep018102-21ssh09:56:280
245532120,1sleep00-21swapper/007:10:190
2363099129,2cyclictest0-21swapper/308:30:203
2363099129,2cyclictest0-21swapper/307:35:113
2363099121,7cyclictest0-21swapper/308:02:593
2363099120,2cyclictest0-21swapper/310:31:133
2363099120,11cyclictest0-21swapper/307:20:213
2362999129,2cyclictest9979-21kworker/u8:108:02:042
2362999129,2cyclictest2030-21kworker/u8:210:21:052
2362999129,2cyclictest17291-21kworker/u8:011:15:052
2362999127,3cyclictest9785-21latency_hist07:55:002
2362999123,6cyclictest0-21swapper/212:15:592
2362999120,2cyclictest0-21swapper/212:27:042
2362999120,2cyclictest0-21swapper/212:14:402
2362999120,2cyclictest0-21swapper/207:39:042
2362999120,10cyclictest19203-21ssh11:41:562
2362899129,2cyclictest221ktimersoftd/111:49:481
2362899129,2cyclictest0-21swapper/111:41:081
2362899129,2cyclictest0-21swapper/110:39:491
2362899129,2cyclictest0-21swapper/109:33:091
2362899129,2cyclictest0-21swapper/108:40:281
2362899129,2cyclictest0-21swapper/108:40:281
2362899127,3cyclictest1971-21grep07:35:131
2362899127,3cyclictest16492-21munin-node1
2362899127,3cyclictest0-21swapper/111:00:171
2362899122,8cyclictest221ktimersoftd/107:25:171
2362899122,7cyclictest0-21swapper/107:20:011
2362899121,9cyclictest27652-21munin-run07:20:001
2362899121,6cyclictest0-21swapper/108:15:001
2362899121,6cyclictest0-21swapper/107:45:001
2362899121,2cyclictest221ktimersoftd/111:50:151
2362899121,2cyclictest221ktimersoftd/111:50:151
2362899121,2cyclictest221ktimersoftd/108:25:141
2362899120,6cyclictest0-21swapper/109:00:001
2362899120,11cyclictest0-21swapper/111:30:221
2362799129,2cyclictest0-21swapper/008:25:170
2362799129,2cyclictest0-21swapper/008:15:170
2362799129,2cyclictest0-21swapper/007:45:130
2362799122,8cyclictest0-21swapper/009:00:190
2362799120,2cyclictest0-21swapper/009:15:210
2362799120,2cyclictest0-21swapper/008:10:160
2362799120,11cyclictest0-21swapper/008:45:250
2362799120,11cyclictest0-21swapper/008:30:240
2362799120,11cyclictest0-21swapper/007:50:090
2362799120,11cyclictest0-21swapper/007:43:520
2362799120,11cyclictest0-21swapper/007:20:130
2363099111,5cyclictest0-21swapper/307:45:003
2363099110,10cyclictest0-21swapper/307:16:093
2362999118,2cyclictest28357-21kworker/u8:211:00:052
2362999110,8cyclictest0-21swapper/207:57:052
2362899118,2cyclictest21-21rcuc/108:05:231
2362899111,2cyclictest21-21rcuc/107:50:001
2362899110,9cyclictest0-21swapper/107:10:151
2362799118,2cyclictest0-21swapper/007:15:190
2362799111,7cyclictest4566-21metacity08:53:580
2362799110,10cyclictest0-21swapper/008:05:190
2363099105,3cyclictest31798-21sort07:30:003
2363099105,3cyclictest2748-21python08:55:273
2363099102,7cyclictest0-21swapper/307:59:593
2362999107,2cyclictest0-21swapper/208:30:182
2362999102,7cyclictest0-21swapper/208:44:022
2362999102,7cyclictest0-21swapper/208:44:022
2362999100,7cyclictest0-21swapper/209:02:012
2362999100,2cyclictest0-21swapper/209:20:022
2362899107,2cyclictest0-21swapper/107:50:161
2362899101,3cyclictest0-21swapper/109:00:111
2362899100,8cyclictest0-21swapper/108:00:141
2362799101,5cyclictest0-21swapper/008:05:000
2362799100,8cyclictest0-21swapper/007:55:180
2362799100,1cyclictest0-21swapper/008:23:090
236309994,3cyclictest8217-21fschecks_time07:50:183
236309994,3cyclictest28541-21fschecks_time08:40:163
236309994,3cyclictest28541-21fschecks_time08:40:163
236309994,3cyclictest24214-21df07:10:153
236309994,3cyclictest22570-21tr08:25:163
236309994,3cyclictest18105-21aten2.4_r3power08:15:123
236309991,3cyclictest27427-21grep08:35:253
236299994,3cyclictest4027-21fschecks_count07:40:162
236289994,3cyclictest6202-21cron09:10:001
236279990,8cyclictest0-21swapper/008:35:560
5831280,2sleep20-21swapper/209:05:152
236309985,2cyclictest0-21swapper/309:05:283
236309983,3cyclictest6205-21fschecks_time07:45:143
236309983,3cyclictest32438-21tune2fs07:30:193
236309980,6cyclictest3691-21chrt09:00:023
236279983,3cyclictest3745-21runrttasks07:36:050
236279981,6cyclictest0-21swapper/007:29:580
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional