You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-07 - 22:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #1, slot #6

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  Intel
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack1slot6s.osadl.org (updated Thu Dec 07, 2023 12:43:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2378121320,7sleep223782-21munin-node07:14:322
312292930,6sleep33066099cyclictest10:44:013
30660998440,38cyclictest20949-21sendmail_mailtr07:09:313
30660998440,38cyclictest20949-21sendmail_mailtr07:09:313
3066099843,12cyclictest0-21swapper/307:14:083
30660998424,9cyclictest17864-21gltestperf07:04:213
3066099823,12cyclictest0-21swapper/309:29:113
30660998224,9cyclictest1274-21runrttasks10:04:113
30660998215,18cyclictest0-21swapper/307:54:313
3066099813,31cyclictest0-21swapper/306:33:543
3066099813,31cyclictest0-21swapper/306:33:543
3066099813,12cyclictest0-21swapper/310:14:033
3065499813,70cyclictest17540-21aten_r1power_po08:09:120
3066099803,12cyclictest0-21swapper/307:24:103
30660998028,10cyclictest30133-21fschecks_count07:29:163
30660998025,8cyclictest13268-21cpuspeed_turbos09:04:093
3065499804,68cyclictest0-21swapper/005:44:180
3065499804,68cyclictest0-21swapper/005:44:180
3066099793,32cyclictest0-21swapper/308:29:163
3066099793,32cyclictest0-21swapper/308:29:163
3066099793,32cyclictest0-21swapper/307:03:533
30660997926,9cyclictest5562-21grep05:34:373
3065499792,60cyclictest13656-21fschecks_count09:04:130
3065499792,57cyclictest13754-21munin-node05:54:290
30660997830,10cyclictest18683-21seq09:14:143
30660997830,10cyclictest18683-21seq09:14:143
30660997821,10cyclictest4194-21needreboot08:44:223
30660997817,11cyclictest0-21swapper/307:34:083
3065499784,55cyclictest0-21swapper/007:49:100
3065499784,55cyclictest0-21swapper/007:49:100
3065499783,60cyclictest13666-21missed_timers07:59:240
3065499783,58cyclictest0-21swapper/010:14:230
3065499782,69cyclictest14902-21munin-node08:04:290
3066099774,12cyclictest0-21swapper/309:44:053
3066099773,32cyclictest0-21swapper/306:54:283
3065499774,66cyclictest0-21swapper/010:04:050
3065499774,66cyclictest0-21swapper/006:54:320
3065499774,65cyclictest0-21swapper/009:44:090
3065499774,55cyclictest0-21swapper/006:39:370
3065499773,67cyclictest15908-21mailstats06:59:350
3065499773,48cyclictest4562-21crond07:43:510
3066099764,33cyclictest0-21swapper/307:19:203
30660997628,9cyclictest22436-21sed09:19:303
30660997623,10cyclictest31181-21cpuspeed_turbos05:24:173
3065499765,62cyclictest0-21swapper/009:39:130
3065499764,55cyclictest0-21swapper/007:34:110
3065499764,54cyclictest0-21swapper/009:14:060
3065499764,54cyclictest0-21swapper/009:14:060
3065499763,65cyclictest19947-21munin-node08:14:120
3065499763,57cyclictest0-21swapper/007:32:550
3066099753,9cyclictest1274-21runrttasks08:53:093
30660997524,12cyclictest0-21swapper/305:54:273
30660997523,12cyclictest0-21swapper/309:39:253
30660997519,8cyclictest7330-21munin-node07:49:323
30660997519,8cyclictest7330-21munin-node07:49:323
3065499754,64cyclictest0-21swapper/008:34:080
3065499754,54cyclictest0-21swapper/005:59:160
3065499753,55cyclictest0-21swapper/009:28:150
3066099749,24cyclictest8813-21fschecks_time08:54:143
30660997424,10cyclictest25782-21fschecks_time08:24:153
30660997423,9cyclictest11764-21needreboot08:59:223
30660997423,10cyclictest23754-21fschecks_count09:24:143
3065499744,63cyclictest0-21swapper/009:49:040
3065499744,63cyclictest0-21swapper/009:19:170
3065499744,63cyclictest0-21swapper/007:44:190
3065499744,62cyclictest0-21swapper/010:24:090
3065499744,53cyclictest0-21swapper/008:58:480
3065499743,64cyclictest0-21swapper/008:44:080
3065499742,64cyclictest10429-21sshd09:59:440
3066099733,29cyclictest17165-21fschecks_time05:59:243
3066099733,26cyclictest0-21swapper/310:33:243
3065499735,60cyclictest0-21swapper/008:20:140
3065499734,62cyclictest0-21swapper/008:29:240
3065499734,62cyclictest0-21swapper/008:29:240
3065499734,62cyclictest0-21swapper/006:09:360
3065499734,61cyclictest0-21swapper/006:14:240
30654997336,32cyclictest29862-21grep06:24:190
3065499733,64cyclictest26507-21taskset07:21:040
3065499733,62cyclictest28765-21fschecks_count09:34:100
3065499733,52cyclictest0-21swapper/007:09:110
3065499733,52cyclictest0-21swapper/007:09:110
30654997323,44cyclictest1222-21snmpd06:19:390
3065499732,65cyclictest31276-21tail10:44:020
30654997318,37cyclictest0-21swapper/008:51:270
30654997312,39cyclictest7150irq/45-ahci06:04:270
3066099724,30cyclictest0-21swapper/306:48:533
3066099723,30cyclictest0-21swapper/307:59:293
3065499723,61cyclictest1585-21munin-node06:34:320
3065499723,55cyclictest0-21swapper/008:28:050
3066099714,28cyclictest1274-21runrttasks06:43:013
3066099713,30cyclictest0-21swapper/306:13:553
30660997126,9cyclictest26529-21aten_r1power_po06:19:133
3065499718,57cyclictest18602-21aten_r1power_cu10:19:040
3065499714,61cyclictest0-21swapper/006:29:360
3065499714,61cyclictest0-21swapper/006:29:360
3065499714,60cyclictest0-21swapper/010:34:060
3065499714,60cyclictest0-21swapper/010:34:060
3065499714,60cyclictest0-21swapper/009:54:050
3065499714,60cyclictest0-21swapper/009:54:050
3065499714,60cyclictest0-21swapper/008:59:320
3065499714,60cyclictest0-21swapper/006:49:250
3065499713,61cyclictest30176-21processes10:39:220
30654997125,40cyclictest1274-21runrttasks08:41:230
30654997120,45cyclictest6665-21df_abs05:39:180
30660997024,9cyclictest24146-21sort08:19:303
3065499704,59cyclictest0-21swapper/006:47:200
3065499703,60cyclictest0-21swapper/005:24:250
3065499703,51cyclictest0-21swapper/007:27:480
3065499703,36cyclictest0-21swapper/007:14:100
30654997014,39cyclictest9852-21munin-node07:54:290
3065499694,53cyclictest0-21swapper/009:29:330
3065499693,60cyclictest13248-21chrt05:50:340
3065499693,38cyclictest0-21swapper/010:09:060
3066099684,15cyclictest0-21swapper/308:34:163
30660996823,9cyclictest23863-21aten_r1power_cu06:14:133
30660996822,14cyclictest0-21swapper/309:34:053
3065499683,59cyclictest0-21swapper/005:33:570
3065499683,48cyclictest19130-21latency_hist07:08:530
30654996814,37cyclictest0-21swapper/005:34:150
30657996727,32cyclictest30247-21sendmail_mailqu10:39:231
3065499674,56cyclictest0-21swapper/010:49:020
30654996722,39cyclictest1274-21runrttasks09:11:220
30654996713,48cyclictest0-21swapper/010:29:100
30660996619,8cyclictest14361-21kworker/3:006:04:143
30660996616,8cyclictest3013-21unixbench_singl05:29:423
30660996513,10cyclictest4112-21unixbench-2d07:39:373
3065999645,53cyclictest0-21swapper/207:29:232
30659996418,23cyclictest4555-21interrupts09:49:152
3065999634,53cyclictest0-21swapper/207:03:532
3065999634,50cyclictest0-21swapper/207:13:522
3065999634,50cyclictest0-21swapper/207:13:522
30657996310,22cyclictest0-21swapper/106:53:541
3065499635,49cyclictest30626-21crond05:23:560
30660996220,7cyclictest21232-21kworker/3:110:49:023
3065999624,51cyclictest0-21swapper/205:38:562
30660996118,7cyclictest18588-21iostat_ios08:09:243
3066099604,15cyclictest0-21swapper/306:49:173
3065999604,38cyclictest0-21swapper/210:33:252
3065999603,51cyclictest0-21swapper/206:44:212
3065999603,49cyclictest24446-21munin-node07:19:382
243699600,18rtkit-daemon2435-21rtkit-daemon09:11:562
3065999594,50cyclictest0-21swapper/210:24:252
3065999594,49cyclictest0-21swapper/208:24:182
3065999593,50cyclictest0-21swapper/209:34:122
30657995919,32cyclictest14446-21fschecks_count05:54:201
3065999588,21cyclictest0-21swapper/210:43:452
3065999587,30cyclictest0-21swapper/210:34:032
3065999587,30cyclictest0-21swapper/210:34:032
3065999584,34cyclictest0-21swapper/205:24:352
3065999583,49cyclictest0-21swapper/207:39:252
3065999583,49cyclictest0-21swapper/206:34:232
3065999583,48cyclictest0-21swapper/209:17:252
3065999583,48cyclictest0-21swapper/209:17:252
3065999574,47cyclictest0-21swapper/206:49:332
30657995716,21cyclictest792-21cut09:44:041
67672565,12sleep10-21swapper/105:39:201
3065999568,28cyclictest0-21swapper/209:19:072
3065999564,45cyclictest0-21swapper/205:44:152
3065999564,45cyclictest0-21swapper/205:44:152
30659995638,11cyclictest3217-21grep08:44:102
3065999563,47cyclictest0-21swapper/206:09:242
3065999563,45cyclictest0-21swapper/206:14:142
3065999555,30cyclictest0-21swapper/208:04:282
3065999554,44cyclictest0-21swapper/209:59:042
3065999554,31cyclictest0-21swapper/205:54:162
3065999554,25cyclictest27393-21unixbench-2d09:29:302
3065999553,45cyclictest13787-21chrt07:59:272
3065999553,45cyclictest0-21swapper/208:39:092
3065999553,43cyclictest0-21swapper/205:39:072
30659995520,11cyclictest0-21swapper/208:54:302
30659995515,21cyclictest0-21swapper/206:04:152
3065799555,31cyclictest0-21swapper/107:29:101
30657995512,14cyclictest0-21swapper/106:23:541
3065999545,29cyclictest0-21swapper/207:44:282
3065999544,30cyclictest0-21swapper/206:54:402
30659995436,11cyclictest4588-21grep06:39:172
3065999543,45cyclictest0-21swapper/209:39:092
3065999543,45cyclictest0-21swapper/209:24:072
30659995411,21cyclictest0-21swapper/205:28:572
3065799546,23cyclictest0-21swapper/110:28:451
3065799545,40cyclictest0-21swapper/108:54:201
3065799544,41cyclictest0-21swapper/108:24:311
3065799544,30cyclictest0-21swapper/107:14:231
30657995436,12cyclictest15671-21needreboot06:59:301
3065799543,45cyclictest0-21swapper/109:29:111
3065799543,45cyclictest0-21swapper/107:49:141
3065799543,45cyclictest0-21swapper/107:49:141
3065799543,45cyclictest0-21swapper/106:44:541
3065799543,45cyclictest0-21swapper/106:24:331
3065799543,45cyclictest0-21swapper/105:59:281
30657995419,29cyclictest27988-21unixbench_multi10:34:261
30657995419,29cyclictest27988-21unixbench_multi10:34:261
3065999535,39cyclictest0-21swapper/209:54:042
3065999535,39cyclictest0-21swapper/209:54:042
3065999535,38cyclictest1274-21runrttasks06:20:202
3065999534,43cyclictest0-21swapper/210:44:222
3065999534,40cyclictest0-21swapper/210:19:122
3065999534,38cyclictest1644-21taskset07:35:172
3065999534,30cyclictest0-21swapper/208:09:312
3065999534,30cyclictest0-21swapper/207:54:272
3065999534,27cyclictest0-21swapper/207:49:332
3065999534,27cyclictest0-21swapper/207:49:332
3065999534,24cyclictest0-21swapper/208:34:082
3065999534,24cyclictest0-21swapper/207:24:162
3065999533,44cyclictest0-21swapper/209:04:142
3065799534,30cyclictest0-21swapper/108:04:171
3065799534,27cyclictest0-21swapper/105:38:571
30657995334,12cyclictest25530-21unixbench_singl10:29:271
30657995334,11cyclictest13674-21grep06:54:381
3065799533,44cyclictest0-21swapper/110:49:241
3065799533,44cyclictest0-21swapper/109:59:161
3065799533,44cyclictest0-21swapper/109:24:251
3065799533,44cyclictest0-21swapper/109:09:111
3065799533,44cyclictest0-21swapper/107:59:261
3065799533,44cyclictest0-21swapper/107:24:121
3065799533,44cyclictest0-21swapper/106:39:141
3065999524,26cyclictest0-21swapper/210:14:242
3065999523,12cyclictest0-21swapper/208:59:262
30659995218,12cyclictest0-21swapper/208:14:332
30659995217,12cyclictest0-21swapper/210:13:462
30659995215,11cyclictest1789-21grep09:44:132
3065799524,41cyclictest0-21swapper/108:19:321
3065799524,29cyclictest0-21swapper/110:19:131
3065799524,29cyclictest0-21swapper/110:04:201
3065799524,28cyclictest0-21swapper/110:09:251
3065799524,28cyclictest0-21swapper/107:07:511
30657995235,11cyclictest32337-21cpuspeed_turbos07:34:111
30657995234,10cyclictest9692-21tail05:44:231
30657995234,10cyclictest9692-21tail05:44:231
3065799523,43cyclictest26396-21yum07:19:381
3065799523,43cyclictest0-21swapper/110:14:031
3065799523,43cyclictest0-21swapper/109:39:061
3065799523,43cyclictest0-21swapper/108:44:071
3065799523,43cyclictest0-21swapper/107:54:121
3065799523,43cyclictest0-21swapper/106:34:171
3065799523,43cyclictest0-21swapper/106:09:241
3065799523,43cyclictest0-21swapper/105:24:321
243699520,5rtkit-daemon2435-21rtkit-daemon09:58:463
243699520,5rtkit-daemon2435-21rtkit-daemon09:58:463
3065999514,39cyclictest0-21swapper/207:04:202
3065999514,39cyclictest0-21swapper/205:59:172
3065999514,29cyclictest0-21swapper/208:29:222
3065999514,29cyclictest0-21swapper/208:29:222
3065999514,29cyclictest0-21swapper/206:24:322
3065999514,24cyclictest0-21swapper/206:29:122
3065999514,24cyclictest0-21swapper/206:29:122
3065999513,25cyclictest0-21swapper/208:19:192
3065999513,24cyclictest0-21swapper/205:49:272
3065799515,39cyclictest0-21swapper/108:35:271
3065799515,15cyclictest0-21swapper/107:39:271
3065799514,40cyclictest0-21swapper/108:59:251
3065799514,38cyclictest0-21swapper/106:14:311
3065799514,29cyclictest0-21swapper/109:04:151
3065799514,28cyclictest0-21swapper/106:04:291
3065799514,26cyclictest0-21swapper/108:53:491
3065799514,26cyclictest0-21swapper/108:09:121
30657995134,10cyclictest32208-21cut06:29:191
30657995134,10cyclictest32208-21cut06:29:191
3065799513,44cyclictest0-21swapper/110:44:031
30657995118,13cyclictest0-21swapper/109:49:151
30660995034,10cyclictest15920-21gltestperf08:04:173
30659995031,11cyclictest7224-21unixbench_multi08:49:322
3065999503,39cyclictest30630-21latency_hist05:23:572
3065799504,39cyclictest0-21swapper/108:40:471
3065799504,28cyclictest0-21swapper/107:48:221
3065799504,27cyclictest0-21swapper/107:09:121
3065799504,27cyclictest0-21swapper/107:09:121
3065799503,41cyclictest0-21swapper/109:54:171
3065799503,41cyclictest0-21swapper/109:54:171
3065799503,41cyclictest0-21swapper/109:19:191
30660994932,10cyclictest9044-21tr05:44:163
30660994932,10cyclictest9044-21tr05:44:163
3065999494,38cyclictest0-21swapper/210:51:222
3065799494,30cyclictest0-21swapper/105:29:361
3065799493,40cyclictest0-21swapper/108:29:191
3065799493,40cyclictest0-21swapper/108:29:191
3065799484,37cyclictest0-21swapper/105:23:571
3065799483,40cyclictest0-21swapper/105:49:231
3065799474,25cyclictest0-21swapper/108:15:291
243699470,5rtkit-daemon2435-21rtkit-daemon10:09:463
30660994610,13cyclictest0-21swapper/305:39:293
3065999464,27cyclictest0-21swapper/210:04:172
3065799464,26cyclictest0-21swapper/109:14:251
3065799464,26cyclictest0-21swapper/109:14:251
3066099453,34cyclictest0-21swapper/309:49:063
30660994528,10cyclictest12290-21hwlatdetect05:49:243
3065799453,37cyclictest0-21swapper/109:34:071
243699450,4rtkit-daemon2435-21rtkit-daemon10:19:313
30660994430,8cyclictest22136-21tail10:24:143
30660994419,8cyclictest34-21rcuc/310:39:233
30660994312,18cyclictest0-21swapper/306:34:203
3066099423,32cyclictest0-21swapper/309:09:103
30660994226,10cyclictest2195-21grep08:39:333
3066099415,21cyclictest30083-21iostat_ios06:24:233
3066099414,9cyclictest10263-21grep09:59:263
30660994121,8cyclictest26162-21munin-node10:34:293
30660994121,8cyclictest26162-21munin-node10:34:293
30660994025,9cyclictest5478-21fschecks_count07:44:183
3066099393,29cyclictest0-21swapper/305:23:573
3066099383,20cyclictest0-21swapper/308:14:283
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional