You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-28 - 20:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot7s.osadl.org (updated Thu Sep 28, 2023 12:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3830210490,9sleep10-21swapper/107:06:111
4014210390,9sleep30-21swapper/307:08:333
385128365,13sleep20-21swapper/207:06:282
199928168,9sleep00-21swapper/007:05:100
86742550,1sleep213876-21kworker/2:211:45:012
4204992925,3cyclictest94-21kswapd009:54:593
420199270,24cyclictest0-21swapper/010:59:260
420199260,1cyclictest0-21swapper/010:27:420
420199259,8cyclictest1416-21munin-run09:39:590
4203992220,1cyclictest18263-21grep11:50:252
420399220,1cyclictest0-21swapper/208:10:242
420299220,22cyclictest0-21swapper/110:48:261
420299220,22cyclictest0-21swapper/110:48:261
420199220,1cyclictest0-21swapper/009:33:140
420199220,1cyclictest0-21swapper/009:26:450
420199220,1cyclictest0-21swapper/009:15:050
420499202,10cyclictest224-21systemd-journal07:41:273
420499200,15cyclictest0-21swapper/311:19:073
4202992017,2cyclictest10334-21aten2_r7power_v10:40:231
420199200,2cyclictest0-21swapper/009:12:110
420199200,1cyclictest0-21swapper/009:23:170
420499190,4cyclictest0-21swapper/311:48:193
420499190,2cyclictest0-21swapper/310:27:413
420499190,18cyclictest0-21swapper/309:45:083
420399195,10cyclictest0-21swapper/208:35:212
420199191,13cyclictest0-21swapper/011:50:200
420499181,2cyclictest0-21swapper/312:01:113
420399182,1cyclictest11099-21ssh09:47:142
4203991813,2cyclictest0-21swapper/212:15:242
4202991816,1cyclictest29214-21sh09:10:001
4202991815,2cyclictest5946-21python10:35:361
420499171,0cyclictest0-21swapper/310:23:433
4203991715,1cyclictest10115-21python11:25:342
420499160,12cyclictest0-21swapper/311:53:223
420399161,15cyclictest0-21swapper/210:10:232
420399161,14cyclictest29217-21diskmemload12:27:202
420399160,15cyclictest30885-21/usr/sbin/munin08:05:362
420399160,15cyclictest0-21swapper/210:45:092
420399160,15cyclictest0-21swapper/210:45:082
420199160,15cyclictest0-21swapper/010:54:520
420499152,5cyclictest0-21swapper/310:55:333
4204991514,1cyclictest0-21swapper/311:26:313
4204991511,2cyclictest12849-21wc07:25:273
420499150,12cyclictest0-21swapper/311:37:393
420399151,13cyclictest18169-21cpu11:10:212
420399150,12cyclictest0-21swapper/210:15:002
420299150,2cyclictest0-21swapper/112:06:031
420199150,15cyclictest0-21swapper/010:09:300
420199150,14cyclictest0-21swapper/011:20:220
420199150,14cyclictest0-21swapper/010:32:430
420499147,4cyclictest4153-21sed09:15:213
420499145,6cyclictest11820-21munin-run10:15:013
4204991411,2cyclictest28429-21wc12:35:353
4204991411,2cyclictest1334-21grep11:40:253
4204991410,4cyclictest29217-21diskmemload12:19:383
4204991410,3cyclictest26515-21grep10:00:243
4204991410,2cyclictest4841-21tr11:00:203
420499140,12cyclictest0-21swapper/309:10:133
420399140,2cyclictest0-21swapper/210:55:212
420399140,1cyclictest29217-21diskmemload09:31:392
420399140,14cyclictest5138-21kworker/2:111:38:292
420399140,13cyclictest0-21swapper/209:10:272
420299141,9cyclictest0-21swapper/108:35:211
4202991410,3cyclictest1686-21idleruntime-cro12:40:001
420299140,1cyclictest0-21swapper/112:15:331
420199148,4cyclictest27126-21runrttasks09:05:230
420199142,11cyclictest1960-21rm09:40:230
4201991410,2cyclictest1586-21ssh12:20:010
420199140,1cyclictest0-21swapper/011:36:140
303442140,0chrt0-21swapper/109:35:441
420499139,3cyclictest29991-21wc09:35:313
420499139,3cyclictest26447-21cut11:55:253
420499139,3cyclictest19494-21cut12:10:363
420499139,3cyclictest14859-21cron08:40:013
420499138,2cyclictest11341-21taskset07:22:343
420499137,2cyclictest224-21systemd-journal09:05:013
420499136,4cyclictest14461-21sh10:15:513
420499133,7cyclictest32654-21ssh11:20:163
420499133,1cyclictest0-21swapper/307:50:213
420499132,2cyclictest224-21systemd-journal10:55:003
4204991312,1cyclictest0-21swapper/312:34:553
4204991310,2cyclictest19627-21wc11:10:343
4204991310,2cyclictest11495-21grep11:05:213
4204991310,2cyclictest10932-21head09:20:313
4204991310,2cyclictest10535-21tr12:25:213
420499130,9cyclictest16614-21sh11:30:203
420399139,3cyclictest29961-21wc08:00:352
420399139,3cyclictest12101-21sed08:30:352
420399138,2cyclictest8428-21ssh11:02:372
420399134,5cyclictest24459-21munin-run09:00:002
420399134,5cyclictest24459-21munin-run09:00:002
4203991310,2cyclictest5161-21grep10:35:272
4203991310,2cyclictest32113-21perf11:20:012
4203991310,2cyclictest28368-21ssh09:35:202
4203991310,2cyclictest27884-21tr12:35:292
4203991310,2cyclictest17271-21cut09:25:272
4203991310,2cyclictest13931-21sh07:30:002
4203991310,2cyclictest12690-21sed11:05:312
420399130,2cyclictest29217-21diskmemload09:22:482
420399130,1cyclictest0-21swapper/211:30:242
420299139,2cyclictest32731-21wc11:40:001
420299137,1cyclictest9974-21grep09:20:231
4202991310,2cyclictest30163-21cat09:10:261
4202991310,2cyclictest24566-21grep10:50:291
4201991310,2cyclictest26904-21grep07:55:240
4201991310,2cyclictest2565-21grep08:10:370
4201991310,2cyclictest13-21ksoftirqd/012:00:250
420499129,2cyclictest5931-21wc07:10:333
420499129,2cyclictest22344-21cstates09:30:213
420499129,2cyclictest11137-21grep12:05:313
420499128,3cyclictest22565-21grep07:45:293
420499124,4cyclictest224-21systemd-journal08:04:593
420499120,11cyclictest17067-21head10:45:253
420499120,11cyclictest17067-21head10:45:243
420499120,11cyclictest0-21swapper/309:56:143
420399129,2cyclictest8683-21cut08:25:532
420399129,2cyclictest7741-21sh08:25:002
420399129,2cyclictest16489-21wc08:40:342
420399128,3cyclictest4903-21grep08:15:402
420399128,3cyclictest4314-21cut09:15:222
420399128,3cyclictest31877-21cat10:30:282
420399128,3cyclictest23360-21taskset07:49:172
420399128,3cyclictest18438-21grep10:20:212
420399127,4cyclictest17380-21ssh09:52:082
420399123,9cyclictest29028-21kworker/2:011:24:372
420399121,9cyclictest29217-21diskmemload11:55:142
4203991212,0cyclictest0-21swapper/210:03:052
420299129,2cyclictest8460-21ssh12:24:051
420299129,2cyclictest5380-21head11:00:221
420299129,2cyclictest31780-21awk10:55:351
420299129,2cyclictest31695-21grep08:05:271
420299129,2cyclictest26434-21wc11:15:321
420299129,2cyclictest25599-21ssh10:25:311
420299129,2cyclictest20403-21grep12:30:401
420299129,2cyclictest19429-21sed11:50:341
420299129,2cyclictest19261-21head08:45:371
420299129,2cyclictest16772-21grep09:25:231
420299129,2cyclictest16638-21tr07:35:211
420299129,2cyclictest13920-21ls07:30:001
420299129,2cyclictest11974-21ssh11:05:241
420299126,2cyclictest224-21systemd-journal09:20:011
420299122,7cyclictest10422-21tr11:45:291
4202991210,1cyclictest11877-21python12:25:331
420299120,1cyclictest0-21swapper/112:13:331
420299120,1cyclictest0-21swapper/109:42:581
420199129,2cyclictest9666-21grep08:26:020
420199129,2cyclictest9642-21sed09:45:360
420199129,2cyclictest31891-21ssh11:19:510
420199129,2cyclictest19800-21grep11:10:340
420199129,2cyclictest17806-21tr08:45:240
420199129,2cyclictest15407-21tr08:40:240
420199129,2cyclictest13-21ksoftirqd/011:55:020
420199129,2cyclictest11116-21grep08:30:230
420199128,3cyclictest9610-21sh11:03:470
420199128,2cyclictest21608-21grep08:50:340
420199128,2cyclictest21047-21ssh10:21:320
420199127,3cyclictest25471-21grep07:50:390
420199124,2cyclictest224-21systemd-journal10:05:000
420199120,11cyclictest22565-21sh10:50:000
420199120,11cyclictest22565-21sh10:50:000
420499118,2cyclictest9478-21sort08:26:023
420499118,2cyclictest8911-21seq07:17:083
420499118,2cyclictest5149-21grep10:35:283
420499118,2cyclictest4585-21ssh09:41:313
420499118,2cyclictest32468-21sort08:05:363
420499118,2cyclictest30579-21ssh10:30:083
420499118,2cyclictest27688-21wc07:55:353
420499118,2cyclictest22247-21taskset08:55:143
420499118,2cyclictest22247-21taskset08:55:143
420499118,2cyclictest2096-21ssh10:06:123
420499118,2cyclictest1928-21ssh12:20:143
420499118,2cyclictest17620-21cut08:45:243
420499118,2cyclictest16787-21sed09:25:233
420499118,2cyclictest1540-21cut08:10:283
420499118,2cyclictest12205-21chrt08:31:273
420499118,2cyclictest12045-21grep10:40:353
420499114,6cyclictest500-21in:imuxsock08:20:003
420499112,5cyclictest5716-21aten2_r7power_e08:20:243
4204991110,1cyclictest0-21swapper/308:42:513
420399118,2cyclictest8707-21ssh11:45:012
420399118,2cyclictest3178-21ssh09:40:322
420399118,2cyclictest27341-21tr09:05:232
420399118,2cyclictest25227-21ssh10:50:382
420399118,2cyclictest20258-21ssh09:55:062
420399118,2cyclictest19537-21cat12:30:322
420399118,2cyclictest19437-21taskset08:47:362
420399118,2cyclictest14955-21cut07:30:282
420399118,2cyclictest13664-21ssh12:07:082
420399118,2cyclictest11436-21ntpq10:40:292
420399116,4cyclictest2501-21expr12:20:202
420399114,6cyclictest0-21swapper/207:50:212
420399110,10cyclictest26029-21ssh10:25:372
420399110,0cyclictest0-21swapper/210:06:082
420299119,1cyclictest17387-21cut11:30:271
420299118,2cyclictest9488-21ssh11:25:271
420299118,2cyclictest6640-21taskset07:13:331
420299118,2cyclictest3881-21ssh11:41:171
420299118,2cyclictest32458-21cat10:30:351
420299118,2cyclictest31980-21ssh10:05:171
420299118,2cyclictest3022-21ssh11:21:081
420299118,2cyclictest27488-21cat11:55:351
420299118,2cyclictest25087-21grep07:50:351
420299118,2cyclictest23917-21wc08:55:361
420299118,2cyclictest23917-21wc08:55:351
420299118,2cyclictest21717-21taskset08:51:231
420299118,2cyclictest20460-21apache_volume09:55:181
420299118,2cyclictest16490-21ssh10:18:251
420299118,2cyclictest14184-21chrt09:50:231
420299118,2cyclictest11965-21ssh09:48:221
420299118,2cyclictest11555-21sleep07:25:001
420299118,2cyclictest11421-21cat08:30:271
420299118,2cyclictest1111-21ssh12:00:011
420299117,2cyclictest7311-21grep07:15:231
420299117,1cyclictest0-21swapper/110:20:181
420299115,5cyclictest13337-21munin-node10:02:091
420299113,7cyclictest0-21swapper/109:30:271
420199119,1cyclictest20065-21cut12:30:370
420199119,1cyclictest164802sleep011:30:160
420199119,1cyclictest11012-21head10:40:260
420199118,2cyclictest8723-21ssh11:45:020
420199118,2cyclictest5297-21sort07:10:260
420199118,2cyclictest17803-21ssh12:10:180
420199118,2cyclictest15158-21ssh10:16:490
420199118,2cyclictest14887-21tail09:50:260
420199118,2cyclictest13931-21ntpq08:35:310
420199118,2cyclictest13389-21ssh12:06:550
420199118,2cyclictest12584-21ssh12:25:510
420199118,2cyclictest11823-21awk10:15:000
420199118,2cyclictest1153-21wget11:40:230
420199118,2cyclictest11108-21ssh11:05:070
420199118,2cyclictest10529-21sh11:25:370
420199112,8cyclictest26866-21perf09:05:000
420199112,6cyclictest4889-21df10:35:250
420499108,1cyclictest27582-21expr09:05:243
420499107,2cyclictest21598-21sed08:50:353
420499107,2cyclictest17891-21fgrep07:35:293
420499105,2cyclictest14156-21taskset07:30:133
420399108,1cyclictest26658-21chrt09:02:482
420399108,1cyclictest19436-21expr07:40:222
420399108,1cyclictest1175-21ssh12:00:042
420399107,2cyclictest27762-21grep07:55:342
420399107,2cyclictest22003-21cron08:55:012
420399107,2cyclictest17916-21apache_accesses12:10:222
420399100,9cyclictest10587-21ntpq07:20:322
420399100,2cyclictest0-21swapper/207:10:232
420299108,2cyclictest8247-21awk08:25:511
420299108,1cyclictest6652-21ssh10:10:251
420299108,1cyclictest6589-21tr08:20:291
420299108,1cyclictest30082-21wc08:00:361
420299108,1cyclictest283802sleep108:00:001
420299108,1cyclictest26419-21grep09:00:561
420299108,1cyclictest21683-21expr07:45:231
420299108,1cyclictest17061-21taskset08:42:351
420299107,2cyclictest3316-21grep08:15:251
420299107,2cyclictest2560-21sed08:10:371
420299107,2cyclictest14478-21expr07:30:251
420299100,9cyclictest18354-21cat11:10:221
420199108,1cyclictest7383-21expr07:15:230
420199108,1cyclictest4007-21tail08:15:310
420199108,1cyclictest27211-21ssh12:35:220
420199108,1cyclictest26681-21sed12:15:220
420199108,1cyclictest20705-21cat09:55:210
420199107,2cyclictest32018-21grep08:05:300
420199107,2cyclictest29692-21ntpq08:00:310
420199107,2cyclictest23399-21cut08:55:300
420199107,2cyclictest23399-21cut08:55:290
420199107,2cyclictest19275-21wc07:40:210
420199107,2cyclictest17167-21cut07:35:240
420199107,2cyclictest12692-21if_enp2s007:25:250
420199106,2cyclictest9552-21grep07:20:260
420199103,5cyclictest21333-21apache_volume07:45:200
420199100,9cyclictest15948-21chrt07:30:430
42039998,1cyclictest17703-21ntp_kernel_pll_07:35:272
42039996,2cyclictest7674-21uniq07:15:262
42029998,1cyclictest209892sleep107:44:171
42019997,1cyclictest6279-21grep08:20:260
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional