You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-26 - 02:20

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Thu Apr 25, 2024 12:44:32)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
89612241164,25sleep30-21swapper/307:09:133
87142241165,24sleep10-21swapper/107:06:011
88322238165,59sleep00-21swapper/007:07:340
87102233163,57sleep20-21swapper/207:05:582
2377221640,4sleep3912799cyclictest08:55:023
138532770,2sleep20-21swapper/211:00:012
256532690,3sleep10-21swapper/111:25:171
167822620,4sleep230-21rcuc/212:20:022
187052570,2sleep20-21swapper/211:10:132
20902560,2sleep30-21swapper/310:30:243
911799441,13cyclictest0-21swapper/110:10:311
9110994210,15cyclictest8812-21munin-run09:35:010
9110994210,15cyclictest8812-21munin-run09:35:010
911799411,10cyclictest0-21swapper/108:10:051
9127994016,8cyclictest0-21swapper/310:20:513
911799401,8cyclictest0-21swapper/111:09:471
9127993926,8cyclictest0-21swapper/308:47:073
9127993918,11cyclictest0-21swapper/311:18:033
911799391,13cyclictest0-21swapper/110:39:391
911799391,12cyclictest0-21swapper/112:18:171
9127993823,9cyclictest0-21swapper/308:14:403
9127993815,3cyclictest0-21swapper/310:10:113
911799381,29cyclictest0-21swapper/110:16:271
911799381,14cyclictest3680-21/usr/sbin/munin11:50:131
911799381,14cyclictest0-21swapper/112:33:271
9127993720,6cyclictest0-21swapper/307:31:353
9127993713,7cyclictest0-21swapper/311:33:593
911799371,14cyclictest0-21swapper/109:50:111
911799371,12cyclictest0-21swapper/109:17:351
911799371,11cyclictest0-21swapper/111:43:131
911799371,11cyclictest0-21swapper/111:43:131
9127993623,3cyclictest0-21swapper/309:54:563
9127993620,6cyclictest0-21swapper/307:27:313
911799361,9cyclictest0-21swapper/110:02:031
911799361,15cyclictest0-21swapper/108:37:051
911799361,14cyclictest0-21swapper/111:59:031
911799361,11cyclictest0-21swapper/112:29:511
911099367,13cyclictest29227-21munin-run11:35:010
9127993524,3cyclictest0-21swapper/312:18:533
911799351,27cyclictest0-21swapper/110:59:211
911799351,14cyclictest0-21swapper/111:45:251
911799351,13cyclictest0-21swapper/112:20:511
911799351,10cyclictest0-21swapper/111:14:571
911799350,11cyclictest0-21swapper/107:39:391
9127993416,3cyclictest0-21swapper/307:11:453
912799341,20cyclictest0-21swapper/311:46:093
911799341,15cyclictest0-21swapper/109:32:551
911799341,15cyclictest0-21swapper/109:32:551
911799340,16cyclictest0-21swapper/111:39:091
9127993316,10cyclictest0-21swapper/307:48:243
911799331,12cyclictest0-21swapper/108:04:591
67652330,4sleep2311ktimersoftd/209:30:012
912799323,8cyclictest0-21swapper/312:20:393
912799321,30cyclictest0-21swapper/311:21:433
911799321,25cyclictest0-21swapper/111:17:331
911799321,15cyclictest22672-21ntp_states08:50:231
911799321,10cyclictest0-21swapper/112:06:071
911099325,13cyclictest3795-21munin-run08:10:000
912799319,9cyclictest12338-21sed12:10:013
912799315,3cyclictest0-21swapper/309:22:253
9127993117,8cyclictest0-21swapper/311:38:233
911799311,6cyclictest0-21swapper/112:00:351
911799311,24cyclictest0-21swapper/110:40:351
911799311,24cyclictest0-21swapper/110:40:351
911799311,11cyclictest0-21swapper/109:57:531
911799311,10cyclictest0-21swapper/109:44:191
911799311,10cyclictest0-21swapper/109:07:591
911799310,19cyclictest0-21swapper/107:15:591
911099314,13cyclictest14460-21munin-run12:15:010
9127993019,3cyclictest0-21swapper/309:36:513
9127993018,3cyclictest0-21swapper/310:06:353
9127993017,3cyclictest0-21swapper/312:35:253
9127993016,9cyclictest0-21swapper/308:16:123
912799301,14cyclictest0-21swapper/308:55:193
912799301,13cyclictest0-21swapper/310:26:473
911799301,20cyclictest0-21swapper/110:29:251
911799301,12cyclictest0-21swapper/110:50:271
911799301,12cyclictest0-21swapper/109:23:111
911799301,10cyclictest0-21swapper/112:12:471
911799300,12cyclictest0-21swapper/107:47:551
911799300,10cyclictest0-21swapper/109:01:351
911099301,13cyclictest7851-21munin-run12:00:010
912799298,3cyclictest30180-21diskmemload12:13:353
9127992917,3cyclictest0-21swapper/309:04:493
9127992917,3cyclictest0-21swapper/307:16:273
9127992912,9cyclictest0-21swapper/308:07:303
9127992910,7cyclictest1393-21ps09:15:233
912299292,9cyclictest0-21swapper/210:20:012
911799291,14cyclictest0-21swapper/107:29:011
911799291,13cyclictest0-21swapper/112:35:311
911799291,13cyclictest0-21swapper/107:21:511
911799291,13cyclictest0-21swapper/107:21:511
911799290,11cyclictest0-21swapper/109:35:271
9110992910,15cyclictest15410-21munin-run09:50:010
911099290,13cyclictest13840-21idleruntime-cro11:00:020
912799289,8cyclictest0-21swapper/312:25:233
912799289,3cyclictest0-21swapper/311:02:433
912799281,8cyclictest0-21swapper/309:47:503
9127992810,3cyclictest18993-21runrttasks08:42:433
9127992810,3cyclictest0-21swapper/307:25:003
9127992810,3cyclictest0-21swapper/307:24:593
911799281,14cyclictest0-21swapper/107:11:221
911799281,12cyclictest0-21swapper/109:27:031
911799281,10cyclictest0-21swapper/110:23:111
911799280,11cyclictest0-21swapper/111:34:231
911799280,11cyclictest0-21swapper/107:31:491
911099281,12cyclictest18247-21perf11:10:010
242312281,3sleep00-21swapper/011:20:280
912799278,3cyclictest5472-21kworker/u8:111:55:453
912799276,9cyclictest0-21swapper/308:00:593
912799275,8cyclictest0-21swapper/307:42:163
9127992715,3cyclictest0-21swapper/309:55:293
9127992714,7cyclictest0-21swapper/311:26:153
9127992712,9cyclictest0-21swapper/308:37:113
9127992712,8cyclictest0-21swapper/309:06:413
9127992712,4cyclictest0-21swapper/311:06:333
912799271,13cyclictest0-21swapper/311:10:213
912799271,13cyclictest0-21swapper/310:57:433
912799271,13cyclictest0-21swapper/310:52:093
912799271,13cyclictest0-21swapper/307:55:143
911799279,4cyclictest836-21runrttasks08:17:171
911799271,9cyclictest0-21swapper/107:44:451
911799270,13cyclictest0-21swapper/110:48:251
911799270,10cyclictest0-21swapper/108:22:491
911099278,16cyclictest9432-21munin-run10:50:010
9127992615,3cyclictest0-21swapper/311:52:453
912799261,11cyclictest0-21swapper/309:10:473
912799260,12cyclictest0-21swapper/310:17:273
912299266,7cyclictest17906-40dnf07:25:322
912299264,12cyclictest7742-21cat09:30:192
912299264,12cyclictest7742-21cat09:30:192
911799266,3cyclictest0-21swapper/108:45:511
911799260,9cyclictest0-21swapper/110:05:111
911799260,14cyclictest0-21swapper/110:31:231
911799260,11cyclictest0-21swapper/109:14:471
911099263,11cyclictest7-21ksoftirqd/007:35:010
911099262,5cyclictest0-21swapper/010:20:010
911099261,12cyclictest13226-21perf09:45:010
912799258,3cyclictest0-21swapper/308:31:153
912799257,3cyclictest0-21swapper/310:46:013
912799255,12cyclictest0-21swapper/310:02:113
912799252,3cyclictest0-21swapper/308:27:283
912799252,3cyclictest0-21swapper/308:27:273
9127992511,7cyclictest0-21swapper/307:51:163
912799251,16cyclictest0-21swapper/312:01:213
9127992510,3cyclictest0-21swapper/309:33:223
9127992510,3cyclictest0-21swapper/309:33:213
912299252,11cyclictest893-21kworker/2:010:39:332
912299251,10cyclictest0-21swapper/208:52:312
911799259,3cyclictest15888-21chrt11:03:171
911799256,3cyclictest0-21swapper/108:09:331
911099251,12cyclictest12260-21munin-run12:10:010
912799246,5cyclictest0-21swapper/310:42:433
912799246,5cyclictest0-21swapper/310:42:433
912799246,10cyclictest0-21swapper/307:39:553
9127992411,3cyclictest496-21jbd2/dm-0-809:44:543
912299248,8cyclictest0-21swapper/211:49:552
912299247,7cyclictest20023-21/usr/sbin/munin10:00:282
912299246,8cyclictest0-21swapper/208:37:052
912299241,12cyclictest0-21swapper/212:23:292
912299241,10cyclictest6210-21df11:55:112
912299241,10cyclictest0-21swapper/211:24:392
911799249,3cyclictest0-21swapper/108:55:151
911799241,8cyclictest0-21swapper/111:24:211
9117992418,3cyclictest12253-40dnf08:25:471
9117992418,3cyclictest12253-40dnf08:25:471
911099241,9cyclictest268072ls07:45:320
911099241,6cyclictest0-21swapper/012:27:390
911099241,11cyclictest27039-21perf11:30:010
911099240,11cyclictest14185-21idleruntime-cro07:20:010
912799239,3cyclictest0-21swapper/309:27:233
912799235,9cyclictest0-21swapper/312:34:233
9127992313,3cyclictest0-21swapper/311:44:313
9127992313,3cyclictest0-21swapper/311:44:313
912299238,9cyclictest0-21swapper/209:52:232
912299238,9cyclictest0-21swapper/209:44:272
912299237,8cyclictest0-21swapper/208:28:012
912299237,8cyclictest0-21swapper/208:28:012
912299234,5cyclictest12050irq/123-ahci[0012:15:022
9122992312,3cyclictest12050irq/123-ahci[0007:40:012
912299231,11cyclictest0-21swapper/207:24:312
912299231,11cyclictest0-21swapper/207:24:312
9122992310,2cyclictest24176-21needreboot07:40:212
911799231,9cyclictest0-21swapper/107:54:491
911799231,12cyclictest0-21swapper/108:34:031
912799221,13cyclictest0-21swapper/310:38:593
912299228,5cyclictest86550irq/126-enp1s0-12:35:262
912299228,3cyclictest88250irq/135-enp2s0-07:58:512
912299228,2cyclictest88050irq/133-enp2s0-08:42:492
912299226,7cyclictest26761-40dnf11:26:482
912299225,8cyclictest24439-21acpi10:10:112
912299225,7cyclictest0-21swapper/209:59:352
912299225,4cyclictest780-21rngd09:15:452
912299224,3cyclictest14661-21expr11:00:162
912299223,9cyclictest27977-21latency_hist09:05:022
912299223,8cyclictest780-21rngd08:05:112
912299222,13cyclictest0-21swapper/208:49:592
912299221,8cyclictest0-21swapper/210:52:032
912299221,11cyclictest0-21swapper/212:30:152
912299221,10cyclictest0-21swapper/210:33:152
912299221,10cyclictest0-21swapper/208:14:232
911799221,11cyclictest0-21swapper/107:55:191
90892220,4sleep29092-21gltestperf08:20:162
912799214,3cyclictest0-21swapper/308:20:323
912299215,7cyclictest780-21rngd10:49:512
912299211,7cyclictest0-21swapper/209:07:372
912299211,11cyclictest0-21swapper/208:31:512
912299211,10cyclictest0-21swapper/212:29:352
912299211,10cyclictest0-21swapper/208:56:472
911799213,3cyclictest0-21swapper/109:46:051
911799211,12cyclictest0-21swapper/108:44:251
911099211,6cyclictest0-21swapper/011:52:260
911099211,10cyclictest30814-21perf10:25:010
268062210,5sleep230-21rcuc/207:45:322
150242210,2sleep011-21rcuc/012:15:140
912299204,9cyclictest0-21swapper/209:49:072
912299204,9cyclictest0-21swapper/207:18:562
912299203,8cyclictest29010-21cstates10:20:112
912299203,7cyclictest0-21swapper/209:37:312
912299201,9cyclictest0-21swapper/211:40:592
912299201,9cyclictest0-21swapper/211:40:592
912299201,8cyclictest30-21rcuc/212:01:592
9122992015,3cyclictest6816-21tail08:15:162
912299201,12cyclictest0-21swapper/211:35:192
912299200,12cyclictest5601-21diskstats10:40:122
912299200,12cyclictest5601-21diskstats10:40:122
911099202,10cyclictest24432-21/usr/sbin/munin10:10:140
9110992013,4cyclictest24209-21munin-run10:10:010
911099201,11cyclictest31412-21sshd07:55:480
912299197,7cyclictest86750irq/128-enp1s0-09:24:342
912299196,4cyclictest86750irq/128-enp1s0-12:05:262
912299195,5cyclictest551-21awk10:30:012
912299195,4cyclictest780-21rngd11:10:012
912299194,6cyclictest88150irq/134-enp2s0-09:12:412
912299192,9cyclictest14634-21snmpd07:32:462
912299192,5cyclictest28209-21missed_timers11:30:222
912299191,7cyclictest22637-21awk11:20:012
912299191,6cyclictest0-21swapper/211:54:502
911099198,3cyclictest14054-21/usr/sbin/munin11:00:210
911099196,4cyclictest30180-21diskmemload11:35:240
911099191,7cyclictest3204-21kworker/u8:212:01:520
911099191,15cyclictest18563-21idleruntime-cro07:30:010
911099191,10cyclictest23189-21/usr/sbin/munin07:40:270
911099190,10cyclictest0-21swapper/009:12:300
912299183,8cyclictest12050irq/123-ahci[0007:15:012
912299182,7cyclictest0-21swapper/207:51:072
912299182,2cyclictest11031-21kworker/u8:010:08:262
911099188,2cyclictest5254-21/usr/sbin/munin10:40:180
911099188,2cyclictest5254-21/usr/sbin/munin10:40:180
911099187,3cyclictest20655-21/usr/sbin/munin11:15:170
911099186,4cyclictest528-21cat11:40:290
911099186,4cyclictest528-21cat11:40:290
911099182,13cyclictest2593-21tr09:20:100
911099181,9cyclictest6270-40dnf09:25:490
911099181,7cyclictest0-21swapper/009:17:420
911099181,6cyclictest30180-21diskmemload09:35:130
911099181,4cyclictest9365-21kworker/u8:210:50:560
9110991813,3cyclictest3036-21grep11:45:310
9110991812,3cyclictest2131-21swap10:30:250
911099181,10cyclictest14634-21snmpd08:53:000
911099181,10cyclictest14399-21/usr/sbin/munin07:20:300
911099181,10cyclictest14399-21/usr/sbin/munin07:20:300
57012181,2sleep00-21swapper/008:11:240
912299172,6cyclictest88050irq/133-enp2s0-08:02:232
911099177,6cyclictest0-21swapper/009:04:560
911099176,3cyclictest15019-21/usr/sbin/munin08:35:260
911099171,8cyclictest0-21swapper/008:00:200
911099171,7cyclictest0-21swapper/009:50:190
911099171,7cyclictest0-21swapper/007:10:170
911099171,6cyclictest3657-21grep10:35:160
911099171,6cyclictest30180-21diskmemload11:10:230
911099171,4cyclictest21275-21/usr/sbin/munin12:30:240
911099171,4cyclictest0-21swapper/009:59:400
911099171,15cyclictest0-21swapper/008:45:520
9110991711,3cyclictest28839-21grep09:05:170
9110991711,3cyclictest20993-21/usr/sbin/munin07:35:160
911099171,10cyclictest0-21swapper/010:00:280
79242170,2sleep00-21swapper/008:16:460
911099161,6cyclictest8418-21/usr/sbin/munin08:20:190
911099161,6cyclictest0-21swapper/008:55:260
911099161,14cyclictest0-21swapper/008:34:550
9110991611,3cyclictest24781-21wc12:35:270
9110991611,3cyclictest17208-21/usr/sbin/munin08:40:230
9110991611,3cyclictest16967-21cat12:20:180
911099161,12cyclictest31132-21cpuspeed_turbos10:25:110
9110991610,3cyclictest28450-21cat07:50:180
911099151,6cyclictest0-21swapper/008:25:130
911099151,6cyclictest0-21swapper/008:25:130
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional