You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-10-01 - 07:31

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rack7slot3s.osadl.org (updated Sat Sep 30, 2023 12:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
78872950,10sleep1221ktimersoftd/108:56:021
63252860,4sleep2147599cyclictest12:27:412
224912760,8sleep00-21swapper/010:45:460
147399766,63cyclictest0-21swapper/007:30:490
147599717,44cyclictest31936-21rm12:19:462
80802690,2sleep30-21swapper/307:26:043
29822690,1sleep2301ktimersoftd/211:42:242
147499677,10cyclictest20673-21ssh10:00:471
147499671,5cyclictest26916-21sed11:31:101
147399676,25cyclictest0-21swapper/009:05:480
147399675,40cyclictest0-21swapper/007:50:450
48792660,3sleep123-21ksoftirqd/111:04:071
147599661,7cyclictest11861-21latency_hist12:35:272
147399656,47cyclictest0-21swapper/008:05:450
1473996519,28cyclictest28745-21aten_rbpower_po12:15:490
147399645,18cyclictest7950irq/56-dwc_otg11:05:460
147399642,35cyclictest24217-21ssh10:05:260
1476996317,22cyclictest21081-21/usr/sbin/munin09:20:523
147599621,4cyclictest8150irq/56-dwc_otg_10:35:262
1474996215,8cyclictest13452-21latency_hist11:15:261
147399626,37cyclictest0-21swapper/007:55:480
147399624,30cyclictest0-21swapper/012:35:530
147399624,30cyclictest0-21swapper/012:35:530
1473996235,5cyclictest8236-21ssh12:30:490
147499618,5cyclictest4466-21unixbench_singl07:16:041
147599605,5cyclictest17420-21ssh11:20:262
147499607,4cyclictest8050irq/56-dwc_otg_10:20:591
1473996022,7cyclictest0-21swapper/007:15:450
1473996011,30cyclictest0-21swapper/011:51:100
1473996010,23cyclictest28069-21ssh10:10:270
55052590,2sleep00-21swapper/010:22:310
147599591,6cyclictest1855-21tune2fs07:10:502
1473995942,5cyclictest8050irq/56-dwc_otg_11:30:490
1473995925,15cyclictest8050irq/56-dwc_otg_08:55:530
1473995913,23cyclictest22211-21if_wlan011:25:560
147599585,8cyclictest7950irq/56-dwc_otg07:16:012
147599585,10cyclictest8150irq/56-dwc_otg_10:51:062
147599585,10cyclictest8150irq/56-dwc_otg_10:51:062
147599581,9cyclictest8150irq/56-dwc_otg_09:25:592
147399586,41cyclictest0-21swapper/011:40:470
1476995718,20cyclictest1706-21/usr/sbin/munin08:40:483
147599572,9cyclictest8150irq/56-dwc_otg_11:33:522
147599571,5cyclictest20451-21latency_hist10:00:262
147599571,4cyclictest0-21swapper/208:35:582
147599571,3cyclictest0-21swapper/210:26:082
147599571,18cyclictest0-21swapper/212:11:432
147599571,14cyclictest22996-21tail11:26:092
147399575,30cyclictest0-21swapper/012:20:450
147399573,26cyclictest0-21swapper/009:55:460
1473995728,12cyclictest8050irq/56-dwc_otg_11:15:490
147399572,30cyclictest8563-21grep10:26:110
147399571,7cyclictest4298-21idleruntime-cro09:40:260
147399571,37cyclictest28633-21wget08:25:490
147599565,8cyclictest8150irq/56-dwc_otg_11:15:262
147599562,5cyclictest8150irq/56-dwc_otg_10:38:062
147599561,4cyclictest16510-21latency_hist09:55:272
147599561,4cyclictest16510-21latency_hist09:55:272
147599561,16cyclictest0-21swapper/210:24:462
147599561,10cyclictest20106-21sshd11:23:062
147399566,20cyclictest8150irq/56-dwc_otg_12:25:450
147399563,32cyclictest131-21systemd-journal08:15:260
1473995626,21cyclictest8150irq/56-dwc_otg_12:00:490
147399562,27cyclictest13-21rcuc/011:25:260
147399561,32cyclictest13-21rcuc/011:36:100
147399561,29cyclictest3752-21fschecks_time08:45:510
11332563,12sleep30-21swapper/307:10:233
1476995522,12cyclictest9528-21processes07:31:013
1476995520,21cyclictest16969-21/usr/sbin/munin12:00:493
1476995517,20cyclictest5464-21ssh10:22:263
1476995510,29cyclictest0-21swapper/311:55:483
147599556,4cyclictest0-21swapper/212:23:382
147599555,9cyclictest21477-21ssh09:20:592
147599555,8cyclictest3821-21ssh11:02:052
147599555,4cyclictest7950irq/56-dwc_otg08:40:572
147599553,4cyclictest8050irq/56-dwc_otg_08:25:272
147599551,9cyclictest14943-21ssh09:12:592
147599551,5cyclictest8150irq/56-dwc_otg_10:11:022
147599551,4cyclictest8050irq/56-dwc_otg_10:05:482
147599551,13cyclictest8150irq/56-dwc_otg_12:06:042
147599551,13cyclictest8150irq/56-dwc_otg_12:06:042
147599551,10cyclictest18661-21tr10:40:462
147599551,10cyclictest0-21swapper/209:38:362
1473995529,5cyclictest8050irq/56-dwc_otg_09:41:000
147399552,28cyclictest9135-21ssh09:45:510
147399551,51cyclictest2667-21head07:11:020
147399551,29cyclictest25612-21runrttasks12:11:000
1476995414,26cyclictest22851-21ssh10:46:013
147599541,9cyclictest17545-21ssh12:00:592
147599541,5cyclictest8968-21diskstats07:30:522
1475995412,6cyclictest7327-21grep11:06:142
147599541,16cyclictest0-21swapper/208:33:212
147599541,12cyclictest8150irq/56-dwc_otg_10:56:472
147599541,12cyclictest0-21swapper/211:54:082
147399541,31cyclictest5177-21cut07:20:440
147399541,28cyclictest0-21swapper/010:55:480
147399541,22cyclictest18540-21taskset10:40:310
147699532,7cyclictest0-21swapper/311:15:463
1476995322,19cyclictest32761-21ls08:35:533
1476995320,11cyclictest4522-21/usr/sbin/munin08:46:053
147599534,4cyclictest8150irq/56-dwc_otg_12:35:502
147599534,4cyclictest8150irq/56-dwc_otg_12:35:502
147499533,20cyclictest0-21swapper/109:20:571
147399531,28cyclictest12417-21unixbench_singl10:31:130
147599528,5cyclictest301ktimersoftd/210:00:472
147599521,7cyclictest0-21swapper/210:47:292
147599521,13cyclictest0-21swapper/209:15:392
147599521,13cyclictest0-21swapper/209:15:392
147599521,11cyclictest30345-21ssh09:31:152
147599521,11cyclictest0-21swapper/208:19:542
1473995227,7cyclictest8150irq/56-dwc_otg_09:51:040
1473995227,7cyclictest8150irq/56-dwc_otg_09:51:040
147399522,35cyclictest0-21swapper/010:10:260
147399521,34cyclictest26769-21fschecks_count10:50:520
147399521,34cyclictest26769-21fschecks_count10:50:520
147399521,31cyclictest2124-21aten_rbpower_vo11:00:500
147399521,16cyclictest16340-21ssh09:15:270
147599514,6cyclictest3240750ssh11:39:102
147599511,9cyclictest0-21swapper/209:48:382
147599511,5cyclictest7653-21ntp_states07:25:592
147599511,12cyclictest8150irq/56-dwc_otg_12:00:002
147599511,11cyclictest11763-21taskset07:36:182
147599511,10cyclictest0-21swapper/208:46:312
147499517,5cyclictest8150irq/56-dwc_otg_08:30:261
1473995127,5cyclictest17686-21timerandwakeup09:16:020
1473995127,5cyclictest17686-21timerandwakeup09:16:020
1473995124,6cyclictest8050irq/56-dwc_otg_11:56:010
147399512,30cyclictest20742-21ssh09:20:480
1473995119,5cyclictest28458-21cron09:30:250
147399511,29cyclictest7042-21cut07:25:480
1476995021,11cyclictest26280-21/usr/sbin/munin11:30:583
1476995019,12cyclictest30207-21sed09:31:053
1476995019,11cyclictest3639-21/usr/sbin/munin07:15:513
1476995017,16cyclictest12493-21latency_hist11:55:263
1476995015,19cyclictest30426-21rm12:16:573
147599502,15cyclictest0-21swapper/210:16:262
147599501,9cyclictest8150irq/56-dwc_otg_07:48:162
147599501,9cyclictest8040-21rm09:44:492
147399504,29cyclictest29800-21kworker/0:109:30:530
1473995027,5cyclictest0-21swapper/011:11:010
1473995026,11cyclictest0-21swapper/007:35:590
1473995025,5cyclictest16583-21sh10:36:520
1473995019,16cyclictest8050irq/56-dwc_otg_09:05:260
1476994912,25cyclictest21685-21/usr/sbin/munin11:26:033
147599491,7cyclictest0-21swapper/209:10:202
147599491,4cyclictest0-21swapper/207:41:432
147599491,4cyclictest0-21swapper/207:41:432
147599491,13cyclictest0-21swapper/211:50:152
147399492,30cyclictest0-21swapper/012:08:510
147399492,30cyclictest0-21swapper/012:08:510
147399491,23cyclictest493350/usr/sbin/munin11:45:480
147499481,17cyclictest832-21ssh10:59:021
147399487,26cyclictest0-21swapper/007:40:470
147399487,26cyclictest0-21swapper/007:40:470
147699478,21cyclictest0-21swapper/307:21:013
147699477,25cyclictest0-21swapper/307:45:473
147699476,25cyclictest8150irq/56-dwc_otg_09:58:043
1476994720,8cyclictest603-21sh12:20:503
1476994718,13cyclictest16113-21ssh09:15:133
1476994718,11cyclictest17702-21ssh11:20:363
1476994717,15cyclictest16280-21ssh09:55:183
1476994717,15cyclictest16280-21ssh09:55:183
1476994717,12cyclictest11751-21taskset07:36:083
1476994712,7cyclictest25658-21ls10:06:023
147599471,5cyclictest8150irq/56-dwc_otg_08:15:062
147599471,5cyclictest0-21swapper/208:09:532
1474994721,5cyclictest28716-21latency_hist09:30:271
1473994721,6cyclictest25718-21/usr/sbin/munin08:15:580
147399471,28cyclictest20323-21ps08:01:000
147699467,24cyclictest0-21swapper/308:55:483
147699467,12cyclictest27269-21chrt10:51:003
147699467,12cyclictest27269-21chrt10:51:003
1476994614,14cyclictest6967-21ssh11:47:253
1476994612,17cyclictest23244-21rm12:08:293
1476994612,17cyclictest23244-21rm12:08:293
147599461,5cyclictest5665-21latency07:20:522
147499462,6cyclictest8050irq/56-dwc_otg_10:45:491
147699457,24cyclictest0-21swapper/308:15:473
1476994515,18cyclictest7716-21ssh11:06:553
1476994515,12cyclictest30996-21ssh10:13:563
1476994511,18cyclictest10454-21ssh10:29:583
1476994511,11cyclictest7509-21ssh12:30:003
147599451,11cyclictest0-21swapper/207:55:492
147499453,6cyclictest0-21swapper/107:40:451
147499453,6cyclictest0-21swapper/107:40:451
147399452,8cyclictest0-21swapper/008:35:490
147399452,17cyclictest27911-21grep08:21:040
1473994515,21cyclictest0-21swapper/010:16:580
8132442,35sleep00-21swapper/007:06:120
147699449,15cyclictest11731-21ssh12:35:263
147699448,13cyclictest2398-21ssh10:19:003
1476994415,7cyclictest39-21ksoftirqd/309:50:263
1476994410,15cyclictest25045-21/usr/sbin/munin09:25:513
147599441,14cyclictest0-21swapper/208:54:252
147499447,24cyclictest0-21swapper/111:40:491
147499446,5cyclictest8150irq/56-dwc_otg_09:35:521
147499446,23cyclictest0-21swapper/108:00:481
147399442,24cyclictest0-21swapper/008:50:360
147399441,25cyclictest0-21swapper/007:47:300
1476994317,11cyclictest2254-21ssh09:36:423
1476994314,14cyclictest2633-21unixbench_multi07:11:023
1476994313,13cyclictest20604-21ssh10:00:403
1476994311,12cyclictest4983-21rm11:04:193
1476994310,13cyclictest1233-21ssh11:40:533
147599431,8cyclictest0-21swapper/208:59:222
147399431,32cyclictest0-21swapper/008:33:130
259172420,2sleep10-21swapper/110:06:181
147699429,13cyclictest31892-21ssh10:56:193
147699426,23cyclictest0-21swapper/309:15:483
147699426,23cyclictest0-21swapper/309:15:483
1476994214,11cyclictest18572-21ssh10:40:343
147499426,30cyclictest0-21swapper/108:40:471
147499422,31cyclictest0-21swapper/107:45:491
1473994223,5cyclictest0-21swapper/008:42:000
1476994112,11cyclictest17197-21taskset07:51:343
1476994110,11cyclictest30546-21expr11:36:063
147499414,32cyclictest8150irq/56-dwc_otg_12:20:431
147499412,6cyclictest9330-21fschecks_time11:50:581
1474994119,6cyclictest221ktimersoftd/109:55:571
306012400,1sleep10-21swapper/110:13:061
147699409,10cyclictest0-21swapper/308:54:213
147699406,9cyclictest0-21swapper/312:10:573
1476994014,14cyclictest6413-21ssh09:41:483
1476994012,11cyclictest13569-21ssh12:36:193
1476994012,11cyclictest13569-21ssh12:36:193
1476994012,10cyclictest8050irq/56-dwc_otg_10:37:403
1476994011,13cyclictest0-21swapper/307:41:393
1476994011,13cyclictest0-21swapper/307:41:393
1476994010,11cyclictest11894-21sh11:12:213
147599406,25cyclictest0-21swapper/208:00:482
147499407,24cyclictest0-21swapper/111:15:311
147499403,5cyclictest7950irq/56-dwc_otg07:30:471
147699397,9cyclictest0-21swapper/310:32:203
147599391,13cyclictest0-21swapper/208:28:042
147499394,8cyclictest8150irq/56-dwc_otg_12:35:501
147499394,8cyclictest8150irq/56-dwc_otg_12:35:501
1474993913,6cyclictest16335-21cron12:00:261
1474993912,6cyclictest30072-21sh11:35:561
256842380,5sleep1147499cyclictest08:15:571
147699387,21cyclictest21047-21runrttasks08:04:403
1476993828,7cyclictest27355-21ls08:20:533
147499388,8cyclictest0-21swapper/110:51:061
147499388,8cyclictest0-21swapper/110:51:061
147499386,23cyclictest0-21swapper/110:40:481
1474993818,5cyclictest32302-21idleruntime-cro12:20:261
147699373,19cyclictest8150irq/56-dwc_otg_09:10:173
1476993726,8cyclictest8852-21apt-get09:00:463
147699371,23cyclictest0-21swapper/308:05:453
147599371,13cyclictest0-21swapper/207:54:102
147499377,19cyclictest1472-21cyclictest11:20:311
147499376,6cyclictest8050irq/56-dwc_otg_08:30:571
147499375,11cyclictest28907-21sh09:30:451
147499372,21cyclictest1341-21snmpd11:29:071
1474993719,5cyclictest14989-21sh09:52:451
1474993719,5cyclictest14989-21sh09:52:451
147499367,13cyclictest8150irq/56-dwc_otg_08:10:261
147499366,4cyclictest24961-21grep12:10:491
147499365,21cyclictest7950irq/56-dwc_otg08:45:471
147499364,9cyclictest8150irq/56-dwc_otg_12:31:041
147499362,7cyclictest1416-21apt-config08:40:261
1474993619,5cyclictest5942-21sensors11:46:081
1474993616,4cyclictest9125-21fschecks_time09:45:501
1474993613,6cyclictest23-21ksoftirqd/107:20:521
147699354,19cyclictest19005-21runrttasks07:56:383
147499359,5cyclictest0-21swapper/112:28:381
147499359,4cyclictest7950irq/56-dwc_otg11:05:571
147499358,5cyclictest12242-21latency_hist09:10:261
147499357,5cyclictest0-21swapper/109:40:571
147499352,20cyclictest0-21swapper/109:10:501
1474993513,5cyclictest12118-21idleruntime-cro07:40:261
147499351,22cyclictest0-21swapper/110:29:341
188132341,3sleep10-21swapper/112:02:331
1476993423,8cyclictest28603-21apt-get08:25:463
147599341,16cyclictest0-21swapper/209:01:592
147499349,5cyclictest0-21swapper/110:30:421
147499349,3cyclictest0-21swapper/109:17:191
147499349,3cyclictest0-21swapper/109:17:191
147499344,7cyclictest7950irq/56-dwc_otg07:30:261
1474993412,6cyclictest0-21swapper/112:05:471
1474993412,6cyclictest0-21swapper/112:05:471
147499341,20cyclictest17804-21wget07:55:491
1474993410,4cyclictest0-21swapper/110:18:561
1476993319,11cyclictest23914-21/usr/sbin/munin08:11:003
147499331,7cyclictest16084-21wc10:36:101
147499315,5cyclictest0-21swapper/107:50:551
147499303,7cyclictest8050irq/56-dwc_otg_08:51:001
147499302,5cyclictest0-21swapper/108:10:551
147699295,5cyclictest8150irq/56-dwc_otg_08:30:513
147499291,14cyclictest7950irq/56-dwc_otg07:10:551
147499284,4cyclictest7950irq/56-dwc_otg09:01:051
10632281,15sleep10-21swapper/107:09:281
321342277,8sleep20-21swapper/207:05:492
1474992721,4cyclictest27122-21fschecks_time08:20:501
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional