You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-23 - 09:51

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack7slot3s.osadl.org (updated Sat Sep 23, 2023 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29712999252,31cyclictest0-21swapper/000:00:420
2971299873,50cyclictest0-21swapper/023:10:400
296082870,2sleep10-21swapper/123:45:461
32342790,2sleep20-21swapper/223:53:322
276682770,5sleep32971699cyclictest23:42:583
276682770,5sleep32971699cyclictest23:42:583
2971299762,33cyclictest0-21swapper/019:35:420
274002760,5sleep02971299cyclictest23:00:440
260742750,6sleep12971499cyclictest21:34:431
275932700,1sleep10-21swapper/122:18:551
2971299683,36cyclictest0-21swapper/020:10:400
29712996819,6cyclictest0-21swapper/023:25:400
29712996815,27cyclictest425-21cron21:40:220
16782670,2sleep10-21swapper/100:35:461
2971299669,38cyclictest10754-21unixbench_singl21:55:580
2971699658,29cyclictest8050irq/56-dwc_otg_22:50:413
29712996422,6cyclictest0-21swapper/000:15:370
2971299641,34cyclictest0-21swapper/022:35:270
2971299641,34cyclictest0-21swapper/020:55:390
2971699634,26cyclictest9504-21ssh21:55:403
29712996342,4cyclictest25104-21uniq22:15:450
29712996310,49cyclictest101ktimersoftd/023:55:560
2971699627,5cyclictest13405-21sshd22:41:533
2971699627,5cyclictest13405-21sshd22:41:533
2971699626,4cyclictest8150irq/56-dwc_otg_21:23:573
2971699625,13cyclictest8150irq/56-dwc_otg_20:00:573
2971699621,7cyclictest8150irq/56-dwc_otg_23:50:543
29712996231,8cyclictest0-21swapper/023:40:220
2971299622,34cyclictest20736-21unixbench_singl21:26:050
20492620,1sleep00-21swapper/023:10:220
2971699616,4cyclictest8150irq/56-dwc_otg_00:30:423
29712996139,9cyclictest101ktimersoftd/022:30:440
29712996139,10cyclictest24888-21ssh00:22:350
2971299611,40cyclictest28024-21awk20:35:210
2971299611,40cyclictest28024-21awk20:35:210
127692610,2sleep30-21swapper/321:15:493
2971699604,15cyclictest8050irq/56-dwc_otg_23:55:373
2971699601,5cyclictest2046-21memory00:35:503
29712996035,13cyclictest0-21swapper/021:53:200
29712996031,2cyclictest0-21swapper/021:10:410
2971299602,28cyclictest21442-21ls20:15:520
2971299601,18cyclictest3058-21grep19:25:480
29716995925,6cyclictest8150irq/56-dwc_otg_00:05:423
2971299595,37cyclictest32293-21latency_hist22:25:220
2971299592,30cyclictest6604-21latency_hist21:05:220
2971299591,37cyclictest0-21swapper/022:10:430
2971699582,6cyclictest131-21systemd-journal19:15:213
2971699581,4cyclictest8050irq/56-dwc_otg_00:28:033
2971299587,48cyclictest101ktimersoftd/000:30:280
29712995834,4cyclictest0-21swapper/021:15:400
29712995831,9cyclictest29312-21basename23:45:420
29712995831,6cyclictest8050irq/56-dwc_otg_22:51:000
29712995810,38cyclictest31726-21ls19:15:460
2971699572,17cyclictest7900-21aten_rbpower_po22:35:423
2971699571,5cyclictest8150irq/56-dwc_otg_22:30:573
2971299573,36cyclictest11522-21sh19:50:220
2971299571,36cyclictest29810-21tr19:10:410
2971299571,28cyclictest0-21swapper/022:40:400
2971299571,28cyclictest0-21swapper/022:40:400
29712995711,26cyclictest0-21swapper/019:50:470
2971699562,9cyclictest0-21swapper/323:19:593
2971699562,5cyclictest8150irq/56-dwc_otg_20:40:513
2971699562,3cyclictest0-21swapper/300:25:223
2971699561,4cyclictest3105-21ssh23:10:543
2971699561,3cyclictest0-21swapper/320:15:223
2971499569,35cyclictest7950irq/56-dwc_otg22:30:391
29712995643,5cyclictest1972-21unixbench_singl23:51:010
29712995641,6cyclictest0-21swapper/023:45:220
29712995641,6cyclictest0-21swapper/023:45:220
29712995637,8cyclictest0-21swapper/021:42:380
29712995625,14cyclictest8050irq/56-dwc_otg_00:25:420
2971299562,41cyclictest15896-21uname20:00:470
2971299562,30cyclictest10302-21tune2fs23:20:470
2971299561,5cyclictest13655-21sort22:00:470
2971299561,52cyclictest101ktimersoftd/000:35:480
2971299561,31cyclictest12727-21ssh00:05:540
2971699554,11cyclictest8050irq/56-dwc_otg_21:40:393
2971699552,9cyclictest17547-21fschecks_time22:05:473
2971699552,19cyclictest0-21swapper/321:45:273
2971699552,19cyclictest0-21swapper/321:45:273
2971699551,4cyclictest0-21swapper/320:35:483
2971499553,7cyclictest0-21swapper/122:10:421
2971499552,10cyclictest0-21swapper/123:35:391
2971299554,30cyclictest0-21swapper/020:05:560
2971699546,5cyclictest8016-21latency_hist19:40:223
2971699543,8cyclictest7950irq/56-dwc_otg21:10:473
2971699542,3cyclictest0-21swapper/300:18:083
2971699541,5cyclictest10603-21apt-config00:03:093
2971699541,4cyclictest8050irq/56-dwc_otg_23:22:493
2971699541,4cyclictest15793-21/usr/sbin/munin00:10:413
29716995412,5cyclictest0-21swapper/321:55:223
2971299544,36cyclictest0-21swapper/021:32:520
29712995432,10cyclictest0-21swapper/022:09:260
29712995430,13cyclictest0-21swapper/022:25:460
29712995425,3cyclictest0-21swapper/022:45:410
2971299541,33cyclictest0-21swapper/019:40:540
2971299541,27cyclictest23503-21fschecks_time22:55:470
29712995410,23cyclictest7063-21ls21:05:440
285712540,5sleep128574-21ntp_kernel_err19:05:521
285712540,5sleep128574-21ntp_kernel_err19:05:521
2971699531,9cyclictest0-21swapper/319:43:013
2971699531,8cyclictest8050irq/56-dwc_otg_23:05:543
2971699531,13cyclictest0-21swapper/319:16:413
2971599534,36cyclictest7950irq/56-dwc_otg22:35:272
2971599532,38cyclictest0-21swapper/222:10:422
2971299532,29cyclictest0-21swapper/021:47:160
2971299532,29cyclictest0-21swapper/021:47:160
29712995313,38cyclictest101ktimersoftd/019:55:280
2971299531,29cyclictest1522-21fschecks_time20:50:470
2971699521,3cyclictest7950irq/56-dwc_otg23:26:523
29712995231,7cyclictest8150irq/56-dwc_otg_19:25:210
29712995224,5cyclictest527-21unixbench_multi20:45:580
2971699512,3cyclictest7950irq/56-dwc_otg22:00:533
2971699511,5cyclictest5158-21grep19:30:573
2971599517,13cyclictest8050irq/56-dwc_otg_21:55:392
2971499514,34cyclictest0-21swapper/119:40:401
29712995128,5cyclictest8050irq/56-dwc_otg_23:30:550
2971299512,28cyclictest30073-21apt-get20:40:400
2971699505,6cyclictest8150irq/56-dwc_otg_21:34:433
2971699502,4cyclictest0-21swapper/322:55:303
2971699501,9cyclictest8150irq/56-dwc_otg_19:55:443
2971699501,5cyclictest0-21swapper/322:16:403
2971699501,4cyclictest26758-21apt-get21:35:413
2971699501,3cyclictest0-21swapper/323:30:563
29712995030,11cyclictest17451-21ssh00:12:150
29712995022,5cyclictest19090-21rm21:25:210
2971299501,18cyclictest7950irq/56-dwc_otg20:35:430
2971699491,4cyclictest8050irq/56-dwc_otg_22:22:553
2971699491,4cyclictest1641-21ssh22:26:293
2971599493,10cyclictest0-21swapper/222:30:402
2971299492,27cyclictest0-21swapper/019:31:010
29716994813,5cyclictest381ktimersoftd/323:47:253
2971599482,4cyclictest0-21swapper/223:15:392
29712994832,10cyclictest101ktimersoftd/020:22:140
29712994831,8cyclictest0-21swapper/023:16:050
2971699474,4cyclictest7950irq/56-dwc_otg22:14:323
2971699472,14cyclictest8150irq/56-dwc_otg_23:35:573
2971499475,8cyclictest0-21swapper/119:45:411
29716994622,4cyclictest3057-21fschecks_time19:25:483
2971699461,3cyclictest0-21swapper/319:51:463
2971499469,10cyclictest8150irq/56-dwc_otg_23:25:401
2971499466,26cyclictest6132-21ssh23:15:401
291352463,36sleep00-21swapper/019:06:250
291352463,36sleep00-21swapper/019:06:250
2971699457,21cyclictest131-21systemd-journal20:20:273
2971699451,4cyclictest0-21swapper/320:27:533
2971599453,9cyclictest8050irq/56-dwc_otg_20:05:382
29715994519,7cyclictest6419-21sh21:50:552
2971599451,7cyclictest8150irq/56-dwc_otg_19:10:382
2971499458,23cyclictest7950irq/56-dwc_otg00:30:261
2971499455,33cyclictest8050irq/56-dwc_otg_21:45:381
2971499455,33cyclictest8050irq/56-dwc_otg_21:45:381
29712994510,22cyclictest25685-21grep20:25:590
2971499448,23cyclictest0-21swapper/120:15:441
2971499447,24cyclictest0-21swapper/120:55:271
2971499441,4cyclictest8050irq/56-dwc_otg_23:05:391
2971499441,32cyclictest8150irq/56-dwc_otg_19:50:451
2971599437,26cyclictest0-21swapper/219:35:422
2971599437,25cyclictest0-21swapper/219:40:422
29715994317,5cyclictest29901-21sh23:04:032
2971499436,24cyclictest2299-21ssh23:10:421
2971599427,25cyclictest0-21swapper/221:20:402
2971599421,9cyclictest8150irq/56-dwc_otg_22:00:422
2971499429,6cyclictest8050irq/56-dwc_otg_19:35:421
2971499429,19cyclictest12253-21ssh21:15:401
2971499421,9cyclictest30006-21apt21:40:221
2971599417,24cyclictest0-21swapper/220:15:442
2971599417,24cyclictest0-21swapper/220:00:402
2971599415,24cyclictest0-21swapper/220:20:282
29715994132,5cyclictest29-21rcuc/200:15:382
2971599412,5cyclictest8050irq/56-dwc_otg_00:05:402
2971699401,21cyclictest27541-21/usr/sbin/munin23:00:473
2971599401,9cyclictest0-21swapper/200:30:392
2971599401,6cyclictest0-21swapper/219:50:392
2971499406,4cyclictest0-21swapper/121:54:191
29714994010,19cyclictest8050irq/56-dwc_otg_20:00:411
271602400,1sleep10-21swapper/123:41:551
271602400,1sleep10-21swapper/123:41:551
2971699397,22cyclictest0-21swapper/320:30:273
2971699397,22cyclictest0-21swapper/320:30:273
2971599396,21cyclictest0-21swapper/220:55:272
2971599395,11cyclictest7950irq/56-dwc_otg23:05:392
29715993927,5cyclictest10413-21ssh21:12:022
2971599391,24cyclictest0-21swapper/219:15:452
2971499397,17cyclictest0-21swapper/122:21:021
2971499396,21cyclictest15582-21ssh00:10:391
29714993929,3cyclictest8150irq/56-dwc_otg_20:20:461
29714993917,4cyclictest7937-21latency_hist00:00:221
2971499391,29cyclictest7950irq/56-dwc_otg00:20:511
2971499391,27cyclictest7950irq/56-dwc_otg20:45:431
29715993820,8cyclictest8050irq/56-dwc_otg_23:30:492
29715993812,5cyclictest1292-21runrttasks21:42:552
2971499382,7cyclictest0-21swapper/123:50:411
29714993819,11cyclictest23-21ksoftirqd/119:55:431
29714993812,17cyclictest0-21swapper/121:55:551
29714993810,19cyclictest8050irq/56-dwc_otg_20:05:551
144362380,2sleep10-21swapper/122:43:591
144362380,2sleep10-21swapper/122:43:591
2971599376,15cyclictest8150irq/56-dwc_otg_22:05:462
2971599374,12cyclictest8150irq/56-dwc_otg_23:10:402
2971599371,27cyclictest1891-21ls19:20:592
29715993711,5cyclictest0-21swapper/222:28:212
2971499379,15cyclictest8988-21fschecks_time21:10:471
2971499374,17cyclictest0-21swapper/123:00:381
2971499371,8cyclictest0-21swapper/122:05:421
29714993716,7cyclictest26343-21latency_hist20:30:231
29716993611,13cyclictest16542-21timerandwakeup22:45:573
2971599368,5cyclictest7950irq/56-dwc_otg23:55:442
2971599361,4cyclictest8050irq/56-dwc_otg_21:35:572
2971599361,22cyclictest8150irq/56-dwc_otg_21:46:082
2971599361,22cyclictest8150irq/56-dwc_otg_21:46:082
2971499369,8cyclictest7950irq/56-dwc_otg22:35:581
2971499369,17cyclictest0-21swapper/123:20:501
2971499368,7cyclictest0-21swapper/122:50:421
2971699356,7cyclictest37-21rcuc/319:20:503
2971699355,15cyclictest8150irq/56-dwc_otg_20:10:223
2971699351,24cyclictest7653-21timerandwakeup21:05:543
2971599355,21cyclictest12286-21ssh21:15:422
2971599353,13cyclictest8050irq/56-dwc_otg_22:20:442
2971599352,18cyclictest6116-21modprobe19:35:222
2971599351,27cyclictest11708-21ls19:50:232
2971499355,8cyclictest19484-21ssh21:25:421
2971499355,10cyclictest13355-21ssh22:00:401
2971499353,9cyclictest0-21swapper/122:45:411
29714993525,7cyclictest19428-21fschecks_time20:10:471
2971499352,23cyclictest7950irq/56-dwc_otg21:40:531
29714993517,6cyclictest8050irq/56-dwc_otg_22:25:511
29714993514,6cyclictest27399-21rt-features00:25:541
2971499351,23cyclictest0-21swapper/100:00:411
29714993510,15cyclictest0-21swapper/122:55:431
2971599347,15cyclictest0-21swapper/219:55:272
2971599341,4cyclictest8150irq/56-dwc_otg_00:35:402
2971599341,4cyclictest0-21swapper/221:30:392
2971599341,24cyclictest28006-21latency_hist20:35:212
2971599341,24cyclictest28006-21latency_hist20:35:212
2971599341,17cyclictest17489-21ls23:30:222
2971499343,18cyclictest0-21swapper/121:20:371
2971499341,15cyclictest18386-21ssh23:30:521
29714993410,5cyclictest27330-21switchtime20:30:561
29714993410,5cyclictest27330-21switchtime20:30:561
196232340,2sleep20-21swapper/221:25:502
2971699335,7cyclictest7950irq/56-dwc_otg20:55:223
2971699331,22cyclictest5750-21switchtime21:00:533
2971599339,4cyclictest7950irq/56-dwc_otg23:35:582
2971599338,20cyclictest0-21swapper/220:10:272
2971599332,6cyclictest8050irq/56-dwc_otg_00:20:462
2971599331,6cyclictest28312-21cron22:20:222
2971499339,14cyclictest32311-21ps19:15:571
2971499337,6cyclictest8050irq/56-dwc_otg_00:20:211
2971499337,4cyclictest8150irq/56-dwc_otg_20:45:221
2971499333,20cyclictest7950irq/56-dwc_otg19:20:581
2971499332,3cyclictest0-21swapper/100:05:571
2971499332,16cyclictest29963-21latency_hist20:40:221
2971699325,7cyclictest3082-21/usr/sbin/munin20:55:533
2971599327,15cyclictest0-21swapper/220:45:422
2971599325,7cyclictest8050irq/56-dwc_otg_19:30:212
2971599321,6cyclictest0-21swapper/223:20:412
29715993215,11cyclictest15548-21/usr/sbin/munin00:10:442
2971599321,22cyclictest26781-21grep00:25:452
2971599321,22cyclictest23545-21uniq22:55:472
29714993215,8cyclictest7950irq/56-dwc_otg19:31:001
2971499321,17cyclictest0-21swapper/121:05:411
293792322,23sleep20-21swapper/219:09:342
293792322,23sleep20-21swapper/219:09:342
2971599319,6cyclictest4750irq/17-3f00b88023:50:082
2971599319,10cyclictest0-21swapper/222:54:152
2971599313,5cyclictest30307-21fschecks_count20:40:462
2971599313,16cyclictest8050irq/56-dwc_otg_22:46:162
29714993124,5cyclictest23-21ksoftirqd/121:05:221
2971499312,3cyclictest0-21swapper/119:25:591
29716993014,5cyclictest20684-21unixbench_multi21:26:053
2971599304,12cyclictest8150irq/56-dwc_otg_20:35:582
2971599302,20cyclictest0-21swapper/220:29:452
29715993015,5cyclictest31-21ksoftirqd/221:10:222
2971499302,21cyclictest8050irq/56-dwc_otg_20:50:461
2971499302,20cyclictest0-21swapper/119:10:311
2971699292,13cyclictest8150irq/56-dwc_otg_19:46:513
2971599294,6cyclictest8150irq/56-dwc_otg_21:00:432
2971599293,6cyclictest8150irq/56-dwc_otg_22:40:472
2971599293,6cyclictest8150irq/56-dwc_otg_22:40:472
2971599293,13cyclictest8150irq/56-dwc_otg_00:02:262
2971599292,5cyclictest26480-21ls23:40:582
2971599292,5cyclictest26480-21ls23:40:582
2971599292,16cyclictest0-21swapper/220:54:132
2971699284,10cyclictest8150irq/56-dwc_otg_20:20:173
2971699262,5cyclictest570-21sed20:45:593
292602263,15sleep30-21swapper/319:08:013
292602263,15sleep30-21swapper/319:08:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional