You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2025-05-13 - 03:09

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.

Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack7slot3s.osadl.org (updated Tue May 13, 2025 00:43:36)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
5562991001,85cyclictest29893-21cat00:20:500
556299962,66cyclictest0-21swapper/000:10:360
556299911,73cyclictest29667-21aten_rbpower_vo22:10:390
556299911,5cyclictest9920-21tr19:20:470
5562999019,51cyclictest7716-21ssh21:40:480
5562999019,51cyclictest7716-21ssh21:40:480
556299882,72cyclictest25331-21ssh23:30:360
556299882,5cyclictest22730-21grep19:55:480
556299881,6cyclictest29195-21awk22:10:210
556299881,67cyclictest878-21runrttasks22:42:290
556299881,67cyclictest878-21runrttasks22:42:290
5562998416,37cyclictest0-21swapper/022:00:350
85182830,2sleep00-21swapper/023:08:030
90872820,2sleep10-21swapper/119:20:221
556299822,69cyclictest0-21swapper/022:29:210
556299822,67cyclictest0-21swapper/019:50:220
556299822,64cyclictest0-21swapper/022:55:410
556299821,8cyclictest26953-21cron22:50:210
556299821,4cyclictest0-21swapper/021:35:130
5562998210,62cyclictest16225-21diskmemload22:53:010
556299817,28cyclictest0-21swapper/020:20:380
556299812,66cyclictest0-21swapper/019:50:410
556299812,60cyclictest0-21swapper/000:15:520
556299812,5cyclictest29116-21aten_rbpower_cu23:35:380
556299811,2cyclictest0-21swapper/023:15:390
556299801,64cyclictest0-21swapper/021:59:580
556299801,5cyclictest16907-21df_inode21:10:390
556299801,2cyclictest0-21swapper/023:52:010
556299801,2cyclictest0-21swapper/023:52:010
556299791,68cyclictest14777-21ssh23:59:390
556299791,66cyclictest0-21swapper/023:01:170
292742790,1sleep30-21swapper/300:20:373
556299781,4cyclictest1172-21ssh22:15:410
556299771,3cyclictest0-21swapper/021:37:580
556299767,24cyclictest0-21swapper/019:30:360
556299762,58cyclictest0-21swapper/020:40:370
556299762,45cyclictest0-21swapper/020:37:470
556299761,60cyclictest21937-21ssh21:16:130
556299761,5cyclictest8069-21ls19:15:460
556299761,5cyclictest7446-21/usr/sbin/munin20:45:460
279182760,5sleep1556399cyclictest00:17:251
556299754,53cyclictest0-21swapper/000:37:590
556299752,61cyclictest0-21swapper/000:31:520
556299751,4cyclictest0-21swapper/020:31:520
556299751,3cyclictest0-21swapper/000:07:330
244012750,5sleep1556399cyclictest22:03:181
556299741,4cyclictest0-21swapper/021:51:380
556299731,4cyclictest0-21swapper/023:30:150
209822730,2sleep10-21swapper/100:06:541
556299722,55cyclictest0-21swapper/021:30:100
556299722,4cyclictest0-21swapper/022:39:430
556299721,54cyclictest0-21swapper/021:46:110
5562997211,44cyclictest18934-21unixbench_singl23:20:520
556299712,54cyclictest0-21swapper/023:11:560
556299711,58cyclictest18018-21basename19:45:210
556299711,56cyclictest14857-21awk21:05:380
556299711,4cyclictest1632-21ssh00:26:100
556299702,56cyclictest0-21swapper/000:02:170
556299702,56cyclictest0-21swapper/000:02:170
5562997013,47cyclictest0-21swapper/019:12:530
41512700,1sleep00-21swapper/019:05:410
556299696,59cyclictest0-21swapper/023:45:380
556299691,5cyclictest12548-21chrt20:59:430
556299691,5cyclictest12548-21chrt20:59:430
556299691,4cyclictest0-21swapper/022:34:240
556299691,3cyclictest122812sleep019:26:250
556299691,3cyclictest122812sleep019:26:250
556299682,51cyclictest0-21swapper/023:41:220
556299681,54cyclictest0-21swapper/022:23:420
556299681,4cyclictest259122sleep021:21:380
556299675,36cyclictest0-21swapper/020:15:230
556299661,50cyclictest29061-21apt-get20:15:310
556299661,4cyclictest0-21swapper/020:26:160
556299651,4cyclictest0-21swapper/020:01:140
556399644,29cyclictest7950irq/56-dwc_otg_23:30:221
556399641,37cyclictest23433-21ssh22:45:261
556399641,34cyclictest23208-21sshd21:18:521
556299642,4cyclictest0-21swapper/020:06:430
556499637,12cyclictest7850irq/56-dwc_otg21:10:452
556499632,5cyclictest31965-21unixbench_multi20:20:512
556399634,49cyclictest31600-21/usr/sbin/munin20:20:471
556399632,45cyclictest27212-21runrttasks22:50:241
556399631,47cyclictest18822-21ssh00:05:151
556399631,47cyclictest18822-21ssh00:05:151
556399631,43cyclictest22742-21munin-node00:10:351
556399631,39cyclictest0-21swapper/122:21:031
556299631,4cyclictest0-21swapper/020:51:240
556399621,49cyclictest15715-21ssh23:16:591
556399621,41cyclictest19682-21ssh23:22:181
556399621,39cyclictest19185-21ssh21:13:241
556399621,37cyclictest31355-21ssh21:30:071
556399621,36cyclictest10514-21chrt20:51:431
556299626,43cyclictest14408-21modprobe21:05:210
556399611,50cyclictest3260-21latency_hist00:30:221
556399611,49cyclictest21031-21ls19:50:511
556299611,4cyclictest0-21swapper/019:36:470
556499605,4cyclictest0-21swapper/221:35:442
556399601,45cyclictest30334-21ls00:20:561
556399601,45cyclictest27265-21apt-get20:10:361
556399601,44cyclictest27918-21runrttasks22:07:411
556399601,42cyclictest2804-21munin-plugin-st21:35:211
556399601,40cyclictest9811-21/usr/sbin/munin19:20:461
556399601,39cyclictest11619-21ssh23:11:361
556399601,39cyclictest1092-21ssh22:58:191
556499596,3cyclictest0-21swapper/223:11:142
556499592,3cyclictest0-21swapper/222:20:402
556499591,6cyclictest21116-21grep19:50:522
556499591,5cyclictest6146-21latency20:40:432
556499591,4cyclictest8109-21unixbench_singl21:40:522
556499591,4cyclictest8109-21unixbench_singl21:40:522
556399591,43cyclictest14929-21ssh23:59:561
556399591,40cyclictest25480-21ssh23:30:371
556399591,36cyclictest7795-21switchtime21:40:491
556399591,36cyclictest7795-21switchtime21:40:491
556399591,36cyclictest1326-21if_wlan022:15:421
556399591,33cyclictest29068-21apt-get23:35:371
556499582,5cyclictest0-21swapper/223:35:422
556499582,4cyclictest23116-21fschecks_time00:10:432
556499581,3cyclictest0-21swapper/221:25:372
556499581,17cyclictest0-21swapper/222:02:262
556399582,35cyclictest17803-21ssh22:37:071
556399581,39cyclictest30424-21sshd20:17:521
556399581,37cyclictest9720-21sh00:39:541
556399581,36cyclictest14642-21chrt21:05:321
556499578,5cyclictest31873-21open_inodes22:55:472
556499574,5cyclictest7850irq/56-dwc_otg00:15:372
556499574,4cyclictest8050irq/56-dwc_otg_22:50:522
556499572,9cyclictest8050irq/56-dwc_otg_19:10:362
556499571,8cyclictest7850irq/56-dwc_otg19:56:192
556499571,6cyclictest6839-21cut00:35:372
556499571,17cyclictest0-21swapper/222:31:502
5564995711,5cyclictest10164-21timerandwakeup19:20:522
556399573,44cyclictest30863-1kworker/u9:023:46:221
556399572,43cyclictest0-21swapper/122:33:471
556399572,34cyclictest7950irq/56-dwc_otg_21:35:411
556399571,46cyclictest23097-21ssh22:45:121
556399571,46cyclictest23097-21ssh22:45:121
556399571,38cyclictest5655-21apt-get20:40:341
556399571,35cyclictest11135-21sh23:54:401
556399571,35cyclictest11135-21sh23:54:401
556399571,34cyclictest7446-21/usr/sbin/munin20:45:531
556499567,5cyclictest7950irq/56-dwc_otg_23:30:442
556499566,4cyclictest0-21swapper/223:54:052
556499566,4cyclictest0-21swapper/223:54:052
556499563,16cyclictest0-21swapper/221:32:412
556499562,11cyclictest0-21swapper/222:40:142
556499561,5cyclictest14211-21ssh23:58:192
556499561,5cyclictest11687-21ls20:55:422
556499561,5cyclictest11687-21ls20:55:422
556499561,4cyclictest7950irq/56-dwc_otg_19:15:382
556499561,4cyclictest0-21swapper/222:45:342
556499561,18cyclictest0-21swapper/223:17:042
556499560,3cyclictest7850irq/56-dwc_otg21:05:432
556399561,36cyclictest5179-21taskset20:37:471
556499555,4cyclictest7950irq/56-dwc_otg_21:20:452
556499553,5cyclictest8050irq/56-dwc_otg_20:00:372
556499552,6cyclictest0-21swapper/222:15:452
556499552,5cyclictest7950irq/56-dwc_otg_23:03:482
556499551,6cyclictest14418-21munin-run21:05:212
556499551,5cyclictest14588-21df21:50:362
556399551,32cyclictest19605-21taskset19:47:591
556499548,4cyclictest0-21swapper/200:04:292
556499548,4cyclictest0-21swapper/200:04:292
556499541,2cyclictest0-21swapper/200:23:322
556399543,31cyclictest29853-21ssh22:10:421
556399541,49cyclictest0-21swapper/119:10:221
556399541,43cyclictest12268-21ssh21:46:381
556399541,37cyclictest20164-21ssh21:57:201
556499532,5cyclictest7850irq/56-dwc_otg22:10:482
556399533,39cyclictest7950irq/56-dwc_otg_20:57:071
556399533,39cyclictest7950irq/56-dwc_otg_20:57:071
556399532,41cyclictest7950irq/56-dwc_otg_23:03:491
556399532,41cyclictest0-21swapper/121:54:421
556399531,41cyclictest0-21swapper/123:07:151
556399531,38cyclictest7850irq/56-dwc_otg21:05:211
556399531,37cyclictest27398-21ssh21:24:451
556599526,11cyclictest7950irq/56-dwc_otg_00:10:373
556499527,5cyclictest5407-21latency_hist20:40:222
556499522,4cyclictest0-21swapper/222:08:072
556499521,8cyclictest8050irq/56-dwc_otg_21:59:212
556499521,5cyclictest0-21swapper/223:40:352
556499521,3cyclictest0-21swapper/222:42:342
556499521,3cyclictest0-21swapper/222:42:342
556499521,3cyclictest0-21swapper/220:15:342
556499516,4cyclictest7850irq/56-dwc_otg23:49:062
556499513,4cyclictest9550irq/48-DMA08:47:302
556499511,6cyclictest7724-21ssh23:06:252
556499511,10cyclictest12470-21chrt19:28:522
556499511,10cyclictest12470-21chrt19:28:522
5563995121,27cyclictest3680-21latency_hist20:35:211
556399511,38cyclictest0-21swapper/120:04:021
556399511,28cyclictest0-21swapper/100:34:591
556499507,7cyclictest8050irq/56-dwc_otg_00:26:202
556499505,3cyclictest7950irq/56-dwc_otg_21:49:402
556499501,3cyclictest7850irq/56-dwc_otg00:07:442
556399502,34cyclictest0-21swapper/119:55:541
556399502,32cyclictest18009-21latency_hist19:45:221
556399500,43cyclictest8366-21ssh22:25:351
556499496,7cyclictest8050irq/56-dwc_otg_21:16:182
556499494,9cyclictest17763-21taskset19:42:232
556499493,4cyclictest0-21swapper/223:26:462
556499492,15cyclictest0-21swapper/220:12:472
556499491,3cyclictest0-21swapper/200:35:042
556499491,2cyclictest0-21swapper/222:30:052
556499491,11cyclictest18227-21chrt19:45:272
60372480,4sleep3381ktimersoftd/322:21:073
556499481,13cyclictest0-21swapper/220:54:402
556399482,41cyclictest11050-21apt-get19:25:321
556399482,41cyclictest11050-21apt-get19:25:321
556399481,32cyclictest0-21swapper/119:36:511
556599477,25cyclictest0-21swapper/320:10:383
556499477,26cyclictest0-21swapper/219:30:362
556499471,5cyclictest0-21swapper/223:21:012
556399471,33cyclictest16225-21diskmemload23:41:011
556399471,29cyclictest7950irq/56-dwc_otg_19:32:291
301362470,1sleep30-21swapper/322:10:443
556599467,25cyclictest29091-21sh23:35:383
45942460,5sleep3556599cyclictest21:36:143
556599456,24cyclictest0-21swapper/321:05:373
556399451,41cyclictest0-21swapper/120:05:341
556399451,33cyclictest1487-21runrttasks20:26:341
556599437,22cyclictest0-21swapper/322:25:373
251892430,2sleep30-21swapper/322:46:403
556599427,23cyclictest0-21swapper/320:45:383
556599427,19cyclictest0-21swapper/320:40:373
556599422,4cyclictest7950irq/56-dwc_otg_20:00:343
556599411,4cyclictest7850irq/56-dwc_otg21:10:353
556499411,11cyclictest0-21swapper/220:46:192
556599407,23cyclictest0-21swapper/323:15:393
556599399,18cyclictest0-21swapper/322:08:323
556599397,23cyclictest0-21swapper/319:25:353
556599397,23cyclictest0-21swapper/319:25:353
556599387,21cyclictest0-21swapper/319:45:363
556599381,27cyclictest8054-21ssh23:50:363
556599381,27cyclictest8054-21ssh23:50:363
131662380,1sleep30-21swapper/321:48:303
556599377,23cyclictest0-21swapper/320:15:333
556599375,21cyclictest0-21swapper/321:24:113
556599371,26cyclictest17999-21cut23:20:413
556499371,12cyclictest0-21swapper/219:36:562
556599367,5cyclictest7850irq/56-dwc_otg21:25:473
556599367,15cyclictest7950irq/56-dwc_otg_00:30:503
5565993616,6cyclictest16225-21diskmemload22:37:553
5565993615,4cyclictest21846-21sh00:08:563
556599361,25cyclictest9763-21latency19:20:453
556599354,3cyclictest8050irq/56-dwc_otg_22:30:343
5565993519,11cyclictest37-21rcuc/319:10:343
556599351,23cyclictest5270-21ssh23:45:503
5565993512,12cyclictest7271-21cat23:05:533
556499351,13cyclictest0-21swapper/220:34:472
556599349,6cyclictest0-21swapper/322:04:093
556599348,4cyclictest10738-21ntpq23:10:483
556599347,24cyclictest0-21swapper/320:30:373
556599345,7cyclictest7950irq/56-dwc_otg_19:30:413
556599342,4cyclictest0-21swapper/300:36:193
556599341,28cyclictest16576-21ls00:00:503
556599341,28cyclictest16576-21ls00:00:503
556599341,23cyclictest12164-21cut23:55:403
5565993411,13cyclictest16225-21diskmemload23:28:153
5565993411,11cyclictest14339-21/usr/sbin/munin21:50:453
556599335,19cyclictest27323-21ssh22:50:363
556599331,23cyclictest8449-21timerwakeupswit19:15:523
5564993313,5cyclictest29-21rcuc/220:25:442
556599329,4cyclictest0-21swapper/300:19:023
556599325,6cyclictest8050irq/56-dwc_otg_21:15:523
556599324,6cyclictest7950irq/56-dwc_otg_19:55:393
556599324,4cyclictest720-21sort23:40:413
556599321,7cyclictest25500-21diskstats23:30:393
556599321,21cyclictest0-21swapper/322:16:563
556599319,10cyclictest0-21swapper/300:28:353
556599317,10cyclictest8050irq/56-dwc_otg_22:40:403
556599317,10cyclictest8050irq/56-dwc_otg_22:40:403
5565993117,5cyclictest381ktimersoftd/319:38:493
5565993113,10cyclictest39-21ksoftirqd/321:05:213
556599311,14cyclictest7950irq/56-dwc_otg_23:00:223
556599309,5cyclictest0-21swapper/321:55:383
556599307,14cyclictest0-21swapper/320:20:363
556599302,3cyclictest0-21swapper/323:00:423
556599302,15cyclictest1737-21ssh21:33:113
556599302,14cyclictest7850irq/56-dwc_otg20:25:523
5565993012,9cyclictest17011-21missed_timers19:40:443
556599295,10cyclictest8050irq/56-dwc_otg_20:55:413
556599295,10cyclictest8050irq/56-dwc_otg_20:55:413
556599292,18cyclictest0-21swapper/320:06:193
556599291,19cyclictest9519-21uniq20:50:413
556599282,5cyclictest7850irq/56-dwc_otg20:35:383
556599282,13cyclictest8050irq/56-dwc_otg_19:50:343
556599281,16cyclictest0-21swapper/321:40:313
556599281,16cyclictest0-21swapper/321:40:313
52532274,14sleep10-21swapper/119:09:521
51882274,15sleep30-21swapper/319:09:013
37272263,13sleep20-21swapper/219:05:282
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional