You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-27 - 03:25

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Sat Jul 27, 2024 00:45:08)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
667399593,38cyclictest26306-21sh23:35:101
6665995620,8cyclictest131rcu_preempt22:35:130
668399521,35cyclictest8310-21ssh21:55:112
6665995237,8cyclictest0-21swapper/022:55:140
668399503,38cyclictest22146irq/40-dwc2_hso22:45:122
6692994818,18cyclictest25866irq/58-eth023:40:123
667399463,3cyclictest131rcu_preempt20:10:101
667399453,33cyclictest22709-21ssh21:45:101
666599457,23cyclictest12-21ksoftirqd/023:20:150
666599457,23cyclictest12-21ksoftirqd/023:20:140
6665994511,4cyclictest131rcu_preempt21:10:120
668399421,31cyclictest0-21swapper/221:40:122
668399410,17cyclictest22146irq/40-dwc2_hso20:10:122
6673994127,9cyclictest23685-21ssh00:10:151
667399401,27cyclictest2863-21ssh22:10:091
668399392,30cyclictest25762-21users22:20:312
669299381,19cyclictest25866irq/58-eth023:45:123
6673993823,12cyclictest21-21ksoftirqd/121:20:101
667399381,25cyclictest8222-21ssh00:20:101
666599371,33cyclictest6384-21sh22:30:120
668399362,15cyclictest8142-21ssh23:25:122
668399362,15cyclictest8142-21ssh23:25:112
6665993613,8cyclictest131rcu_preempt23:00:120
666599352,29cyclictest22518-21rm23:15:120
6692993425,5cyclictest25866irq/58-eth021:30:163
6692993416,12cyclictest25866irq/58-eth021:15:113
666599344,7cyclictest131rcu_preempt20:20:130
6683993314,14cyclictest131rcu_preempt22:35:132
6683993313,14cyclictest6377-21ssh22:30:122
6673993320,6cyclictest522-21usb-storage22:40:101
6665993317,13cyclictest12-21ksoftirqd/019:30:100
666599331,28cyclictest19212-21ssh21:25:120
668399327,20cyclictest22146irq/40-dwc2_hso22:40:112
668399323,18cyclictest22146irq/40-dwc2_hso20:30:122
666599322,25cyclictest12-21ksoftirqd/021:35:110
669299312,18cyclictest25866irq/58-eth022:20:113
668399314,17cyclictest22146irq/40-dwc2_hso21:15:132
668399312,22cyclictest0-21swapper/219:20:122
667399312,22cyclictest4528-21ssh23:05:131
666599313,23cyclictest28253-21ssh21:30:160
666599312,20cyclictest12-21ksoftirqd/019:20:120
666599311,28cyclictest24398-21ssh22:40:120
666599311,22cyclictest13784-21ssh21:40:120
669299304,17cyclictest0-21swapper/321:40:123
668399303,15cyclictest22146irq/40-dwc2_hso20:50:112
666599303,18cyclictest8241-21ssh00:20:110
6665993017,11cyclictest12-21ksoftirqd/020:50:120
668399295,14cyclictest22146irq/40-dwc2_hso19:30:112
668399291,19cyclictest0-21swapper/222:15:112
667399292,19cyclictest14964-21ssh00:05:111
6665992915,6cyclictest0-21swapper/019:50:110
669299285,9cyclictest25866irq/58-eth021:45:093
669299282,18cyclictest25866irq/58-eth022:40:123
668399284,14cyclictest22146irq/40-dwc2_hso23:55:132
668399283,20cyclictest0-21swapper/200:23:342
668399282,21cyclictest3413-21sed21:50:322
668399282,21cyclictest30059-21ssh21:48:312
668399282,16cyclictest22146irq/40-dwc2_hso23:00:122
668399281,18cyclictest22146irq/40-dwc2_hso21:05:132
667399282,21cyclictest30275-21ssh22:05:321
666599283,21cyclictest1272-21ssh21:15:120
666599281,19cyclictest12-21ksoftirqd/019:25:140
669299272,21cyclictest25866irq/58-eth000:30:103
668399273,20cyclictest0-21swapper/221:38:322
668399273,19cyclictest22146irq/40-dwc2_hso00:35:102
668399273,15cyclictest22146irq/40-dwc2_hso19:35:122
668399271,21cyclictest13516-21ssh23:10:122
6683992712,12cyclictest14621-21cron21:00:022
667399272,19cyclictest28233-21rm22:23:311
667399271,21cyclictest24299-21diskmemload22:25:331
666599272,19cyclictest28232-21sh22:23:320
666599271,15cyclictest958-21ssh00:35:120
669299262,18cyclictest0-21swapper/321:39:313
6692992614,8cyclictest25866irq/58-eth023:10:113
669299261,15cyclictest0-21swapper/323:30:163
668399263,13cyclictest22146irq/40-dwc2_hso22:00:122
668399262,9cyclictest0-21swapper/219:25:162
668399262,16cyclictest4134-21find20:15:122
667399263,15cyclictest25229-21ssh00:30:101
666599262,20cyclictest29804-21ssh23:55:150
666599261,21cyclictest10250-21ssh21:20:120
6665992612,11cyclictest20528-21sendmail-msp22:20:010
669299253,18cyclictest25866irq/58-eth023:55:143
668399253,16cyclictest22146irq/40-dwc2_hso00:00:122
668399252,8cyclictest1495-21ssh21:31:332
668399250,13cyclictest15171-21uniq00:05:142
667399252,14cyclictest21-21ksoftirqd/120:45:131
667399251,19cyclictest32218-21ssh00:15:131
667399251,19cyclictest32218-21ssh00:15:121
666599251,4cyclictest0-21swapper/019:15:110
666599251,15cyclictest12-21ksoftirqd/020:25:110
669299243,16cyclictest0-21swapper/323:05:123
669299242,16cyclictest25866irq/58-eth020:25:123
6692992413,7cyclictest25866irq/58-eth021:25:133
668399243,16cyclictest0-21swapper/200:34:332
668399242,16cyclictest0-21swapper/200:10:152
668399241,7cyclictest12594-21ls20:20:302
668399241,19cyclictest25969-21apt-config23:35:012
668399240,20cyclictest6842-21ls19:40:302
667399242,18cyclictest31734-21ssh21:50:131
667399242,16cyclictest20944-21ssh23:30:291
6673992414,7cyclictest6904-21ssh21:35:211
6673992411,10cyclictest27929-21cron20:40:011
666599244,17cyclictest12-21ksoftirqd/020:45:100
666599242,15cyclictest14934-21ssh00:05:110
669299234,10cyclictest0-21swapper/322:50:093
669299233,13cyclictest25866irq/58-eth022:35:133
668399239,12cyclictest0-21swapper/220:25:242
668399232,16cyclictest0-21swapper/200:26:332
6683992312,9cyclictest4530-21ssh23:21:072
6683992312,9cyclictest4530-21ssh23:21:072
668399231,13cyclictest26388-21ssh22:05:152
6683992310,10cyclictest0-21swapper/220:35:292
667399232,15cyclictest4613-21ssh22:45:281
6673992314,7cyclictest20490-21cron22:20:021
6673992311,10cyclictest23286-21ssh23:15:201
666599234,11cyclictest12-21ksoftirqd/023:10:110
666599231,17cyclictest29889-21ssh22:25:140
666599231,17cyclictest2887-21ssh23:40:100
666599231,15cyclictest17290-21ssh23:30:160
669299222,16cyclictest25866irq/58-eth000:20:103
669299222,13cyclictest25866irq/58-eth021:50:133
669299221,18cyclictest25866irq/58-eth022:30:113
6692992211,5cyclictest0-21swapper/300:05:113
668399228,10cyclictest29798-21kthreadcore20:40:212
668399222,6cyclictest0-21swapper/222:28:332
668399222,15cyclictest0-21swapper/222:11:332
6683992216,3cyclictest13233-21ssh21:20:202
6683992211,9cyclictest8287-21ssh23:41:112
6683992211,9cyclictest17762-21ssh22:54:062
668399221,16cyclictest0-21swapper/219:15:112
668399221,12cyclictest19220-21rm21:25:132
667399224,15cyclictest17310-21ssh22:00:111
667399222,4cyclictest8958-21apt-get20:20:131
667399221,18cyclictest17526-21ls23:10:291
667399220,15cyclictest1257-21ssh21:15:121
666599222,16cyclictest9915-21ssh22:50:100
6665992215,4cyclictest27234-21usage_EthcTask21:45:300
6665992210,3cyclictest12-21ksoftirqd/000:20:020
6665992210,3cyclictest12-21ksoftirqd/000:20:010
669299212,13cyclictest25866irq/58-eth022:25:143
6692992111,6cyclictest0-21swapper/300:00:123
669299211,13cyclictest25866irq/58-eth019:30:133
669299211,12cyclictest25866irq/58-eth022:15:113
668399219,5cyclictest24193-21cut23:15:212
668399212,5cyclictest0-21swapper/223:35:122
668399212,11cyclictest0-21swapper/219:10:122
668399212,10cyclictest19101-21kthreadcore19:55:222
6683992116,3cyclictest27544-21ssh23:00:012
668399211,16cyclictest22554-21ps20:00:182
668399211,11cyclictest6507-21kthreadcore23:05:212
6683992110,5cyclictest6378-21ssh00:18:002
6683992110,5cyclictest6378-21ssh00:18:002
667399215,3cyclictest131rcu_preempt23:00:281
667399211,7cyclictest14106-21grep22:50:301
667399211,17cyclictest10386-21ls22:30:291
667399211,15cyclictest22503-21ls21:45:011
666599212,6cyclictest52246usb-storage21:00:050
666599211,2cyclictest131rcu_preempt19:40:130
6665992112,6cyclictest0-21swapper/021:55:120
666599211,17cyclictest7415-21latency20:15:230
666599211,15cyclictest11808-21ssh23:45:130
666599211,14cyclictest25236-21sh00:30:100
666599211,12cyclictest31866-21ssh22:06:180
6665992110,8cyclictest478-21cat21:50:220
669299209,8cyclictest0-21swapper/322:05:263
669299202,11cyclictest25866irq/58-eth023:50:103
6692992011,7cyclictest0-21swapper/322:46:273
6692992011,6cyclictest25866irq/58-eth023:25:133
6692992011,6cyclictest25866irq/58-eth023:25:123
669299201,10cyclictest25866irq/58-eth023:00:113
668399209,9cyclictest25629-21ssh21:10:192
668399208,9cyclictest23378-21cut23:50:212
668399203,7cyclictest30699-21timerandwakeup20:05:302
668399201,9cyclictest12864-21hwlatdetect19:50:162
668399201,15cyclictest1336-21kthreadcore20:45:202
668399201,10cyclictest8833-21kthreadcore19:45:182
667399202,15cyclictest22333-21fschecks_count20:00:141
667399201,15cyclictest23934-21munin-run22:40:021
667399201,15cyclictest13741-21apt-get20:25:121
666599202,7cyclictest12-21ksoftirqd/019:35:180
6665992012,6cyclictest21962-21sshd19:57:310
666599201,15cyclictest3156-21grep20:10:330
666599201,14cyclictest13015-21users23:25:330
666599201,14cyclictest13015-21users23:25:320
669299196,8cyclictest25866irq/58-eth021:10:123
669299192,12cyclictest0-21swapper/322:10:323
669299192,12cyclictest0-21swapper/322:02:333
669299192,11cyclictest0-21swapper/323:35:103
669299192,11cyclictest0-21swapper/323:20:153
669299192,11cyclictest0-21swapper/323:20:143
6692991912,4cyclictest33-21ksoftirqd/322:55:193
669299191,12cyclictest0-21swapper/321:05:243
668399191,15cyclictest11915-21df23:45:152
668399191,10cyclictest15795-21kthreadcore21:00:212
667399195,3cyclictest131rcu_preempt00:35:181
667399192,13cyclictest18625-21kthreadcore00:25:191
667399192,12cyclictest31857-21sh21:30:331
6673991911,6cyclictest30501-21sshd19:34:321
667399191,15cyclictest897-21ls23:55:251
6673991911,5cyclictest11763-21/usr/sbin/munin23:45:251
667399190,17cyclictest12279-21ls21:55:341
666599195,11cyclictest8084-21wc19:45:150
666599192,14cyclictest4915-21ps23:05:170
666599192,11cyclictest0-21swapper/021:05:130
666599191,9cyclictest28166-21/usr/sbin/munin20:40:210
666599191,5cyclictest26780-21idleruntime-cro20:05:010
666599191,2cyclictest131rcu_preempt00:25:100
6665991912,4cyclictest0-21swapper/000:00:120
6665991911,6cyclictest29309-21sshd00:11:330
6665991910,6cyclictest18079-21kthreadcore22:00:200
669299182,11cyclictest0-21swapper/300:36:333
669299182,11cyclictest0-21swapper/300:25:123
669299182,10cyclictest0-21swapper/323:15:133
669299182,10cyclictest0-21swapper/321:00:053
669299182,10cyclictest0-21swapper/320:20:143
669299182,10cyclictest0-21swapper/319:40:153
669299182,10cyclictest0-21swapper/319:35:123
669299181,13cyclictest0-21swapper/320:10:323
6692991810,6cyclictest0-21swapper/321:20:043
6692991810,6cyclictest0-21swapper/300:11:593
667399189,6cyclictest11839-21ssh23:25:261
667399189,6cyclictest11839-21ssh23:25:251
667399188,8cyclictest19198-21apt-get21:25:121
667399185,10cyclictest14530-21diskmemload00:05:021
667399184,11cyclictest25094-21irqcore21:10:161
667399183,12cyclictest22274-21kthreadcore22:55:261
667399181,9cyclictest29764-21tr20:40:211
667399181,14cyclictest8228-21irqcore19:45:161
667399181,14cyclictest30848-21cut19:35:141
6673991811,4cyclictest0-21swapper/119:15:261
667399180,10cyclictest6655-21cyclictest19:10:011
666599184,12cyclictest1371-21irqcore22:45:190
666599183,12cyclictest22783-21ps23:50:190
666599182,2cyclictest0-21swapper/020:55:140
666599182,13cyclictest30184-21kthreadcore20:05:220
666599181,14cyclictest3565-21irqstats22:10:160
666599181,14cyclictest27524-21kthreadcore23:35:180
666599181,12cyclictest28054-21ls20:40:010
669299177,8cyclictest0-21swapper/321:55:033
669299172,9cyclictest0-21swapper/320:45:123
669299172,9cyclictest0-21swapper/319:10:133
669299172,10cyclictest0-21swapper/320:50:133
6692991712,3cyclictest33-21ksoftirqd/320:15:283
6692991711,4cyclictest0-21swapper/300:15:313
6692991711,4cyclictest0-21swapper/300:15:313
669299171,12cyclictest0-21swapper/320:35:023
669299171,11cyclictest0-21swapper/319:20:153
669299171,10cyclictest0-21swapper/320:55:133
668399171,7cyclictest6702-21latency_hist19:10:022
667399179,5cyclictest14870-21kthreadcore19:50:221
667399174,3cyclictest6655-21cyclictest20:15:191
667399174,10cyclictest9034-21kthreadcore19:10:211
667399172,12cyclictest2897-21ssh23:40:111
667399171,9cyclictest22684-21kthreadcore23:50:201
667399171,5cyclictest0-21swapper/119:20:251
6673991712,3cyclictest17733-21if_eth119:55:181
6673991711,4cyclictest0-21swapper/123:20:341
6673991711,4cyclictest0-21swapper/123:20:331
666599171,3cyclictest131rcu_preempt20:30:050
666599171,12cyclictest11506-21latency_hist19:15:020
669299161,12cyclictest0-21swapper/319:25:133
669299161,12cyclictest0-21swapper/319:15:103
667399169,5cyclictest14608-21sshd20:55:321
667399163,10cyclictest6740-21threads19:40:311
667399163,10cyclictest0-21swapper/121:00:131
667399162,9cyclictest632-21Async17:44:201
667399162,12cyclictest7979-21kthreadcore20:50:221
667399161,12cyclictest21176-21apt-get19:25:131
667399161,12cyclictest20139-21cut21:05:191
6692991510,3cyclictest33-21ksoftirqd/319:55:243
667399153,9cyclictest18661-21cpuspeed20:30:151
666599151,11cyclictest6710-21latency_hist19:10:010
669299149,3cyclictest33-21ksoftirqd/319:46:053
669299141,11cyclictest0-21swapper/320:35:143
669299138,3cyclictest0-21swapper/320:00:163
669299138,3cyclictest0-21swapper/319:50:243
669299131,7cyclictest0-21swapper/320:40:123
669299125,5cyclictest33-21ksoftirqd/320:05:113
669299111,7cyclictest0-21swapper/319:10:013
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional