You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-23 - 18:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot4s.osadl.org (updated Tue Apr 23, 2024 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
19575996020,19cyclictest27-21ksoftirqd/211:25:132
19575995417,27cyclictest27-21ksoftirqd/207:35:132
1956699482,38cyclictest0-21swapper/109:20:141
1956699481,5cyclictest0-21swapper/108:45:131
1956699473,40cyclictest17827-21apt-get11:25:131
1955799471,3cyclictest20781-21ssh09:20:140
19575994611,17cyclictest27-21ksoftirqd/211:10:112
1955799462,3cyclictest11846-21apt-get11:00:140
1956699442,37cyclictest25429-21apt-get09:45:111
1956699442,37cyclictest25429-21apt-get09:45:111
19575994314,19cyclictest27-21ksoftirqd/210:55:132
1955799431,4cyclictest0-21swapper/008:45:130
19575994215,10cyclictest8361-21ssh09:31:462
1956699421,3cyclictest0-21swapper/109:50:121
1956699421,35cyclictest0-21swapper/111:55:121
1955799424,3cyclictest131rcu_preempt10:35:120
1955799422,4cyclictest0-21swapper/008:50:110
1955799421,8cyclictest0-21swapper/008:05:130
19575994116,13cyclictest0-21swapper/211:40:112
1956699411,36cyclictest29537-21apt-get10:30:111
1957599402,32cyclictest2151-21ssh10:52:462
1957599392,19cyclictest10541-21cat10:15:312
1955799391,3cyclictest0-21swapper/009:15:120
19586993818,13cyclictest26866irq/58-eth009:45:123
19586993818,13cyclictest26866irq/58-eth009:45:123
1957599381,21cyclictest10378-21ssh09:35:122
1956699383,31cyclictest5397-21ssh09:10:131
1956699382,3cyclictest12593-21apt-get08:10:131
1956699381,3cyclictest0-21swapper/107:10:151
1956699381,33cyclictest0-21swapper/107:55:081
1955799381,4cyclictest0-21swapper/011:35:230
1957599373,19cyclictest0-21swapper/209:46:452
1957599373,19cyclictest0-21swapper/209:46:452
19575993718,12cyclictest27-21ksoftirqd/208:40:132
1956699373,28cyclictest30558-21apt-get10:10:121
1955799373,3cyclictest0-21swapper/007:55:080
1955799372,4cyclictest0-21swapper/008:35:140
1955799371,4cyclictest8947-21apt-get08:40:130
1955799371,4cyclictest0-21swapper/009:10:120
1955799371,2cyclictest0-21swapper/012:05:110
1957599362,18cyclictest18100-21cat10:20:332
19575993615,7cyclictest19548-21cyclictest12:25:112
1957599361,28cyclictest29222-21ssh11:51:462
1956699360,2cyclictest0-21swapper/108:40:131
1956699360,2cyclictest0-21swapper/108:35:131
1955799362,4cyclictest0-21swapper/010:40:140
1955799362,4cyclictest0-21swapper/010:40:140
1955799362,3cyclictest0-21swapper/010:05:130
1955799362,31cyclictest26018-21wc12:10:330
1955799361,3cyclictest0-21swapper/008:10:130
1955799361,29cyclictest1645-21apt-get07:25:120
1958699350,19cyclictest26866irq/58-eth009:55:103
1957599354,16cyclictest22046irq/40-dwc2_hso11:35:132
19575993523,7cyclictest27-21ksoftirqd/212:10:152
1957599351,10cyclictest630-21Async19:56:212
1956699352,28cyclictest11291-21apt-get07:35:131
1956699351,3cyclictest0-21swapper/111:00:141
1956699351,3cyclictest0-21swapper/108:20:111
1955799355,27cyclictest6591-21sed08:00:290
1955799355,27cyclictest1-21systemd07:19:570
1955799352,4cyclictest0-21swapper/009:30:110
1955799351,2cyclictest397-21apt-get09:50:120
1958699342,25cyclictest26866irq/58-eth010:00:453
1956699341,4cyclictest0-21swapper/108:05:131
1956699341,3cyclictest0-21swapper/110:45:161
1956699341,2cyclictest19379-21apt-get11:05:131
1956699341,2cyclictest0-21swapper/109:30:111
1956699341,21cyclictest92566ssh11:13:151
1956699341,17cyclictest10372-21apt-get09:35:111
1956699340,2cyclictest22266-21ssh12:10:161
1955799347,24cyclictest19283-21cut11:45:230
1955799342,3cyclictest0-21swapper/010:30:120
1955799341,3cyclictest7961-21apt-get11:40:040
1955799341,3cyclictest0-21swapper/011:55:120
1955799341,3cyclictest0-21swapper/007:35:130
1955799341,2cyclictest0-21swapper/010:10:120
1955799341,2cyclictest0-21swapper/008:30:100
1955799341,12cyclictest1341-21ntp_states12:15:280
1955799340,10cyclictest28114-21apt-get09:00:130
1958699331,21cyclictest26866irq/58-eth007:10:153
1957599332,15cyclictest28233-21ssh10:05:462
1957599331,25cyclictest0-21swapper/212:00:132
19575993311,16cyclictest19548-21cyclictest11:30:112
1956699331,3cyclictest0-21swapper/110:15:111
1956699331,28cyclictest5567-21usage_Anybus07:25:311
1955799337,24cyclictest17575-21sed11:25:010
1955799337,24cyclictest17575-21sed11:25:010
1955799331,8cyclictest23032-21tr08:20:190
1955799331,10cyclictest6572-21df_inode07:30:140
1955799330,4cyclictest17686-21apt-get09:40:110
1957599327,20cyclictest630-21Async19:56:212
1957599324,18cyclictest22046irq/40-dwc2_hso07:10:172
1957599322,25cyclictest0-21swapper/210:29:462
1956699321,27cyclictest25970-21sed11:30:171
1956699321,11cyclictest5564-21ssh12:20:121
1956699320,2cyclictest0-21swapper/112:15:131
1956699320,2cyclictest0-21swapper/110:55:121
1956699320,18cyclictest31744-21apt-get08:30:101
1955799327,22cyclictest21869-21fschecks_count10:25:180
1955799326,23cyclictest775-21sd-accept10:50:390
1955799322,4cyclictest0-21swapper/011:05:110
1955799321,9cyclictest9925-21ssh10:57:260
1955799321,8cyclictest19411-21ntp_states10:00:320
1955799321,7cyclictest30842-21ls11:10:300
1955799321,7cyclictest23988-21missed_timers10:45:280
1955799321,7cyclictest19682-21proc_pri07:40:280
1955799321,6cyclictest29102-21cat07:50:230
1955799321,11cyclictest11651-21ntp_states09:55:300
1955799321,10cyclictest23122-21ntp_states07:10:300
1955799321,10cyclictest1154-21sed09:05:210
1955799320,9cyclictest20804-21ntp_states08:15:280
1958699313,19cyclictest26866irq/58-eth011:30:123
1956699312,25cyclictest19684-21proc_pri07:40:281
1956699311,8cyclictest1565766ssh11:21:521
1956699311,8cyclictest1565766ssh11:21:521
1956699311,27cyclictest11167-21latency_hist07:35:021
1956699310,3cyclictest29353-21apt-get12:35:111
1956699310,2cyclictest21481-21ssh12:30:121
1956699310,2cyclictest0-21swapper/108:15:121
1956699310,11cyclictest21717-21ssh12:09:461
1955799315,23cyclictest13851-21ls09:35:230
1955799314,24cyclictest23929-21irqrtprio08:55:170
1955799311,7cyclictest7407-21kernelversion12:00:200
1955799311,7cyclictest14168-21irqcore10:20:200
1955799311,6cyclictest7242-21kthreadcore10:15:190
1955799311,6cyclictest6174-21irqrtprio12:20:190
1955799311,6cyclictest24308-21missed_timers07:45:250
1955799311,6cyclictest23945-21fschecks_time11:50:160
1955799311,4cyclictest26974-21apt-get08:25:120
1955799311,4cyclictest25512-21grep12:30:310
1955799311,4cyclictest0-21swapper/012:35:130
1955799310,9cyclictest6419-21uptime11:15:310
19586993013,11cyclictest26866irq/58-eth010:28:463
1957599306,12cyclictest0-21swapper/209:15:112
1957599303,18cyclictest22046irq/40-dwc2_hso07:25:132
1957599302,21cyclictest11563-21ntp_192.168.11409:55:282
1957599302,20cyclictest29375-21ssh12:35:132
1956699301,7cyclictest848-21irqcore09:05:201
1956699301,7cyclictest25773-21kthreadcore10:05:231
1956699301,7cyclictest13531-21apt-get12:25:121
1956699301,6cyclictest11860-21missed_timers11:40:241
1956699301,25cyclictest18512-21/usr/sbin/munin08:50:161
1956699301,25cyclictest14169-21irqcore10:20:201
1956699300,2cyclictest0-21swapper/110:35:121
1955799301,7cyclictest13917-21ssh12:25:170
1955799301,6cyclictest27099-21cut11:30:200
1955799301,4cyclictest26185-21ssh09:45:200
1955799301,4cyclictest26185-21ssh09:45:190
1955799301,11cyclictest17747-21ssh11:25:050
1955799300,8cyclictest1094-21ssh09:26:040
1957599293,20cyclictest0-21swapper/209:50:462
19575992922,4cyclictest261rcuc/210:30:122
19575992916,8cyclictest20635-21ssh10:00:452
1956699291,7cyclictest32661-21cat09:25:321
1956699291,2cyclictest6679-21apt-get12:00:121
1956699291,2cyclictest12754-21apt-get10:40:141
1956699291,2cyclictest12754-21apt-get10:40:141
1956699291,2cyclictest0-21swapper/110:25:141
1956699291,25cyclictest31991-21ntp_states10:50:261
1956699291,24cyclictest5800-21ssh11:15:241
1956699290,10cyclictest7996-21/usr/sbin/munin09:55:101
1955799291,4cyclictest32674-21ntp_192.168.11407:20:220
1958699283,18cyclictest0-21swapper/310:30:353
1958699283,16cyclictest0-21swapper/310:53:463
1958699283,16cyclictest0-21swapper/309:31:473
1957599283,20cyclictest0-21swapper/211:57:462
1957599282,19cyclictest0-21swapper/208:10:052
1957599282,16cyclictest0-21swapper/209:05:132
1957599282,15cyclictest19398-21rm11:05:132
1956699281,7cyclictest24809-21cut11:50:191
1956699281,6cyclictest32715-21cat09:05:011
1956699281,6cyclictest23299-21/usr/sbin/munin08:55:271
1956699281,6cyclictest19629-21latency_hist07:10:021
1956699281,6cyclictest16458-21grep10:00:251
1956699281,3cyclictest0-21swapper/108:25:131
1956699281,3cyclictest0-21swapper/107:45:121
1956699281,2cyclictest13075-21apt-get09:15:121
1956699281,24cyclictest4345-21ssh11:35:281
1956699281,24cyclictest1410-21cron07:25:011
1956699280,2cyclictest17684-21ssh09:40:101
1956699280,2cyclictest0-21swapper/111:45:151
1958699273,15cyclictest0-21swapper/310:15:323
1957599273,19cyclictest0-21swapper/207:30:472
1957599273,16cyclictest27-21ksoftirqd/212:05:112
1957599273,15cyclictest22046irq/40-dwc2_hso10:40:142
1957599273,15cyclictest22046irq/40-dwc2_hso10:40:142
1957599273,14cyclictest22046irq/40-dwc2_hso09:00:142
1957599273,13cyclictest22046irq/40-dwc2_hso10:35:112
1957599272,20cyclictest0-21swapper/212:32:472
1957599272,19cyclictest0-21swapper/208:25:142
1957599271,17cyclictest22165-21apt-get08:20:112
1956699271,7cyclictest24657-21fschecks_time07:15:161
1956699271,6cyclictest2937-21grep08:00:121
1956699271,23cyclictest25987-21hwlatdetect07:50:151
1958699269,9cyclictest26866irq/58-eth012:35:133
1957599262,19cyclictest9563-21ssh11:19:462
1957599261,12cyclictest86866snmpd10:49:412
1958699254,11cyclictest321rcuc/307:30:123
1958699253,14cyclictest0-21swapper/310:20:323
1958699253,13cyclictest0-21swapper/312:00:133
19586992510,10cyclictest26866irq/58-eth012:05:123
1957599252,19cyclictest24376-21ssh09:43:462
1955799251,7cyclictest19548-21cyclictest07:10:020
19575992412,9cyclictest4161-21ssh12:18:212
1957599240,14cyclictest27860-21cron09:00:012
1958699232,13cyclictest26866irq/58-eth011:35:123
1958699228,9cyclictest26866irq/58-eth011:20:123
1958699228,9cyclictest26866irq/58-eth011:20:113
1958699226,10cyclictest26866irq/58-eth011:51:463
1957599227,13cyclictest27-21ksoftirqd/211:20:232
1957599227,13cyclictest27-21ksoftirqd/211:20:222
1957599225,9cyclictest2662-21cron08:00:002
1957599221,8cyclictest20417-21kthreadcore08:15:232
1957599221,17cyclictest5051-21diskmemload11:49:462
1957599221,16cyclictest8657-21kthreadcore12:20:242
1957599221,12cyclictest28203-21grep07:15:312
1958699219,8cyclictest26866irq/58-eth011:05:133
1958699218,8cyclictest321rcuc/308:20:113
19586992113,5cyclictest0-21swapper/311:55:433
19586992112,4cyclictest0-21swapper/309:14:233
19586992110,7cyclictest26866irq/58-eth010:55:143
1957599214,8cyclictest27-21ksoftirqd/208:00:122
1957599214,8cyclictest14561-21kthreadcore08:45:202
1957599214,14cyclictest21708-21kthreadcore07:45:212
1957599211,15cyclictest31922-21fschecks_time08:30:142
1957599211,11cyclictest73-21kswapd011:00:152
1957599210,17cyclictest29008-21latency07:50:222
1958699206,8cyclictest26866irq/58-eth009:15:113
1958699203,11cyclictest0-21swapper/312:30:463
1958699202,13cyclictest0-21swapper/311:25:263
19586992011,5cyclictest26866irq/58-eth010:10:113
19586992010,7cyclictest0-21swapper/310:05:173
19586992010,7cyclictest0-21swapper/309:20:153
19586992010,6cyclictest26866irq/58-eth011:15:123
19586992010,6cyclictest0-21swapper/312:20:233
1957599207,10cyclictest25619-21awk09:20:382
1957599202,9cyclictest19610-21kthreadcore08:50:192
1957599201,10cyclictest7577-21cut09:10:222
1958699193,11cyclictest0-21swapper/312:29:463
1958699192,11cyclictest0-21swapper/308:10:053
19586991913,4cyclictest0-21swapper/311:02:253
19586991913,4cyclictest0-21swapper/309:40:563
19586991913,4cyclictest0-21swapper/309:36:143
19586991912,5cyclictest0-21swapper/311:45:123
19586991912,5cyclictest0-21swapper/311:11:543
1958699191,12cyclictest0-21swapper/312:15:253
19586991910,6cyclictest26866irq/58-eth011:40:123
19586991910,6cyclictest0-21swapper/309:25:223
1957599191,9cyclictest7468-21missed_timers08:35:262
1957599191,8cyclictest8200-21irqcore08:05:182
1957599191,10cyclictest30014-21kthreadcore07:20:182
1957599190,5cyclictest16267-21ls07:40:142
1958699189,6cyclictest0-21swapper/310:36:313
1958699182,10cyclictest0-21swapper/308:55:123
1958699182,10cyclictest0-21swapper/308:40:153
1958699182,10cyclictest0-21swapper/308:25:133
1958699182,10cyclictest0-21swapper/307:50:123
1958699182,10cyclictest0-21swapper/307:15:133
19586991812,4cyclictest0-21swapper/309:50:133
1958699181,13cyclictest0-21swapper/309:10:013
1958699181,13cyclictest0-21swapper/307:40:303
1958699181,12cyclictest0-21swapper/308:00:233
19586991810,5cyclictest0-21swapper/310:45:273
1958699179,5cyclictest26866irq/58-eth012:10:143
1958699172,10cyclictest0-21swapper/310:40:143
1958699172,10cyclictest0-21swapper/310:40:143
1958699172,10cyclictest0-21swapper/309:00:153
1958699172,10cyclictest0-21swapper/308:50:133
1958699171,13cyclictest0-21swapper/308:05:233
1958699171,12cyclictest0-21swapper/307:35:303
1958699171,10cyclictest0-21swapper/307:20:113
1957599171,12cyclictest19624-21latency_hist07:10:022
1958699161,9cyclictest0-21swapper/308:35:013
1958699151,11cyclictest0-21swapper/308:00:013
19586991510,3cyclictest0-21swapper/308:15:273
1958699149,3cyclictest0-21swapper/308:45:273
1958699141,9cyclictest0-21swapper/307:45:143
1958699141,7cyclictest0-21swapper/307:25:283
1958699138,3cyclictest0-21swapper/308:35:233
195869991,6cyclictest0-21swapper/307:10:023
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional