You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-01-28 - 08:14

x86 Intel Xeon E3-1220L V2 @2300 MHz, Linux 4.9.47-rt37 (Profile)

Latency plot of system in rack #a, slot #0
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Characteristics of the highest latencies:
System rackaslot0.osadl.org (updated Sat Jan 28, 2023 00:44:38)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
1668621070,0sleep016689-21id21:45:050
297862900,2sleep22260499cyclictest22:26:052
227692830,0sleep20-21swapper/221:16:512
59232810,0sleep25925-21id00:32:052
290522650,1sleep315420-21diskmemload23:46:393
205822640,0sleep041ktimersoftd/021:35:230
264172630,1sleep126418-21sshd21:26:271
113502630,1sleep311328-21sshd00:21:563
8042610,1sleep0802-21bash23:18:030
33302600,1sleep03301-21sshd21:21:080
196062590,1sleep219588-21sshd21:55:162
72492580,1sleep30-21swapper/300:32:143
73162570,2sleep041ktimersoftd/000:07:460
2222925736,18sleep00-21swapper/019:08:320
2222725735,18sleep30-21swapper/319:08:313
116192570,0sleep111620-21sshd23:38:351
212782560,0sleep021274-21bash00:19:370
174942560,1sleep215420-21diskmemload22:11:432
135842550,1sleep113554-21sshd22:01:161
165702540,5sleep230-21ksoftirqd/223:03:352
2210825327,9sleep20-21swapper/219:07:022
136972530,1sleep20-21swapper/222:18:002
116932520,0sleep311692-21bash00:01:043
307402510,1sleep030742-21unixbench_singl20:30:260
2229025125,9sleep10-21swapper/119:09:221
126132510,0sleep10-21swapper/121:54:381
60042500,1sleep015420-21diskmemload23:02:320
315172500,0sleep30-21swapper/323:40:303
230942500,0sleep00-21swapper/000:23:140
173382500,1sleep00-21swapper/022:37:490
139982500,1sleep213993-21sshd22:24:372
111072500,0sleep015420-21diskmemload21:54:300
271482490,0sleep20-21swapper/223:23:542
249962490,1sleep20-21swapper/221:35:502
237672490,0sleep00-21swapper/022:48:100
172032490,0sleep20-21swapper/223:42:162
135112480,0sleep00-21swapper/023:22:310
141852470,1sleep31787-21runrttasks23:28:593
106402460,0sleep10-21swapper/122:56:371
8522440,0sleep10-21swapper/123:30:511
275842410,0sleep230-21ksoftirqd/222:55:052
189782380,1sleep115420-21diskmemload21:19:421
34422230,1sleep33439-21bash21:53:413
22602992321,1cyclictest7661-21sshd23:54:140
2260599220,21cyclictest0-21swapper/322:38:183
22604992119,1cyclictest0-21swapper/200:20:052
2260399215,15cyclictest22910-21sshd00:30:231
22603992010,5cyclictest23196-21sshd21:23:091
2260399200,14cyclictest3641-21sshd21:56:551
2260299200,14cyclictest0-21swapper/022:10:440
22605991917,1cyclictest0-21swapper/300:25:483
2260399190,1cyclictest6385-21sshd22:46:231
2260399190,1cyclictest10107-21sshd22:17:371
302982180,0sleep130242-21sshd21:36:251
22605991817,1cyclictest0-21swapper/323:23:523
22605991814,1cyclictest22664-21bash21:39:053
22604991817,1cyclictest0-21swapper/222:09:292
2260399185,5cyclictest0-21swapper/123:55:051
22603991816,1cyclictest12832-21sshd23:48:171
2260399181,1cyclictest120050irq/25-eth000:19:501
2260399180,2cyclictest3749-21sshd00:28:211
2260399180,17cyclictest0-21swapper/100:24:251
2260299180,18cyclictest0-21swapper/019:55:200
117232180,0sleep38-21rcu_preempt22:50:143
319622170,0sleep031948-21sshd22:29:360
2260599170,16cyclictest120050irq/25-eth022:17:153
22604991715,1cyclictest27496-21sshd00:10:012
2260499171,15cyclictest0-21swapper/222:33:092
2260499171,13cyclictest3173-21bash23:27:522
22604991710,5cyclictest10325-21sshd22:40:202
2260399178,8cyclictest3321-21sshd21:33:411
2260399178,5cyclictest120050irq/25-eth023:13:411
2260399177,5cyclictest16576-21sshd23:03:351
2260399173,1cyclictest120050irq/25-eth022:13:021
2260399172,1cyclictest0-21swapper/121:47:501
2260399171,2cyclictest0-21swapper/122:35:171
2260399171,2cyclictest0-21swapper/122:29:131
2260399171,2cyclictest0-21swapper/100:10:151
2260399171,15cyclictest0-21swapper/123:16:171
2260399171,10cyclictest0-21swapper/123:29:531
2260399170,1cyclictest4168-21sshd23:40:581
2260399170,1cyclictest21954-21sshd21:10:291
2260399170,1cyclictest1632-21sshd22:50:281
2260399170,1cyclictest120050irq/25-eth022:23:421
2260399170,16cyclictest9453-21sshd22:07:321
2260399170,16cyclictest3390-21sshd00:35:241
2260399170,16cyclictest10336-21bash22:40:211
2260399170,10cyclictest29699-21sshd00:03:101
2260299170,1cyclictest31731-21sshd22:42:300
2260499169,4cyclictest0-21swapper/222:48:262
2260499161,2cyclictest23626-21sshd23:55:412
22604991610,5cyclictest6140-21sshd21:24:392
2260499160,13cyclictest0-21swapper/221:29:112
2260499160,11cyclictest26895-21sshd21:42:452
2260399169,6cyclictest25303-21ntp_states19:15:011
2260399161,14cyclictest0-21swapper/123:22:271
2260399160,2cyclictest0-21swapper/123:50:031
2260399160,2cyclictest0-21swapper/122:30:121
2260399160,15cyclictest5132-21sshd23:05:371
2260399160,15cyclictest0-21swapper/100:05:441
2260399160,10cyclictest2657-21rm21:40:181
2260299163,12cyclictest0-21swapper/021:41:040
2260299160,15cyclictest0-21swapper/021:16:550
96102150,6sleep230-21ksoftirqd/223:15:372
2260499159,4cyclictest26697-21sshd00:13:152
2260499153,5cyclictest29208-21sshd00:16:552
2260499153,3cyclictest22444-21sshd21:52:172
2260499152,6cyclictest0-21swapper/221:11:092
2260499152,5cyclictest23844-21bash22:54:422
2260499151,13cyclictest12254-21sshd21:47:512
22604991511,3cyclictest0-21swapper/223:35:572
2260499150,14cyclictest0-21swapper/223:45:142
2260399151,13cyclictest0-21swapper/119:25:151
2260399150,11cyclictest0-21swapper/119:40:281
2260299159,3cyclictest0-21swapper/022:55:040
2260299158,4cyclictest0-21swapper/022:03:160
22602991513,1cyclictest16181-21sshd22:53:570
119712150,0sleep041ktimersoftd/021:57:470
2260599140,5cyclictest0-21swapper/322:27:043
2260599140,2cyclictest25972-21sshd22:05:513
2260599140,1cyclictest15420-21diskmemload21:21:013
2260599140,13cyclictest0-21swapper/323:32:193
2260599140,13cyclictest0-21swapper/321:29:073
2260599140,13cyclictest0-21swapper/321:10:583
2260499149,4cyclictest0-21swapper/223:13:442
2260499148,2cyclictest0-21swapper/222:37:502
2260499142,4cyclictest31816-21sshd23:30:412
2260499142,4cyclictest21450-21sshd21:32:142
2260499142,3cyclictest22671-21sshd00:26:432
2260499141,3cyclictest25961-21bash23:52:502
2260399140,10cyclictest0-21swapper/120:03:561
2260299149,4cyclictest0-21swapper/023:34:530
2260299149,4cyclictest0-21swapper/022:30:230
2260299148,5cyclictest14621-21bash00:01:260
2260299142,2cyclictest32586-21sshd23:40:350
22602991412,1cyclictest25524-21awk23:40:000
2260299140,3cyclictest23785-21sshd23:10:380
2260299140,13cyclictest403-21sshd23:27:360
2260299140,13cyclictest0-21swapper/023:56:400
2260299140,13cyclictest0-21swapper/022:06:170
2260299140,13cyclictest0-21swapper/021:10:020
2260599139,2cyclictest0-21swapper/322:57:483
2260599139,1cyclictest0-21swapper/322:44:153
2260599139,1cyclictest0-21swapper/321:45:423
2260599139,1cyclictest0-21swapper/300:38:423
2260599138,4cyclictest32660-21bash22:49:063
2260599138,4cyclictest1760-21sshd21:30:203
2260599138,2cyclictest30596-21sshd23:56:243
2260599138,2cyclictest15172-21sshd00:08:403
2260599138,2cyclictest0-21swapper/300:18:223
2260599131,9cyclictest599-21sshd22:32:543
22605991311,2cyclictest4990-21sshd22:00:253
22605991310,2cyclictest17281-21sshd22:11:413
2260599130,2cyclictest11789-21sshd22:21:003
2260599130,2cyclictest0-21swapper/323:13:093
2260599130,1cyclictest0-21swapper/323:36:133
2260599130,1cyclictest0-21swapper/323:08:183
2260599130,12cyclictest18861-21sshd23:00:363
2260599130,12cyclictest16279-21sshd21:16:133
2260599130,12cyclictest0-21swapper/323:51:563
2260599130,12cyclictest0-21swapper/300:10:393
2260499132,3cyclictest2637-21sshd22:00:132
2260499132,2cyclictest17260-21bash00:36:592
22604991310,2cyclictest31964-21sshd23:05:092
22604991310,2cyclictest0-21swapper/200:00:092
2260399131,2cyclictest0-21swapper/119:50:001
2260299139,3cyclictest0-21swapper/023:46:460
2260299137,3cyclictest0-21swapper/022:23:070
2260299133,1cyclictest7506-21sshd21:30:510
2260299132,1cyclictest5129-21sshd00:35:350
2260299131,1cyclictest18797-21sshd23:07:010
2260299130,3cyclictest7025-21sshd00:28:440
2260299130,12cyclictest0-21swapper/022:16:400
2260299130,12cyclictest0-21swapper/021:25:580
2260599129,2cyclictest0-21swapper/321:40:133
2260599120,1cyclictest22021-21sshd21:55:283
2260599120,11cyclictest0-21swapper/323:15:093
22604991210,1cyclictest25392-21apache_volume19:15:022
2260399124,7cyclictest14245-21munin-node21:05:251
2260399120,11cyclictest0-21swapper/120:47:441
2260299129,3cyclictest0-21swapper/020:47:050
2260299128,3cyclictest6414-21sshd00:32:090
2260299120,11cyclictest29784-21sshd00:10:140
2260599110,1cyclictest0-21swapper/320:37:453
2260499118,2cyclictest0-21swapper/220:15:162
22603991110,1cyclictest0-21swapper/119:35:281
2260399110,1cyclictest0-21swapper/120:32:451
2260399110,1cyclictest0-21swapper/120:23:341
2260399110,11cyclictest0-21swapper/120:07:041
2260299111,8cyclictest0-21swapper/020:29:490
2260299110,1cyclictest0-21swapper/020:51:300
2260299110,11cyclictest0-21swapper/019:32:370
2260299110,10cyclictest0-21swapper/019:15:010
2260599109,1cyclictest0-21swapper/321:07:103
2260599109,1cyclictest0-21swapper/320:51:503
2260599109,1cyclictest0-21swapper/320:10:423
2260599109,1cyclictest0-21swapper/319:32:353
2260599109,1cyclictest0-21swapper/319:27:493
2260599100,9cyclictest25011-21munin-node19:15:023
2260599100,1cyclictest0-21swapper/320:07:473
2260599100,1cyclictest0-21swapper/319:40:083
2260599100,10cyclictest0-21swapper/321:03:273
2260599100,10cyclictest0-21swapper/320:44:573
2260599100,10cyclictest0-21swapper/320:29:033
2260599100,10cyclictest0-21swapper/320:17:063
2260599100,10cyclictest0-21swapper/320:04:393
2260599100,10cyclictest0-21swapper/319:59:113
2260499109,1cyclictest0-21swapper/219:25:152
2260499102,3cyclictest8872-21df20:55:192
2260399109,1cyclictest0-21swapper/120:29:091
2260399109,1cyclictest0-21swapper/119:51:441
2260399109,1cyclictest0-21swapper/119:10:441
2260399104,5cyclictest8889-21df_abs20:55:181
2260399101,3cyclictest2671-21cut20:40:211
2260399101,3cyclictest0-21swapper/120:50:191
2260399101,3cyclictest0-21swapper/120:15:201
2260399101,2cyclictest0-21swapper/120:10:201
2260399101,2cyclictest0-21swapper/119:55:171
2260399100,3cyclictest0-21swapper/121:02:051
2260399100,2cyclictest28062-21munin-node19:20:131
2260399100,2cyclictest0-21swapper/120:35:161
2260399100,2cyclictest0-21swapper/119:30:131
2260299109,1cyclictest0-21swapper/020:55:550
2260299109,1cyclictest0-21swapper/020:21:170
2260299109,1cyclictest0-21swapper/020:13:450
2260299109,1cyclictest0-21swapper/019:49:410
2260299108,1cyclictest14028-21fschecks_time21:05:200
2260299107,2cyclictest0-21swapper/019:20:260
2260299103,4cyclictest0-21swapper/019:25:200
2260299102,8cyclictest0-21swapper/020:09:360
2260299101,8cyclictest0-21swapper/020:35:140
2260299100,8cyclictest0-21swapper/019:42:230
2260299100,7cyclictest0-21swapper/020:40:130
2260299100,5cyclictest0-21swapper/020:00:120
2260299100,10cyclictest0-21swapper/020:18:440
2260299100,10cyclictest0-21swapper/019:53:170
2260299100,10cyclictest0-21swapper/019:11:280
226059999,0cyclictest0-21swapper/320:23:283
226059990,1cyclictest0-21swapper/320:45:153
226049996,2cyclictest0-21swapper/220:40:112
226049991,7cyclictest0-21swapper/219:53:502
226029990,6cyclictest0-21swapper/019:36:210
226059987,1cyclictest0-21swapper/319:54:053
226059987,1cyclictest0-21swapper/319:20:583
226059982,2cyclictest0-21swapper/319:45:203
226059981,6cyclictest0-21swapper/320:55:223
226059980,0cyclictest0-21swapper/320:30:153
226049982,5cyclictest32474-21strings20:35:202
226049982,3cyclictest5858-21chrt20:49:542
226049981,6cyclictest0-21swapper/220:30:172
226049981,6cyclictest0-21swapper/220:10:222
226049981,6cyclictest0-21swapper/219:45:182
226049981,1cyclictest0-21swapper/219:40:132
226049980,8cyclictest0-21swapper/220:51:342
226049980,7cyclictest0-21swapper/220:28:062
226049980,5cyclictest13423-21strings19:55:202
226029980,1cyclictest10886-21apache_processe21:00:150
226059971,5cyclictest0-21swapper/319:35:263
226059970,6cyclictest0-21swapper/319:10:253
226049977,0cyclictest0-21swapper/220:00:132
226049972,4cyclictest1787-21runrttasks20:22:322
226049971,5cyclictest28206-21grep19:20:142
226049971,5cyclictest13741-21cut21:05:202
226049971,5cyclictest0-21swapper/221:00:202
226049971,5cyclictest0-21swapper/219:35:152
226049970,2cyclictest0-21swapper/219:30:142
226049963,2cyclictest17187-21munin-node20:05:152
226049960,4cyclictest0-21swapper/219:10:122
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional