You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-20 - 19:31

x86 Intel Celeron N3350 @1100 MHz, Linux 5.10.90-rt60 (Profile)

Latency plot of system in rack #c, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot4.osadl.org (updated Mon Mar 20, 2023 12:45:16)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
40698762265190,28sleep10-21swapper/107:09:421
40696002248198,27sleep00-21swapper/007:06:440
407002699107101,5cyclictest4089553-21kworker/u8:008:05:581
4070022995627,12cyclictest38265-21apt-get11:40:130
4070022995524,16cyclictest4178414-21ssh10:27:450
41851052540,4sleep10-21swapper/110:35:581
159199540,9rtkit-daemon0-21swapper/011:13:520
4070022995322,16cyclictest4138263-40apt-check09:35:100
159199530,4rtkit-daemon8037-21apt-get11:00:201
159199530,3rtkit-daemon1590-21rtkit-daemon07:34:081
41338922520,4sleep10-21swapper/109:30:091
407002699528,12cyclictest4130194-21apt-get09:25:131
407002299529,6cyclictest13-21ksoftirqd/011:10:000
4070022995212,7cyclictest13-21ksoftirqd/011:20:230
4070022995212,5cyclictest13-21ksoftirqd/010:20:290
4070022995211,6cyclictest13-21ksoftirqd/008:40:020
159199510,3rtkit-daemon0-21swapper/007:25:130
407002299509,6cyclictest13-21ksoftirqd/010:30:240
407002299498,7cyclictest13-21ksoftirqd/011:40:000
407002299498,7cyclictest13-21ksoftirqd/009:25:010
407002299498,7cyclictest13-21ksoftirqd/007:45:280
4070022994911,5cyclictest13-21ksoftirqd/012:20:210
4070022994911,5cyclictest13-21ksoftirqd/012:20:210
41889812480,2sleep10-21swapper/110:41:461
41754752480,7sleep1251rcuc/110:24:341
407002299486,9cyclictest60552-21sh12:09:400
4070022994819,15cyclictest21512-21ssh11:17:210
407002299480,21cyclictest13-21ksoftirqd/008:50:010
41912142470,4sleep10-21swapper/110:45:251
407002699471,9cyclictest75573-21latency_hist12:30:001
407002699464,12cyclictest7872-21latency_hist11:00:011
407002699464,12cyclictest76415-21/usr/sbin/munin12:30:261
407002699464,11cyclictest38265-21apt-get11:40:141
407002699463,10cyclictest23905-21ssh11:20:261
407002699462,12cyclictest4106556-21latency_hist08:40:011
407002299466,6cyclictest13-21ksoftirqd/009:10:000
159199460,4rtkit-daemon1590-21rtkit-daemon10:44:500
159199460,3rtkit-daemon1590-21rtkit-daemon09:54:231
41536032450,3sleep0161rcuc/009:55:150
407002699455,9cyclictest79438-21apt-get12:35:211
407002699455,9cyclictest20123-21cut11:15:241
407002699454,12cyclictest4164794-21apt-get10:10:141
407002699452,10cyclictest4139230-21/usr/sbin/munin09:35:301
407002299459,7cyclictest13-21ksoftirqd/009:25:140
407002299456,7cyclictest13-21ksoftirqd/010:17:580
407002299455,6cyclictest13-21ksoftirqd/007:50:270
4070022994517,6cyclictest4161408-21sed10:05:210
4070022994511,5cyclictest161rcuc/012:02:100
407002299450,13cyclictest63950irq/130-enp2s009:15:180
407002699444,10cyclictest4113514-21processes08:55:251
407002699443,11cyclictest53633-21df_inode12:00:201
407002299448,12cyclictest50613-21mailstats11:55:270
407002299448,12cyclictest50613-21mailstats11:55:270
407002299446,6cyclictest13-21ksoftirqd/008:55:250
407002299446,16cyclictest4165041-21ssh10:10:180
4070022994412,6cyclictest13-21ksoftirqd/008:40:260
4070022994411,3cyclictest141rcu_preempt09:10:140
159199440,6rtkit-daemon0-21swapper/010:03:580
159199440,3rtkit-daemon1590-21rtkit-daemon08:15:341
407002699433,11cyclictest64569-21apt-get12:15:161
407002699432,12cyclictest4073976-21sed07:15:311
407002699432,11cyclictest4146729-21proc_pri09:45:271
407002699432,10cyclictest4164600-21latency_hist10:10:011
407002299435,5cyclictest161rcuc/010:47:230
407002299435,3cyclictest13-21ksoftirqd/009:40:190
4070022994318,9cyclictest73210-21ssh12:25:340
4070022994312,7cyclictest13-21ksoftirqd/009:50:240
4070022994312,6cyclictest161rcuc/007:15:300
4070022994311,18cyclictest82952-21munin-run12:39:590
407002699424,8cyclictest4125559-21runrttasks09:18:441
407002699422,10cyclictest4157146-21latency_hist10:00:001
407002699421,9cyclictest34573-21/usr/sbin/munin11:35:181
407002699420,10cyclictest16662-21processes11:10:331
407002299428,7cyclictest13-21ksoftirqd/011:00:180
407002299428,3cyclictest141rcu_preempt08:50:150
407002299425,15cyclictest7666-21sshd10:59:560
4070022994212,10cyclictest64511-21ssh12:15:080
4070022994211,6cyclictest13-21ksoftirqd/007:44:340
159199420,3rtkit-daemon0-21swapper/011:31:170
407002699413,9cyclictest4176089-21tr10:25:161
407002699413,9cyclictest31004-21/usr/sbin/munin11:30:391
407002699413,10cyclictest4160785-21latency_hist10:05:011
407002699412,9cyclictest4089193-21cut07:55:201
407002699412,15cyclictest40704072sleep107:10:171
407002699412,10cyclictest4076536-21latency_hist07:25:011
407002299416,6cyclictest13-21ksoftirqd/007:35:180
407002299415,7cyclictest13-21ksoftirqd/008:05:180
407002299415,6cyclictest13-21ksoftirqd/011:49:580
407002299415,10cyclictest141rcu_preempt08:00:190
159199410,3rtkit-daemon0-21swapper/011:29:290
407002699404,10cyclictest4096542-21latency_hist08:15:021
407002699403,8cyclictest4108776-21cpu08:45:171
407002699403,11cyclictest4082923-21apt-get07:40:131
407002699402,12cyclictest63828-21sshd12:13:561
407002699402,10cyclictest4099570-21latency08:20:251
4070026994012,16cyclictest4171070-21ssh10:17:411
407002299406,6cyclictest161rcuc/010:36:490
407002299405,6cyclictest13-21ksoftirqd/010:50:220
407002299404,6cyclictest13-21ksoftirqd/009:45:270
407002299403,8cyclictest13-21ksoftirqd/009:35:010
407002299403,7cyclictest13-21ksoftirqd/007:25:010
4070022994010,3cyclictest141rcu_preempt08:10:220
407002699397,6cyclictest26-21ksoftirqd/107:25:141
407002699396,10cyclictest4127889-21head09:20:301
407002699395,13cyclictest4117150-21/usr/sbin/munin09:05:261
407002699394,9cyclictest4102091-21sed08:25:281
407002699394,16cyclictest0-21swapper/112:06:381
407002699394,14cyclictest71896-21/usr/sbin/munin12:25:011
407002699394,14cyclictest71896-21/usr/sbin/munin12:25:011
407002699394,14cyclictest4114802-21sh09:00:151
407002699394,10cyclictest4087025-21df_inode07:50:161
407002699393,12cyclictest4118293-21diskmemload11:05:461
407002699392,10cyclictest46874-21open_files11:50:281
407002299399,6cyclictest161rcuc/012:10:310
407002299399,3cyclictest141rcu_preempt08:18:400
407002299398,6cyclictest0-21swapper/008:21:050
407002299397,7cyclictest141rcu_preempt09:00:140
407002299395,6cyclictest13-21ksoftirqd/012:30:240
159199390,3rtkit-daemon0-21swapper/111:54:380
407002699385,6cyclictest26-21ksoftirqd/111:55:161
407002699385,6cyclictest26-21ksoftirqd/111:55:161
407002699384,14cyclictest4143016-21grep09:40:271
407002699383,9cyclictest42170-21cpu11:45:201
407002699383,11cyclictest668-21apt-get10:50:181
407002699383,10cyclictest4107568-21open_files08:40:241
407002699382,9cyclictest4119911-21sed09:10:281
407002699382,9cyclictest4085133-21fschecks_count07:45:181
407002699381,9cyclictest4111616-21wc08:50:241
407002299388,3cyclictest141rcu_preempt07:30:270
407002299382,3cyclictest13-21ksoftirqd/007:10:220
4070022993816,7cyclictest4100916-21apt-get08:25:120
407002699373,11cyclictest4179809-21apt-get10:30:151
407002699372,9cyclictest4103234-21grep08:30:181
4070026993713,2cyclictest26-21ksoftirqd/111:30:011
407002299378,6cyclictest13-21ksoftirqd/007:56:220
4070022993711,2cyclictest141rcu_preempt08:30:230
407002699361,10cyclictest4091457-21netstat08:00:301
407002699353,11cyclictest4081784-21processes07:35:261
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional