You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-01-29 - 09:40

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Core i3-6100U @2300 MHz, Linux 5.15.79-rt54 (Profile)

Latency plot of shadow in rack #7, slot #7
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Jan 29, 2023 00:43:58)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2892210288,10sleep10-21swapper/119:08:071
89129784,9sleep30-21swapper/319:05:033
279228068,8sleep00-21swapper/019:06:490
276427666,6sleep20-21swapper/219:06:272
165582560,0sleep30-21swapper/320:45:213
312099256,3cyclictest141rcu_preempt21:54:542
3111992519,2cyclictest29047-21apache_processe00:10:180
311199250,24cyclictest0-21swapper/000:02:240
3120992421,2cyclictest29851-21diskstats21:30:232
3120992420,3cyclictest32128-21users23:10:302
3120992419,4cyclictest14495-21irqstats22:20:242
3115992421,2cyclictest23380-21hddtemp_smartct23:45:261
3111992421,2cyclictest31694-21ntp_states23:30:240
3111992421,2cyclictest30065-21apache_processe22:50:180
3111992421,2cyclictest14161-21apache_accesses23:20:190
3125992320,2cyclictest5952-21df22:35:203
3125992320,2cyclictest5459-21diskstats21:15:243
3125992320,2cyclictest5333-21apache_processe22:15:183
3125992320,2cyclictest5333-21apache_processe22:15:183
3125992320,2cyclictest30225-21df22:30:203
3125992320,2cyclictest30084-21apache_volume22:50:193
3125992320,2cyclictest29989-21diskstats21:50:203
3125992320,2cyclictest26549-21ntp_states00:30:293
3125992320,2cyclictest22262-21apache_processe23:25:173
3125992320,2cyclictest21438-21apache_accesses00:05:183
3125992320,2cyclictest21438-21apache_accesses00:05:173
3125992320,2cyclictest14328-21hddtemp_smartct22:20:223
3120992320,2cyclictest5816-21apache_accesses22:55:162
3120992320,2cyclictest4861-21/usr/sbin/munin21:35:232
3120992320,2cyclictest15536-21ntp_states23:20:302
3120992320,2cyclictest15084-21users21:40:302
3120992319,3cyclictest7443-21memory23:55:242
3115992320,2cyclictest6502-21memory22:15:271
3115992320,2cyclictest6502-21memory22:15:271
3115992320,2cyclictest31656-21memory23:50:241
3115992320,2cyclictest30751-21irqstats22:30:241
3115992320,2cyclictest30046-21df22:10:201
3115992320,2cyclictest22694-21ntp_states22:05:251
3115992320,2cyclictest21566-21apache_accesses22:25:161
3115992320,2cyclictest18902-21diskstats00:25:211
3115992320,2cyclictest12297-21munin-run21:20:001
3111992321,1cyclictest5694-21perl21:35:200
3111992320,2cyclictest5417-21apache_accesses22:35:160
3111992320,2cyclictest457-21perl00:35:220
3111992320,2cyclictest31832-21ntp_states23:50:260
3111992320,2cyclictest30831-21ntp_states22:10:270
3111992320,2cyclictest29667-21apache_accesses22:30:160
3111992320,2cyclictest22903-21hddtemp_smartct23:05:210
3111992320,2cyclictest20440-21munin-run21:25:000
3111992320,2cyclictest15511-21memory23:40:260
3111992320,2cyclictest14079-21df_inode22:20:200
3111992320,2cyclictest13213-21apache_volume22:00:160
311199230,22cyclictest0-21swapper/000:15:230
3125992220,1cyclictest7997-21users23:15:353
3125992220,1cyclictest21578-21apache_processe22:25:163
3125992220,1cyclictest13618-21df21:40:183
3125992220,1cyclictest13220-21apache_volume22:00:163
3125992219,2cyclictest6464-21diskstats22:55:223
3125992219,2cyclictest5545-21df_inode21:35:193
3125992219,2cyclictest302-21df_inode00:35:203
3125992219,2cyclictest21987-21apache_accesses23:05:143
3120992220,1cyclictest6771-21diskstats23:35:192
3120992220,1cyclictest30157-21diskstats22:10:212
3120992220,1cyclictest29053-21apache_volume00:10:182
3120992220,1cyclictest22219-21/usr/sbin/munin23:25:232
3120992220,1cyclictest21994-21apache_accesses23:05:152
3120992220,1cyclictest21083-21apache_volume21:25:192
3120992220,1cyclictest19572-21ntp_states00:25:282
3120992220,1cyclictest14122-21df22:40:212
3120992219,2cyclictest7121-21memory23:15:272
3120992219,2cyclictest5990-21df_inode22:15:222
3120992219,2cyclictest5990-21df_inode22:15:222
3120992219,2cyclictest30475-21apache_accesses23:50:152
3120992219,2cyclictest21572-21apache_accesses22:25:172
3120992219,2cyclictest13973-21irqstats21:20:242
3120992219,2cyclictest11317-21/usr/sbin/munin00:20:322
3115992220,1cyclictest22813-21irqstats22:45:281
3115992219,2cyclictest6100-21apache_accesses23:35:151
3115992219,2cyclictest29839-21df_inode21:30:231
3115992219,2cyclictest11441-21apache_volume00:20:251
3111992220,1cyclictest30544-21ntp_states21:30:290
3111992220,1cyclictest30244-21ntp_states21:10:240
3111992220,1cyclictest13087-21apache_volume21:40:140
3111992219,2cyclictest6108-21apache_processe23:35:140
3111992219,2cyclictest30758-21df_inode23:10:190
3125992118,2cyclictest15009-21diskstats23:40:213
3115992119,1cyclictest6231-21/usr/sbin/munin23:55:231
3115992119,1cyclictest5973-21apache_volume23:15:201
3115992119,1cyclictest14638-21irqstats22:40:251
3115992119,1cyclictest14395-21memory22:00:241
3115992118,2cyclictest6476-21irqstats22:35:241
3115992118,2cyclictest15673-21ntp_states23:40:271
3111992119,1cyclictest5841-21apache_volume22:55:180
3111992119,1cyclictest5281-21/usr/sbin/munin22:15:240
3111992119,1cyclictest5281-21/usr/sbin/munin22:15:230
3111992118,2cyclictest23582-21ntp_states23:25:250
3111992118,2cyclictest21380-21apache_processe22:05:150
3111992118,2cyclictest12765-21ntp_states00:20:360
3111992118,1cyclictest14234-21diskstats22:40:200
3125992018,1cyclictest5236-21irqstats00:15:333
3120992018,1cyclictest6410-21sensors_fan21:15:312
3120992017,2cyclictest21366-21apache_accesses22:05:162
3120992017,2cyclictest14336-21apache_accesses23:40:172
3115992018,1cyclictest30057-21/usr/sbin/munin23:10:321
3115992017,2cyclictest21277-21apache_accesses21:45:191
3115992017,2cyclictest15054-21memory23:00:231
3111992017,2cyclictest29286-21/usr/sbin/munin21:50:250
312599192,2cyclictest0-21swapper/323:35:393
312599190,1cyclictest0-21swapper/321:31:583
3115991917,1cyclictest5072-21apache_processe21:55:171
3120991816,1cyclictest22177-21hddtemp_smartct21:45:252
3120991816,1cyclictest14004-21perl22:00:202
3115991816,1cyclictest6216-21ntp_states21:35:251
311599180,0cyclictest0-21swapper/122:24:341
3111991815,2cyclictest22433-21/usr/sbin/munin23:45:290
3125991714,2cyclictest28855-21/usr/sbin/munin21:10:223
312599171,16cyclictest20697-21ssh00:25:593
312599171,0cyclictest0-21swapper/321:49:143
312599170,1cyclictest0-21swapper/321:56:013
3120991715,1cyclictest30029-21/usr/sbin/munin21:10:212
312099170,15cyclictest0-21swapper/220:00:302
3111991715,1cyclictest22341-21irqstats21:45:260
3111991714,2cyclictest21531-21/usr/sbin/munin22:25:180
3111991714,2cyclictest21394-21/usr/sbin/munin00:05:240
3111991714,2cyclictest21394-21/usr/sbin/munin00:05:230
311199170,2cyclictest0-21swapper/023:18:010
3125991613,2cyclictest29458-21/usr/sbin/munin22:10:173
3125991613,2cyclictest29041-21apache_processe00:10:173
3120991614,1cyclictest603-21hddtemp_smartct00:35:232
3120991613,2cyclictest5027-21/usr/sbin/munin21:55:172
3115991613,2cyclictest25179-21apache_processe00:30:181
311599160,0cyclictest0-21swapper/100:19:401
3111991613,2cyclictest21017-21/usr/sbin/munin21:25:310
3125991512,2cyclictest12890-21/usr/sbin/munin21:20:233
3125991510,3cyclictest19143-21tr20:50:193
3120991514,1cyclictest0-21swapper/222:50:162
312099150,1cyclictest0-21swapper/220:30:172
3115991512,2cyclictest15354-21/usr/sbin/munin00:00:261
311599150,0cyclictest0-21swapper/121:20:021
311199150,14cyclictest0-21swapper/021:55:350
3125991413,1cyclictest0-21swapper/300:00:243
3120991412,1cyclictest21710-21/usr/sbin/munin22:45:272
311599141,6cyclictest0-21swapper/120:35:161
311599140,5cyclictest0-21swapper/123:34:531
3111991410,3cyclictest24680-21gpgv19:55:000
311199140,12cyclictest9841-21grep19:20:310
312599137,4cyclictest7522-21df20:25:183
3125991312,1cyclictest0-21swapper/321:29:533
3125991310,2cyclictest28057-21grep20:00:213
3125991310,2cyclictest27157-21cat20:00:003
3125991310,2cyclictest16916-21grep19:35:373
3125991310,2cyclictest16196-21grep23:20:363
3125991310,2cyclictest14180-21grep23:00:183
3125991310,2cyclictest13832-21basename00:00:003
312599130,0cyclictest0-21swapper/323:46:543
3120991310,2cyclictest914-21grep20:10:242
3120991310,2cyclictest19349-21grep19:40:342
3120991310,2cyclictest18598-21sh20:49:592
3120991310,2cyclictest12825-21grep20:35:232
312099130,1cyclictest0-21swapper/222:31:132
312099130,12cyclictest23644-21ls19:50:272
311599137,5cyclictest29888-21sed21:50:191
3115991310,2cyclictest25398-21wc21:00:301
311199139,2cyclictest30048-21tr20:05:170
3111991310,2cyclictest7019-21cut23:55:200
3111991310,2cyclictest25999-21grep00:30:230
312599129,2cyclictest5697-21wc19:15:003
312599129,2cyclictest24635-21grep21:00:203
312599129,2cyclictest22082-21grep22:05:213
312599129,2cyclictest21171-21tr19:45:293
312599129,2cyclictest19804-21latency_hist19:45:003
312599128,3cyclictest26853-21cut21:05:203
312099129,2cyclictest31609-21cron00:35:012
312099129,2cyclictest31480-21cat23:30:222
312099129,2cyclictest22765-21tr23:45:232
312099129,2cyclictest14721-21head23:00:212
312099128,3cyclictest21567-21ssh00:05:212
312099128,3cyclictest21567-21ssh00:05:212
312099121,1cyclictest17221-21kworker/2:119:45:212
3120991210,1cyclictest15228-21grep00:00:252
311599129,2cyclictest7823-21grep22:55:331
311599129,2cyclictest301-21ssh00:35:201
311599129,2cyclictest25817-21grep19:55:201
311599129,2cyclictest24235-21grep23:25:321
311599128,3cyclictest21741-21tr00:05:221
311599128,3cyclictest21741-21tr00:05:221
311599127,4cyclictest1693-21ssh00:13:301
311599125,5cyclictest21017-21/usr/sbin/munin21:25:331
311599120,11cyclictest24984-21taskset23:06:171
311199129,2cyclictest4520-21wc20:20:000
311199129,2cyclictest23483-21wc21:00:000
311199129,2cyclictest19024-21wc19:40:300
311199129,2cyclictest16017-21grep19:35:270
311199129,2cyclictest14263-21tr20:40:200
311199129,2cyclictest12956-21head20:35:250
311199129,2cyclictest12283-21turbostat.cron21:20:000
311199129,2cyclictest10284-21uname20:30:230
311199128,3cyclictest11264-21grep19:25:210
311199123,6cyclictest500-21in:imuxsock20:55:010
311199120,11cyclictest14666-21sort23:00:200
312599118,2cyclictest884-21ssh23:31:123
312599118,2cyclictest6511-21tail19:15:243
312599118,2cyclictest4246-21taskset20:19:193
312599118,2cyclictest30644-21ssh23:10:183
312599118,2cyclictest21536-21ssh22:45:103
312599118,2cyclictest14713-21ssh00:21:593
312599118,2cyclictest14398-21grep22:40:223
312599118,2cyclictest14247-21wc19:30:273
312599118,2cyclictest12135-21expr20:35:173
312599118,2cyclictest10274-21cron19:25:003
312599110,11cyclictest0-21swapper/323:51:583
312099119,1cyclictest9083-21apt20:30:002
312099118,2cyclictest7713-21ssh00:17:062
312099118,2cyclictest4780-21cat19:10:272
312099118,2cyclictest31520-21sed20:05:312
312099118,2cyclictest27623-21smart_sda21:05:282
312099118,2cyclictest25464-21expr19:55:182
312099118,2cyclictest23915-21cut21:00:182
312099118,2cyclictest15719-21date20:40:322
312099118,2cyclictest12808-21ls19:30:002
312099110,2cyclictest6493-21cut22:35:242
312099110,11cyclictest0-21swapper/219:30:172
311599118,2cyclictest6382-21taskset20:20:591
311599118,2cyclictest382-21cut20:10:211
311599118,2cyclictest31241-21ssh21:10:441
311599118,2cyclictest22027-21sed20:55:211
311599118,2cyclictest19646-21grep20:50:221
311599118,2cyclictest14968-21grep21:40:291
311599118,2cyclictest10107-21tail20:30:211
311599117,3cyclictest30749-21cut22:50:231
311599115,2cyclictest224-21systemd-journal20:30:001
311599115,2cyclictest224-21systemd-journal19:30:001
311599112,6cyclictest0-21swapper/121:06:381
311199118,2cyclictest4775-21wc19:10:270
311199118,2cyclictest26506-21tr21:05:180
311199118,2cyclictest26123-21cat21:05:000
311199118,2cyclictest23914-21ssh22:45:420
311199118,2cyclictest1459-21grep20:10:310
311199117,3cyclictest7882-21cron19:20:000
311199117,3cyclictest17236-21grep20:45:240
311199117,2cyclictest29130-21sed20:00:310
311199110,10cyclictest22133-21ssh00:27:290
312599108,1cyclictest45622sleep320:20:083
312599108,1cyclictest10149-21hostname20:30:213
312599107,2cyclictest30074-21cut20:05:173
312599107,2cyclictest24060-21grep19:50:303
312599105,3cyclictest22567-21grep20:55:273
312599103,6cyclictest16243-21idleruntime-cro20:45:003
312099108,1cyclictest8528-21expr19:20:192
312099108,1cyclictest4651-21hostname20:20:172
312099108,1cyclictest21589-21expr20:55:192
312099108,1cyclictest20500-21unixbench_multi20:50:312
312099108,1cyclictest16185-21chrt19:35:282
312099107,2cyclictest5819-21wget19:15:222
312099107,2cyclictest2197-21chrt20:15:192
311599108,1cyclictest30069-21expr20:05:171
311599108,1cyclictest28737-21cut20:00:271
311599108,1cyclictest2419-21tr20:15:211
311599108,1cyclictest18356-21uniq19:40:241
311599108,1cyclictest10111-21chrt19:22:571
311599107,2cyclictest5933-21wc19:15:221
311599107,2cyclictest3323-21taskset19:10:111
311599107,2cyclictest24135-21sed19:50:311
311599107,2cyclictest17270-21cut20:45:241
311599107,2cyclictest16049-21head19:35:271
311599107,2cyclictest14043-21ssh23:20:151
311599105,3cyclictest20085-21apt-get19:45:201
311199108,1cyclictest14975-21chrt19:34:260
311199107,2cyclictest8667-21head20:25:300
311199107,2cyclictest25325-21tr19:55:170
311199107,2cyclictest20570-21grep19:45:230
31259997,1cyclictest1801-21taskset20:13:323
31259997,1cyclictest10549-21apache_processe19:25:153
31159997,1cyclictest14113-21cat20:40:171
31159990,4cyclictest224-21systemd-journal19:35:001
31119996,2cyclictest5652-21sed20:20:230
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional