You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-20 - 16:26

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 4.16.15-rt7 (Profile)

Latency plot of shadow in rack #5, slot #3
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the alll highest latencies:
System rack5slot3s.osadl.org (updated Mon Mar 20, 2023 12:44:33)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
42742207170,24sleep10-21swapper/107:07:581
42742207170,24sleep10-21swapper/107:07:581
42582203171,21sleep20-21swapper/207:07:472
42582203171,21sleep20-21swapper/207:07:472
41802203165,25sleep30-21swapper/307:06:473
41802203165,25sleep30-21swapper/307:06:473
42882202165,24sleep00-21swapper/007:08:100
42882202165,24sleep00-21swapper/007:08:090
277621770,5sleep2452599cyclictest11:55:282
173321720,3sleep1451999cyclictest10:40:251
452599107102,4cyclictest0-21swapper/208:50:212
56082980,1sleep00-21swapper/012:05:030
124852800,3sleep00-21swapper/011:05:210
4525997774,2cyclictest0-21swapper/210:08:582
101392590,1sleep10-21swapper/107:20:141
16162550,2sleep30-21swapper/310:40:243
4525995349,3cyclictest0-21swapper/207:40:222
451399426,8cyclictest32682-40dnf10:38:500
451399413,8cyclictest823-21snmpd07:37:320
183872410,2sleep00-21swapper/008:50:250
451999381,28cyclictest0-21swapper/108:25:141
451999365,21cyclictest28839-21cstates10:30:171
451999361,26cyclictest0-21swapper/110:53:301
451399368,13cyclictest7659-21sadc12:10:020
451399356,13cyclictest27161-21munin-run08:00:000
451399349,12cyclictest20845-21munin-run12:40:020
451399341,12cyclictest19543-21kworker/0:110:24:220
451399341,12cyclictest19543-21kworker/0:110:24:220
453199331,12cyclictest0-21swapper/312:34:563
453199331,12cyclictest0-21swapper/312:34:563
451999331,24cyclictest0-21swapper/110:57:501
451999331,24cyclictest0-21swapper/107:20:001
451399331,9cyclictest4481-21kworker/0:207:17:100
453199321,23cyclictest0-21swapper/307:48:373
451999323,20cyclictest27655-21runrttasks09:13:341
451999321,25cyclictest0-21swapper/110:04:541
451999316,11cyclictest2030-21awk09:30:011
451399311,14cyclictest4218-21munin-run09:35:010
451999304,11cyclictest9481-21kworker/u8:210:08:121
451999301,13cyclictest0-21swapper/109:35:321
4519993010,17cyclictest88150irq/134-enp2s0-07:45:121
451399303,2cyclictest88150irq/134-enp2s0-09:40:160
453199298,7cyclictest27491-21cpuspeed_turbos08:00:193
453199290,11cyclictest0-21swapper/307:25:233
451999296,8cyclictest8235-21df_inode12:10:181
451999294,23cyclictest0-21swapper/107:59:461
451999294,16cyclictest25032-21interrupts10:20:211
451999294,16cyclictest25032-21interrupts10:20:211
451999292,9cyclictest0-21swapper/111:50:221
451999291,6cyclictest0-21swapper/112:23:041
451999291,13cyclictest0-21swapper/107:35:521
451999291,12cyclictest2093-21kernelversion11:55:201
451999291,12cyclictest0-21swapper/109:16:221
451999291,12cyclictest0-21swapper/109:16:221
451999291,12cyclictest0-21swapper/108:52:501
451999291,10cyclictest0-21swapper/109:21:541
451999290,8cyclictest0-21swapper/112:25:221
451999290,3cyclictest0-21swapper/107:11:291
451399290,14cyclictest17313-21kworker/0:010:14:100
453199281,17cyclictest0-21swapper/311:18:533
451999284,17cyclictest0-21swapper/108:42:551
451999281,11cyclictest0-21swapper/109:06:441
451999281,10cyclictest0-21swapper/111:40:261
451999281,10cyclictest0-21swapper/108:45:081
451999280,22cyclictest0-21swapper/110:37:001
451999280,11cyclictest0-21swapper/111:46:581
451399280,12cyclictest20231-21kworker/0:207:41:500
451999277,11cyclictest0-21swapper/111:14:581
451999273,10cyclictest0-21swapper/111:35:221
451999271,9cyclictest0-21swapper/108:59:341
451999271,9cyclictest0-21swapper/108:39:481
451999271,14cyclictest0-21swapper/108:14:021
451999271,14cyclictest0-21swapper/108:14:021
451999271,12cyclictest0-21swapper/108:24:261
451999271,12cyclictest0-21swapper/107:40:401
451999271,12cyclictest0-21swapper/107:31:501
451999271,11cyclictest0-21swapper/109:04:501
451399274,11cyclictest2644-21awk10:45:010
451399273,11cyclictest10201-21munin-run08:35:010
4513992711,2cyclictest22056-21kworker/0:111:45:350
453199261,10cyclictest0-21swapper/312:26:043
451999267,6cyclictest19839-21/usr/sbin/munin10:10:131
451999262,8cyclictest0-21swapper/108:05:421
451999261,12cyclictest0-21swapper/108:01:441
451999261,11cyclictest0-21swapper/107:51:121
451999261,11cyclictest0-21swapper/107:29:441
451999261,10cyclictest0-21swapper/111:23:301
451999261,10cyclictest0-21swapper/108:16:321
451999260,10cyclictest0-21swapper/110:25:261
451399267,16cyclictest6422-21munin-run09:40:010
451399262,12cyclictest8922-21kworker/0:107:23:060
451399262,10cyclictest19543-21kworker/0:110:16:500
4513992611,3cyclictest30818-21kworker/0:310:55:240
104062260,2sleep20-21swapper/208:35:022
453199258,9cyclictest23668-21hddtemp_smartct07:50:173
453199254,14cyclictest86550irq/126-enp1s0-12:00:173
453199251,11cyclictest0-21swapper/311:26:403
451999254,18cyclictest23992-21latency_hist10:20:011
4519992510,8cyclictest9051-21kworker/u8:011:34:471
451399251,11cyclictest11092-21kworker/0:111:01:500
451399251,10cyclictest4768-21kworker/0:110:52:340
193382250,2sleep2311ktimersoftd/208:55:132
453199241,11cyclictest7434-21/usr/sbin/munin10:55:173
453199240,13cyclictest0-21swapper/309:35:213
451999248,8cyclictest836-21runrttasks12:35:351
451999242,17cyclictest86750irq/128-enp1s0-10:45:121
451999242,16cyclictest780-21rngd11:28:581
451999242,13cyclictest13237-21/usr/sbin/munin09:55:251
451399249,2cyclictest13352-21kworker/0:211:16:220
451399242,10cyclictest5546-21kworker/0:008:26:240
451399241,10cyclictest27497-21kworker/0:210:28:220
4531992310,3cyclictest0-21swapper/311:15:023
453199230,21cyclictest87950irq/132-enp2s0-08:59:073
452599235,10cyclictest0-21swapper/210:03:322
451999238,4cyclictest0-21swapper/112:00:221
451999238,3cyclictest0-21swapper/111:07:041
451999234,11cyclictest8844-21/usr/sbin/munin09:45:271
451999234,10cyclictest86550irq/126-enp1s0-12:15:181
451999231,10cyclictest0-21swapper/112:08:481
451999230,11cyclictest0-21swapper/112:30:101
451999230,11cyclictest0-21swapper/112:30:101
451399232,14cyclictest23773-21kworker/0:007:11:480
451399232,10cyclictest14498-21kworker/0:008:45:280
451399231,11cyclictest1972-21kworker/0:110:00:280
451399231,10cyclictest28955-21kworker/0:011:51:390
453199229,3cyclictest27935-21latency11:40:203
453199224,7cyclictest0-21swapper/310:20:173
453199224,7cyclictest0-21swapper/310:20:173
453199223,3cyclictest0-21swapper/311:58:203
453199221,3cyclictest9481-21kworker/u8:210:45:183
453199221,10cyclictest0-21swapper/312:16:063
453199221,10cyclictest0-21swapper/307:33:593
453199220,14cyclictest0-21swapper/312:36:123
451999227,7cyclictest0-21swapper/111:04:181
451999227,3cyclictest0-21swapper/108:31:041
4519992215,4cyclictest2780-21cut09:30:211
451999221,10cyclictest0-21swapper/111:17:281
451999220,7cyclictest823-21snmpd09:53:191
451399225,3cyclictest28955-21kworker/0:011:55:180
451399225,15cyclictest11709-21kworker/0:007:30:240
451399224,4cyclictest11709-21kworker/0:007:26:240
451399222,17cyclictest14852-21kworker/0:210:05:520
451399222,12cyclictest5189-21kworker/0:312:20:230
451399221,9cyclictest22470-21kworker/0:107:50:580
451399221,8cyclictest14121-21kworker/0:212:26:130
4513992215,2cyclictest24762-21kworker/0:208:05:260
451399221,14cyclictest22470-21kworker/0:107:46:560
4513992211,3cyclictest27508-21kworker/0:209:12:100
453199216,8cyclictest5750irq/9-acpi11:50:283
453199215,2cyclictest0-21swapper/310:10:303
453199214,9cyclictest0-21swapper/309:21:553
453199213,6cyclictest86550irq/126-enp1s0-10:38:493
453199213,5cyclictest948150kworker/u8:209:48:303
453199212,7cyclictest0-21swapper/310:50:593
453199212,3cyclictest0-21swapper/310:27:033
453199212,14cyclictest88050irq/133-enp2s0-10:01:213
453199212,14cyclictest88050irq/133-enp2s0-08:39:093
453199211,2cyclictest87950irq/132-enp2s0-07:20:423
453199211,10cyclictest0-21swapper/309:15:103
453199211,10cyclictest0-21swapper/309:15:103
452599215,7cyclictest0-21swapper/209:23:482
452599212,5cyclictest32682-40dnf10:38:482
4525992116,3cyclictest13144-21grep11:05:302
451999211,9cyclictest0-21swapper/109:40:541
451399219,2cyclictest13352-21kworker/0:211:13:180
451399214,2cyclictest86850irq/129-enp1s0-12:34:430
451399214,2cyclictest86850irq/129-enp1s0-12:34:430
451399212,6cyclictest14852-21kworker/0:209:59:360
451399211,8cyclictest25232-21kworker/0:109:15:240
451399211,8cyclictest25232-21kworker/0:109:15:240
451399211,5cyclictest0-21swapper/012:16:260
451399211,14cyclictest18927-21kworker/0:009:05:200
451399210,9cyclictest24762-21kworker/0:208:21:580
261522210,2sleep341-21ksoftirqd/311:35:263
105542211,17sleep2823-21snmpd09:46:522
453199208,3cyclictest0-21swapper/307:43:033
453199207,3cyclictest0-21swapper/309:42:113
453199207,2cyclictest5750irq/9-acpi10:30:243
453199204,2cyclictest0-21swapper/309:02:313
453199203,3cyclictest0-21swapper/312:11:083
453199202,3cyclictest0-21swapper/310:06:573
453199201,18cyclictest0-21swapper/308:52:393
453199201,12cyclictest0-21swapper/312:20:463
453199201,11cyclictest823-21snmpd08:28:053
453199201,11cyclictest22657-21/usr/sbin/munin11:30:303
453199200,11cyclictest88050irq/133-enp2s0-08:20:133
452599209,3cyclictest30-21rcuc/209:40:232
452599207,3cyclictest177392ntp_offset12:30:272
452599207,3cyclictest177392ntp_offset12:30:272
452599201,7cyclictest9051-21kworker/u8:011:35:282
452599201,3cyclictest0-21swapper/210:10:542
451399207,3cyclictest18248-21kworker/0:011:23:240
451399204,3cyclictest10596-21kworker/0:009:50:540
451399201,8cyclictest3272-21kworker/0:108:18:440
451399201,8cyclictest24762-21kworker/0:208:05:000
451399200,8cyclictest18927-21kworker/0:009:01:060
293442201,2sleep20-21swapper/208:04:562
59322190,1sleep20-21swapper/209:35:312
453199196,3cyclictest0-21swapper/308:16:233
453199195,2cyclictest2433-21cstates09:30:193
453199194,3cyclictest41-21ksoftirqd/312:05:363
453199193,4cyclictest41-21ksoftirqd/308:45:233
453199193,13cyclictest13006-21latency_hist09:55:023
453199191,8cyclictest0-21swapper/309:13:203
453199191,7cyclictest10146-21hddtemp_smartct11:00:253
453199191,2cyclictest0-21swapper/308:11:013
453199191,2cyclictest0-21swapper/308:11:013
453199191,10cyclictest32405-21/usr/sbin/munin09:25:173
453199191,10cyclictest22032-21/usr/sbin/munin10:15:193
452599194,6cyclictest13596-21users07:25:272
452599192,14cyclictest13365-21tr09:55:132
452599191,8cyclictest5327-21gltestperf07:10:162
452599191,5cyclictest0-21swapper/212:24:522
4525991915,3cyclictest15466-21sensors07:30:262
4525991914,3cyclictest7925-21grep10:55:202
452599191,10cyclictest10967-21awk07:20:242
452599191,10cyclictest0-21swapper/211:27:142
452599191,10cyclictest0-21swapper/211:18:282
452599191,10cyclictest0-21swapper/211:00:352
451399194,3cyclictest24762-21kworker/0:208:12:480
451399194,3cyclictest24762-21kworker/0:208:12:480
451399194,2cyclictest86550irq/126-enp1s0-10:34:500
451399191,9cyclictest18927-21kworker/0:008:57:220
451399191,8cyclictest9853-21kworker/0:308:43:340
451399191,8cyclictest32107-21kworker/0:209:27:380
4513991912,4cyclictest0-21swapper/011:30:240
451399191,12cyclictest1972-21kworker/0:109:47:460
451399191,10cyclictest22056-21kworker/0:111:41:440
315952190,2sleep29-21rcu_preempt08:10:012
200072191,2sleep30-21swapper/311:22:273
453199185,3cyclictest0-21swapper/307:15:183
453199185,2cyclictest4725-21/usr/sbin/munin07:10:143
453199183,12cyclictest12622-21/usr/sbin/munin08:40:163
453199181,9cyclictest823-21snmpd07:38:433
453199181,9cyclictest0-21swapper/311:05:253
453199181,11cyclictest9856-21sshd08:30:483
453199181,11cyclictest0-21swapper/309:58:223
453199180,3cyclictest0-21swapper/308:06:113
452599185,10cyclictest581-21systemd-journal12:10:012
452599184,3cyclictest86550irq/126-enp1s0-09:55:012
452599183,7cyclictest30-21rcuc/207:50:252
452599183,6cyclictest30-21rcuc/211:50:302
452599182,2cyclictest86750irq/128-enp1s0-09:25:242
452599181,8cyclictest0-21swapper/209:15:242
452599181,8cyclictest0-21swapper/209:15:242
452599181,5cyclictest24294-21taskset11:31:022
452599181,5cyclictest0-21swapper/212:16:572
4525991812,3cyclictest27576-21awk10:25:242
4525991812,3cyclictest14974-21open_inodes11:10:302
452599181,15cyclictest22720-21switchtime09:00:262
452599181,15cyclictest22265-21sed07:45:272
452599181,11cyclictest0-21swapper/209:32:422
452599181,11cyclictest0-21swapper/207:35:292
451399187,3cyclictest2286-21kworker/0:010:47:480
451399186,9cyclictest9853-21kworker/0:308:35:380
451399182,4cyclictest22056-21kworker/0:111:26:220
451399181,9cyclictest22056-21kworker/0:111:38:260
4513991812,3cyclictest7394-21kworker/0:012:10:290
312692180,14sleep00-21swapper/009:20:250
261762181,2sleep20-21swapper/210:24:532
261762181,2sleep20-21swapper/210:24:532
143932181,3sleep20-21swapper/208:42:412
56872171,3sleep20-21swapper/208:23:472
453199173,3cyclictest88250irq/135-enp2s0-09:09:293
453199171,10cyclictest0-21swapper/311:46:463
452599177,3cyclictest86850irq/129-enp1s0-08:15:152
452599171,7cyclictest7895-21/usr/sbin/munin12:10:232
452599171,7cyclictest0-21swapper/210:30:322
452599171,6cyclictest5839-21if_err_enp1s010:50:222
452599171,6cyclictest3856-21needreboot10:45:272
452599171,6cyclictest26419-21grep09:10:172
452599171,6cyclictest18878-21/usr/sbin/munin12:35:252
452599171,6cyclictest0-21swapper/212:28:212
452599171,6cyclictest0-21swapper/210:17:352
4525991712,3cyclictest7712-21sort07:15:162
4525991712,3cyclictest15635-21kernelversion08:45:222
452599171,14cyclictest0-21swapper/212:00:082
453199165,3cyclictest0-21swapper/307:59:113
452599169,4cyclictest11483-21ntpq08:35:212
452599163,3cyclictest86550irq/126-enp1s0-10:41:392
452599161,8cyclictest0-21swapper/211:40:232
452599161,5cyclictest0-21swapper/209:09:452
4525991612,3cyclictest0-21swapper/211:22:252
4525991611,3cyclictest32397-21gltestperf08:10:162
4525991611,3cyclictest32397-21gltestperf08:10:162
4525991611,3cyclictest29249-21/usr/sbin/munin11:45:272
4525991611,3cyclictest25514-21expr07:55:142
4525991610,3cyclictest6427-21cut08:25:152
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional