You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2022-12-08 - 21:12

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 5.10.41-rt42 (Profile)

Latency plot of shadow in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackaslot8s.osadl.org (updated Wed Dec 07, 2022 12:45:09)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
257112181119,30sleep30-21swapper/307:07:203
256012180102,28sleep00-21swapper/007:05:510
259082178103,28sleep20-21swapper/207:09:522
257882177102,29sleep10-21swapper/107:08:181
1860821410,2sleep00-21swapper/012:00:290
1771121250,2sleep00-21swapper/010:36:410
239732740,2sleep30-21swapper/312:33:483
323042610,3sleep10-21swapper/111:45:201
292982550,3chrt0-21swapper/109:50:291
215292550,1sleep20-21swapper/209:44:542
234422520,9sleep122-21ksoftirqd/110:41:361
199862520,1sleep20-21swapper/209:15:202
152322510,4sleep215234-21cpuspeed_turbos10:35:192
133072490,1sleep10-21swapper/110:05:191
112702490,1sleep30-21swapper/311:55:153
26200994722,2cyclictest0-21swapper/310:30:483
128512460,2sleep20-21swapper/211:55:302
313242450,2sleep00-21swapper/008:35:200
286502450,1chrt0-21swapper/109:22:331
286502450,1chrt0-21swapper/109:22:321
277082440,1sleep00-21swapper/010:17:350
277082440,1sleep00-21swapper/010:17:340
230202430,1sleep00-21swapper/009:17:460
71732390,1sleep20-21swapper/210:55:252
26200993730,5cyclictest30913-21latency_hist12:40:003
2619999372,12cyclictest0-21swapper/210:05:152
2619999366,12cyclictest27307-21/usr/sbin/munin07:10:252
2619999354,28cyclictest5036-21ls10:25:312
26198993529,4cyclictest211rcuc/109:29:141
26198993415,5cyclictest22-21ksoftirqd/111:19:591
42742330,2sleep20-21swapper/209:30:002
2619999337,12cyclictest0-21swapper/211:10:212
2619999336,9cyclictest15545-21ssh11:30:272
2619999336,12cyclictest22021-21sh12:04:272
2619999335,9cyclictest13626-21sed11:00:302
2619999334,15cyclictest1723-21wc12:15:012
2619999334,13cyclictest21869-21sed10:40:312
2619999334,12cyclictest25173-21sed08:20:282
2619999333,13cyclictest26895-21sed11:40:242
2619999330,19cyclictest0-21swapper/209:45:112
26197993327,4cyclictest13598-21ssh11:56:140
2619999326,11cyclictest9484-21expr10:30:152
2619999326,10cyclictest31173-21perl07:20:212
2619999325,15cyclictest4519-21rm10:53:462
2619999325,10cyclictest29364-21ssh09:50:312
2619999324,11cyclictest13643-21cpu09:10:142
2619999324,10cyclictest13212-21diskmemload12:15:222
2619999323,10cyclictest26280-21sed12:35:232
2619999315,10cyclictest6168-21sed11:50:242
2619999313,11cyclictest9209-21sed12:20:282
2619999313,10cyclictest32681-21latency_hist08:40:012
2619999312,9cyclictest28053-21sh09:21:482
2619999312,9cyclictest28053-21sh09:21:482
2619999312,9cyclictest0-21swapper/212:32:002
2619999312,3cyclictest28-21ksoftirqd/209:59:412
2619799310,28cyclictest20368-21ssh12:30:220
225592310,1chrt0-21swapper/211:09:142
26200993021,6cyclictest13212-21diskmemload10:21:523
2620099301,26cyclictest28532-21latency_hist08:30:003
2620099301,26cyclictest21350-21ssh09:15:423
2619999304,9cyclictest24642-21ls10:15:172
2619999304,9cyclictest24642-21ls10:15:172
2619999304,11cyclictest28728-21ssh10:45:592
2619999304,10cyclictest17712-21/usr/sbin/munin08:05:182
2619999303,11cyclictest4409-21sed11:20:312
2619999303,10cyclictest8568-21rm09:33:132
2619999303,10cyclictest5255-21tune2fs08:50:162
2619999302,12cyclictest13604-21cstates07:55:142
2619999300,16cyclictest0-21swapper/208:40:102
2619899308,12cyclictest22-21ksoftirqd/109:50:011
2619899304,2cyclictest141rcu_preempt11:59:591
2619899301,7cyclictest25664-21latency_hist11:40:001
2619899301,4cyclictest13212-21diskmemload09:37:421
26197993024,4cyclictest23453-21ssh11:37:200
26197993023,6cyclictest0-21swapper/009:37:460
2619799301,26cyclictest20792-21ssh10:11:060
2619799300,28cyclictest16751-21ssh10:08:140
2620099291,4cyclictest16176-21sh10:07:323
2620099291,25cyclictest15226-21ssh11:30:233
2620099290,26cyclictest27155-21ssh10:16:543
2620099290,26cyclictest27155-21ssh10:16:543
2619999294,9cyclictest16794-21sed08:00:272
2619999294,12cyclictest7676-21sed08:55:192
2619999294,12cyclictest12524-21sed07:50:312
2619999294,10cyclictest28298-21sleep12:10:002
2619999293,9cyclictest30113-21apt-get10:20:142
2619999293,8cyclictest22990-21ssh10:13:532
2619999293,11cyclictest2953-21sort07:30:172
2619999291,16cyclictest0-21swapper/208:35:002
2619899297,9cyclictest22-21ksoftirqd/112:33:241
2619899297,7cyclictest22-21ksoftirqd/110:00:321
2619899296,6cyclictest22-21ksoftirqd/107:15:001
26198992924,3cyclictest211rcuc/110:16:081
26198992924,3cyclictest211rcuc/110:16:071
2619899292,2cyclictest22-21ksoftirqd/110:11:221
2619899291,7cyclictest22-21ksoftirqd/110:50:011
2619899291,26cyclictest2849-21apt-get11:20:151
2619899290,5cyclictest22-21ksoftirqd/107:15:021
2619899290,5cyclictest0-21swapper/111:40:311
26197992924,3cyclictest23906-21apt-get08:20:010
2619799291,26cyclictest28647-21runrttasks11:41:330
2619799291,26cyclictest0-21swapper/010:51:140
2619799291,26cyclictest0-21swapper/009:54:390
2619799290,26cyclictest2685-21ssh11:20:030
26200992824,2cyclictest32572-21ssh11:17:153
26200992822,4cyclictest22719-21ssh10:13:343
2620099281,25cyclictest24370-21sh11:38:283
2620099281,24cyclictest30912-21ssh09:52:143
2620099281,24cyclictest13610-21ssh12:25:143
2620099281,23cyclictest0-21swapper/311:54:103
2619999284,10cyclictest1939-21ssh11:19:272
2619999283,9cyclictest15156-21sed12:25:292
2619999283,9cyclictest1447-21awk07:25:232
2619999283,8cyclictest13212-21diskmemload11:49:562
2619999282,9cyclictest13212-21diskmemload10:02:362
2619999282,16cyclictest11901-21ssh09:35:372
2619999280,4cyclictest23199-21ssh11:37:012
2619999280,15cyclictest0-21swapper/207:35:202
2619899287,8cyclictest22-21ksoftirqd/112:29:591
2619899282,4cyclictest22-21ksoftirqd/111:30:011
2619899281,2cyclictest12593-21ssh10:32:311
2619899281,25cyclictest0-21swapper/112:39:261
2619899281,25cyclictest0-21swapper/109:31:061
2619899281,25cyclictest0-21swapper/108:59:131
2619899281,25cyclictest0-21swapper/108:38:401
2619899281,25cyclictest0-21swapper/108:27:011
2619899281,25cyclictest0-21swapper/108:02:441
26197992824,3cyclictest0-21swapper/010:31:400
26197992823,3cyclictest9815-21ssh11:53:580
26197992823,3cyclictest22772-21ssh10:40:440
26197992823,2cyclictest8539-21ssh11:25:080
26197992822,4cyclictest29053-21/usr/sbin/munin08:30:150
26197992820,6cyclictest17381-21ssh12:28:020
2619799281,3cyclictest30790-21ssh12:39:590
2619799281,25cyclictest2053-21taskset10:23:470
2619799281,25cyclictest0-21swapper/011:15:580
2619799281,25cyclictest0-21swapper/009:28:200
2619799281,25cyclictest0-21swapper/008:50:460
26200992722,3cyclictest28878-21cstates08:30:143
26200992721,4cyclictest2856-21ssh09:28:203
2620099271,3cyclictest30289-21ssh11:43:353
2620099271,24cyclictest0-21swapper/309:55:043
2620099271,24cyclictest0-21swapper/309:20:143
2620099271,24cyclictest0-21swapper/309:20:143
2620099271,24cyclictest0-21swapper/309:00:123
2620099271,23cyclictest19047-21ls08:05:303
2620099270,3cyclictest4286-21ls11:20:303
2619999273,9cyclictest1-21systemd11:25:272
2619999273,8cyclictest0-21swapper/208:10:212
2619999272,10cyclictest22015-21apt-get08:15:112
2619999271,24cyclictest0-21swapper/209:01:312
2619999271,24cyclictest0-21swapper/208:48:112
2619999271,24cyclictest0-21swapper/208:25:312
26198992725,1cyclictest0-21swapper/111:12:251
2619899272,4cyclictest22-21ksoftirqd/111:35:011
26198992724,2cyclictest32225-21ssh10:21:011
26198992723,2cyclictest2597-21apt-get07:30:131
26198992723,2cyclictest0-21swapper/109:15:071
2619899271,4cyclictest22-21ksoftirqd/108:55:011
2619899271,25cyclictest0-21swapper/110:49:181
2619899271,25cyclictest0-21swapper/110:35:441
2619899271,25cyclictest0-21swapper/110:26:111
2619899271,25cyclictest0-21swapper/109:58:431
2619899271,24cyclictest0-21swapper/112:00:131
2619899271,24cyclictest0-21swapper/111:54:051
2619899271,24cyclictest0-21swapper/109:40:571
2619899271,24cyclictest0-21swapper/109:10:061
2619899271,24cyclictest0-21swapper/109:01:071
2619899271,24cyclictest0-21swapper/108:15:351
2619899271,24cyclictest0-21swapper/108:06:371
26198992711,3cyclictest22-21ksoftirqd/112:10:551
2619799271,25cyclictest0-21swapper/008:26:110
2619799271,24cyclictest0-21swapper/011:30:550
2619799271,24cyclictest0-21swapper/010:25:210
2619799271,24cyclictest0-21swapper/010:00:460
2619799271,24cyclictest0-21swapper/009:41:340
2619799271,24cyclictest0-21swapper/009:00:540
2619799271,24cyclictest0-21swapper/008:59:320
2619799271,24cyclictest0-21swapper/008:16:140
2619799271,24cyclictest0-21swapper/008:05:040
2619799271,24cyclictest0-21swapper/007:50:350
2619799271,24cyclictest0-21swapper/007:47:270
2619799271,19cyclictest0-21swapper/009:48:350
2620099261,2cyclictest0-21swapper/310:36:023
2620099261,24cyclictest0-21swapper/311:12:593
2620099261,24cyclictest0-21swapper/310:00:403
2620099261,24cyclictest0-21swapper/307:48:363
2620099261,23cyclictest0-21swapper/311:25:593
2620099261,23cyclictest0-21swapper/310:40:033
2620099261,23cyclictest0-21swapper/309:40:483
2620099261,23cyclictest0-21swapper/309:10:413
2620099261,23cyclictest0-21swapper/308:55:273
2620099261,23cyclictest0-21swapper/308:50:143
2620099261,23cyclictest0-21swapper/308:48:013
2620099261,23cyclictest0-21swapper/308:37:153
2620099261,23cyclictest0-21swapper/308:15:293
2620099261,1cyclictest0-21swapper/310:49:513
2620099260,25cyclictest55350irq/142-enp3s0-09:37:453
2619999265,7cyclictest12261-21ntp_states09:05:322
2619999264,7cyclictest29759-21sed07:15:262
2619999261,1cyclictest0-21swapper/207:47:042
2619899269,9cyclictest22-21ksoftirqd/110:59:401
26198992624,1cyclictest0-21swapper/108:40:041
26198992623,2cyclictest0-21swapper/107:50:151
2619899261,6cyclictest19184-21python311:05:321
2619899261,2cyclictest0-21swapper/108:47:591
2619899261,1cyclictest0-21swapper/108:30:231
2619899260,24cyclictest0-21swapper/108:14:061
26197992624,1cyclictest0-21swapper/011:12:010
26197992624,1cyclictest0-21swapper/009:30:010
26197992623,2cyclictest0-21swapper/009:58:520
26197992623,2cyclictest0-21swapper/008:40:240
2619799261,24cyclictest0-21swapper/010:48:580
2619799261,1cyclictest0-21swapper/008:47:540
2619799261,1cyclictest0-21swapper/008:03:050
2619799261,1cyclictest0-21swapper/007:30:200
2619799260,24cyclictest13454-21ssh09:10:010
2619799260,24cyclictest0-21swapper/008:13:140
253132260,2chrt0-21swapper/309:47:273
26200992523,1cyclictest0-21swapper/308:03:173
2620099251,23cyclictest0-21swapper/310:50:443
2620099251,23cyclictest0-21swapper/310:25:353
2620099251,1cyclictest0-21swapper/309:30:133
2620099250,23cyclictest0-21swapper/308:40:203
2619899256,8cyclictest22-21ksoftirqd/111:04:101
2619899255,9cyclictest22-21ksoftirqd/107:39:311
2619899254,10cyclictest22-21ksoftirqd/107:40:141
2619899254,10cyclictest22-21ksoftirqd/107:40:131
26198992523,1cyclictest0-21swapper/107:49:331
26198992514,3cyclictest211rcuc/112:05:291
26197992523,1cyclictest0-21swapper/009:20:430
26197992523,1cyclictest0-21swapper/009:20:420
2619799251,1cyclictest0-21swapper/011:47:150
26200992422,1cyclictest0-21swapper/308:14:283
2619899249,4cyclictest22-21ksoftirqd/112:21:211
2619899241,4cyclictest22-21ksoftirqd/109:09:591
2619899241,3cyclictest22-21ksoftirqd/112:15:171
26197992422,1cyclictest0-21swapper/007:55:290
2620099236,14cyclictest141rcu_preempt12:20:173
26200992318,3cyclictest3150-21hddtemp_smartct07:30:183
26200992314,6cyclictest2363-21df12:15:173
26198992319,3cyclictest211rcuc/108:00:001
2619999223,6cyclictest8210-21switchtime07:40:262
2619999223,6cyclictest8210-21switchtime07:40:252
2619899226,8cyclictest22-21ksoftirqd/108:23:251
2619899223,4cyclictest22-21ksoftirqd/107:30:001
2620099211,3cyclictest0-21swapper/307:20:293
26200992113,6cyclictest31199-21rm12:11:293
26200992113,6cyclictest29327-21memory07:15:223
26200992113,6cyclictest11793-21fw_packets09:05:253
26200992112,7cyclictest24691-21df08:20:213
26198992112,6cyclictest31891-21sensors_temp07:20:281
26197992110,8cyclictest12-21ksoftirqd/012:10:300
88982201,2sleep00-21swapper/010:56:490
26200992014,4cyclictest8480-21sh10:56:173
26200992013,5cyclictest12808-21users07:50:333
26200992013,4cyclictest19805-21sh12:01:433
26200991912,5cyclictest7262-21expr07:40:163
26200991912,5cyclictest7262-21expr07:40:163
26200991912,4cyclictest17922-21cpuspeed_turbos11:05:203
26200991911,5cyclictest32640-21interrupts11:45:243
2620099191,13cyclictest11970-21ssh11:00:153
2619799191,5cyclictest20347-21ssh11:06:320
26197991913,4cyclictest12767-21sh12:24:280
26197991911,6cyclictest6568-21seq07:39:250
2619799190,16cyclictest426-21chrt07:25:140
2620099189,7cyclictest26438-21cat07:10:163
2620099188,2cyclictest34-21ksoftirqd/307:40:003
26200991812,3cyclictest22817-21sh12:05:103
26200991810,5cyclictest14230-21missed_timers07:55:203
26197991813,3cyclictest11302-21/usr/sbin/munin09:05:340
26197991811,5cyclictest29990-21grep07:15:290
26197991811,5cyclictest23225-21cat12:05:190
26197991810,6cyclictest8997-21wc07:45:010
26197991810,6cyclictest8997-21wc07:45:010
26197991810,5cyclictest12331-21sh11:00:160
2620099178,6cyclictest1369-21ntp_offset07:25:223
2619799173,8cyclictest26242-21apt-get07:10:140
26197991710,5cyclictest32728-21sed07:25:000
26197991710,5cyclictest2736-21cat12:15:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional