You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-23 - 04:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  

x86 Intel Atom E3940 @1600 MHz, Linux 5.10.41-rt42 (Profile)

Latency plot of shadow in rack #a, slot #8
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackaslot8s.osadl.org (updated Thu Mar 23, 2023 00:45:11)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
113962175101,29sleep30-21swapper/319:09:453
113962175101,29sleep30-21swapper/319:09:443
111012165113,27sleep10-21swapper/119:05:541
111012165113,27sleep10-21swapper/119:05:541
112062162101,30sleep20-21swapper/219:07:162
112062162101,30sleep20-21swapper/219:07:162
111272160103,29sleep00-21swapper/019:06:130
111272160103,29sleep00-21swapper/019:06:130
132042700,2sleep10-21swapper/123:24:231
218202670,2sleep30-21swapper/321:30:353
320812650,4sleep130788-21diskmemload23:40:361
17612570,2sleep10-21swapper/100:14:361
17612570,2sleep10-21swapper/100:14:361
173672520,2sleep00-21swapper/023:57:130
172532490,1sleep30-21swapper/323:27:303
46962420,1sleep00-21swapper/022:45:330
207002420,2sleep10-21swapper/120:45:221
177732410,1sleep20-21swapper/223:57:472
52632390,1sleep10-21swapper/123:15:511
18972360,1chrt0-21swapper/222:12:562
214522340,1sleep20-21swapper/221:30:292
1169199341,30cyclictest19994-21cat00:00:201
1169199322,4cyclictest19583-21ssh23:00:201
1169299311,23cyclictest0-21swapper/223:05:202
11693993022,6cyclictest24966-21apt-get23:35:193
1169299301,5cyclictest30788-21diskmemload00:05:292
1169099301,3cyclictest14452-21diskstats23:25:180
1169099301,26cyclictest17554-21ntp_states00:30:270
11693992824,2cyclictest10387-21ls23:50:313
1169299281,24cyclictest8238-21apt-get23:50:022
1169099281,25cyclictest25918-21missed_timers00:05:280
1169099281,25cyclictest0-21swapper/023:45:120
1169399271,23cyclictest25003-21apt-get23:05:183
1169299270,24cyclictest2751-21ssh00:15:172
11691992725,1cyclictest0-21swapper/100:06:001
1169199270,25cyclictest0-21swapper/123:05:311
1169099271,24cyclictest0-21swapper/000:00:150
9972260,1sleep30-21swapper/322:42:183
1169399261,23cyclictest0-21swapper/323:10:323
11692992622,2cyclictest0-21swapper/223:05:012
1169299261,2cyclictest31771-21ls23:10:322
1169299261,23cyclictest0-21swapper/223:35:222
1169299261,23cyclictest0-21swapper/200:30:202
1169199266,6cyclictest11281-21sh20:25:011
11691992624,1cyclictest0-21swapper/100:20:341
1169199262,17cyclictest10804-21kworker/u8:122:55:161
1169099261,2cyclictest0-21swapper/023:10:290
1169099261,1cyclictest0-21swapper/000:26:340
1169099260,24cyclictest0-21swapper/023:50:250
1169099260,24cyclictest0-21swapper/023:05:260
42532250,2sleep122-21ksoftirqd/100:15:521
1169399250,24cyclictest0-21swapper/300:00:333
1169399250,23cyclictest0-21swapper/323:45:153
11692992523,1cyclictest0-21swapper/223:45:192
11692992522,2cyclictest0-21swapper/223:25:202
1169299250,1cyclictest19974-21ssh00:00:202
11691992523,1cyclictest0-21swapper/123:45:191
11691992523,1cyclictest0-21swapper/123:25:231
11691992514,7cyclictest55450irq/143-enp3s0-23:50:041
11690992523,1cyclictest0-21swapper/023:00:360
11690992523,1cyclictest0-21swapper/000:40:010
11690992523,1cyclictest0-21swapper/000:20:290
11690992515,8cyclictest19851-21cron21:59:590
68732240,2sleep20-21swapper/222:17:332
3112240,2sleep20-21swapper/221:40:332
269352240,7sleep012-21ksoftirqd/022:35:420
11693992421,2cyclictest0-21swapper/300:15:183
11693992414,7cyclictest9131-21df22:50:203
1169399241,1cyclictest0-21swapper/323:03:513
11692992422,1cyclictest0-21swapper/221:50:202
11692992418,4cyclictest19126-21munin-run23:00:012
11691992422,1cyclictest0-21swapper/122:05:271
1169199242,16cyclictest20288-21apt-get22:00:161
11690992421,2cyclictest0-21swapper/023:35:270
11690992416,6cyclictest11359-21sshd21:20:430
297932230,2sleep10-21swapper/121:05:231
11693992321,1cyclictest0-21swapper/323:20:223
11693992321,1cyclictest0-21swapper/300:39:473
11693992321,1cyclictest0-21swapper/300:05:313
11692992321,1cyclictest0-21swapper/200:10:202
11692992321,1cyclictest0-21swapper/200:10:192
1169199232,14cyclictest25869-21runrttasks21:35:151
1169199232,14cyclictest14934-21ssh21:55:141
11691992321,1cyclictest0-21swapper/123:10:331
11691992311,3cyclictest22-21ksoftirqd/122:10:271
11691992310,8cyclictest55450irq/143-enp3s0-23:55:151
1169399222,14cyclictest19801-21apt-get22:30:183
11693992220,1cyclictest0-21swapper/323:55:023
11693992220,1cyclictest0-21swapper/321:50:203
11691992220,1cyclictest0-21swapper/100:26:301
1169199221,8cyclictest19349-21munin-run22:30:001
11691992213,6cyclictest55450irq/143-enp3s0-21:40:311
11690992215,5cyclictest12-21ksoftirqd/022:05:000
55042211,17sleep00-21swapper/021:45:330
1169399218,11cyclictest8276-21mailstats00:20:303
1169399215,11cyclictest34-21ksoftirqd/322:25:033
11693992114,5cyclictest34-21ksoftirqd/323:00:013
11693992113,5cyclictest6883-21sh22:17:343
11692992112,6cyclictest24607-21/usr/sbin/munin19:40:152
1169299210,4cyclictest0-21swapper/200:36:122
11691992119,1cyclictest0-21swapper/123:35:071
11690992119,1cyclictest0-21swapper/023:20:160
11690992115,4cyclictest2879-21munin-run23:45:010
11690992113,6cyclictest22782-21diskstats20:50:160
11690992113,6cyclictest22782-21diskstats20:50:150
11690992113,6cyclictest12000-21fw_packets20:25:170
11690992112,6cyclictest14335-21cat22:55:180
1169399208,7cyclictest31049-21apt-get19:55:123
11693992013,4cyclictest5126-21sh23:15:413
11693992012,6cyclictest32164-21ntpq21:10:213
11693992012,6cyclictest13013-21sed19:10:333
11693992012,6cyclictest11711-21awk00:25:153
11693992011,6cyclictest30235-21cat00:10:163
11693992011,6cyclictest30235-21cat00:10:163
1169299202,13cyclictest14474-21taskset22:25:152
11692992013,5cyclictest1233-21perl20:00:162
11692992012,6cyclictest31311-21ls23:40:292
11692992011,6cyclictest8103-21pluginstate00:20:282
11692992011,6cyclictest10064-21ntpq22:20:262
1169299201,12cyclictest8747-21ssh23:20:152
1169199203,5cyclictest0-21swapper/121:25:301
11691992014,4cyclictest22-21ksoftirqd/120:59:591
1169199201,3cyclictest22-21ksoftirqd/121:50:261
11691992012,6cyclictest21107-21sed19:30:201
11691992012,6cyclictest15803-21cat20:35:111
11691992011,6cyclictest1-21systemd21:30:051
11691992010,8cyclictest31497-21proc_pri22:40:251
11691992010,7cyclictest8773-21apt-get22:50:181
11690992018,1cyclictest0-21swapper/023:34:340
11690992013,5cyclictest7295-21ls20:15:140
11690992012,6cyclictest26648-21ntpq22:05:270
11690992012,6cyclictest13469-21seq22:54:400
11690992012,6cyclictest11293-21basename20:25:010
11690992012,6cyclictest11079-21ls21:50:270
1169399199,7cyclictest4276-21grep21:15:183
1169399198,7cyclictest0-21swapper/320:35:113
1169399192,11cyclictest20321-21apt-get19:30:133
1169399191,5cyclictest0-21swapper/321:41:263
11693991915,3cyclictest22347-21ssh22:01:043
11693991913,4cyclictest4314-21missed_timers22:45:283
11693991913,4cyclictest28755-21sh22:38:193
11693991913,3cyclictest25138-21gltestperf19:40:173
11693991912,5cyclictest21755-21seq20:46:533
11693991912,4cyclictest31257-21sh22:10:183
11693991911,6cyclictest5229-21awk21:45:283
11693991911,6cyclictest1933-21tr20:00:243
11693991910,6cyclictest9387-21apt20:20:173
11693991910,6cyclictest26707-21cat21:00:013
11693991910,6cyclictest17003-21cut00:30:223
1169299199,7cyclictest17324-21users20:35:262
1169299198,7cyclictest26775-21apt-get19:45:132
1169299192,11cyclictest2938-21apt-get20:05:152
11692991914,3cyclictest4884-21ssh23:15:332
11692991913,4cyclictest13927-21sh00:26:242
11692991913,4cyclictest12317-21sh22:53:032
11692991912,5cyclictest19117-21awk19:25:252
11692991911,6cyclictest7880-21cut20:15:192
11692991911,6cyclictest26656-21ls21:35:212
11692991911,6cyclictest15968-21memory21:25:242
11692991910,6cyclictest9309-21cat21:20:012
1169199198,8cyclictest498-21dbus-daemon19:39:591
11691991912,5cyclictest9951-21sh22:20:231
11691991912,5cyclictest30522-21seq19:51:391
11691991912,5cyclictest20226-21sed19:30:001
11691991911,6cyclictest21126-21wc22:30:311
11691991911,6cyclictest20185-21idleruntime23:30:241
1169099199,8cyclictest29755-21grep19:50:220
1169099199,7cyclictest3020-21sort20:05:150
1169099192,12cyclictest3327-21apt-get23:15:180
11690991914,4cyclictest12-21ksoftirqd/022:20:000
11690991913,4cyclictest3457-21sh21:14:410
11690991912,5cyclictest20341-21seq21:30:030
11690991912,5cyclictest19203-21ps19:25:250
11690991912,5cyclictest17199-21sensors19:20:230
11690991911,6cyclictest28784-21ls21:05:010
11690991911,6cyclictest22739-21expr19:35:150
11690991911,6cyclictest15903-21cat22:25:290
11690991910,7cyclictest908-21grep20:00:140
11690991910,7cyclictest3139-21cat22:45:010
11690991910,6cyclictest5360-21expr20:10:160
36632180,2sleep00-21swapper/021:44:520
278532181,14sleep10-21swapper/122:37:001
1169399182,12cyclictest14520-21apt-get21:25:023
1169399181,8cyclictest16632-21users21:55:323
1169399181,7cyclictest30806-21tune2fs23:40:223
11693991813,3cyclictest28887-21/usr/sbin/munin19:50:243
11693991813,3cyclictest23571-21sendmail_mailtr20:50:263
11693991813,3cyclictest23571-21sendmail_mailtr20:50:263
11693991813,3cyclictest18158-21/usr/sbin/munin19:25:303
11693991813,3cyclictest12237-21sh22:22:403
11693991812,4cyclictest25547-21sh22:05:113
11693991812,3cyclictest23836-21sh23:34:083
11693991811,5cyclictest10897-21sed21:20:293
11693991810,6cyclictest12744-21wc20:25:253
11693991810,5cyclictest8251-21cut20:15:233
1169299182,13cyclictest5461-21fschecks_count20:10:162
11692991813,3cyclictest7151-21sh21:47:152
11692991812,4cyclictest19563-21sh21:59:362
11692991812,4cyclictest14450irq/133-ahci[0020:35:002
11692991812,4cyclictest11247-21sh21:20:332
11692991811,5cyclictest30457-21cat22:40:162
11692991811,5cyclictest25572-21grep20:55:232
11692991811,5cyclictest23784-21cat20:50:282
11692991811,5cyclictest23784-21cat20:50:282
11692991811,5cyclictest21698-21sed19:30:262
11692991811,5cyclictest21273-21tr20:45:282
11692991811,4cyclictest30788-21diskmemload22:38:002
1169299181,12cyclictest19820-21ssh22:30:192
11692991810,6cyclictest21017-21wc23:30:332
1169199189,7cyclictest0-21swapper/119:15:001
1169199182,9cyclictest3430-21ssh22:45:181
11691991812,4cyclictest27084-21df21:00:191
11691991812,4cyclictest22-21ksoftirqd/119:59:591
11691991812,4cyclictest17522-21grep19:20:271
11691991811,5cyclictest22261-21awk00:35:231
11691991811,5cyclictest18230-21expr20:40:151
11691991811,4cyclictest30788-21diskmemload21:13:011
1169199181,12cyclictest16560-21apt-get00:30:191
11691991810,6cyclictest9523-21apt21:20:161
11691991810,6cyclictest15121-21awk20:30:311
11691991810,5cyclictest12236-21latency20:25:201
1169099189,7cyclictest24749-21cut22:35:000
1169099189,7cyclictest24707-21ls19:40:140
1169099182,12cyclictest9076-21apt-get22:20:180
11690991813,3cyclictest16957-21sh21:25:430
11690991812,4cyclictest29100-21sh21:37:400
11690991812,4cyclictest20678-21forks19:30:150
11690991811,5cyclictest5493-21seq21:15:300
11690991811,5cyclictest5142-21seq00:17:280
11690991811,5cyclictest30233-21sed21:05:290
11690991811,5cyclictest25400-21cat20:55:200
11690991810,5cyclictest55250irq/141-enp3s0-22:10:280
1169399178,6cyclictest13657-21/usr/sbin/munin20:30:153
1169399172,10cyclictest17981-21apt-get20:40:143
1169399171,2cyclictest141rcu_preempt19:35:263
11693991711,3cyclictest30569-21sh21:39:413
11693991711,3cyclictest15223-21timerwakeupswit19:15:263
11693991710,5cyclictest6300-21cat20:10:263
11693991710,5cyclictest27141-21uniq21:00:213
11693991710,5cyclictest16816-21ls19:20:203
1169299179,6cyclictest20462-21grep22:00:172
1169299175,3cyclictest141rcu_preempt21:10:152
1169299174,3cyclictest141rcu_preempt22:05:332
11692991713,2cyclictest11709-21/usr/sbin/munin19:10:212
11692991711,4cyclictest29507-21basename19:50:182
1169299171,13cyclictest4834-21find22:45:352
11692991710,5cyclictest23275-21cat19:35:212
11692991710,5cyclictest18149-21cat20:40:152
11692991710,5cyclictest16636-21basename19:20:182
1169199178,7cyclictest0-21swapper/120:00:231
1169199172,4cyclictest4211-21apt-get21:45:191
11691991712,3cyclictest5191-21sh21:15:281
1169199171,12cyclictest23370-21ntp_states20:50:241
1169199171,12cyclictest23370-21ntp_states20:50:241
1169199171,11cyclictest7220-21apt-get20:15:131
1169199171,11cyclictest5239-21ssh22:15:301
1169099173,8cyclictest11735-21apt-get19:10:120
11690991712,3cyclictest31426-21fschecks_count19:55:160
11690991711,3cyclictest30788-21diskmemload00:11:250
11690991711,3cyclictest30788-21diskmemload00:11:250
11690991710,5cyclictest28022-21grep19:45:280
11690991710,5cyclictest16743-21seq20:35:210
1169399169,5cyclictest28957-21sort21:05:173
1169399168,5cyclictest4058-21mailstats20:05:253
11693991611,3cyclictest26750-21/usr/sbin/munin19:45:223
1169299168,6cyclictest12836-21cat20:25:272
1169299168,5cyclictest32256-21switchtime19:55:242
1169299168,5cyclictest28948-21awk21:05:172
1169299167,6cyclictest15065-21switchtime19:15:252
1169299161,10cyclictest9388-21apt-get20:20:162
11692991610,4cyclictest27329-21hostname21:00:222
1169199168,6cyclictest25060-21cut19:40:171
11691991611,3cyclictest3551-21if_enp3s020:05:191
1169199161,11cyclictest5080-21apt-get20:10:131
11691991610,4cyclictest28574-21seq19:49:051
11691991610,4cyclictest13989-21expr19:15:121
1169099169,5cyclictest14668-21sed19:15:200
1169099168,6cyclictest20214-21grep20:45:160
11690991611,3cyclictest19872-21latency_hist20:45:000
1169099161,10cyclictest13682-21apt-get20:30:140
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional