You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-03-31 - 05:25

x86 Intel Celeron N3350 @1100 MHz, Linux 5.10.90-rt60 (Profile)

Latency plot of system in rack #c, slot #4
Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Command line: cyclictest -l100000000 -m -Sp99 -i200 -h400 -q
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot4.osadl.org (updated Fri Mar 31, 2023 00:44:57)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9204622268192,27sleep10-21swapper/119:07:481
9202572248196,28sleep00-21swapper/019:05:360
92079199227222,3cyclictest979931-21kworker/u8:121:42:571
920791995525,15cyclictest1042265-21ssh22:52:141
10067062550,8sleep1251rcuc/122:00:531
10067062550,8sleep1251rcuc/122:00:521
92079199548,3cyclictest0-21swapper/120:15:221
10499432540,5sleep00-21swapper/023:02:560
92079199536,4cyclictest1110975-21grep00:25:281
920791995222,7cyclictest1029510-21store22:35:021
920791995210,2cyclictest0-21swapper/121:45:261
920791995116,21cyclictest986963-21ssh21:35:191
920791995112,7cyclictest1099422-21df_inode00:10:271
92079199508,3cyclictest0-21swapper/100:38:351
92079199507,3cyclictest0-21swapper/121:51:191
920791995017,15cyclictest1010888-21ssh22:07:261
920791995015,21cyclictest1051709-21ssh23:05:201
920791995012,11cyclictest1058252-21ssh23:15:001
920791994918,2cyclictest1059021-21cut23:15:251
92079199491,3cyclictest0-21swapper/122:37:091
920791994818,16cyclictest1115349-21rm00:31:471
920791994812,3cyclictest0-21swapper/100:22:201
92079199478,2cyclictest0-21swapper/123:55:211
92079199476,3cyclictest0-21swapper/121:55:141
920791994719,3cyclictest1021045-21wc22:20:281
920791994719,3cyclictest1021045-21wc22:20:281
920789994713,7cyclictest942604-21tail20:05:180
920789994618,5cyclictest1024792-21unixbench_singl22:25:420
920791994517,10cyclictest1063339-21awk23:20:261
920791994512,4cyclictest26-21ksoftirqd/120:14:181
920791994510,17cyclictest1013878-21ssh22:10:481
920789994514,7cyclictest1099076-40apt-check00:10:190
92079199446,5cyclictest1074455-21grep23:35:271
92078999442,4cyclictest1117286-21apt-key00:35:040
9974932430,2sleep00-21swapper/021:50:010
10243202430,6sleep1251rcuc/122:25:381
92078999421,2cyclictest18550irq/39-mmc000:00:010
159199420,5rtkit-daemon1076124-21rm23:38:320
159199420,3rtkit-daemon0-21swapper/120:46:341
92079199416,5cyclictest1050959-21wc23:05:011
920789994017,8cyclictest985771-21cron21:35:000
920789994014,4cyclictest954577-21needreboot20:35:210
159199390,7rtkit-daemon1069384-21latency_hist23:30:010
92079199381,3cyclictest141rcu_preempt23:50:001
92078999388,26cyclictest1116861-21ssh00:35:020
92078999383,9cyclictest1005158-21apt-get22:00:160
92078999383,9cyclictest1005158-21apt-get22:00:150
92079199376,2cyclictest141rcu_preempt20:30:241
92078999373,12cyclictest935686-21apt-get19:45:180
920789993715,17cyclictest934474-21head19:40:390
159199370,5rtkit-daemon960816-21ls20:50:280
159199370,3rtkit-daemon1590-21rtkit-daemon20:25:011
920791993620,7cyclictest938636-21grep19:55:191
920791993617,10cyclictest960593-21ls20:50:251
92078999362,3cyclictest0-21swapper/022:35:010
920789993618,9cyclictest1058903-21fschecks_time23:15:240
159199360,9rtkit-daemon1066083-21diskstats23:25:181
159199360,6rtkit-daemon1065081-21ssh23:23:530
92078999354,2cyclictest141rcu_preempt21:40:020
920789993524,3cyclictest1047261-21cron23:00:000
920789993522,3cyclictest1087821-21cron23:55:000
92078999350,3cyclictest1029510-21store22:35:030
920789993425,4cyclictest3147515-20systemd-journal22:09:560
920789993415,4cyclictest1092425-21interrupts00:00:240
92079199338,5cyclictest0-21swapper/121:33:381
92079199330,2cyclictest141rcu_preempt23:45:011
92078999334,17cyclictest1113123-21latency_hist00:30:010
920789993326,3cyclictest1058271-21perf23:14:590
92078999332,5cyclictest13-21ksoftirqd/000:09:570
920789993322,2cyclictest958173-21gltestperf20:45:190
920789993322,2cyclictest1080344-21latency_hist23:45:000
920789993322,2cyclictest0-21swapper/000:20:340
920789993320,3cyclictest944533-21tail20:10:190
920789993314,10cyclictest1039859-21awk22:50:000
920789993312,6cyclictest1020973-21switchtime22:20:280
920789993312,6cyclictest1020973-21switchtime22:20:280
92078999330,4cyclictest956253-21head20:40:200
92078999330,4cyclictest956253-21head20:40:200
9361822327,2sleep1936247-21sort19:45:251
92079199329,4cyclictest0-21swapper/119:43:481
92079199329,4cyclictest0-21swapper/100:05:201
92079199328,3cyclictest0-21swapper/122:45:251
920791993222,4cyclictest1016380-21tune2fs22:15:181
92079199322,14cyclictest1043799-21apt-get22:55:141
920791993210,3cyclictest0-21swapper/122:44:481
92078999326,4cyclictest1054540-21grep23:10:000
92078999325,6cyclictest951238-21unixbench_multi20:25:360
920789993212,4cyclictest0-21swapper/020:20:160
159199320,5rtkit-daemon1092403-21if_enp2s000:00:241
92079199319,3cyclictest0-21swapper/123:30:241
92079199311,6cyclictest1105870-21sendmail-msp00:20:001
920791993113,5cyclictest940875-21cat20:00:211
920791993113,5cyclictest0-21swapper/123:50:231
920791993110,2cyclictest0-21swapper/120:40:181
920791993110,2cyclictest0-21swapper/120:40:181
920789993124,2cyclictest1004842-21cron22:00:000
920789993118,3cyclictest1016203-21tr22:15:160
92079199306,2cyclictest0-21swapper/120:55:141
920791993016,3cyclictest954925-21grep20:35:271
92078999309,2cyclictest0-21swapper/000:15:210
92078999308,4cyclictest938940-21needreboot19:55:220
92078999307,4cyclictest990595-21grep21:40:180
920789993022,3cyclictest953698-21dump-pmu-power20:35:010
920789993017,5cyclictest1069606-21ssh23:30:170
920789993013,4cyclictest961810-21cpuspeed_turbos20:55:150
920789993010,4cyclictest1040827-21needreboot22:50:220
92079199292,3cyclictest0-21swapper/120:25:281
92079199292,3cyclictest0-21swapper/120:05:151
92078999296,4cyclictest1081249-21latency23:45:280
920789992923,4cyclictest1035759-21turbostat.cron22:45:000
920789992922,2cyclictest940674-21tail20:00:190
92078999290,4cyclictest997824-21tr21:50:210
159199290,3rtkit-daemon1590-21rtkit-daemon21:29:201
92079199287,2cyclictest0-21swapper/119:50:301
92079199286,3cyclictest141rcu_preempt21:20:011
920791992811,4cyclictest3147515-20systemd-journal21:00:231
920791992811,2cyclictest0-21swapper/119:25:151
920789992821,2cyclictest933358-21store19:40:010
92078999281,4cyclictest947890-21grep20:20:010
92078999272,2cyclictest0-21swapper/021:10:020
920789992716,6cyclictest964101-40apt-check21:00:140
92078999271,3cyclictest0-21swapper/022:10:210
92078999271,3cyclictest0-21swapper/019:15:020
920791992614,3cyclictest26-21ksoftirqd/119:21:401
920789992618,3cyclictest938061-21cron19:54:590
920789992615,2cyclictest0-21swapper/019:32:310
920789992614,2cyclictest13-21ksoftirqd/019:25:150
920789992611,2cyclictest0-21swapper/019:10:270
92079199254,6cyclictest933350-21sort19:40:021
92079199252,15cyclictest966106-21apt-get21:05:131
920789992510,10cyclictest970831-21apt-get21:15:140
92079199242,14cyclictest974816-21apt-get21:20:151
920791992412,2cyclictest0-21swapper/121:12:241
92078999244,15cyclictest3147515-20systemd-journal21:30:000
920789992412,2cyclictest0-21swapper/021:10:010
92079199225,3cyclictest0-21swapper/119:33:371
920789992212,2cyclictest63950irq/130-enp2s019:20:120
92079199215,2cyclictest0-21swapper/119:19:591
92079199203,3cyclictest0-21swapper/119:10:071
92078999200,3cyclictest975154-21fschecks_time21:20:190
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:533
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
none09:06:532
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional