You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-24 - 19:39

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot4s.osadl.org (updated Wed Apr 24, 2024 12:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
4302993111,5cyclictest9931-21perl12:35:151
4302993011,4cyclictest30675-21expr08:40:131
430299301,13cyclictest25845-21ls10:05:011
430299298,5cyclictest12547-21expr07:45:091
430299298,5cyclictest12547-21expr07:45:091
4302992917,10cyclictest27461-21fschecks_count08:30:171
430299291,10cyclictest9257-21grep09:15:011
430299287,5cyclictest302-21systemd-journal07:30:001
430299277,7cyclictest0-21swapper/112:05:301
430299271,10cyclictest2772-21latency_hist08:55:001
430299269,4cyclictest5349-21cat10:45:001
430299268,5cyclictest7754-21dpkg12:30:141
430299268,5cyclictest7214-21ls10:50:151
4302992613,2cyclictest14168-21fschecks_time11:10:151
4302992612,3cyclictest1980-21dump-pmu-power10:35:001
4302992611,3cyclictest23452-21timerandwakeup08:15:261
4302992610,4cyclictest17585-21tr08:00:131
430799253,7cyclictest12161-21perf07:44:592
430799252,15cyclictest17189-21snmpd10:00:132
430299258,4cyclictest16921-21grep07:55:291
4302992513,2cyclictest24201-21df08:20:131
430299251,12cyclictest12964-40apt-check09:25:081
430299251,10cyclictest12510-21lsb_release09:25:001
4295992514,7cyclictest22145-21wc08:15:020
4302992413,2cyclictest28374-21latency10:10:181
4302992413,2cyclictest19474-21cpu09:45:131
4302992412,2cyclictest25033-21latency_hist11:45:011
430299241,10cyclictest19636-21ntp_states08:05:201
4295992415,4cyclictest31521-21latency_hist12:05:000
4312992310,9cyclictest17257-21apt-get08:00:103
431299230,7cyclictest12984-21apt-get09:25:153
4302992310,2cyclictest27614-21sh10:10:001
4295992316,4cyclictest23099-21sed09:55:140
4295992315,4cyclictest32473-21sh10:25:000
4295992314,6cyclictest10123-21basename11:00:000
4312992215,5cyclictest10163-21http11:00:013
430799222,13cyclictest22675-21apt-get08:15:142
4307992214,6cyclictest31450-21awk12:00:282
4307992214,6cyclictest25337-21users08:20:292
430799220,7cyclictest25-21ksoftirqd/208:45:162
430299225,4cyclictest0-21swapper/112:00:231
429599228,12cyclictest302-21systemd-journal08:34:590
429599222,14cyclictest4341-21apt-get07:10:120
429599221,14cyclictest18528-21lsb_release11:24:590
4312992115,4cyclictest10324-21mandb07:35:013
4312992114,5cyclictest2343-21snmp_o5powerctl08:50:263
431299211,3cyclictest3651-21lsb_release10:39:593
431299211,3cyclictest25752-21python310:05:013
431299211,3cyclictest21910-21python308:15:003
431299211,3cyclictest10616-40apt-check11:00:053
431299211,17cyclictest14078-21grep07:50:133
431299211,17cyclictest12511-21python309:25:003
431299210,3cyclictest9822-21df12:35:133
431299210,18cyclictest6121-40apt-check12:25:043
431299210,18cyclictest5665-21python312:25:003
430799212,13cyclictest31091-21apt-get10:20:112
430799212,13cyclictest1336-21apt-get08:50:142
4307992112,6cyclictest5672-21df10:45:152
430799210,6cyclictest17191-21cat08:00:012
4302992116,3cyclictest29727-21processes08:35:291
4302992116,3cyclictest11496-21processes11:00:211
4302992114,5cyclictest7605-21sed12:30:001
4302992113,6cyclictest22840-21expr09:55:121
4302992113,6cyclictest15966-21sshd09:34:481
4302992112,2cyclictest8715-21/usr/sbin/munin10:55:261
430299211,10cyclictest20489-21apt-get08:10:131
429599212,13cyclictest4241-21apt-get12:20:150
4295992114,5cyclictest13234-21mailstats11:05:250
4295992113,6cyclictest2749-21cut10:35:120
429599211,13cyclictest24018-21apt-get08:20:130
431299203,15cyclictest5267-21latency_hist10:45:003
431299202,12cyclictest21881-21apt-get08:15:133
4312992015,3cyclictest17611-21munin-run09:40:003
431299201,4cyclictest4342-21cron09:00:013
4312992013,5cyclictest8729-21basename12:30:283
4312992013,5cyclictest8528-21ls07:25:123
4312992013,5cyclictest28010-21sed11:50:333
4312992013,5cyclictest25112-21cat08:20:283
4312992013,5cyclictest19645-21expr09:45:143
4312992013,5cyclictest15848-21awk09:30:273
4312992012,5cyclictest10409-21grep09:15:233
431299201,13cyclictest30134-21apt-get12:00:123
431299200,6cyclictest18530-21basename11:25:003
431299200,4cyclictest693-21cron12:10:013
431299200,3cyclictest7630-21lsb_release09:10:003
431299200,18cyclictest20147-21awk11:30:003
430799203,4cyclictest25-21ksoftirqd/211:15:162
430799202,6cyclictest25-21ksoftirqd/208:35:122
430799202,13cyclictest30499-21apt-get08:40:112
430799202,12cyclictest23652-21apt-get11:40:122
430799202,12cyclictest17122-21apt-get11:20:112
430799201,6cyclictest8956-21fschecks_count10:55:172
4307992012,6cyclictest27471-21sed11:50:242
4307992012,6cyclictest20651-21perl11:30:112
4307992012,5cyclictest22705-21grep09:55:112
430799201,14cyclictest9457-21apt-get09:15:112
430799201,14cyclictest17852-21apt-get09:40:122
430799201,13cyclictest9636-21apt-get12:35:132
430799201,13cyclictest25637-21apt-get08:25:142
4307992010,7cyclictest1719-21lsb_release10:30:002
4302992016,2cyclictest19655-21sendmail_mailqu11:25:201
4302992012,6cyclictest18103-21grep11:20:221
4302992011,6cyclictest25448-21df11:45:131
4302992011,6cyclictest25448-21df11:45:131
429599203,9cyclictest26214-40apt-check10:05:060
429599202,13cyclictest3959-21kworker/u8:012:25:060
429599202,12cyclictest25743-21lsb_release10:05:000
4295992013,5cyclictest7239-21expr10:50:160
4295992012,6cyclictest3388-21grep08:55:170
4295992012,5cyclictest15265-21awk07:50:350
4295992011,5cyclictest771-21dbus-daemon11:00:010
429599201,14cyclictest32115-21apt-get08:45:140
429599201,14cyclictest16922-21latency_hist11:20:000
431299196,11cyclictest19262-21awk09:45:003
431299192,15cyclictest652-17auditd10:25:003
4312991912,5cyclictest28088-21awk08:30:253
4312991912,5cyclictest2789-21df12:15:143
4312991911,6cyclictest29773-21perl10:15:143
431299191,16cyclictest0-21swapper/311:35:153
4312991911,5cyclictest29008-21expr08:35:133
4312991910,7cyclictest31307-21uniq10:20:153
4312991910,7cyclictest23184-21cat09:55:153
431299190,3cyclictest10293-1kworker/3:2H07:40:013
431299190,2cyclictest27013-21cron08:30:003
430799192,13cyclictest13494-21sed11:05:302
4307991912,5cyclictest5268-21cat09:00:192
4307991912,5cyclictest4987-21cat07:10:172
4307991912,5cyclictest389-21sed12:05:302
4307991912,5cyclictest14426-21sed11:10:172
4307991912,4cyclictest8076-21grep07:20:232
4307991911,6cyclictest4844-21tr10:40:242
4307991911,6cyclictest20533-21cat09:45:282
4307991911,5cyclictest29072-21grep10:10:382
430799191,13cyclictest4049-21sed12:20:002
430799191,13cyclictest2963-21apt-get08:55:112
430799191,13cyclictest26206-40apt-check10:05:092
430799191,13cyclictest21970-21apt-get11:35:162
430799191,13cyclictest18873-21apt-get08:05:112
430799191,13cyclictest12964-40apt-check09:25:052
430799191,12cyclictest9177-21turbostat12:35:002
430799191,12cyclictest28496-21apt-get11:55:102
430799191,12cyclictest21490-21snmp_o5powerctl08:10:312
430799191,12cyclictest11094-21apt-get09:20:152
430299192,13cyclictest23344-21apt-get11:38:491
4302991915,2cyclictest27377-21interrupts11:50:241
4302991915,2cyclictest16577-21gltestperf09:35:151
4302991912,5cyclictest29643-21df10:15:131
4302991912,5cyclictest26108-21cat08:25:181
4302991911,6cyclictest4435-21df_inode12:20:151
4302991911,5cyclictest21815-21ntp_kernel_pll_09:50:201
4295991913,3cyclictest6252-21ntp_states10:45:230
4295991913,3cyclictest25037-21munin-plugin-st11:44:590
4295991912,5cyclictest31937-21sed08:45:010
4295991912,5cyclictest28815-21perl11:55:140
4295991912,5cyclictest20475-21expr11:30:100
4295991912,5cyclictest14921-21grep09:30:130
4295991912,5cyclictest14474-21grep11:10:170
4295991911,6cyclictest21505-21sed09:50:150
429599191,13cyclictest7839-21apt-get09:10:110
429599191,13cyclictest4607-21apt-get09:00:120
429599190,13cyclictest29452-21apt-get10:15:100
431299189,5cyclictest21836-21cut11:34:593
4312991815,2cyclictest0-21swapper/311:40:273
4312991814,2cyclictest17013-21latency_hist08:00:013
4312991814,2cyclictest12860-21if_enp2s007:45:113
4312991814,2cyclictest12860-21if_enp2s007:45:113
4312991812,4cyclictest6961-21expr07:20:093
4312991812,4cyclictest32083-21cat08:45:003
4312991811,5cyclictest9276-21cut09:15:003
4312991811,5cyclictest6157-21uname10:45:223
4312991811,5cyclictest4752-21expr09:00:143
4312991811,5cyclictest30032-21sed12:00:003
4312991811,5cyclictest2366-21grep12:15:013
4312991811,5cyclictest15932-21basename11:15:153
4312991811,5cyclictest14154-21tune2fs11:10:153
431299181,12cyclictest21082-21apt-get09:50:123
4312991810,6cyclictest32587-21cat08:45:183
4312991810,5cyclictest73550irq/132-enp2s011:45:133
4312991810,5cyclictest73550irq/132-enp2s011:45:133
430799189,6cyclictest10754-21tune2fs07:35:102
430799189,6cyclictest10187-21find07:35:002
430799183,9cyclictest8463-21apt-get07:25:112
4307991813,3cyclictest17630-21sed08:00:132
4307991811,5cyclictest19877-21wc11:25:262
430799181,13cyclictest27250-21apt-get08:30:132
430799181,13cyclictest15591-21snmp_o5powerctl09:30:252
430799181,12cyclictest7103-21apt-get10:50:152
430799181,12cyclictest6121-40apt-check12:25:082
430799181,12cyclictest1970-21snmp_o5powerctl12:10:242
430799181,11cyclictest25265-21apt-get11:45:132
430799181,11cyclictest25265-21apt-get11:45:122
430799181,11cyclictest12390-21apt-get07:45:092
430799181,11cyclictest12390-21apt-get07:45:082
4307991810,6cyclictest31067-21sed10:20:012
4307991810,5cyclictest7640-21grep09:10:002
4302991814,2cyclictest9968-21kernelversion09:15:181
4302991814,2cyclictest2724-21kernelversion10:35:121
4302991814,2cyclictest12744-21interrupts11:05:181
4302991812,4cyclictest2717-21expr12:15:121
4302991811,5cyclictest6809-21users10:45:321
4302991811,5cyclictest6764-21date07:20:001
4302991811,5cyclictest32568-21cat08:45:191
4302991811,5cyclictest21079-21ntp_kernel_err11:30:161
4302991811,5cyclictest1858-21cat10:30:001
4302991811,5cyclictest18018-21expr09:40:131
429599189,5cyclictest0-21swapper/012:35:000
429599183,9cyclictest10474-21apt-get07:35:080
429599183,8cyclictest12474-21apt-get09:25:120
429599182,12cyclictest19254-21munin-run09:45:000
429599182,11cyclictest22376-21apt-get08:15:090
4295991814,2cyclictest6601-21latency_hist07:20:000
4295991814,2cyclictest2369-21latency_hist12:15:000
4295991812,4cyclictest5379-21sed10:45:000
4295991811,5cyclictest19212-21head08:05:150
4295991811,5cyclictest15621-21dpkg07:55:100
4295991811,4cyclictest10372-21latency_hist07:35:000
429599181,12cyclictest9466-21apt-key12:35:000
429599181,12cyclictest6846-21apt-get07:20:090
429599181,12cyclictest28864-21apt-get08:35:130
429599181,12cyclictest19471-21apt-get09:45:120
429599181,11cyclictest12525-21latency_hist09:25:000
431299179,6cyclictest28157-21tune2fs10:10:163
4312991713,2cyclictest26025-21apt-config10:05:013
4312991713,2cyclictest18870-21/usr/sbin/munin08:05:153
4312991713,2cyclictest12150-21cron07:44:593
431299171,12cyclictest8098-21snmp_o5powerctl10:50:303
4312991710,5cyclictest4535-21df07:10:133
431299170,1cyclictest0-21swapper/307:19:593
431299170,16cyclictest0-21swapper/311:10:003
4307991715,1cyclictest0-21swapper/210:33:502
4307991713,2cyclictest21079-21/usr/sbin/munin09:50:162
4307991712,3cyclictest7946-21ls09:10:122
4307991712,3cyclictest4923-21needreboot12:20:202
430799171,12cyclictest17208-21snmp_o5powerctl09:35:252
430799171,12cyclictest14004-21apt-get07:50:122
430299178,6cyclictest3604-21cat08:55:201
4302991713,2cyclictest6641-21hwlatdetect09:05:181
4302991712,3cyclictest5006-21needreboot07:10:171
430299171,12cyclictest1969-21snmp_o5powerctl12:10:231
430299171,12cyclictest15427-21sort07:55:011
4302991710,5cyclictest8435-21cut07:25:001
4302991710,5cyclictest4801-21df09:00:141
4302991710,5cyclictest10921-21basename07:35:111
429599173,11cyclictest18648-21idleruntime-cro08:05:010
4295991713,2cyclictest19740-21swap11:25:260
4295991713,2cyclictest13783-21latency_hist07:50:010
4295991713,2cyclictest13783-21latency_hist07:50:010
4295991711,4cyclictest25402-21expr11:45:130
4295991711,4cyclictest25402-21expr11:45:130
4295991711,4cyclictest22091-21expr11:35:160
429599171,11cyclictest2599-21apt-get12:15:110
4295991710,5cyclictest8845-21expr07:25:130
4295991710,5cyclictest6609-21sed09:05:190
4295991710,5cyclictest16814-21memory09:35:180
4295991710,5cyclictest10428-21tr09:15:230
431299169,5cyclictest2115-21cut10:35:003
4307991612,2cyclictest2321-21cstates10:35:082
4307991611,3cyclictest6082-21ntp_states07:15:022
4307991610,4cyclictest10435-21mktemp11:00:002
430299169,5cyclictest16150-21grep11:15:171
4302991610,4cyclictest28605-21expr11:55:111
429599169,5cyclictest9-21ksoftirqd/010:25:160
429599169,5cyclictest31947-21entropy12:05:150
429599169,5cyclictest12290-21sed07:45:010
429599163,7cyclictest2749-21lsb_release08:55:000
4295991612,2cyclictest29228-21latency_hist10:15:010
429599161,11cyclictest25902-21fschecks_time08:25:160
4312991511,2cyclictest4117-21latency_hist07:10:013
4302991511,2cyclictest31955-21processes10:20:201
429599159,4cyclictest27381-21irqstats11:50:250
4295991513,1cyclictest0-21swapper/010:30:210
4302991412,1cyclictest0-21swapper/107:32:291
430799132,8cyclictest23765-21munin-node07:10:012
430299138,4cyclictest12320-21latency_hist07:45:001
429599121,7cyclictest4299-21latency_hist07:10:000
430299118,2cyclictest4117-21latency_hist07:10:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional