You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-05-27 - 18:29

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot4s.osadl.org (updated Mon May 27, 2024 12:44:21)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
13480993912,10cyclictest14374-21timerwakeupswit11:55:282
13480993812,11cyclictest32445-21ntp_states08:00:222
13480993811,11cyclictest28694-21awk07:50:162
13480993811,11cyclictest16799-21unixbench_singl10:25:362
13480993810,12cyclictest18527-21cat07:20:162
13480993714,8cyclictest8557-21mailstats08:25:242
13480993712,8cyclictest5836-21snmp_o5powerctl09:55:292
13480993711,9cyclictest963-21snmp_o5powerctl09:40:312
13480993711,9cyclictest963-21snmp_o5powerctl09:40:302
13480993710,10cyclictest9854-21sed08:30:202
13480993613,7cyclictest3615-21snmp_o5powerctl08:10:292
13480993611,8cyclictest20479-21ntpq12:15:302
1348099357,16cyclictest0-21swapper/212:00:152
13480993512,9cyclictest7590-21ps11:35:262
13480993511,11cyclictest16507-21awk08:50:242
13480993511,11cyclictest16507-21awk08:50:232
13480993511,10cyclictest19318-21sed10:35:232
13480993511,10cyclictest19318-21sed10:35:222
1348099348,14cyclictest0-21swapper/209:20:112
13480993412,8cyclictest22336-21mailstats12:20:302
13480993411,9cyclictest21495-21tr09:05:262
13480993411,9cyclictest17345-21processes07:16:322
13480993411,8cyclictest25497-21irqstats07:40:192
13480993411,7cyclictest29929-21snmp_o5powerctl11:05:322
13480993411,10cyclictest2076-21sed08:05:302
13480993312,7cyclictest31493-21ntp_states09:35:232
13480993312,7cyclictest31493-21ntp_states09:35:232
13480993311,8cyclictest6913-21mailstats08:20:252
13480993311,8cyclictest21977-21perl07:30:162
13480993310,8cyclictest19714-21open_files09:00:252
13480993310,10cyclictest26337-21sed10:55:272
1348099330,18cyclictest24215-21sed12:25:322
1348099327,15cyclictest0-21swapper/207:10:252
13480993211,7cyclictest22734-21ntpq10:45:212
13480993112,6cyclictest4819-21users11:25:352
13480993111,6cyclictest20777-21ntp_states07:25:242
1348099308,8cyclictest0-21swapper/212:05:162
1348099308,8cyclictest0-21swapper/209:45:142
1348099298,8cyclictest0-21swapper/211:45:152
1348099298,8cyclictest0-21swapper/211:00:322
1348099298,7cyclictest28264-21ntp_states09:25:282
1348099298,7cyclictest18563-21if_enp2s012:10:152
1348099297,8cyclictest0-21swapper/211:25:002
13480992918,8cyclictest5638-21sshd08:19:002
13480992916,10cyclictest25221-21sendmail-msp09:20:002
13480992916,10cyclictest11820-21apt-config10:15:002
1348099287,6cyclictest0-21swapper/207:35:172
13480992815,10cyclictest9776-21sleep11:45:002
13480992814,10cyclictest25016-21python310:55:012
13480992810,7cyclictest0-21swapper/211:15:232
13480992716,8cyclictest31483-21rm08:00:002
13480992714,8cyclictest0-21swapper/207:45:122
13480992710,10cyclictest0-21swapper/212:35:102
1346999271,9cyclictest9-21ksoftirqd/012:05:220
1346999271,8cyclictest9-21ksoftirqd/008:50:200
1346999271,8cyclictest9-21ksoftirqd/008:50:200
1348099268,6cyclictest13875-21/usr/sbin/munin10:20:272
1348099267,5cyclictest0-21swapper/210:05:182
13480992617,6cyclictest3951-21awk09:50:222
13480992615,8cyclictest13444-21/usr/sbin/munin08:40:282
13480992610,10cyclictest0-21swapper/210:40:112
13480992518,4cyclictest18136-21grep08:55:232
13480992518,4cyclictest11735-21sendmail_mailqu08:35:252
13480992516,7cyclictest11645-21grep11:50:142
13480992516,6cyclictest29849-21ntpq09:30:272
13480992515,7cyclictest31534-21sensors_temp11:10:272
13480992515,7cyclictest23565-21dpkg09:15:012
13480992514,9cyclictest7646-21awk10:00:232
13480992514,8cyclictest17474-21grep10:30:162
13480992514,8cyclictest15445-21cron08:50:002
13480992513,9cyclictest13771-21cat10:20:002
1346999257,6cyclictest9-21ksoftirqd/008:40:270
1346999252,9cyclictest9-21ksoftirqd/009:20:200
1348499246,16cyclictest16246-21idleruntime-cro12:05:003
1348499245,17cyclictest28398-21cstates07:50:113
13484992416,5cyclictest18405-21apt-get07:20:133
1348499241,4cyclictest31964-21python311:15:003
13480992413,9cyclictest5354-21grep11:30:172
13480992413,9cyclictest5354-21grep11:30:172
13480992413,8cyclictest24972-21tail12:30:142
1346999242,8cyclictest9-21ksoftirqd/011:45:230
13484992312,9cyclictest10578-21turbostat08:35:003
13475992315,6cyclictest19924-21awk12:15:171
1346999236,6cyclictest9-21ksoftirqd/009:05:190
1346999231,8cyclictest9-21ksoftirqd/008:55:250
1346999230,9cyclictest9-21ksoftirqd/010:45:300
1348499221,4cyclictest32424-40apt-check11:15:053
13484992214,6cyclictest7430-21date08:25:003
13484992214,6cyclictest14881-21snmp_o5powerctl10:20:323
1348499221,18cyclictest22758-21lsb_release12:25:003
1348499220,5cyclictest12975-21cron11:55:003
1348499220,19cyclictest2400-21munin-run08:10:003
1347599222,14cyclictest900-21apt-get08:05:111
13475992214,6cyclictest5746-21sed11:30:231
13475992214,6cyclictest5746-21sed11:30:231
1346999222,14cyclictest5626-21check-new-relea08:19:010
13469992218,3cyclictest101rcuc/010:40:210
13469992218,3cyclictest101rcuc/010:30:260
1346999221,7cyclictest9-21ksoftirqd/008:10:120
1346999220,6cyclictest25676-21apt-get10:55:130
1348499212,16cyclictest24363-21turbostat.cron12:29:593
13484992116,3cyclictest7441-40apt-check10:00:243
13484992116,3cyclictest5718-21idleruntime-cro08:20:013
1348499211,3cyclictest19494-21munin-run12:15:003
1348499211,3cyclictest13640-21munin-run10:20:003
13484992113,6cyclictest17624-21ls12:05:363
13484992112,6cyclictest789-21awk09:40:273
13484992112,6cyclictest789-21awk09:40:273
1348499211,18cyclictest25017-21python310:55:003
1348499210,4cyclictest6238-21plymouth10:00:003
1348499210,3cyclictest14628-21munin-run12:00:003
1348499210,18cyclictest27606-21perf12:40:003
1348499210,18cyclictest11588-21python310:15:003
13475992114,5cyclictest346-21df11:15:191
13475992113,6cyclictest39-21kauditd11:30:001
1346999212,7cyclictest9-21ksoftirqd/012:25:280
1346999212,7cyclictest9-21ksoftirqd/010:35:210
1346999212,7cyclictest9-21ksoftirqd/010:35:210
1346999212,6cyclictest9-21ksoftirqd/011:05:200
1346999211,7cyclictest9-21ksoftirqd/010:13:050
13469992113,6cyclictest9207-21sed10:05:200
13469992113,6cyclictest6103-21sensors_temp11:30:310
13469992113,6cyclictest6103-21sensors_temp11:30:310
13469992113,6cyclictest12130-21users08:35:300
1346999211,14cyclictest7758-21pmu-power10:00:250
1346999210,7cyclictest9-21ksoftirqd/012:30:130
1346999210,7cyclictest9-21ksoftirqd/010:15:360
1346999210,7cyclictest9-21ksoftirqd/007:16:330
1348499202,16cyclictest644-21latency_hist08:05:003
13484992016,2cyclictest26045-21gltestperf10:55:223
13484992015,3cyclictest20134-21munin-run10:40:013
13484992015,3cyclictest20134-21munin-run10:40:013
13484992015,2cyclictest28011-21munin-run07:50:003
13484992013,5cyclictest31056-21perl09:35:153
13484992013,5cyclictest31056-21perl09:35:153
13484992013,5cyclictest30667-21cut09:35:003
13484992013,5cyclictest20184-21cat12:15:203
13484992013,5cyclictest13645-21expr07:10:103
13484992012,6cyclictest16962-21awk08:50:303
13484992012,6cyclictest16962-21awk08:50:303
1348499200,3cyclictest13861-21lsb_release08:45:003
1348499200,18cyclictest17979-21dpkg07:20:013
13480992011,7cyclictest13495-21sed07:10:002
1347599209,8cyclictest771-21dbus-daemon08:19:001
1347599202,12cyclictest30545-21apt-get11:10:111
13475992016,2cyclictest30496-21dump-pmu-power09:35:001
13475992016,2cyclictest21001-21missed_timers10:40:201
13475992015,3cyclictest6395-21idleruntime08:20:171
13475992013,5cyclictest26524-21fw_packets12:35:151
13475992013,5cyclictest26003-21fw_packets10:55:221
13475992013,5cyclictest21991-21grep10:45:131
13475992013,5cyclictest20860-21proc_pri07:25:251
13475992012,6cyclictest24358-21snmp_o5powerctl07:35:281
13475992012,6cyclictest16734-21sed08:50:281
13475992012,6cyclictest16734-21sed08:50:271
13475992012,6cyclictest12578-21basename11:50:251
13475992012,5cyclictest22614-21awk12:20:371
13475992012,5cyclictest16906-21basename10:30:001
1347599201,14cyclictest8365-21apt-get11:40:121
1347599201,13cyclictest27505-21apt-get09:25:131
13475992010,8cyclictest23542-21cron09:15:011
1346999203,6cyclictest9-21ksoftirqd/009:25:130
1346999203,5cyclictest9-21ksoftirqd/008:50:000
1346999202,7cyclictest9-21ksoftirqd/010:20:270
1346999202,7cyclictest9-21ksoftirqd/009:15:190
1346999201,7cyclictest9-21ksoftirqd/007:20:160
13469992016,2cyclictest10582-21uname08:35:000
13469992016,2cyclictest101rcuc/007:25:270
13469992013,5cyclictest8487-21expr11:40:140
13469992013,5cyclictest5465-21sed09:55:210
13469992013,5cyclictest30240-21df_inode07:55:140
13469992013,5cyclictest24325-21ntpq10:50:200
13469992013,5cyclictest15591-21ntpq12:00:210
13469992012,6cyclictest7357-21sed11:35:210
1346999201,14cyclictest31688-21apt-get08:00:110
1346999200,7cyclictest9-21ksoftirqd/012:35:270
1346999200,7cyclictest9-21ksoftirqd/009:47:360
1346999200,7cyclictest9-21ksoftirqd/008:25:190
1346999200,7cyclictest9-21ksoftirqd/007:14:020
1346999200,6cyclictest9-21ksoftirqd/011:10:250
1346999200,6cyclictest9-21ksoftirqd/009:35:130
1346999200,6cyclictest9-21ksoftirqd/009:35:130
1348499191,8cyclictest9546-21snmp_o5powerctl10:05:263
1348499191,5cyclictest6518-21idleruntime-cro11:35:013
1348499191,5cyclictest6518-21idleruntime-cro11:35:003
13484991913,3cyclictest8972-21latency_hist08:30:003
13484991912,5cyclictest9027-21needreboot11:40:213
13484991912,5cyclictest3234-21ls08:10:223
13484991912,5cyclictest30231-21awk11:05:353
13484991912,5cyclictest24413-21ls07:35:323
13484991912,5cyclictest23344-21sshd10:48:233
13484991912,5cyclictest22317-21sed07:30:213
13484991912,5cyclictest17530-21cat10:30:153
13484991912,5cyclictest17496-21diskstats08:55:133
13484991911,6cyclictest25188-21df_inode07:40:153
13484991911,6cyclictest24964-21cat09:15:343
1348499191,16cyclictest0-21swapper/308:00:003
1348499190,9cyclictest28501-21snmp_o5powerctl09:25:343
1348499190,3cyclictest16891-21turbostat10:29:593
1348499190,2cyclictest21552-21snmp_o5powerctl09:05:293
1348499190,2cyclictest15439-21cron08:50:003
1348499190,16cyclictest8155-21sendmail-msp11:40:003
13475991915,2cyclictest8071-21interrupts08:25:181
13475991915,2cyclictest24073-21timerandwakeup12:25:311
13475991914,3cyclictest29941-21swap11:05:331
13475991914,3cyclictest18510-21ls10:35:011
13475991913,4cyclictest30695-21sed07:55:201
13475991912,5cyclictest31690-21dpkg08:00:091
13475991912,5cyclictest18971-21grep10:35:181
13475991912,5cyclictest18971-21grep10:35:181
13475991912,5cyclictest18077-21dpkg07:20:021
13475991912,5cyclictest1705-21expr09:45:141
13475991912,5cyclictest14448-21grep10:20:191
13475991912,5cyclictest14318-21sed11:55:281
13475991911,6cyclictest24816-21tune2fs12:30:131
13475991911,5cyclictest6468-21cat10:00:011
13475991911,5cyclictest11237-21grep11:46:131
1347599191,14cyclictest13519-21apt-get07:10:091
1347599191,13cyclictest6728-21apt-get11:35:131
1347599191,13cyclictest16476-21apt-get12:05:151
1347599191,12cyclictest302-21systemd-journal10:05:111
1347599191,12cyclictest23785-21apt-get09:15:101
1346999192,11cyclictest31928-21apt-get11:15:130
1346999191,6cyclictest9-21ksoftirqd/012:15:400
13469991915,3cyclictest101rcuc/011:25:330
13469991915,2cyclictest6006-21cpu08:20:130
13469991915,2cyclictest4129-21rt-features09:50:260
13469991915,2cyclictest29728-21vmstat07:50:290
13469991915,2cyclictest29630-21kernelversion09:30:240
13469991912,5cyclictest27259-21cat07:45:240
13469991912,5cyclictest2598-21awk11:20:190
13469991912,5cyclictest16249-21cat10:25:230
13469991912,5cyclictest12023-21grep11:50:160
1346999190,7cyclictest9-21ksoftirqd/012:20:380
1346999190,7cyclictest9-21ksoftirqd/009:00:190
1346999190,7cyclictest9-21ksoftirqd/007:40:300
1348499181,2cyclictest24594-1kworker/3:1H11:04:563
13484991812,4cyclictest26407-21mailstats09:20:253
13484991811,5cyclictest4596-21cron09:55:003
13484991811,5cyclictest3494-21apt11:25:103
13484991811,5cyclictest24584-21dpkg12:30:093
13484991811,5cyclictest20123-21cat07:25:123
1348499181,12cyclictest20368-21apt-get10:40:133
1348499181,12cyclictest1871-21apt-get11:20:113
1348499180,8cyclictest11804-21snmp_o5powerctl08:35:273
1348499180,16cyclictest23560-21latency_hist09:15:013
1348499180,16cyclictest0-21swapper/311:45:203
1347599189,7cyclictest7678-21ntpq10:00:241
1347599189,7cyclictest13812-21cat10:20:001
13475991814,2cyclictest28277-21sendmail_mailtr11:00:261
13475991812,4cyclictest26606-21sed09:20:311
13475991812,4cyclictest10949-21awk10:10:241
13475991811,5cyclictest3415-21df09:50:151
13475991811,5cyclictest27425-21cut07:45:261
13475991811,5cyclictest19083-21sensors_temp12:10:261
13475991811,5cyclictest10035-21awk08:30:251
1347599181,12cyclictest2620-21apt-get08:10:141
13475991810,6cyclictest15157-21cat08:45:281
1346999181,6cyclictest9-21ksoftirqd/011:55:120
13469991814,3cyclictest101rcuc/008:05:260
13469991811,5cyclictest27888-21cat11:00:180
1346999181,13cyclictest23164-21snmp_o5powerctl09:10:270
13484991713,2cyclictest1593-21/usr/sbin/munin09:45:233
1348499171,12cyclictest19934-21snmp_o5powerctl09:00:303
1347599178,6cyclictest17679-21cron07:20:001
13475991713,2cyclictest21462-21rt-features09:05:261
13475991713,2cyclictest20321-21latency_hist09:05:011
13475991711,4cyclictest28327-21expr07:50:111
13475991711,4cyclictest25683-21strings07:40:211
13475991711,4cyclictest12613-21expr08:40:131
1347599171,12cyclictest32368-21apt-get09:40:121
1347599171,12cyclictest32368-21apt-get09:40:121
13475991710,5cyclictest15985-21sed12:00:331
1346999171,11cyclictest21656-21apt-get07:30:120
1346999171,11cyclictest18091-21apt-get12:10:090
1346999170,6cyclictest9-21ksoftirqd/007:35:330
1347599168,6cyclictest30926-21df_inode09:35:131
1347599168,6cyclictest30926-21df_inode09:35:131
13475991610,4cyclictest23779-21expr10:50:141
13475991610,4cyclictest22266-21grep07:30:201
13469991611,3cyclictest32420-21ls09:40:140
13469991611,3cyclictest32420-21ls09:40:130
1347599158,5cyclictest17887-21cat08:55:181
1347599157,6cyclictest12398-21cut08:40:001
1347599147,5cyclictest19-21ksoftirqd/111:20:221
1348499139,3cyclictest13503-21cut07:10:013
134759991,6cyclictest13478-21latency_hist07:10:001
134699992,5cyclictest13464-21cyclictest07:10:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional