You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-05-21 - 17:56

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot4s.osadl.org (updated Tue May 21, 2024 12:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
143099307,21cyclictest16009-21gpgv09:30:013
143099291,24cyclictest22018-40apt-check11:25:063
141999262,16cyclictest21372-21python311:25:010
142699253,19cyclictest13770-21grep07:45:172
1426992511,10cyclictest15937-21cron09:30:012
143099240,7cyclictest27774-21apt10:05:003
1426992416,5cyclictest10085-21cut09:10:162
1426992416,5cyclictest10085-21cut09:10:152
1426992416,4cyclictest25034-21idleruntime-cro11:35:002
1426992413,7cyclictest302-21systemd-journal12:15:002
141999242,14cyclictest30130-21apt-get11:50:120
1419992419,2cyclictest8341-21xargs07:30:000
1430992315,6cyclictest610-21df10:20:133
1426992315,5cyclictest31767-21apt-key08:40:012
1424992314,5cyclictest31548-21grep11:55:001
141999231,6cyclictest16002-21sed09:30:010
1430992213,5cyclictest73550irq/132-enp2s012:20:093
143099221,18cyclictest718-21munin-run12:00:013
142699226,3cyclictest25-21ksoftirqd/210:25:012
142699222,16cyclictest16926-21tune2fs09:30:142
142699221,17cyclictest17384-21strings11:10:222
1424992214,6cyclictest615-21df10:20:141
141999224,9cyclictest4308-21apt-get08:55:110
141999223,13cyclictest11196-21apt-get12:30:110
141999222,14cyclictest787-21latency_hist12:00:000
141999222,13cyclictest10209-21apt-get10:50:110
1419992214,4cyclictest9180-21grep12:20:430
141999221,14cyclictest18202-21sort08:00:000
143099213,16cyclictest9984-21awk10:49:593
143099213,16cyclictest13338-21cat11:00:013
143099211,4cyclictest14224-21awk12:40:003
143099211,3cyclictest8337-21basename07:30:013
143099211,3cyclictest31161-21python308:40:003
1430992113,6cyclictest24239-21snmp_o5powerctl08:15:253
1430992113,6cyclictest17183-21cat11:10:203
1430992113,6cyclictest13471-21grep11:00:143
1430992113,6cyclictest11401-21cat12:30:133
1430992113,5cyclictest21155-21expr08:05:313
1430992112,6cyclictest8834-21ntpq09:05:233
143099210,4cyclictest4103-21turbostat08:55:003
143099210,3cyclictest5719-1kworker/3:1H09:19:593
143099210,19cyclictest6767-21cat07:25:013
143099210,18cyclictest23024-21munin-run08:15:003
142699213,9cyclictest23125-21apt-get09:50:122
142699213,14cyclictest31764-21latency10:15:232
142699213,13cyclictest22030-21apt-get11:25:162
142699213,10cyclictest13328-21lsb_release07:45:002
142699212,15cyclictest3091-21cut08:50:192
142699212,14cyclictest25263-21apt-get11:35:142
142699212,13cyclictest7345-21python309:03:022
142699212,13cyclictest31163-21python308:40:002
142699212,12cyclictest27935-21latency_hist08:30:012
1426992113,6cyclictest11916-21awk12:30:182
1426992111,6cyclictest24537-21grep08:15:292
142699211,14cyclictest19954-21apt-get11:20:122
142699211,14cyclictest12770-21latency_hist09:20:012
1424992116,3cyclictest8462-21interrupts12:20:161
1424992116,3cyclictest4989-21uptime07:15:321
1424992114,5cyclictest24739-21cut08:20:001
1424992113,6cyclictest9340-21awk09:06:031
1424992113,6cyclictest5662-21fw_packets10:35:131
1424992113,6cyclictest23837-21df_inode11:30:131
1424992113,5cyclictest30460-21ntpq08:35:281
1424992113,5cyclictest20636-21ntp_kernel_pll_09:40:241
1424992113,5cyclictest15515-21grep09:25:241
1424992112,6cyclictest25791-21rt-features08:20:251
141999212,14cyclictest8073-21apt-get09:05:110
141999212,13cyclictest31828-21apt-get08:40:150
141999212,13cyclictest28517-21apt-get11:45:140
141999212,13cyclictest26899-21apt-get11:40:090
141999212,13cyclictest21832-40apt-check11:25:090
141999212,12cyclictest18396-21apt-get08:00:130
1419992116,3cyclictest302-21systemd-journal08:40:000
143099203,15cyclictest2604-21cat08:50:003
1430992014,3cyclictest29001-21switchtime10:05:313
1430992013,5cyclictest9962-21expr12:25:193
1430992013,4cyclictest20425-21cat11:20:173
1430992012,6cyclictest7743-21ntp_states10:40:223
1430992012,6cyclictest3416-21snmp_o5powerctl07:15:023
1430992012,6cyclictest16078-21snmp_o5powerctl11:05:333
1430992012,6cyclictest16078-21snmp_o5powerctl11:05:323
143099201,13cyclictest21510-21apt-get09:45:113
143099200,6cyclictest7598-21apt-config09:03:033
143099200,18cyclictest14350-21munin-run09:25:003
143099200,18cyclictest10270-21package-data-do07:35:013
143099200,17cyclictest4050-21python312:10:003
143099200,16cyclictest0-21swapper/312:10:053
142699207,3cyclictest25-21ksoftirqd/210:50:142
142699206,3cyclictest25-21ksoftirqd/209:45:112
142699204,9cyclictest2100-21apt-get10:25:132
142699203,9cyclictest29603-21apt-get10:10:132
142699203,9cyclictest29603-21apt-get10:10:132
142699202,13cyclictest11827-21apt-get10:55:112
142699202,10cyclictest3073-21munin-node07:55:132
1426992013,5cyclictest29550-21sed11:45:352
1426992013,4cyclictest2403-21latency_hist12:05:002
1426992012,5cyclictest8365-21anacron07:30:002
1426992012,5cyclictest28828-21pmu-power10:05:262
1426992012,5cyclictest11068-21awk07:35:272
1426992011,6cyclictest5033-21grep10:30:312
1426992011,6cyclictest32139-21sed11:55:182
142699201,14cyclictest5340-21apt-get10:35:092
142699201,13cyclictest15816-21ntp_kernel_pll_11:05:232
142699201,13cyclictest15816-21ntp_kernel_pll_11:05:232
1424992015,3cyclictest14861-21vmstat07:45:321
1424992013,5cyclictest4177-21if_enp2s010:30:151
1424992013,5cyclictest19251-21snmp_o5powerctl09:35:291
1424992013,5cyclictest18510-21diskstats11:15:141
1424992013,5cyclictest18251-21cut09:35:011
1424992013,5cyclictest14132-21users12:35:301
1424992012,6cyclictest9017-21cat10:45:161
1424992012,6cyclictest5138-21grep12:10:231
1424992012,6cyclictest21313-21sshd11:23:441
1424992012,6cyclictest10304-21ls09:10:201
1424992012,6cyclictest10304-21ls09:10:191
142499201,13cyclictest21625-21apt-get08:10:141
141999208,8cyclictest10278-21apt-get07:35:150
141999203,9cyclictest8566-21apt-get07:30:120
141999202,10cyclictest4707-21apt-get12:10:150
141999201,8cyclictest111rcu_preempt10:40:120
1419992013,5cyclictest20296-21sed09:40:180
1419992012,6cyclictest4919-21grep07:15:310
1419992012,6cyclictest25919-21sed11:35:240
1419992012,6cyclictest25360-21sed09:55:210
1419992012,6cyclictest1695-21grep10:20:280
141999201,13cyclictest23060-21date08:15:000
143099192,15cyclictest29709-21cat08:35:003
1430991915,2cyclictest6089-21ntp_offset10:35:193
1430991915,2cyclictest10196-21cat07:35:013
1430991913,3cyclictest32155-21mailstats10:15:293
1430991912,5cyclictest28695-21cat11:45:143
1430991912,5cyclictest2214-21expr10:25:153
1430991912,5cyclictest1967-21which12:00:263
1430991912,5cyclictest16080-21readlink07:50:293
1430991911,6cyclictest20474-21cat09:40:213
1430991911,5cyclictest19510-21ls08:00:313
143099191,13cyclictest3186-21snmpd11:35:133
1430991910,6cyclictest31390-21ubuntu-distro-i08:40:003
143099190,2cyclictest27905-21lsb_release08:30:003
143099190,17cyclictest15982-21python309:30:023
143099190,16cyclictest25036-21turbostat11:35:003
143099190,16cyclictest11095-21lsb_release09:15:003
143099190,16cyclictest11095-21lsb_release09:15:003
142699197,2cyclictest25-21ksoftirqd/207:25:002
142699193,9cyclictest26376-21apt-get10:00:122
142699193,9cyclictest1045-21apt-get08:45:102
1426991914,3cyclictest1965-21ntpd11:43:502
1426991912,5cyclictest15016-21sed11:05:002
1426991911,6cyclictest7997-21grep10:40:292
1426991911,5cyclictest4051-21python312:10:002
142699191,13cyclictest7966-21apt-get12:20:112
142699191,13cyclictest20013-21apt-get08:05:112
142699191,13cyclictest12811-21apt-get12:35:102
142699191,12cyclictest21319-21latency_hist09:44:592
142699190,17cyclictest19035-21grep08:00:222
142499199,7cyclictest10071-21cat12:25:211
142499192,12cyclictest31745-21apt-get11:55:121
1424991914,3cyclictest10932-21nfs4_client10:50:211
1424991912,5cyclictest3484-21pluginstate08:50:261
1424991912,5cyclictest29550-21cut10:10:001
1424991912,5cyclictest17186-21interrupts11:10:211
1424991912,5cyclictest1700-21grep08:45:191
1424991912,5cyclictest15887-21ntpq07:50:241
1424991912,4cyclictest3889-21awk12:06:171
1424991911,5cyclictest28340-21cut08:30:141
1424991911,5cyclictest11625-21idleruntime12:30:151
142499191,13cyclictest31832-21apt-get08:40:131
142499191,13cyclictest11303-21apt-get09:15:121
141999193,9cyclictest23244-21apt-get08:15:090
141999192,11cyclictest13463-21apt-get11:00:130
1419991915,2cyclictest6642-21vmstat07:20:340
1419991914,3cyclictest13233-21awk11:00:000
1419991912,5cyclictest3049-21sed12:05:200
1419991912,5cyclictest28503-21cut10:05:200
1419991912,5cyclictest25877-21snmp_o5powerctl08:20:270
1419991912,5cyclictest17120-21sed11:10:200
1419991912,5cyclictest14126-21sed09:20:300
1419991912,5cyclictest12226-21cut07:40:170
1419991911,5cyclictest23638-21cut09:50:200
1419991911,5cyclictest2317-21grep08:45:290
141999191,14cyclictest10583-21snmp_o5powerctl12:25:330
141999191,13cyclictest9689-21apt-get09:10:120
141999191,13cyclictest9689-21apt-get09:10:120
1430991814,2cyclictest3723-21apt-get10:30:103
1430991814,2cyclictest18769-21if_enp2s011:15:163
143099181,2cyclictest850-1kworker/3:0H09:00:003
1430991812,4cyclictest25596-21ps09:55:273
1430991812,3cyclictest19653-21latency_hist09:40:013
1430991812,3cyclictest0-21swapper/310:10:133
1430991812,3cyclictest0-21swapper/310:10:123
1430991811,5cyclictest14749-21sed07:45:313
1430991810,5cyclictest11343-21cat10:50:323
142699183,8cyclictest18313-21apt-get11:15:112
142699183,7cyclictest9584-21apt-get12:25:122
142699182,12cyclictest5081-21lsb_release07:20:012
142699181,2cyclictest111rcu_preempt08:20:132
142699181,13cyclictest19255-21snmp_o5powerctl09:35:312
142699181,12cyclictest8594-21apt-get10:45:102
142699181,12cyclictest24739-21apt-get09:55:132
142699181,12cyclictest1455-21apt-get07:10:112
142499183,9cyclictest5311-21apt-get07:20:141
1424991814,2cyclictest7229-21processes12:15:251
1424991814,2cyclictest27304-21sendmail_mailst10:00:261
1424991814,2cyclictest21614-21cstates09:45:131
1424991813,3cyclictest2209-21ntp_offset07:10:231
1424991813,3cyclictest12011-21sh10:55:111
1424991812,4cyclictest25374-21ls11:35:151
1424991812,4cyclictest15286-21forks11:05:171
1424991812,4cyclictest15286-21forks11:05:171
1424991811,5cyclictest3671-21cat10:30:011
1424991811,5cyclictest27034-21expr11:40:131
1424991811,5cyclictest23557-21ls09:50:191
1424991811,5cyclictest14013-21cat09:20:291
1424991810,6cyclictest27038-21wc08:25:191
1424991810,6cyclictest1586-21cat12:00:191
1424991810,6cyclictest10719-21cat07:35:221
141999182,14cyclictest2485-21sed10:25:170
1419991814,2cyclictest0-21swapper/012:35:220
1419991812,4cyclictest30997-21sh10:15:010
1419991812,4cyclictest30997-21sh10:15:010
1419991812,3cyclictest772-21in:imuxsock07:55:000
1419991811,5cyclictest29202-21cat08:30:290
141999181,13cyclictest7315-21apt-get09:03:020
141999181,13cyclictest32222-21snmp_o5powerctl10:15:340
141999181,12cyclictest6324-21apt-get12:15:120
1419991810,6cyclictest26739-21grep10:00:170
1419991810,6cyclictest16514-21sed11:10:010
1419991810,6cyclictest16514-21sed11:10:000
1419991810,5cyclictest8994-21cut10:45:160
143099172,11cyclictest16755-21apt-get07:55:113
1430991711,4cyclictest11919-21dpkg07:40:123
1430991710,5cyclictest25058-21expr08:20:153
1430991710,5cyclictest23538-21expr09:50:173
1430991710,4cyclictest30283-21expr11:50:133
142699178,6cyclictest30503-21sed11:50:192
142699172,11cyclictest22369-21ntpq08:10:222
1426991713,2cyclictest6585-21fschecks_time12:15:172
1426991713,2cyclictest4546-21fschecks_count08:55:142
1426991713,2cyclictest15142-21/usr/sbin/munin07:50:322
1426991713,2cyclictest0-21swapper/208:30:172
1426991712,3cyclictest9117-21switchtime09:06:022
142699171,11cyclictest0-21swapper/209:20:122
142699171,10cyclictest9985-21awk07:35:002
142499179,5cyclictest7347-21gltestperf10:40:171
142499178,6cyclictest5078-21awk08:55:211
142499178,6cyclictest26116-21cron10:00:001
1424991713,2cyclictest23043-21switchtime11:25:321
1424991712,3cyclictest18722-21gltestperf08:00:191
1424991711,4cyclictest14730-21unixbench_singl11:00:351
142499171,12cyclictest7927-21snmp_o5powerctl07:25:261
1424991710,5cyclictest6787-21awk09:00:231
141999179,6cyclictest25045-21pluginstate11:35:000
141999173,7cyclictest13530-21apt-get07:45:120
1419991712,3cyclictest26908-21sed08:25:170
1419991711,4cyclictest22051-21grep09:45:190
141999171,12cyclictest21445-21latency_hist08:10:000
141999171,12cyclictest2001-21snmp_o5powerctl12:00:280
1419991710,5cyclictest11417-21expr09:15:120
1419991710,4cyclictest0-21swapper/007:15:010
143099169,5cyclictest7281-21mailstats12:15:263
1430991610,4cyclictest1576-21expr07:10:123
142699162,9cyclictest1429-21latency_hist07:10:002
142699162,9cyclictest1429-21latency_hist07:10:002
142499169,5cyclictest31436-21grep10:15:191
142499169,5cyclictest21390-21cron08:10:001
142499169,5cyclictest12020-21cat07:40:151
1424991612,2cyclictest17830-21threads07:55:281
142499161,11cyclictest10171-21cat07:35:011
1424991610,4cyclictest31178-21date10:15:011
1424991610,4cyclictest31178-21date10:15:001
141999169,5cyclictest6463-21sed10:35:280
141999168,5cyclictest19157-21proc_pri11:15:230
1419991613,2cyclictest4019-21fschecks_time10:30:140
141999161,2cyclictest111rcu_preempt09:30:040
141999161,12cyclictest19473-21sed09:35:330
143099158,4cyclictest27952-21cat11:40:283
142499159,4cyclictest28860-21grep11:45:191
141999158,5cyclictest2830-21expr08:50:160
143099119,1cyclictest0-21swapper/307:10:003
143099100,8cyclictest1435-21cat07:10:013
141999101,7cyclictest1428-21latency_hist07:10:010
141999101,7cyclictest1428-21latency_hist07:10:000
14249990,6cyclictest1427-21latency_hist07:10:001
14249990,6cyclictest1427-21latency_hist07:10:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional