You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-25 - 18:58

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot4s.osadl.org (updated Thu Apr 25, 2024 00:44:20)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2976399370,19cyclictest31234-21ntp_states21:00:212
2976499351,31cyclictest3551-40apt-check22:55:053
2976499350,32cyclictest5650-21sh23:00:143
2976499340,32cyclictest0-21swapper/319:15:003
2976499340,31cyclictest29129-21fschecks_time20:55:153
2976399340,19cyclictest24925-21sshd23:58:032
2976399340,18cyclictest31576-21ntp_states22:40:212
29763993310,20cyclictest11881-21latency21:40:212
2976499322,27cyclictest20106-21grep20:25:273
2976499320,29cyclictest17049-21apt-key20:20:003
2976499320,17cyclictest7009-21awk19:49:593
29763993212,17cyclictest24398-21tr22:20:102
2976399321,15cyclictest26819-21apt-get00:05:112
2976499312,22cyclictest3091-21python322:54:593
2976499311,28cyclictest14371-21lsb_release21:50:003
2976499311,27cyclictest24397-21grep22:20:093
29764993111,18cyclictest20708-21awk22:05:293
29764993111,17cyclictest4645-21dpkg21:20:013
29764993110,19cyclictest28328-21sed00:10:003
2976499310,29cyclictest0-21swapper/300:35:073
2976499310,28cyclictest28181-21mailstats20:50:223
2976399311,26cyclictest5178-21cat23:00:012
29761993122,5cyclictest2942-21apt-get21:15:130
2976499301,25cyclictest23378-21latency_hist23:54:593
29764993011,17cyclictest24736-21sort23:55:273
2976399307,20cyclictest3504-21latency_hist19:35:002
2976399302,25cyclictest30056-21/usr/sbin/munin00:15:302
2976399302,21cyclictest3092-21python322:54:592
2976399301,25cyclictest24719-21ntpq20:40:182
2976399300,27cyclictest28499-21unixbench_singl20:50:262
2976499299,18cyclictest20468-21expr23:45:133
2976499298,19cyclictest18899-21sed22:00:263
2976499298,19cyclictest10609-21expr20:00:163
2976499298,18cyclictest409-21awk21:05:233
2976499298,18cyclictest25886-21grep00:00:223
29764992914,13cyclictest3853-21cat19:40:003
2976499291,21cyclictest0-21swapper/320:40:103
2976499290,27cyclictest3471-21lsb_release19:35:013
2976499290,26cyclictest16524-21sed20:15:323
2976499290,1cyclictest0-21swapper/300:30:013
2976499290,15cyclictest9824-21snmp_o5powerctl19:55:283
2976399292,20cyclictest31190-21python319:15:002
2976399291,25cyclictest17976-21ntpq20:20:272
2976399291,23cyclictest29435-21fschecks_time22:35:162
2976399291,21cyclictest7214-21apt-get19:50:132
2976399291,15cyclictest0-21swapper/221:35:322
2976399290,26cyclictest18353-21uname22:00:162
29761992916,8cyclictest2549-21apt-get00:30:090
2976499289,17cyclictest31800-21cat00:20:133
2976499281,23cyclictest0-21swapper/319:40:063
2976499281,22cyclictest0-21swapper/320:30:133
2976499281,22cyclictest0-21swapper/319:50:123
2976499281,20cyclictest9707-21apt-get21:35:143
2976499280,25cyclictest15957-21cron21:55:003
2976499280,24cyclictest14843-21ls20:10:333
2976499280,24cyclictest11740-21sed21:40:193
2976499280,23cyclictest7787-21sed21:25:303
2976499280,15cyclictest0-21swapper/321:00:233
2976399283,17cyclictest21069-21apt-get22:10:122
2976399283,17cyclictest21069-21apt-get22:10:122
2976399282,22cyclictest20277-21http20:30:012
2976399282,19cyclictest13373-21python323:25:012
2976399281,21cyclictest20732-21apt-get20:30:122
2976399281,19cyclictest5598-21apt-get19:45:112
2976399281,18cyclictest22344-21apt-get20:35:092
2976399281,18cyclictest10465-21apt-get20:00:142
2976399280,1cyclictest0-21swapper/200:30:002
2976499278,16cyclictest0-21swapper/319:15:053
2976499276,19cyclictest28922-21cat00:10:183
2976499276,19cyclictest28922-21cat00:10:183
2976499271,23cyclictest3976-21wc00:35:013
2976499271,23cyclictest23739-21latency_hist20:40:003
2976499271,23cyclictest17453-21gltestperf23:35:163
2976499271,22cyclictest26141-21ls20:45:203
2976499271,22cyclictest0-21swapper/322:45:103
2976499271,22cyclictest0-21swapper/322:15:103
2976499271,22cyclictest0-21swapper/321:30:123
2976499270,24cyclictest29433-21grep22:35:163
2976499270,22cyclictest28124-21latency22:30:183
2976499270,22cyclictest2052-21ntpq21:10:233
2976499270,22cyclictest2052-21ntpq21:10:233
2976399279,15cyclictest22279-21grep23:50:172
2976399275,18cyclictest17451-21sed23:35:152
2976399272,18cyclictest32435-21apt-get22:45:092
2976399272,18cyclictest2549-21apt-get00:30:082
2976399272,17cyclictest31845-40apt-check19:15:082
29763992714,11cyclictest10163-21awk19:55:322
2976399271,23cyclictest376-21ntpq21:05:222
2976399271,23cyclictest16797-21gpgv20:20:002
2976399271,23cyclictest1076-21find19:20:182
2976399271,22cyclictest14844-21sort20:10:332
2976399271,21cyclictest26313-21gltestperf22:25:182
29763992712,13cyclictest5117-21sed19:40:222
2976399271,19cyclictest12080-21apt-get20:05:112
2976399270,25cyclictest20671-21sed22:05:292
2976399270,24cyclictest7785-21grep21:25:302
2976399270,24cyclictest28165-21cat22:30:192
2976399270,24cyclictest1713-21sed21:10:182
2976399270,24cyclictest1713-21sed21:10:182
2976399270,23cyclictest8766-21cpuspeed_turbos23:10:102
2976399270,23cyclictest8766-21cpuspeed_turbos23:10:102
2976399270,22cyclictest8410-21vmstat23:05:262
2976399270,13cyclictest25595-21/usr/sbin/munin20:45:282
2976499269,15cyclictest11536-21unixbench_multi23:15:303
2976499266,17cyclictest9687-21/usr/sbin/munin23:10:243
2976499266,17cyclictest9687-21/usr/sbin/munin23:10:233
2976499263,20cyclictest7126-21grep23:05:113
2976499262,19cyclictest0-21swapper/322:10:123
2976499262,19cyclictest0-21swapper/322:10:123
2976499261,22cyclictest4383-21sh21:20:003
2976499261,22cyclictest26311-21gltestperf22:25:183
2976499261,22cyclictest19506-21/usr/sbin/munin23:40:203
2976499261,21cyclictest726-21missed_timers19:20:143
2976499261,21cyclictest0-21swapper/323:25:063
2976499261,20cyclictest0-21swapper/320:05:113
2976499260,23cyclictest32066-21grep22:40:313
2976499260,23cyclictest13158-21sed23:20:293
2976499260,22cyclictest31472-21dpkg00:20:003
2976399269,14cyclictest14865-21cut21:50:142
2976399262,18cyclictest22684-21apt-get22:15:102
2976399261,22cyclictest3807-21proc_pri21:15:252
2976399261,21cyclictest12962-21/usr/sbin/munin21:45:172
29763992612,12cyclictest5369-21which00:35:292
2976399261,18cyclictest675-21lsb_release00:25:012
2976399261,17cyclictest20333-21apt-get23:45:102
2976399261,16cyclictest16195-21apt-get21:55:112
29763992610,14cyclictest10482-21cat23:15:142
2976399260,23cyclictest28766-21grep00:10:172
2976399260,23cyclictest28766-21grep00:10:172
2976399260,1cyclictest0-21swapper/223:00:252
2976499251,22cyclictest2051-21latency_hist19:30:013
2976399253,16cyclictest8092-21apt-get21:30:132
2976399252,17cyclictest4839-40apt-check21:20:082
2976399251,20cyclictest26434-21grep00:00:302
2976399250,22cyclictest3735-21perf19:40:002
2976399250,22cyclictest30172-21vmstat20:55:312
2976399250,1cyclictest0-21swapper/223:40:242
29761992516,5cyclictest13694-21apt-get20:10:140
2976499241,19cyclictest0-21swapper/323:30:103
2976499240,20cyclictest0-21swapper/321:55:103
2976399242,16cyclictest13826-40apt-check23:25:062
2976399241,19cyclictest1861-21latency_hist19:30:012
2976399240,21cyclictest15458-21apt-get23:30:102
2976199242,15cyclictest3036-21kworker/u8:122:55:130
2976199242,15cyclictest17230-21apt-get20:20:120
29762992315,6cyclictest6596-21snmp_o5powerctl19:45:251
29761992313,7cyclictest31524-21sed19:15:010
2976299221,14cyclictest1408-21kworker/u8:022:55:061
2976199224,13cyclictest20463-21latency_hist20:30:010
2976199223,13cyclictest28436-21apt-get00:10:140
2976199223,13cyclictest28436-21apt-get00:10:140
2976199222,14cyclictest17076-21apt-get23:35:110
29761992214,4cyclictest19230-21sh22:05:000
29762992116,3cyclictest31539-21nfs4_client22:40:201
29762992114,5cyclictest544-21ls19:20:121
29762992113,6cyclictest6664-21df_inode21:25:141
29762992113,6cyclictest5466-21cat00:35:331
29762992113,6cyclictest1528-21memory00:25:201
29762992113,6cyclictest10781-21tune2fs20:00:181
29762992112,7cyclictest27996-21sed00:05:291
2976299211,14cyclictest21949-21apt-get23:50:121
2976199212,13cyclictest7078-21apt-get23:05:100
2976199212,13cyclictest31164-21apt-get19:15:130
2976199212,13cyclictest25942-21apt-get22:25:140
2976199212,13cyclictest1313-21apt-get21:10:150
2976199212,13cyclictest1313-21apt-get21:10:150
2976199212,12cyclictest13826-40apt-check23:25:080
2976199212,11cyclictest16756-21python320:20:000
29761992112,6cyclictest1277-21sed19:20:200
2976199211,15cyclictest29171-21apt-get22:35:130
2976199211,14cyclictest27214-21apt-get20:50:110
29761992110,8cyclictest26819-21apt-get00:05:120
29762992015,3cyclictest26589-21idleruntime-cro00:05:001
29762992013,5cyclictest31620-21ls21:00:271
29762992013,5cyclictest24719-21expr22:20:131
29762992012,6cyclictest32730-21perl22:45:161
29762992012,6cyclictest28757-21cat20:55:001
29762992012,6cyclictest16918-21ntp_kernel_err21:55:211
29762992012,6cyclictest14615-21sendmail_mailqu20:10:291
29762992012,6cyclictest11305-21snmp_o5powerctl23:15:291
29762992012,5cyclictest21479-21expr22:10:171
29762992012,5cyclictest21479-21expr22:10:161
2976299200,18cyclictest28202-21sed22:30:191
2976199202,12cyclictest30818-21apt-get22:40:130
29761992014,3cyclictest1436-21ls22:50:000
29761992013,5cyclictest10021-21sed19:55:300
29761992012,5cyclictest6351-21ntpq19:45:200
29761992012,5cyclictest11689-21cron23:20:010
2976199201,13cyclictest4839-40apt-check21:20:050
2976199201,13cyclictest28829-21apt-get20:55:110
2976199201,13cyclictest12965-21apt-get21:45:130
2976199201,13cyclictest12080-21apt-get20:05:130
29762991915,2cyclictest30205-21idleruntime19:10:151
29762991912,5cyclictest7816-21ntp_kernel_pll_23:05:171
29762991912,5cyclictest32420-21ntp_kernel_err00:20:201
29762991912,5cyclictest32203-21cat21:05:151
29762991912,5cyclictest29801-21cat22:35:181
29762991912,5cyclictest27090-21ls22:25:311
29762991912,5cyclictest20595-21grep23:45:171
29762991912,5cyclictest20496-21dpkg20:30:011
29762991912,5cyclictest12007-21sed21:40:211
29762991912,4cyclictest15479-21expr20:15:151
29762991911,6cyclictest24313-21ntpq23:55:181
29762991911,6cyclictest22766-21sed20:35:161
29762991911,6cyclictest11184-21cut21:40:011
29762991911,6cyclictest10435-21sed20:00:001
29762991911,5cyclictest13033-21expr23:20:291
2976199194,9cyclictest4423-40apt-check00:35:080
2976199192,12cyclictest31698-21apt-get00:20:120
2976199192,10cyclictest808-21thermald21:00:120
29761991916,2cyclictest19455-21apt22:05:090
29761991913,4cyclictest24600-21sed20:40:180
29761991912,5cyclictest22370-21sed23:50:160
29761991912,5cyclictest1788-21ls22:50:130
29761991912,5cyclictest17198-21snmp_o5powerctl21:55:270
29761991912,5cyclictest15531-21grep21:50:220
29761991912,5cyclictest11622-21sed20:00:310
29761991911,5cyclictest6728-21grep23:00:270
29761991911,5cyclictest31828-21kworker/u8:021:10:000
29761991911,5cyclictest26941-21sshd20:46:480
2976199191,13cyclictest3983-21apt-get19:40:080
2976199191,13cyclictest20333-21apt-get23:45:130
2976199191,12cyclictest30059-21apt-get00:15:140
29761991910,6cyclictest302-21systemd-journal00:30:000
2976199190,16cyclictest22067-21snmp_o5powerctl22:10:280
2976199190,16cyclictest22067-21snmp_o5powerctl22:10:280
2976299183,13cyclictest3330-21tail19:30:221
2976299182,11cyclictest25598-21apt-get20:45:151
29762991814,2cyclictest3313-21sh00:30:191
29762991814,2cyclictest30690-21missed_timers00:15:231
29762991813,3cyclictest17491-21hwlatdetect23:35:151
29762991812,4cyclictest2943-21dpkg21:15:131
29762991811,5cyclictest7216-21dpkg19:50:111
29762991811,5cyclictest29273-21cat20:55:161
29762991811,5cyclictest22992-21cut22:15:141
29762991811,5cyclictest19457-21dpkg22:05:101
29762991811,5cyclictest19059-21forks20:25:141
29762991811,5cyclictest18862-21expr23:40:111
29762991811,5cyclictest18164-21grep22:00:141
2976299181,12cyclictest13843-21apt-get23:25:131
2976199183,9cyclictest7214-21apt-get19:50:120
2976199182,13cyclictest2057-21apt-get19:30:070
29761991813,3cyclictest23614-21mailstats22:15:210
29761991812,4cyclictest10947-21grep21:35:330
29761991811,5cyclictest17044-21cut23:35:000
2976199181,13cyclictest8693-21apt-get23:10:110
2976199181,13cyclictest8693-21apt-get23:10:110
2976199181,12cyclictest27558-21apt-get22:30:110
2976199181,12cyclictest25204-21apt-get00:00:120
2976199181,12cyclictest20243-21apt-get20:30:050
2976199181,12cyclictest18716-21apt-get23:40:120
29762991713,2cyclictest2430-21sh22:50:211
29762991711,4cyclictest8526-21basename21:30:181
29762991711,4cyclictest28830-21tail00:10:181
29762991711,4cyclictest28830-21tail00:10:171
29762991710,5cyclictest9730-21sed23:10:261
29762991710,5cyclictest9730-21sed23:10:251
29762991710,5cyclictest756-21cupsd21:20:091
29762991710,5cyclictest1890-21cut19:30:001
29762991710,5cyclictest15849-21expr23:30:151
29762991710,5cyclictest13136-21cat20:05:281
2976299170,2cyclictest16716-21apt-get20:20:121
2976199179,5cyclictest771-21dbus-daemon20:40:000
29761991712,3cyclictest8485-21sed21:30:180
29761991711,4cyclictest12559-21grep23:20:180
2976199171,11cyclictest24323-21apt-get22:20:100
2976299168,6cyclictest15223-21sed21:50:181
29762991614,1cyclictest0-21swapper/119:36:061
29762991612,2cyclictest25261-21sed20:40:271
29762991611,3cyclictest1534-21fschecks_count21:10:171
29762991611,3cyclictest1534-21fschecks_count21:10:171
29762991611,3cyclictest13467-21kernelversion21:45:211
2976299161,10cyclictest32241-21apt-get19:15:131
29762991610,4cyclictest5523-21grep23:00:131
29762991610,4cyclictest4432-21basename19:40:141
29761991610,4cyclictest25174-21sed00:00:000
29761991511,2cyclictest6686-21fschecks_count21:25:160
2976199151,11cyclictest12725-21cron21:44:590
2976199151,10cyclictest3765-21lsb_release19:40:010
2976199147,5cyclictest1815-21cron19:30:000
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional