You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-16 - 13:47

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of all highest latencies:
System rackcslot4s.osadl.org (updated Mon Jul 15, 2024 12:44:19)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
3117899302,23cyclictest31503-40apt-check07:10:053
3117899251,6cyclictest1130-21apt-get08:40:123
3117899241,6cyclictest18240-21lsb_release11:09:593
31178992415,7cyclictest21026-21lsb_release08:05:423
3117899240,6cyclictest31-21ksoftirqd/307:45:173
31177992416,6cyclictest987-40apt-check08:40:082
31177992413,7cyclictest25211-21sort11:30:002
31177992413,7cyclictest25211-21sort11:30:002
31168992415,6cyclictest771-21dbus-daemon08:06:120
3117899231,19cyclictest27226-21lsb_release10:00:013
3117899230,14cyclictest31-21ksoftirqd/310:24:113
31172992313,7cyclictest10757-21cat10:45:271
3116899232,15cyclictest19339-21apt-get09:35:160
31178992214,6cyclictest16292-21grep07:55:323
3117899221,18cyclictest8025-21python307:35:013
3117899221,17cyclictest14281-21lsb_release12:35:003
3117899220,7cyclictest10113-21awk07:39:593
3117899220,7cyclictest10113-21awk07:39:593
3117899220,20cyclictest6002-21apt-config08:55:023
3117799222,14cyclictest15135-21apt-get11:00:142
31177992214,6cyclictest7899-21awk10:36:142
31177992212,7cyclictest1-21systemd08:06:472
31172992214,6cyclictest19047-21users09:30:401
31172992214,6cyclictest15233-21ntp_kernel_pll_09:20:211
3117299220,18cyclictest22247-21lsb_release08:07:141
3116899222,15cyclictest889-21latency_hist11:55:010
31168992213,5cyclictest431-21latency_hist08:40:010
31168992213,5cyclictest23342-21apt-config11:25:000
31168992212,7cyclictest7364-21snmp_o5powerctl08:55:280
3117899214,5cyclictest31-21ksoftirqd/311:44:013
3117899212,7cyclictest31-21ksoftirqd/310:13:013
31178992116,3cyclictest6000-21apt-get12:10:123
3117899211,3cyclictest14003-21irqstats10:55:233
31178992113,5cyclictest19127-21lsb_release09:35:013
31178992113,5cyclictest1540-21awk07:15:203
31178992112,6cyclictest5423-21cat12:05:343
3117899211,17cyclictest23211-21python311:25:013
3117899211,16cyclictest486-21latency_hist08:40:003
3117899210,19cyclictest9515-40apt-check12:20:063
3117899210,19cyclictest13453-21cat10:55:013
3117899210,18cyclictest877-21perf11:55:013
3117899210,18cyclictest11679-21lsb_release10:50:013
3117799219,8cyclictest10340-21apt-get07:40:142
31177992113,6cyclictest30197-21cat10:05:372
31177992113,6cyclictest21491-21lsb_release11:20:002
31177992113,6cyclictest16525-21hostname09:25:172
3117799211,13cyclictest6029-21python307:30:002
31172992116,3cyclictest5754-21latency_hist08:55:011
31172992116,3cyclictest2000-21processes08:40:311
31172992114,5cyclictest26721-21uptime11:30:391
31172992114,5cyclictest23585-21snmp_o5powerctl09:45:351
31172992113,6cyclictest22495-21cut09:45:001
31172992113,6cyclictest20251-21df11:15:141
31172992113,6cyclictest12588-21users12:25:351
31172992113,6cyclictest12588-21users12:25:341
31172992113,5cyclictest32223-21ntp_kernel_pll_08:35:251
31172992113,5cyclictest23287-21sed08:10:021
31172992112,6cyclictest9758-21expr09:05:141
3116899212,3cyclictest14894-21tail09:20:160
3116899212,14cyclictest20975-21apt-get09:40:120
3116899212,14cyclictest17780-21apt-get08:00:210
3116899212,14cyclictest17722-21apt-get09:30:130
3116899212,13cyclictest9515-40apt-check12:20:080
31168992116,3cyclictest23189-21apt-get08:10:000
31168992116,3cyclictest15882-21idleruntime-cro12:40:000
31168992113,6cyclictest10959-21sed07:40:230
31168992112,6cyclictest771-21dbus-daemon09:20:000
3116899211,13cyclictest9674-21python310:45:010
3116899211,13cyclictest6139-21cat07:30:010
3117899203,4cyclictest302-21systemd-journal10:34:553
3117899202,12cyclictest24202-21apt-get09:50:113
3117899201,4cyclictest2505-21cron12:00:003
3117899201,3cyclictest7753-1kworker/3:1H09:00:013
31178992013,5cyclictest6611-21entropy10:35:153
31178992013,5cyclictest4586-21apt07:25:103
31178992013,5cyclictest4055-21grep07:20:373
31178992013,5cyclictest3799-21grep08:45:283
31178992013,5cyclictest23784-21basename08:10:193
31178992013,5cyclictest15480-21grep12:35:293
31178992013,5cyclictest10452-21awk09:05:283
31178992012,6cyclictest20658-21cat11:15:213
31178992012,6cyclictest14291-21awk07:50:213
31178992012,5cyclictest22382-21dpkg09:45:003
31178992012,5cyclictest19237-21open_inodes11:10:253
31178992011,6cyclictest28429-21munin-run11:40:013
3117899200,7cyclictest31-21ksoftirqd/309:38:093
3117899200,3cyclictest1329-21munin-run10:20:003
31177992013,5cyclictest32135-21tune2fs11:50:162
31177992013,5cyclictest3148-21expr07:20:142
31177992013,5cyclictest26444-21sed09:55:172
31177992013,5cyclictest24313-21ls09:50:132
31177992012,6cyclictest5920-21sed10:30:372
31177992012,6cyclictest25748-21ntpq08:15:232
31177992012,6cyclictest14697-21cat07:50:332
31177992011,6cyclictest9851-21diskstats12:20:182
31177992011,6cyclictest9375-21cat10:40:332
31177992011,6cyclictest3342-21sed08:45:192
3117799201,14cyclictest15183-21apt-get07:55:112
31172992015,3cyclictest20741-21sendmail09:40:001
31172992013,5cyclictest28978-21ntpq08:25:241
31172992013,5cyclictest2885-21expr12:00:121
31172992012,6cyclictest5219-21sed07:25:191
31172992012,6cyclictest4704-21tune2fs12:05:171
31172992012,6cyclictest3447-21grep10:25:151
31172992012,6cyclictest1860-21ntp_kernel_pll_11:55:281
31172992012,5cyclictest7900-21basename10:36:141
31172992012,5cyclictest11970-21ntp_kernel_pll_09:10:261
3116899202,14cyclictest13352-21munin-run07:50:000
3116899202,13cyclictest423-21apt-get08:40:130
3116899202,13cyclictest31520-21apt-get07:10:130
31168992012,6cyclictest25976-21grep11:30:180
31168992012,6cyclictest24171-21sed09:50:010
31168992012,6cyclictest18836-21sed11:10:190
31168992011,5cyclictest30449-21sh10:10:000
3116899201,14cyclictest7612-21apt-get12:15:100
3116899201,14cyclictest16836-21apt-get11:05:130
3117899192,5cyclictest31-21ksoftirqd/309:49:133
3117899192,15cyclictest9394-21latency_hist09:05:013
3117899192,15cyclictest31363-21cat08:35:003
3117899192,15cyclictest17501-21latency_hist09:30:003
3117899192,15cyclictest11723-21latency_hist07:45:013
31178991915,2cyclictest14990-21interrupts09:20:183
31178991913,4cyclictest27271-21cat08:20:233
3117899191,2cyclictest23079-1kworker/3:0H11:35:003
31178991912,5cyclictest24241-21cat11:25:213
31178991912,5cyclictest24241-21cat11:25:203
31178991911,5cyclictest16835-21https08:00:013
3117899191,13cyclictest2767-21apt-get12:00:103
3117899190,7cyclictest31-21ksoftirqd/309:11:033
3117899190,6cyclictest9710-21latency_hist10:45:013
3117799195,11cyclictest32178-21needreboot08:35:232
31177991915,2cyclictest28987-21gltestperf11:40:162
31177991913,3cyclictest29624-21cron08:29:592
31177991912,5cyclictest4504-21expr12:05:152
31177991912,5cyclictest4362-21sed10:25:332
31177991912,5cyclictest28190-21sed11:35:442
31177991912,5cyclictest27749-21ls08:20:342
31177991912,5cyclictest26892-21sed11:35:012
31177991912,5cyclictest21938-21cut11:20:162
31177991912,5cyclictest19754-21awk11:10:372
31177991912,5cyclictest15763-21grep12:35:352
31177991911,6cyclictest9080-21expr07:35:192
31177991911,6cyclictest9080-21expr07:35:192
31177991911,6cyclictest2905-21expr12:00:112
31177991911,6cyclictest15811-21awk09:20:352
31177991911,6cyclictest12075-21df_inode10:50:122
3117799191,13cyclictest31383-40apt-check07:10:092
3117799191,13cyclictest13500-21apt-get10:55:152
31177991910,6cyclictest25-21ksoftirqd/208:00:102
31172991915,2cyclictest9633-21sendmail_mailst07:35:281
31172991915,2cyclictest9633-21sendmail_mailst07:35:281
31172991915,2cyclictest350-21interrupts10:15:211
31172991914,3cyclictest31147-21processes11:45:221
31172991914,3cyclictest2052-21cat10:20:211
31172991913,4cyclictest8729-21ntp_kernel_err09:00:251
31172991913,4cyclictest30648-21latency_hist10:10:001
31172991912,5cyclictest9569-21uptime10:40:341
31172991912,5cyclictest9262-21cut12:20:001
31172991912,5cyclictest26636-21dpkg08:20:111
31172991912,5cyclictest25289-21perl08:15:151
31172991912,5cyclictest18577-21expr11:10:131
31172991912,5cyclictest13681-21ls07:50:121
31172991912,5cyclictest12003-21expr10:50:111
31172991912,5cyclictest10505-21expr07:40:161
31172991911,6cyclictest3424-21grep08:45:211
31172991911,6cyclictest1447-21expr07:15:161
31172991910,7cyclictest15348-21expr07:55:121
31168991912,5cyclictest6564-21sed12:10:200
31168991912,5cyclictest3611-21grep07:20:220
31168991912,5cyclictest24558-21grep09:50:160
31168991912,5cyclictest2228-21sed11:55:370
3116899191,15cyclictest31527-21unixbench_multi11:45:310
31168991911,5cyclictest1759-21ls07:15:210
3116899191,13cyclictest9647-21apt-get10:45:120
3116899191,13cyclictest27456-21apt-get10:00:130
3116899191,13cyclictest26635-21apt-get08:20:130
31178991813,3cyclictest9017-21latency_hist12:20:013
31178991812,4cyclictest31474-21sed11:45:313
31178991811,5cyclictest4336-21cat10:25:333
31178991811,4cyclictest14258-21munin-run09:20:003
31178991810,5cyclictest27783-21gltestperf10:00:173
3117899180,6cyclictest31-21ksoftirqd/312:25:053
3117899180,6cyclictest31-21ksoftirqd/312:25:053
3117899180,6cyclictest31-21ksoftirqd/308:19:593
3117899180,2cyclictest30468-21munin-run10:10:003
3117899180,15cyclictest15880-21tr11:00:263
3117799182,11cyclictest23335-21apt-get08:10:132
31177991814,2cyclictest2762-21sendmail07:20:002
31177991813,3cyclictest6609-21/usr/sbin/munin07:30:222
31177991812,4cyclictest21598-21sed09:40:192
31177991811,5cyclictest28109-21cat10:00:202
3117799181,13cyclictest901-21snmp_o5powerctl10:15:312
3117799181,13cyclictest12262-21snmp_o5powerctl12:25:302
3117799181,13cyclictest12262-21snmp_o5powerctl12:25:302
3117799181,12cyclictest9620-21apt-get09:05:142
3117799181,12cyclictest30274-21apt-get11:45:122
3117799181,12cyclictest29868-21apt-get08:30:132
31177991810,6cyclictest9253-21date12:20:012
31177991810,6cyclictest13536-21sed07:50:012
3117799180,15cyclictest7528-21sed08:55:312
3117299189,6cyclictest27747-21tune2fs10:00:161
31172991814,2cyclictest22627-21sendmail_mailqu11:20:281
31172991814,2cyclictest17069-21mailstats09:25:281
31172991813,3cyclictest16042-21rt-features11:00:301
31172991813,3cyclictest15002-21kernelversion12:35:191
31172991813,3cyclictest13894-21switchtime12:30:311
31172991812,4cyclictest6403-21sed12:10:181
31172991812,4cyclictest29147-21cat11:40:171
31172991812,4cyclictest14197-21smbd09:16:471
31172991812,4cyclictest12721-21open_files07:45:261
31172991811,5cyclictest8001-21pluginstate07:35:011
31172991811,5cyclictest7899-21cat09:00:001
31172991811,5cyclictest32694-21pluginstate11:50:261
31172991811,4cyclictest18132-21mailstats08:00:291
31172991810,6cyclictest3460-21cat07:20:191
31172991810,6cyclictest14293-21pluginstate10:55:291
3116899189,6cyclictest32377-21wc10:15:150
3116899183,9cyclictest29646-21munin-run08:30:000
3116899183,8cyclictest5790-21lsb_release12:10:000
3116899182,12cyclictest20066-21apt-get11:15:120
3116899182,12cyclictest11884-21apt-get10:50:110
31168991813,3cyclictest14272-21ls12:35:000
31168991811,5cyclictest7151-21grep07:30:190
31168991811,5cyclictest4811-21cat08:50:150
31168991811,5cyclictest14083-21sed10:55:230
31168991811,3cyclictest9710-21grep09:05:150
3116899181,13cyclictest6407-21apt-get10:35:120
3116899181,13cyclictest12654-21latency_hist09:15:010
3116899181,12cyclictest8005-21apt-get09:00:150
3116899181,11cyclictest26401-21idleruntime-cro08:19:590
31168991810,6cyclictest28994-21grep11:40:150
31168991810,5cyclictest9083-21sed07:35:190
31168991810,5cyclictest9083-21sed07:35:190
31168991810,5cyclictest16439-21grep07:55:340
3116899180,15cyclictest2957-21latency_hist10:25:000
3117899170,15cyclictest29694-21cat08:30:013
3117799179,6cyclictest1793-21needreboot11:55:262
3117799179,6cyclictest14114-21awk09:15:362
3117799179,5cyclictest11634-21expr09:10:202
31177991713,2cyclictest20302-21mailstats09:35:312
31177991711,4cyclictest10931-21cat10:45:312
3117799171,13cyclictest18718-21snmp_o5powerctl09:30:342
3117799171,12cyclictest9000-21snmp_o5powerctl09:00:552
3117799171,11cyclictest22590-21apt-get09:45:152
31177991710,5cyclictest4577-21awk08:50:142
31177991710,5cyclictest3013-21cat10:25:002
31177991710,5cyclictest17952-21ls11:05:322
3117299179,6cyclictest9621-21sed12:20:011
3117299179,6cyclictest4981-21df10:30:151
3117299179,6cyclictest31455-21sh10:10:261
31172991713,2cyclictest26570-21ntp_offset09:55:201
31172991713,2cyclictest24005-21fschecks_count11:25:191
31172991713,2cyclictest24005-21fschecks_count11:25:181
31172991712,3cyclictest27808-21open_inodes11:35:291
31172991711,4cyclictest25760-21cat09:55:011
31172991711,4cyclictest16947-21expr11:05:141
3117299171,11cyclictest30909-21apt-get07:10:131
3116899179,6cyclictest5311-21cut10:30:210
3116899179,6cyclictest14514-21tr07:50:240
3116899179,5cyclictest32110-21latency_hist10:15:000
3116899172,12cyclictest2753-21apt-get08:45:090
3116899171,11cyclictest16104-21apt-get09:25:120
31168991710,5cyclictest30528-21sed08:30:300
31168991710,4cyclictest27418-21sed11:35:160
31168991710,4cyclictest27031-21sed09:55:340
3117799167,7cyclictest31182-21cat10:10:202
31177991612,2cyclictest12876-21/usr/sbin/munin12:30:212
31177991610,4cyclictest6269-21uniq12:10:182
3116899163,11cyclictest12480-21sed12:25:340
3116899163,11cyclictest12480-21sed12:25:340
3116899161,11cyclictest23180-21apt-get11:25:050
3116899161,11cyclictest23180-21apt-get11:25:050
31168991610,4cyclictest4343-21sed10:25:330
31168991610,4cyclictest15717-21sed11:00:230
31172991510,3cyclictest30499-21missed_timers08:30:291
3116899150,1cyclictest0-21swapper/012:00:330
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional