You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-12-02 - 22:17

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackcslot7s.osadl.org (updated Sat Dec 02, 2023 12:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
28726519911133,46cyclictest141rcu_preempt09:46:193
295480921040,6sleep20-21swapper/211:06:212
162399980,32rtkit-daemon1387-21kerneloops11:15:192
287147629161,13sleep00-21swapper/007:05:280
162399910,5rtkit-daemon0-21swapper/007:43:070
287236428964,17sleep20-21swapper/207:08:122
29455182880,6sleep10-21swapper/110:40:321
29751572860,16sleep342-21ksoftirqd/312:10:213
2872651998578,4cyclictest0-21swapper/308:17:583
287224128565,14sleep30-21swapper/307:06:363
287102328260,14sleep10-21swapper/107:05:051
162399820,9rtkit-daemon881251-21Softwar~cThread08:09:193
162399800,9rtkit-daemon881251-21Softwar~cThread08:59:243
162399780,8rtkit-daemon880726-21firefox11:31:163
162399770,4rtkit-daemon0-21swapper/311:27:153
162399770,4rtkit-daemon0-21swapper/311:27:143
162399740,7rtkit-daemon961-21snmpd11:55:053
162399730,8rtkit-daemon1775-21gvfs-afc-volume07:48:163
162399720,8rtkit-daemon1785-21gmain11:09:393
287265199702,10cyclictest881331-21Privileged14:31:443
287264199692,23cyclictest2943505-21BgIOThr~0
2872645996617,44cyclictest26-21ksoftirqd/110:01:191
2872641996658,5cyclictest0-21swapper/008:17:590
287264199661,38cyclictest2940013-21seq10:21:030
2872645996416,8cyclictest26-21ksoftirqd/110:31:191
287264199643,57cyclictest0-21swapper/009:02:590
287264199641,36cyclictest2949657-21taskset10:53:260
287264199632,56cyclictest0-21swapper/010:35:580
287264199632,56cyclictest0-21swapper/010:35:580
287264199632,53cyclictest881335-21WRScene~ilder#112:16:190
287264199631,35cyclictest2954786-40apt-check11:06:220
287264199631,35cyclictest2904226-21python308:35:380
287265199622,19cyclictest1753-21gnome-shell12:39:593
287265199621,57cyclictest1753-21gnome-shell08:21:113
287265199621,57cyclictest1753-21gnome-shell08:21:113
2872651996215,11cyclictest2922970-21cat09:30:263
287264799622,19cyclictest1753-21gnome-shell09:51:592
287264799622,19cyclictest1753-21gnome-shell09:33:582
287264799622,17cyclictest881253-21Renderer08:16:202
287264599622,18cyclictest1753-21gnome-shell12:14:581
287264199623,55cyclictest0-21swapper/007:31:590
287264199623,55cyclictest0-21swapper/007:31:590
287264199622,56cyclictest0-21swapper/009:46:190
287264199621,34cyclictest2930445-21python309:50:430
287264199611,35cyclictest2925076-21awk09:35:380
287264599602,17cyclictest1753-21gnome-shell10:08:581
287264199601,32cyclictest2909897-21taskset08:53:100
28989862590,5sleep10-21swapper/108:21:101
28989862590,5sleep10-21swapper/108:21:091
162399590,8rtkit-daemon1622-21rtkit-daemon07:14:503
287265199580,53cyclictest1622-21rtkit-daemon11:11:083
287264799581,51cyclictest1753-21gnome-shell10:36:592
287264799581,51cyclictest1753-21gnome-shell10:36:592
2872647995810,7cyclictest0-21swapper/209:49:592
287264599582,13cyclictest880966-21IPC1
287264599581,15cyclictest1753-21gnome-shell11:51:591
2872645995810,44cyclictest26-21ksoftirqd/111:31:201
287264199581,52cyclictest1753-21gnome-shell09:06:590
287265199578,45cyclictest881338-21IPC3
2872651995718,34cyclictest2977825-21cat12:20:163
2872645995710,43cyclictest26-21ksoftirqd/109:46:201
287264199572,50cyclictest2875063-21systemd-udevd07:15:190
162399570,9rtkit-daemon1622-21rtkit-daemon12:26:032
162399570,9rtkit-daemon1622-21rtkit-daemon12:05:403
162399570,9rtkit-daemon1622-21rtkit-daemon08:04:173
287265199563,46cyclictest880726-21firefox10:31:203
287265199561,27cyclictest1753-21gnome-shell10:01:203
287264199561,34cyclictest2926679-21missed_timers09:40:280
287264199561,34cyclictest2926679-21missed_timers09:40:270
162399560,9rtkit-daemon1622-21rtkit-daemon08:12:352
162399560,9rtkit-daemon1622-21rtkit-daemon08:05:072
162399560,10rtkit-daemon1622-21rtkit-daemon11:17:433
2872651995522,28cyclictest2880778-40lsb_release07:30:243
2872651995522,28cyclictest2880778-40lsb_release07:30:243
287264599551,11cyclictest1753-21gnome-shell11:07:591
287264199551,30cyclictest2905211-21cat08:40:210
287264199551,11cyclictest1753-21gnome-shell10:59:580
162399550,8rtkit-daemon1622-21rtkit-daemon09:07:382
287265199542,48cyclictest0-21swapper/308:32:593
287265199542,47cyclictest1753-21gnome-shell07:35:593
2872651995416,11cyclictest2966489-21cat11:40:353
2872651995415,15cyclictest881251-21Softwar~cThread09:17:583
2872651995415,11cyclictest2971795-21cat12:00:143
287264799541,46cyclictest1753-21gnome-shell08:21:592
287264799541,46cyclictest1753-21gnome-shell08:21:582
287264599542,47cyclictest1753-21gnome-shell12:15:581
287264199543,46cyclictest1753-21gnome-shell08:01:200
287264199542,47cyclictest1753-21gnome-shell11:36:590
287264199541,48cyclictest1753-21gnome-shell08:25:590
287264199541,30cyclictest2956885-21cut11:15:020
287264199541,11cyclictest1753-21gnome-shell10:49:590
162399540,8rtkit-daemon1622-21rtkit-daemon12:22:592
287265199537,12cyclictest880726-21firefox11:50:023
287265199532,10cyclictest1753-21gnome-shell10:07:593
2872651995316,11cyclictest2887683-21cat07:50:213
287265199531,10cyclictest859-21NetworkManager12:28:583
287265199531,10cyclictest1753-21gnome-shell07:22:583
287264799531,10cyclictest1753-21gnome-shell09:11:592
287264599531,47cyclictest1753-21gnome-shell10:16:591
287264599531,10cyclictest1753-21gnome-shell09:17:591
2872641995340,8cyclictest881331-21Privileged14:31:440
2872641995330,19cyclictest2899028-21nm-dispatcher08:21:110
2872641995330,19cyclictest2899028-21nm-dispatcher08:21:110
287264199532,47cyclictest1753-21gnome-shell09:16:590
2872641995321,27cyclictest2894630-21expr08:10:240
162399530,8rtkit-daemon1622-21rtkit-daemon12:09:122
287265199522,45cyclictest2981276-40lsb_release12:30:243
2872651995223,25cyclictest1753-21gnome-shell10:47:593
2872651995216,12cyclictest2903750-21hddtemp_smartct08:35:273
287264799521,10cyclictest1753-21gnome-shell10:46:582
287264599521,46cyclictest1753-21gnome-shell10:11:591
287264199523,8cyclictest0-21swapper/011:31:200
2872641995220,27cyclictest2941518-21wc10:25:450
2872641995219,29cyclictest2917023-21seq09:13:450
287264199521,46cyclictest2975566-21smartctl12:10:300
287265199516,5cyclictest0-21swapper/308:12:583
287265199512,7cyclictest0-21swapper/311:21:593
2872651995120,27cyclictest2945435-21ntp_kernel_err10:40:293
287265199511,46cyclictest1753-21gnome-shell07:26:593
2872647995138,8cyclictest1753-21gnome-shell10:30:592
287264799511,10cyclictest1753-21gnome-shell07:38:592
287264199512,45cyclictest0-21swapper/007:38:590
2872641995121,26cyclictest2966433-21memory11:40:350
287264199511,28cyclictest0-21swapper/011:00:340
287265199502,7cyclictest0-21swapper/310:21:593
287265199502,44cyclictest0-21swapper/307:55:593
287265199502,42cyclictest2874817-21python307:15:053
2872651995015,12cyclictest2927604-21cut09:45:013
2872651995015,12cyclictest2927604-21cut09:45:013
2872651995012,10cyclictest2969911-21cat11:50:343
2872647995038,8cyclictest1753-21gnome-shell09:43:592
2872647995038,8cyclictest1753-21gnome-shell09:43:592
2872647995037,9cyclictest1753-21gnome-shell07:10:592
287264799501,45cyclictest34-21ksoftirqd/212:35:582
287264199502,7cyclictest0-21swapper/010:42:590
287264199502,6cyclictest0-21swapper/008:59:580
287264199502,44cyclictest0-21swapper/012:30:590
287264199502,44cyclictest0-21swapper/007:45:300
162399500,6rtkit-daemon1622-21rtkit-daemon11:42:382
287265199492,43cyclictest0-21swapper/310:36:583
287265199492,43cyclictest0-21swapper/310:36:583
2872651994923,22cyclictest1069-21runrttasks10:17:183
2872651994912,11cyclictest2922050-21seq09:26:283
2872651994911,11cyclictest2951928-21cpuspeed_turbos11:00:263
2872645994938,7cyclictest1753-21gnome-shell07:49:591
2872645994932,6cyclictest0-21swapper/109:01:201
2872641994940,6cyclictest0-21swapper/011:58:580
2872641994939,6cyclictest0-21swapper/011:15:590
287264199492,6cyclictest0-21swapper/007:12:590
287264199492,44cyclictest0-21swapper/009:26:590
287264199492,44cyclictest0-21swapper/007:23:590
287264199492,25cyclictest0-21swapper/009:30:240
2872641994917,27cyclictest2934721-21cat10:05:410
2872641994916,28cyclictest2961275-21cat11:25:380
2872641994916,28cyclictest2961275-21cat11:25:380
162399490,6rtkit-daemon1622-21rtkit-daemon09:24:592
287265199486,13cyclictest2884156-21cut07:40:203
287265199482,6cyclictest0-21swapper/309:56:593
287265199482,43cyclictest0-21swapper/309:51:583
287265199482,42cyclictest0-21swapper/309:20:583
287265199482,42cyclictest0-21swapper/308:50:593
2872651994822,21cyclictest2950455-21diskstats10:55:283
2872651994819,24cyclictest2936001-21missed_timers10:10:323
2872651994817,27cyclictest2924337-21/usr/sbin/munin09:35:353
2872651994813,12cyclictest2907837-21cat08:45:413
2872651994813,12cyclictest2907837-21cat08:45:413
2872647994838,7cyclictest2887236-21kworker/u8:208:01:592
287264799482,42cyclictest0-21swapper/212:17:592
287264599482,43cyclictest0-21swapper/109:11:581
287264599482,42cyclictest0-21swapper/110:46:591
2872641994840,5cyclictest0-21swapper/011:50:580
287264199482,6cyclictest0-21swapper/008:07:590
287264199482,5cyclictest0-21swapper/007:25:590
287264199482,43cyclictest0-21swapper/012:26:580
2872641994819,24cyclictest2978002-21sshd12:20:220
2872641994814,28cyclictest2907119-21cat08:45:220
2872641994814,28cyclictest2907119-21cat08:45:220
2872651994735,8cyclictest1753-21gnome-shell08:42:593
287265199472,6cyclictest0-21swapper/310:53:593
287265199472,6cyclictest0-21swapper/309:07:593
287265199472,5cyclictest0-21swapper/310:27:593
287265199472,42cyclictest0-21swapper/311:39:593
287264799472,7cyclictest0-21swapper/207:31:242
287264799472,7cyclictest0-21swapper/207:31:232
2872645994738,5cyclictest0-21swapper/107:31:241
2872645994738,5cyclictest0-21swapper/107:31:231
287264599472,5cyclictest0-21swapper/109:51:591
287264599472,5cyclictest0-21swapper/108:10:591
287264599472,42cyclictest0-21swapper/107:38:591
2872641994740,5cyclictest0-21swapper/011:49:590
2872641994725,19cyclictest2899322-21kworker/u8:308:30:210
287264199472,42cyclictest0-21swapper/012:39:590
287264199472,42cyclictest0-21swapper/010:19:590
287264199471,43cyclictest2920255-21awk09:20:420
287265199462,39cyclictest2916075-21hddtemp_smartct09:10:263
287265199462,38cyclictest2899614-21/usr/sbin/munin08:25:303
287264799462,6cyclictest0-21swapper/210:01:202
287264799462,5cyclictest0-21swapper/207:49:592
287264799462,41cyclictest0-21swapper/211:46:592
287264799462,41cyclictest0-21swapper/210:27:582
287264799462,41cyclictest0-21swapper/209:01:202
2872645994639,4cyclictest0-21swapper/112:35:581
2872641994640,4cyclictest0-21swapper/012:05:590
2872641994611,30cyclictest2888552-21cat07:50:400
2872645994539,4cyclictest0-21swapper/110:36:591
2872645994539,4cyclictest0-21swapper/110:36:591
287264599453,36cyclictest1753-21gnome-shell11:23:581
2872645994527,13cyclictest438-20systemd-journal07:29:311
2872641994538,5cyclictest0-21swapper/011:21:590
2872641994538,4cyclictest0-21swapper/010:11:580
2872641994510,30cyclictest2972141-21cat12:00:220
29655812440,8sleep1251rcuc/111:40:001
2872647994438,4cyclictest0-21swapper/211:24:592
2872647994435,6cyclictest0-21swapper/211:31:202
287264799442,40cyclictest0-21swapper/211:55:592
2872645994436,5cyclictest26-21ksoftirqd/108:16:201
287264799431,37cyclictest1753-21gnome-shell08:53:592
287264599432,12cyclictest1753-21gnome-shell09:30:591
2872641994335,5cyclictest0-21swapper/007:55:590
162399430,37rtkit-daemon0-21swapper/212:12:452
2872645994234,6cyclictest231irq_work/110:27:581
2872645994223,14cyclictest2926459-21df_inode09:40:191
2872645994223,14cyclictest2926459-21df_inode09:40:181
2872641994226,11cyclictest1753-21gnome-shell09:57:590
2872647994135,4cyclictest0-21swapper/208:36:592
2872645994114,22cyclictest2901907-40lsb_release08:30:271
287264599411,15cyclictest29723642head12:00:251
29709692400,16chrt34-21ksoftirqd/211:55:032
287264799401,19cyclictest854-21avahi-daemon11:35:032
2872645994029,8cyclictest2874306-21cron07:14:591
2872645994027,8cyclictest2838385-21kworker/1:107:56:591
287264599402,33cyclictest1753-21gnome-shell11:04:591
162399400,31rtkit-daemon1622-21rtkit-daemon07:43:232
29196372390,5sleep10-21swapper/109:20:311
2872647993933,4cyclictest0-21swapper/210:10:592
2872647993932,4cyclictest0-21swapper/210:54:592
2872647993928,7cyclictest1753-21gnome-shell07:23:592
2872647993924,12cyclictest0-21swapper/210:40:192
2872647993924,10cyclictest1753-21gnome-shell10:24:582
287264799392,32cyclictest880726-21firefox10:09:232
2872647993923,12cyclictest2907944-21cat08:45:412
2872647993923,12cyclictest2907944-21cat08:45:402
2872647993922,12cyclictest438-20systemd-journal08:00:012
2872645993932,4cyclictest0-21swapper/111:55:591
2872645993928,7cyclictest1753-21gnome-shell12:06:591
2872645993923,12cyclictest2951537-21/usr/sbin/munin11:00:001
2872645993921,13cyclictest2910589-21ls08:55:231
2872647993831,5cyclictest0-21swapper/208:27:592
2872647993821,13cyclictest2922311-21seq09:29:512
2872645993832,4cyclictest0-21swapper/111:46:581
2872645993831,4cyclictest0-21swapper/108:06:591
2872645993824,10cyclictest2982661-21sh12:35:001
2872645993821,12cyclictest2961034-21hddtemp_smartct11:25:341
2872645993821,12cyclictest2961034-21hddtemp_smartct11:25:331
2872645993820,13cyclictest880970-21Timer09:08:441
2872647993725,8cyclictest1753-21gnome-shell07:19:592
2872647993721,12cyclictest2961387-21ntpq11:25:412
2872647993721,12cyclictest2961387-21ntpq11:25:412
2872647993721,11cyclictest438-20systemd-journal08:33:302
2872647993718,7cyclictest2950290-21cstates10:55:262
2872647993718,13cyclictest881251-21Softwar~cThread09:55:102
2872645993721,11cyclictest2924741-21diskstats09:35:291
2872645993720,12cyclictest2939119-21cat10:20:251
2872645993720,12cyclictest2899637-21apt08:25:181
2872645993717,13cyclictest881322-21Timer08:45:241
2872645993717,13cyclictest881322-21Timer08:45:231
2872647993621,11cyclictest2911803-21/usr/sbin/munin09:00:012
2872647993621,10cyclictest2938833-21sed10:20:022
287264799361,30cyclictest1753-21gnome-shell08:44:582
2872645993623,10cyclictest2883990-21awk07:40:191
2872645993621,11cyclictest2949020-21ntp_kernel_pll_10:50:321
2872645993621,10cyclictest2956140-21ntp_kernel_err11:10:391
2872645993619,13cyclictest2921929-21sed09:25:441
2872645993619,12cyclictest2903562-21cstates08:35:231
2872645993619,12cyclictest2887793-21expr07:50:211
287264599361,30cyclictest2966521-21strings11:40:341
2872647993520,11cyclictest2981796-21memory12:30:342
2872647993520,11cyclictest2971923-21expr12:00:152
2872647993519,12cyclictest2887824-21expr07:50:212
2872647993518,12cyclictest2917809-21fw_packets09:15:262
287264799351,30cyclictest0-21swapper/211:13:582
2872645993524,8cyclictest231irq_work/111:19:581
2872645993519,11cyclictest2874990-21swap07:15:061
287264599351,8cyclictest0-21swapper/112:27:251
287264599351,8cyclictest0-21swapper/112:21:051
287264599351,8cyclictest0-21swapper/108:45:001
2872647993423,7cyclictest2924736-21df_inode09:35:272
2872647993420,10cyclictest880971-21Netlink14:31:442
2872645993424,5cyclictest26-21ksoftirqd/107:23:431
2872645993419,11cyclictest2910211-21wc08:55:011
2872647993322,8cyclictest2952235-21idleruntime11:00:352
2872645993318,10cyclictest2931838-21cat10:00:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional