You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-07-25 - 08:24

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #8, slot #4

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack8slot4s.osadl.org (updated Thu Jul 25, 2024 00:44:43)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
7420522243178,52sleep70-21swapper/719:07:377
7419942232195,24sleep60-21swapper/619:06:476
7421122227171,19sleep20-21swapper/219:08:292
7419442222182,27sleep30-21swapper/319:06:033
7420282216159,17sleep40-21swapper/419:07:174
7420682215177,25sleep50-21swapper/519:07:505
7420162212179,22sleep10-21swapper/119:07:071
7421192210180,19sleep00-21swapper/019:08:340
74245199290,24cyclictest911108-21munin-run00:00:011
74246899280,3cyclictest806186-21idleruntime-cro21:00:017
742451992621,4cyclictest902711-21apt-get23:45:101
74245199260,0cyclictest0-21swapper/119:25:101
742448992620,5cyclictest754479-21idleruntime-cro19:30:000
742468992522,2cyclictest853336-21apt-get22:20:117
742468992520,4cyclictest0-21swapper/700:00:017
74245799250,21cyclictest0-21swapper/320:40:133
74245199250,21cyclictest0-21swapper/121:10:141
74244899250,24cyclictest0-21swapper/000:38:290
74246899240,23cyclictest812556-21apt-get21:10:137
74246899240,0cyclictest0-21swapper/722:38:297
742457992422,1cyclictest922147-21munin-run00:40:003
742457992420,3cyclictest884418-21apt-config23:15:003
742455992421,2cyclictest861484-21awk22:35:002
74245599240,23cyclictest872943-21perf22:55:002
74245599240,23cyclictest0-21swapper/221:35:002
74245199240,23cyclictest0-21swapper/121:45:121
74245199240,1cyclictest0-21swapper/120:25:111
742448992419,4cyclictest0-21swapper/019:20:010
74244899240,21cyclictest0-21swapper/023:45:100
742468992322,1cyclictest0-21swapper/723:35:127
74246899230,22cyclictest876230-21apt-get23:00:107
74246899230,22cyclictest791868-21idleruntime-cro20:35:007
742466992321,1cyclictest766296-21/usr/sbin/munin19:50:126
742466992321,1cyclictest0-21swapper/622:40:116
742466992320,3cyclictest745187-21awk19:15:016
742462992323,0cyclictest0-21swapper/523:10:145
74246299230,22cyclictest0-21swapper/523:25:095
74246099230,23cyclictest0-21swapper/419:20:004
742457992321,1cyclictest921125-21turbostat00:35:003
742457992321,1cyclictest771616-21munin-run20:00:013
742457992319,4cyclictest0-21swapper/300:15:003
742457992316,6cyclictest905149-21basename23:50:003
74245799230,22cyclictest852900-21munin-run22:19:593
74245799230,22cyclictest0-21swapper/323:40:093
74245599230,22cyclictest873357-21apt-get22:55:112
742451992321,1cyclictest0-21swapper/120:25:011
742451992317,5cyclictest748761-21sendmail19:20:001
74245199230,22cyclictest0-21swapper/122:50:111
74245199230,1cyclictest0-21swapper/120:50:121
742448992321,1cyclictest749202-21apt-get19:20:080
742448992321,1cyclictest0-21swapper/023:40:090
742448992321,1cyclictest0-21swapper/021:30:000
74244899230,22cyclictest911197-21latency_hist00:00:000
74244899230,22cyclictest872939-21munin-run22:54:590
74244899230,22cyclictest0-21swapper/022:25:080
74244899230,22cyclictest0-21swapper/020:50:000
74244899230,22cyclictest0-21swapper/020:25:000
742468992222,0cyclictest0-21swapper/721:45:127
742468992221,1cyclictest0-21swapper/700:20:117
742468992220,2cyclictest0-21swapper/719:25:007
74246899220,22cyclictest783030-21apt-get20:20:007
74246899220,21cyclictest0-21swapper/719:25:117
74246899220,21cyclictest0-21swapper/700:35:017
74246899220,1cyclictest889372-21ntp_kernel_err23:20:177
74246899220,1cyclictest875805-21systemctl22:59:597
74246899220,1cyclictest0-21swapper/719:10:127
742466992222,0cyclictest0-21swapper/619:30:106
742466992221,1cyclictest752052-21apt-get19:25:116
742466992221,1cyclictest0-21swapper/622:38:306
742466992221,1cyclictest0-21swapper/620:55:126
74246699220,22cyclictest0-21swapper/622:59:596
74246699220,21cyclictest0-21swapper/621:00:136
74246699220,21cyclictest0-21swapper/620:10:136
74246699220,21cyclictest0-21swapper/620:10:126
74246699220,1cyclictest0-21swapper/621:15:006
742462992222,0cyclictest0-21swapper/520:05:105
742462992221,0cyclictest0-21swapper/523:55:105
742462992220,1cyclictest809026-21awk21:05:015
74246299220,22cyclictest0-21swapper/523:15:125
74246299220,21cyclictest921131-21munin-run00:35:005
74246299220,21cyclictest916020-21perf00:15:015
74246299220,21cyclictest0-21swapper/523:30:115
74246299220,21cyclictest0-21swapper/522:50:115
74246299220,21cyclictest0-21swapper/522:25:085
74246299220,21cyclictest0-21swapper/522:15:125
74246299220,21cyclictest0-21swapper/520:10:125
74246299220,21cyclictest0-21swapper/520:10:125
74246299220,21cyclictest0-21swapper/519:30:095
74246299220,0cyclictest0-21swapper/523:04:595
742460992221,1cyclictest852919-21sleep22:19:594
742460992221,1cyclictest0-21swapper/422:05:004
74246099220,22cyclictest0-21swapper/421:10:124
74246099220,21cyclictest870508-21apt-get22:50:114
74246099220,21cyclictest777753-21apt-get20:10:134
74246099220,21cyclictest777753-21apt-get20:10:124
74246099220,21cyclictest0-21swapper/423:30:114
74246099220,21cyclictest0-21swapper/422:45:114
74246099220,21cyclictest0-21swapper/422:25:104
74246099220,21cyclictest0-21swapper/419:45:124
74246099220,1cyclictest826404-21apt-get21:35:014
742457992222,0cyclictest0-21swapper/319:40:113
742457992220,1cyclictest885040-21apt-get23:15:103
742457992220,1cyclictest0-21swapper/323:05:103
742457992220,1cyclictest0-21swapper/300:00:113
742457992218,3cyclictest875805-21systemctl23:00:003
74245799220,1cyclictest835515-21munin-run21:50:003
74245799220,1cyclictest0-21swapper/323:55:013
74245799220,1cyclictest0-21swapper/319:25:013
74245799220,0cyclictest0-21swapper/320:30:093
742455992221,1cyclictest768761-21munin-run19:55:002
742455992221,1cyclictest0-21swapper/220:40:122
742455992220,1cyclictest829968-21apt-get21:40:122
742455992220,1cyclictest803718-21apt-get20:55:112
74245599220,21cyclictest0-21swapper/223:35:112
74245599220,21cyclictest0-21swapper/222:40:112
74245599220,21cyclictest0-21swapper/220:30:092
74245599220,1cyclictest888175-21apt-get23:20:092
74245599220,1cyclictest823565-21munin-run21:30:002
74245599220,1cyclictest0-21swapper/223:05:002
74245599220,1cyclictest0-21swapper/220:00:112
74245599220,1cyclictest0-21swapper/219:25:122
74245599220,1cyclictest0-21swapper/200:00:012
74245599220,0cyclictest0-21swapper/200:10:012
742451992221,1cyclictest0-21swapper/119:40:111
74245199220,22cyclictest777335-21munin-run20:10:011
74245199220,21cyclictest0-21swapper/120:30:091
74245199220,21cyclictest0-21swapper/120:15:121
742448992222,0cyclictest0-21swapper/019:40:120
742448992221,1cyclictest0-21swapper/021:35:010
742448992221,0cyclictest0-21swapper/020:30:100
742448992220,1cyclictest852936-21sendmail-msp22:20:000
74244899220,21cyclictest835519-21perf21:50:010
74244899220,0cyclictest0-21swapper/022:20:100
742468992121,0cyclictest0-21swapper/721:35:017
742468992121,0cyclictest0-21swapper/720:10:127
742468992121,0cyclictest0-21swapper/720:10:117
74246899210,20cyclictest0-21swapper/720:50:127
74246899210,0cyclictest0-21swapper/722:40:107
74246899210,0cyclictest0-21swapper/722:10:107
74246899210,0cyclictest0-21swapper/722:00:127
742466992121,0cyclictest0-21swapper/621:35:146
742466992121,0cyclictest0-21swapper/621:20:126
742466992120,1cyclictest815405-21apt-get21:15:136
742466992119,2cyclictest887523-21basename23:20:006
74246699210,21cyclictest922137-21apt-get00:40:016
74246699210,21cyclictest0-21swapper/623:35:126
74246699210,21cyclictest0-21swapper/623:20:016
742462992121,0cyclictest0-21swapper/523:50:095
742462992120,1cyclictest0-21swapper/520:15:125
742462992119,1cyclictest760629-21apt-get19:40:135
742462992119,1cyclictest0-21swapper/520:00:005
74246299210,20cyclictest0-21swapper/500:40:015
74246299210,1cyclictest0-21swapper/521:25:015
742460992121,0cyclictest0-21swapper/421:40:114
742460992121,0cyclictest0-21swapper/421:30:124
742460992121,0cyclictest0-21swapper/420:55:114
742460992120,1cyclictest811894-21basename21:10:014
74246099210,21cyclictest0-21swapper/423:15:124
74246099210,21cyclictest0-21swapper/421:50:134
74246099210,1cyclictest1096759-21systemd-journal23:45:004
74246099210,0cyclictest0-21swapper/423:10:134
742457992120,1cyclictest0-21swapper/323:30:113
742457992120,1cyclictest0-21swapper/321:20:113
742457992120,1cyclictest0-21swapper/320:20:023
742457992118,3cyclictest0-21swapper/321:00:113
742457992118,3cyclictest0-21swapper/319:15:003
742457992118,2cyclictest760214-21perf-exec19:40:003
74245799210,19cyclictest0-21swapper/322:05:013
742455992121,0cyclictest0-21swapper/220:15:122
742455992120,0cyclictest0-21swapper/220:35:132
742455992117,3cyclictest919628-21latency_hist00:25:012
74245599210,21cyclictest0-21swapper/221:55:002
74245599210,20cyclictest0-21swapper/219:10:112
74245599210,1cyclictest0-21swapper/223:25:102
74245599210,0cyclictest0-21swapper/221:05:122
74245599210,0cyclictest0-21swapper/220:25:112
742451992120,1cyclictest0-21swapper/123:20:001
742451992120,1cyclictest0-21swapper/119:25:001
742451992119,1cyclictest911571-21apt-get00:00:121
742451992119,1cyclictest766329-21apt-get19:50:111
742451992119,1cyclictest0-21swapper/122:19:591
742451992118,3cyclictest0-21swapper/120:40:121
742451992117,3cyclictest0-21swapper/100:40:001
742451992113,7cyclictest0-21swapper/123:24:591
74245199210,0cyclictest0-21swapper/121:05:121
742448992119,1cyclictest650-21dbus-daemon20:20:000
742448992119,1cyclictest0-21swapper/020:40:120
742448992118,2cyclictest0-21swapper/023:15:010
742448992117,3cyclictest0-21swapper/023:30:120
74244899210,20cyclictest0-21swapper/021:10:000
74244899210,0cyclictest0-21swapper/021:55:140
742468992020,0cyclictest839040-21apt-get21:55:157
742468992020,0cyclictest0-21swapper/721:50:017
742468992020,0cyclictest0-21swapper/721:05:117
742468992019,1cyclictest777353-21awk20:10:017
742468992019,1cyclictest0-21swapper/720:25:117
742468992019,0cyclictest0-21swapper/722:05:127
74246899201,18cyclictest819038-21gltestperf21:20:157
74246899200,7cyclictest0-21swapper/723:08:147
74246899200,20cyclictest885040-21apt-get23:15:117
74246899200,0cyclictest0-21swapper/723:40:107
74246899200,0cyclictest0-21swapper/720:45:107
742466992020,0cyclictest0-21swapper/623:40:106
742466992020,0cyclictest0-21swapper/620:35:126
742466992019,1cyclictest911576-21apt-get00:00:136
742466992019,1cyclictest908672-21apt-get23:55:106
742466992019,1cyclictest0-21swapper/621:05:126
742466992018,1cyclictest823565-21munin-run21:30:016
742462992020,0cyclictest0-21swapper/521:55:145
742462992020,0cyclictest0-21swapper/521:10:135
742462992019,1cyclictest0-21swapper/519:40:005
742462992019,1cyclictest0-21swapper/500:05:025
74246299200,20cyclictest0-21swapper/521:35:145
74246299200,0cyclictest0-21swapper/522:05:115
742460992020,0cyclictest0-21swapper/423:35:114
742460992020,0cyclictest0-21swapper/423:25:094
742460992020,0cyclictest0-21swapper/422:10:094
742460992020,0cyclictest0-21swapper/419:15:004
742460992019,1cyclictest905163-21awk23:50:014
742460992019,1cyclictest0-21swapper/419:55:014
742460992018,2cyclictest916771-21munin-run00:20:014
74246099200,20cyclictest0-21swapper/423:20:024
74246099200,20cyclictest0-21swapper/420:00:014
74246099200,0cyclictest0-21swapper/422:40:104
74246099200,0cyclictest0-21swapper/419:40:124
74246099200,0cyclictest0-21swapper/419:30:004
74246099200,0cyclictest0-21swapper/400:20:114
742457992020,0cyclictest0-21swapper/321:05:123
742457992018,1cyclictest0-21swapper/322:35:123
742457992017,3cyclictest890601-21turbostat23:25:003
74245799200,1cyclictest783030-21apt-get20:20:013
742455992020,0cyclictest0-21swapper/220:50:112
742455992020,0cyclictest0-21swapper/220:45:112
742455992020,0cyclictest0-21swapper/200:00:122
742455992019,1cyclictest0-21swapper/220:20:132
74245599200,0cyclictest0-21swapper/223:40:092
74245599200,0cyclictest0-21swapper/220:10:202
74245599200,0cyclictest0-21swapper/220:10:202
742451992020,0cyclictest0-21swapper/121:35:021
742451992020,0cyclictest0-21swapper/121:30:001
742451992019,1cyclictest0-21swapper/123:40:091
742448992020,0cyclictest0-21swapper/021:15:000
742448992020,0cyclictest0-21swapper/020:50:120
742448992020,0cyclictest0-21swapper/019:50:110
742448992018,1cyclictest0-21swapper/022:10:090
742448992018,1cyclictest0-21swapper/020:00:000
742448992018,1cyclictest0-21swapper/000:20:120
742448992014,5cyclictest0-21swapper/023:24:590
742466991919,0cyclictest0-21swapper/619:15:126
742466991918,1cyclictest905560-21apt-get23:50:096
742466991918,1cyclictest0-21swapper/620:50:126
742466991917,2cyclictest844345-21munin-run22:05:006
74246699190,1cyclictest0-21swapper/620:45:106
742462991919,0cyclictest0-21swapper/523:45:025
742462991919,0cyclictest0-21swapper/522:30:105
742462991919,0cyclictest0-21swapper/521:15:125
742462991919,0cyclictest0-21swapper/520:55:115
742462991919,0cyclictest0-21swapper/520:50:005
742462991919,0cyclictest0-21swapper/520:20:125
742462991919,0cyclictest0-21swapper/519:10:115
742462991917,1cyclictest841911-21apt-get22:00:125
742462991917,1cyclictest0-21swapper/522:40:205
742460991919,0cyclictest0-21swapper/421:55:154
742460991919,0cyclictest0-21swapper/421:45:014
742460991919,0cyclictest0-21swapper/420:40:124
742460991919,0cyclictest0-21swapper/420:05:104
742460991918,1cyclictest749202-21apt-get19:20:094
742460991918,1cyclictest0-21swapper/400:40:014
742460991917,1cyclictest0-21swapper/420:50:004
742457991919,0cyclictest0-21swapper/300:20:013
742457991918,1cyclictest0-21swapper/320:35:133
742457991918,0cyclictest0-21swapper/320:50:113
742457991918,0cyclictest0-21swapper/319:45:123
742457991917,2cyclictest0-21swapper/320:10:113
742457991917,2cyclictest0-21swapper/320:10:113
742457991917,1cyclictest0-21swapper/321:50:013
742457991914,4cyclictest919633-21turbostat00:25:013
742457991911,7cyclictest817844-21munin-run21:20:003
74245799190,1cyclictest774482-21munin-run20:05:013
742455991919,0cyclictest0-21swapper/223:45:102
742455991918,1cyclictest887489-21systemctl23:19:592
742455991918,1cyclictest833082-21apt-get21:45:122
742451991918,1cyclictest774491-21turbostat20:05:011
742451991917,1cyclictest763479-21apt-get19:45:121
742451991916,2cyclictest921131-21munin-run00:35:001
742451991916,2cyclictest893878-21apt-get23:30:121
742448991919,0cyclictest0-21swapper/000:00:130
742448991917,1cyclictest0-21swapper/023:15:010
742448991917,1cyclictest0-21swapper/020:05:000
742448991916,2cyclictest916033-21awk00:15:000
742468991818,0cyclictest0-21swapper/722:50:117
742468991818,0cyclictest0-21swapper/719:30:097
742468991816,1cyclictest0-21swapper/723:45:107
74246899180,3cyclictest0-21swapper/719:15:117
74246899180,2cyclictest760555-21date19:40:017
74246899180,17cyclictest0-21swapper/720:41:207
742466991818,0cyclictest0-21swapper/623:45:016
742466991818,0cyclictest0-21swapper/622:10:106
742466991818,0cyclictest0-21swapper/621:50:126
742466991817,1cyclictest0-21swapper/620:40:126
742466991816,1cyclictest855775-21perf22:24:596
742462991818,0cyclictest0-21swapper/521:35:015
742462991818,0cyclictest0-21swapper/520:00:115
742462991816,1cyclictest899160-21turbostat23:39:595
742462991815,2cyclictest914010-21perf00:05:005
742460991818,0cyclictest0-21swapper/420:50:114
742460991816,1cyclictest820725-21basename21:25:004
74246099180,0cyclictest0-21swapper/400:05:014
742457991817,1cyclictest876230-21apt-get23:00:093
742457991817,1cyclictest0-21swapper/323:50:103
742457991817,1cyclictest0-21swapper/319:50:113
742457991816,2cyclictest0-21swapper/322:20:113
742457991816,1cyclictest0-21swapper/322:45:113
742457991816,1cyclictest0-21swapper/322:40:103
742457991815,2cyclictest0-21swapper/322:10:083
742457991813,4cyclictest0-21swapper/320:10:003
742455991817,1cyclictest864196-21fwupd22:38:292
742451991817,1cyclictest0-21swapper/122:10:091
742451991817,1cyclictest0-21swapper/121:25:001
742451991816,2cyclictest915007-21apt-get00:10:011
742448991818,0cyclictest0-21swapper/021:20:120
742448991818,0cyclictest0-21swapper/019:45:120
742448991818,0cyclictest0-21swapper/000:35:010
742448991817,1cyclictest836188-21apt-get21:50:120
742448991817,0cyclictest0-21swapper/023:05:110
742448991816,1cyclictest0-21swapper/000:20:000
742448991815,3cyclictest0-21swapper/019:15:010
742448991815,2cyclictest875806-21awk22:59:590
742448991813,4cyclictest0-21swapper/023:25:100
74244899180,15cyclictest0-21swapper/020:55:170
742468991717,0cyclictest0-21swapper/722:25:087
742468991717,0cyclictest0-21swapper/720:20:127
742468991715,2cyclictest0-21swapper/719:51:447
742468991715,1cyclictest0-21swapper/719:40:127
742468991714,2cyclictest0-21swapper/700:29:357
742468991714,2cyclictest0-21swapper/700:19:417
74246899171,2cyclictest850721-21grep22:15:127
74246899170,2cyclictest0-21swapper/722:45:097
74246899170,15cyclictest0-21swapper/723:29:197
74246899170,0cyclictest0-21swapper/721:40:127
742466991717,0cyclictest0-21swapper/621:55:236
742466991717,0cyclictest0-21swapper/619:20:086
742466991716,1cyclictest0-21swapper/620:20:006
742462991717,0cyclictest0-21swapper/521:05:135
742462991717,0cyclictest0-21swapper/520:50:115
742462991715,1cyclictest766329-21apt-get19:50:125
742460991717,0cyclictest0-21swapper/420:20:014
742457991717,0cyclictest0-21swapper/323:25:103
742457991717,0cyclictest0-21swapper/321:35:153
742457991716,1cyclictest501ktimers/322:08:003
742457991716,1cyclictest0-21swapper/319:30:093
742457991714,2cyclictest0-21swapper/322:30:283
742457991714,2cyclictest0-21swapper/320:58:053
74245799170,16cyclictest0-21swapper/319:28:053
74245799170,15cyclictest0-21swapper/321:40:023
74245799170,14cyclictest0-21swapper/321:12:103
74245799170,0cyclictest0-21swapper/321:33:073
742455991717,0cyclictest0-21swapper/221:10:022
742455991716,1cyclictest0-21swapper/219:35:012
742455991715,1cyclictest0-21swapper/219:35:112
742451991717,0cyclictest0-21swapper/122:20:111
742451991716,1cyclictest0-21swapper/123:10:131
742451991716,1cyclictest0-21swapper/120:00:001
742451991714,2cyclictest844323-21cron22:05:001
742451991714,2cyclictest0-21swapper/121:00:111
74245199170,16cyclictest0-21swapper/123:37:501
74245199170,16cyclictest0-21swapper/100:16:561
74245199170,0cyclictest0-21swapper/123:02:421
742448991715,2cyclictest0-21swapper/021:00:110
742448991715,1cyclictest0-21swapper/022:05:000
742448991715,1cyclictest0-21swapper/019:40:010
742448991714,2cyclictest777753-21apt-get20:10:120
742448991714,2cyclictest777753-21apt-get20:10:110
74244899170,16cyclictest817769-21sshd21:19:210
74244899170,16cyclictest0-21swapper/022:35:590
74244899170,16cyclictest0-21swapper/020:07:360
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional