You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-04-21 - 06:46

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #7, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack7slot7s.osadl.org (updated Sun Apr 21, 2024 00:43:47)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
32509210288,9sleep10-21swapper/119:08:441
32406210086,9sleep30-21swapper/319:07:253
231442970,0sleep30-21swapper/321:30:373
3218928471,9sleep00-21swapper/019:05:290
3249328264,13sleep20-21swapper/219:08:322
215222570,1sleep10-21swapper/100:10:001
32681992725,1cyclictest0-21swapper/100:35:151
3268099260,26cyclictest0-21swapper/021:57:570
32689992423,1cyclictest0-21swapper/223:14:212
3268999234,17cyclictest18901-21/usr/sbin/munin23:00:102
32689992318,4cyclictest32-21ksoftirqd/223:55:002
3269399220,19cyclictest0-21swapper/323:15:243
3268099223,18cyclictest0-21swapper/023:02:200
3268099220,3cyclictest0-21swapper/021:42:210
3268099210,20cyclictest0-21swapper/023:21:290
3268099210,18cyclictest0-21swapper/022:21:000
3268199206,10cyclictest26626-21/usr/sbin/munin22:20:131
3269399190,2cyclictest0-21swapper/323:11:383
3269399190,1cyclictest0-21swapper/323:08:413
3269399190,1cyclictest0-21swapper/323:08:413
3268199194,7cyclictest224-21systemd-journal23:05:001
32681991918,1cyclictest0-21swapper/123:18:081
3268099195,4cyclictest25523-21diskmemload22:30:230
3268099190,0cyclictest0-21swapper/023:06:100
3268099190,0cyclictest0-21swapper/023:06:100
3268999180,18cyclictest0-21swapper/222:46:162
3268999180,18cyclictest0-21swapper/222:26:142
32681991817,1cyclictest0-21swapper/123:09:071
32681991817,1cyclictest0-21swapper/123:09:071
3268199181,2cyclictest0-21swapper/123:39:261
3268199180,2cyclictest0-21swapper/121:16:581
3268099184,7cyclictest13624-21munin-run21:25:000
3268099180,2cyclictest0-21swapper/023:32:390
32693991716,1cyclictest0-21swapper/321:51:483
32693991716,1cyclictest0-21swapper/300:06:563
3269399171,3cyclictest6561-21grep23:35:293
3269399170,0cyclictest0-21swapper/321:22:323
3268999175,3cyclictest224-21systemd-journal22:22:172
3268999175,10cyclictest6699-21grep22:05:232
32689991715,1cyclictest22174-21ssh22:16:132
3268199170,3cyclictest0-21swapper/122:48:091
3268199170,3cyclictest0-21swapper/122:34:331
32680991711,5cyclictest5203-21ssh23:57:150
3268099170,13cyclictest0-21swapper/021:36:440
3268099170,13cyclictest0-21swapper/021:36:440
3268099170,0cyclictest0-21swapper/023:13:360
3269399162,8cyclictest8762-21/usr/sbin/munin00:25:303
32681991616,0cyclictest0-21swapper/122:44:041
32681991614,1cyclictest0-21swapper/123:13:571
3268099161,2cyclictest0-21swapper/021:30:130
3269399150,2cyclictest0-21swapper/300:37:323
32689991514,0cyclictest0-21swapper/221:46:522
32689991513,1cyclictest0-21swapper/222:59:552
32689991512,2cyclictest0-21swapper/200:01:462
32689991511,3cyclictest9140-21grep21:20:312
3268999150,15cyclictest0-21swapper/200:05:292
3268199158,2cyclictest0-21swapper/119:55:121
32681991512,2cyclictest7283-21ssh21:20:151
3268199150,14cyclictest0-21swapper/121:49:191
3268099151,2cyclictest0-21swapper/020:35:130
32680991511,2cyclictest3630-21grep19:15:200
3268099150,1cyclictest0-21swapper/023:29:030
3268099150,1cyclictest0-21swapper/000:28:170
86142140,4sleep339-21ksoftirqd/320:35:003
3269399146,7cyclictest16300-21diskstats23:20:173
32693991410,4cyclictest23892-21apache222:52:013
32693991410,3cyclictest20864-21chrt00:34:543
32693991410,2cyclictest21752-21cut21:00:163
32693991410,2cyclictest10525-21grep22:30:323
3269399140,0cyclictest0-21swapper/323:04:303
3268999148,5cyclictest17533-21sed21:50:202
32689991410,3cyclictest1676-21tr23:55:162
3268199149,4cyclictest11468-21awk22:55:001
3268199143,5cyclictest15999-21apt-get20:50:121
32681991410,3cyclictest23816-21grep00:10:311
32681991410,2cyclictest15896-21grep19:40:231
3268199140,14cyclictest0-21swapper/100:26:411
3268099146,4cyclictest0-21swapper/020:35:000
3268099144,7cyclictest20563-21idleruntime-cro21:30:000
3268099141,7cyclictest16911-21munin-run19:45:000
32680991410,3cyclictest17410-21tr19:45:220
32680991410,3cyclictest17410-21tr19:45:220
3268099140,8cyclictest0-21swapper/000:30:150
3268099140,1cyclictest0-21swapper/000:36:210
3268099140,13cyclictest1560-21tr22:25:180
3269399139,3cyclictest9146-21cron21:45:003
3269399139,3cyclictest27779-21uname23:50:213
3269399139,3cyclictest11981-21cron22:10:003
3269399139,2cyclictest28630-21tr00:15:153
3269399135,6cyclictest8855-21apt-get20:35:133
32693991312,1cyclictest0-21swapper/323:59:083
32693991310,2cyclictest4465-21ls22:50:003
32693991310,2cyclictest3294-21grep22:25:323
32693991310,2cyclictest23637-21head23:25:183
32693991310,2cyclictest22673-21grep19:55:173
32693991310,2cyclictest19087-21tail20:55:203
32693991310,2cyclictest17657-21grep20:50:273
3269399130,13cyclictest0-21swapper/321:11:443
3269399130,0cyclictest0-21swapper/300:20:083
3268999139,3cyclictest7738-21wc20:30:242
3268999139,3cyclictest32291-21cron21:15:002
3268999138,2cyclictest4382-21tr20:25:152
3268999131,7cyclictest15779-21awk21:25:302
32689991312,1cyclictest0-21swapper/223:20:122
32689991310,2cyclictest3092-21grep20:20:282
32689991310,2cyclictest30734-21uniq23:30:162
32689991310,2cyclictest2759-21grep21:40:162
32689991310,2cyclictest25581-21sed21:55:292
32689991310,2cyclictest22069-21basename00:10:172
3268999130,3cyclictest0-21swapper/223:46:542
3268199139,4cyclictest15603-21/usr/sbin/munin23:20:171
3268199139,3cyclictest3354-21sed23:55:311
32681991310,2cyclictest9778-21basename19:30:001
32681991310,2cyclictest4397-21wc19:15:311
32681991310,2cyclictest21283-21grep22:15:291
3268199130,12cyclictest1595-21grep22:25:181
3268099139,3cyclictest1792-21tr21:15:270
3268099137,6cyclictest11726-21packagekitd23:19:070
3268099133,3cyclictest0-21swapper/020:40:120
32680991312,1cyclictest0-21swapper/022:57:060
32680991310,2cyclictest7966-21wc19:25:150
32680991310,2cyclictest6082-21wc23:35:240
32680991310,2cyclictest26101-21awk23:50:000
32680991310,2cyclictest23629-21grep19:55:280
32680991310,2cyclictest18063-21wc21:50:260
3268099130,12cyclictest25556-21wc21:10:000
3269399129,2cyclictest9461-21sed00:00:203
3269399129,2cyclictest3486-21cut19:15:203
3269399129,2cyclictest27520-21grep20:05:203
3269399129,2cyclictest2194-21grep20:20:183
3269399129,2cyclictest20446-21ssh22:37:583
3269399129,2cyclictest1760-21chrt21:39:393
3269399129,2cyclictest1760-21chrt21:39:393
3269399129,2cyclictest15521-21grep19:40:193
3269399128,3cyclictest25080-21grep20:00:213
3269399128,3cyclictest12739-21tr22:10:153
3269399128,2cyclictest24163-21sed21:05:183
3269399125,5cyclictest30068-21apt-get23:30:133
32693991211,1cyclictest0-21swapper/321:47:023
3268999129,2cyclictest6016-21grep19:20:172
3268999129,2cyclictest31568-21head22:00:192
3268999129,2cyclictest21205-21tr21:00:142
3268999128,3cyclictest4086-21cat00:20:282
3268999127,2cyclictest22130-21ssh22:39:332
3268999127,2cyclictest15356-21sh22:35:002
3268999125,5cyclictest9632-21ntpq23:15:202
3268999124,5cyclictest4762-21ssh22:50:122
32689991211,1cyclictest19416-21kworker/u8:300:16:552
32689991210,2cyclictest37850irq/131-snd_hda23:39:582
3268999120,2cyclictest0-21swapper/221:30:132
3268999120,1cyclictest141rcu_preempt00:33:092
3268199129,2cyclictest31175-21taskset00:16:331
3268199129,2cyclictest27839-21ntp_states23:50:221
3268199129,2cyclictest26392-21cat21:10:131
3268199129,2cyclictest24127-21grep21:05:181
3268199129,2cyclictest23972-21grep21:55:141
3268199129,2cyclictest23708-21head23:25:191
3268199129,2cyclictest21246-21tr21:00:141
3268199129,2cyclictest20963-21cron00:35:001
3268199129,2cyclictest13738-21grep22:55:271
3268199129,2cyclictest13024-21grep19:35:171
3268199129,2cyclictest12789-21cat20:40:261
3268199129,2cyclictest11989-21latency_hist22:10:011
3268199129,2cyclictest1196-21run-parts20:20:001
3268199128,3cyclictest20551-21cron21:30:001
3268199128,2cyclictest18369-21ls19:45:301
3268199128,2cyclictest18369-21ls19:45:301
3268199123,6cyclictest32123-21sh21:37:331
3268199123,6cyclictest32123-21sh21:37:331
3268099129,2cyclictest30809-21grep20:10:310
3268099129,2cyclictest2738-21chrt22:02:520
3268099129,2cyclictest23887-21grep00:10:330
3268099129,2cyclictest16082-21tr22:35:130
3268099128,3cyclictest26979-21uname21:10:170
3268099128,3cyclictest21217-21grep19:50:320
3268099128,2cyclictest15042-21wc20:45:280
3268099127,3cyclictest9643-21apt-get21:45:140
32680991212,0cyclictest0-21swapper/022:42:290
3269399119,1cyclictest701-21sort19:10:203
3269399118,2cyclictest7658-21taskset19:25:123
3269399118,2cyclictest32651-21grep22:00:313
3269399118,2cyclictest30525-21sh22:22:213
3269399118,2cyclictest2751-21ssh21:15:543
3269399118,2cyclictest26382-21ssh00:13:233
3269399118,2cyclictest25377-21sh22:40:443
3269399118,2cyclictest24338-21cut21:55:183
3269399118,2cyclictest21293-21sed22:15:293
3269399118,2cyclictest18733-21ssh21:28:103
3269399118,2cyclictest18067-21ssh22:59:363
3269399118,2cyclictest13050-21sort19:35:173
3269399114,3cyclictest224-21systemd-journal23:45:573
3268999119,1cyclictest28091-21chrt21:35:132
3268999119,1cyclictest28091-21chrt21:35:132
3268999119,1cyclictest1221-21head21:15:202
3268999118,2cyclictest37850irq/131-snd_hda21:05:202
3268999118,2cyclictest3581-21cut19:15:202
3268999118,2cyclictest31225-21cron20:15:002
3268999118,2cyclictest28190-21ssh23:05:292
3268999118,2cyclictest28190-21ssh23:05:292
3268999118,2cyclictest26416-21taskset20:04:192
3268999118,2cyclictest23497-21cat22:40:152
3268999118,2cyclictest20485-21dump-pmu-power21:00:002
3268999118,2cyclictest19652-21aten2_r7power_v19:50:162
3268999118,2cyclictest16179-21cut19:40:262
3268999118,2cyclictest14708-21chrt22:10:472
3268999118,2cyclictest14353-21taskset19:38:112
3268999118,2cyclictest12521-21expr23:40:132
3268999118,2cyclictest11847-21taskset19:31:272
3268999118,2cyclictest11177-21ssh00:26:002
3268999117,3cyclictest1359-21wc20:20:012
3268999110,10cyclictest22597-21chrt23:25:012
3268199118,2cyclictest883-21ssh23:31:381
3268199118,2cyclictest8087-21grep20:30:281
3268199118,2cyclictest7145-21ssh21:43:001
3268199118,2cyclictest6996-21grep19:20:271
3268199118,2cyclictest477-21sh22:00:421
3268199118,2cyclictest29672-21df20:10:181
3268199118,2cyclictest2882-21expr00:20:151
3268199118,2cyclictest28426-21sed20:05:301
3268199118,2cyclictest18724-21ssh22:36:231
3268199118,2cyclictest18618-21ssh22:14:401
3268199118,2cyclictest12674-21ssh00:02:491
3268199118,2cyclictest10263-21tr19:30:161
3268199117,3cyclictest26473-21cron20:05:001
3268199117,2cyclictest12119-21tr23:40:111
3268199113,4cyclictest224-21systemd-journal21:52:081
3268199113,4cyclictest224-21systemd-journal19:51:371
3268199110,10cyclictest23115-21rm21:30:361
3268099118,2cyclictest7090-21ssh00:23:300
3268099118,2cyclictest6050-21entropy22:05:170
3268099118,2cyclictest4857-21aten2_r7power_e22:50:140
3268099118,2cyclictest3283-21ssh22:48:560
3268099118,2cyclictest32264-21head20:15:190
3268099118,2cyclictest26412-21ssh23:50:070
3268099118,2cyclictest25290-21interrupts20:00:230
3268099118,2cyclictest1812-21idleruntime-cro00:20:000
3268099118,2cyclictest15707-21df00:05:180
3268099118,2cyclictest15467-21ssh23:41:500
3268099118,2cyclictest13960-21grep22:10:260
3268099118,2cyclictest10665-21ssh00:00:350
3268099117,3cyclictest6218-21ls20:30:000
3269399108,1cyclictest20942-21python19:50:313
3269399108,1cyclictest14998-21basename20:45:253
3269399108,1cyclictest12678-21sh23:40:143
3269399107,2cyclictest4075-21taskset20:25:113
3269399107,2cyclictest10785-21cut19:30:193
3269399102,6cyclictest0-21swapper/319:20:153
3269399100,9cyclictest880-21taskset20:16:173
3268999108,1cyclictest88192sleep220:35:112
3268999108,1cyclictest276832sleep220:05:222
3268999108,1cyclictest21208-21ssh00:35:012
3268999108,1cyclictest18671-21cat19:45:352
3268999108,1cyclictest18671-21cat19:45:352
3268999108,1cyclictest15648-21taskset20:49:062
3268999107,2cyclictest899-21cat19:10:212
3268999107,2cyclictest8744-21grep19:25:222
3268999107,2cyclictest16479-21df_abs20:50:152
3268999107,2cyclictest11975-21head20:40:162
3268199108,1cyclictest6068-21taskset20:28:181
3268199108,1cyclictest19293-21uname20:55:231
3268199108,1cyclictest19241-21ssh23:45:121
3268199107,2cyclictest718-21wc19:10:201
3268199107,2cyclictest10248-21sensors20:35:271
3268099108,1cyclictest5280-21chrt19:20:120
3268099108,1cyclictest32992sleep020:21:260
3268099108,1cyclictest21648-21fschecks_time21:00:160
3268099108,1cyclictest18644-21cut20:55:160
3268099108,1cyclictest13290-21cat19:35:190
3268099108,1cyclictest11423-21python19:30:270
3268099107,2cyclictest19796-21wc22:15:160
3268099104,4cyclictest1675-21open_files19:10:280
326939996,2cyclictest30050-21grep20:10:223
326939996,2cyclictest17321-21grep19:45:223
326939996,2cyclictest17321-21grep19:45:223
326939996,2cyclictest13106-21chrt20:41:573
326899997,1cyclictest22554-21tune2fs19:55:162
326819997,1cyclictest1433-21date20:20:011
326819997,1cyclictest13954-21expr20:45:151
326809997,1cyclictest27545-21which20:05:200
326809997,1cyclictest17247-21proc_pri20:50:220
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional