You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-24 - 01:22

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rackaslot8s.osadl.org (updated Sat Sep 23, 2023 12:45:13)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
245012203178,17sleep00-21swapper/007:07:270
245822176107,28sleep30-21swapper/307:08:303
243652162111,28sleep10-21swapper/107:05:381
243562160108,30sleep20-21swapper/207:05:322
1725921120,2sleep20-21swapper/211:44:012
826421040,1sleep20-21swapper/212:05:282
2470427657,11sleep20-21swapper/207:10:012
91842730,4chrt9183-21grep07:45:312
302702720,1chrt30271-21/usr/sbin/munin11:25:323
108412620,2sleep20-21swapper/210:37:192
188352570,6sleep228-21ksoftirqd/211:45:202
34462560,2sleep30-21swapper/310:00:313
321872560,2sleep30-21swapper/310:57:013
18872550,1sleep20-21swapper/209:30:212
4802540,1sleep10-21swapper/108:40:311
6552530,2sleep20-21swapper/208:40:332
24975995312,30cyclictest25040-21sendmail_mailqu11:50:270
242532500,2sleep00-21swapper/012:20:260
110762500,2sleep10-21swapper/111:07:561
81102490,3sleep20-21swapper/210:05:202
291442490,2sleep029146-21fschecks_count09:25:250
305062470,1sleep10-21swapper/108:35:311
24978994411,23cyclictest27237-21sh12:23:313
24975994411,19cyclictest21525-21sh10:47:120
2497599440,4cyclictest11931-21diskmemload09:24:130
24978994310,17cyclictest11602-21sh11:38:403
2497599438,23cyclictest11931-21diskmemload12:15:170
2497599438,23cyclictest11931-21diskmemload12:15:170
2497599436,23cyclictest27748-21seq10:23:110
2497599434,29cyclictest16945-21sh10:13:200
24975994310,19cyclictest11605-21sh11:38:400
2497599430,3cyclictest5470-21ssh11:32:360
2497599430,3cyclictest5470-21ssh11:32:350
2497899427,19cyclictest5607-21sh12:03:303
2497599427,22cyclictest18849-21kworker/u8:112:03:060
2497599424,27cyclictest0-21swapper/010:00:470
2497599423,27cyclictest11931-21diskmemload09:50:220
2497599421,6cyclictest14420-21ssh09:41:120
184852420,1sleep20-21swapper/210:45:062
2497899419,15cyclictest17231-21sh10:43:433
24978994111,19cyclictest12019-21sh12:09:593
2497599411,5cyclictest31524-21sh12:27:100
2497599409,18cyclictest639-21runrttasks11:21:060
2497599404,21cyclictest7653-21seq07:44:500
2497599403,21cyclictest8381-21seq09:00:250
2497599401,5cyclictest23250-21seq11:49:480
2497599401,25cyclictest0-21swapper/010:36:050
2497599400,25cyclictest0-21swapper/010:09:040
2497899393,25cyclictest27840-21ssh10:53:243
2497599395,20cyclictest22879-21sh10:19:000
2497599395,20cyclictest14072-21seq07:59:530
2497599391,6cyclictest17175-21ssh12:14:500
2497599391,5cyclictest11931-21diskmemload09:11:580
2497599390,6cyclictest9528-21ssh09:36:530
2497799380,4cyclictest0-21swapper/210:25:202
2497599381,6cyclictest21864-21seq09:49:080
2497599381,28cyclictest0-21swapper/011:05:200
24975993811,24cyclictest18409-21seq11:15:120
2497899371,2cyclictest0-21swapper/310:23:323
2497899370,23cyclictest3070-21sh10:30:243
2497599371,4cyclictest11931-21diskmemload11:41:010
24975993713,22cyclictest26546-21seq07:10:340
2497599371,2cyclictest11931-21diskmemload09:31:020
2497599370,6cyclictest32148-21ssh09:57:360
2497599370,3cyclictest31309-21ssh10:25:410
2497599370,3cyclictest23855-21seq10:50:050
2497899361,8cyclictest16882-21ssh09:14:223
2497799360,7cyclictest28889-21cpuspeed_turbos12:25:202
2497599362,20cyclictest0-21swapper/008:40:300
2497599361,5cyclictest639-21runrttasks07:33:260
24975993612,22cyclictest5656-21sed08:54:590
2497599360,8cyclictest15018-21/usr/sbin/munin08:00:290
2497599354,18cyclictest0-21swapper/012:09:060
2497599351,3cyclictest11931-21diskmemload11:11:550
2497599351,2cyclictest11931-21diskmemload10:44:020
2497599351,2cyclictest0-21swapper/011:27:140
2497599351,2cyclictest0-21swapper/011:02:520
2497599351,20cyclictest0-21swapper/012:35:430
2497599350,5cyclictest22358-21seq08:16:120
2497899340,2cyclictest0-21swapper/312:11:443
2497799341,9cyclictest1717-21latency_hist11:29:592
2497599343,26cyclictest0-21swapper/008:45:200
2497599341,5cyclictest11931-21diskmemload10:56:390
307172330,2chrt0-21swapper/209:26:042
24978993313,18cyclictest29502-21cat07:20:233
2497899330,8cyclictest8522-21cat10:05:243
2497799338,22cyclictest26990-21ssh12:23:112
2497799332,3cyclictest0-21swapper/208:35:182
24978993213,16cyclictest27415-21cat07:15:203
2497899320,6cyclictest30702-21ssh11:55:323
2497799321,8cyclictest18348-21/usr/sbin/munin12:15:232
2497799321,8cyclictest18348-21/usr/sbin/munin12:15:222
2497599327,22cyclictest27788-21cat07:15:250
2497599322,24cyclictest0-21swapper/008:20:190
2497599322,24cyclictest0-21swapper/008:20:180
24975993212,17cyclictest19208-21hddtemp_smartct08:10:230
24975993210,19cyclictest30465-21mailstats07:20:340
2497599320,6cyclictest3606-21ssh12:31:070
2497599320,3cyclictest4153-21/usr/sbin/munin07:35:240
2497599320,2cyclictest0-21swapper/007:50:410
24978993112,16cyclictest18308-21cat12:15:223
24978993112,16cyclictest18308-21cat12:15:213
2497899310,7cyclictest570-21ssh09:29:243
2497899310,7cyclictest31122-21ssh12:26:363
2497899310,7cyclictest28205-21ssh11:54:233
2497899310,4cyclictest30303-21ntp_states08:35:293
2497899310,4cyclictest20624-21latency_hist08:15:003
2497899310,3cyclictest5415-21ssh11:32:323
2497899310,3cyclictest5415-21ssh11:32:313
2497799318,5cyclictest22689-21sendmail09:20:012
24977993111,17cyclictest12778-21sed11:40:002
2497799311,10cyclictest8188-21cut11:05:242
24977993110,18cyclictest24566-21sed09:20:322
2497599311,9cyclictest5359-21ssh10:32:100
2497599311,5cyclictest639-21runrttasks08:09:080
2497599311,4cyclictest5755-21apt-get08:55:200
2497599310,4cyclictest10369-21cat09:05:210
2497599310,3cyclictest29550-21cat08:35:200
2497599310,2cyclictest940-21ssh11:59:240
24978993013,15cyclictest7167-21expr09:35:183
2497899301,25cyclictest24359-21grep08:20:323
2497899301,25cyclictest24359-21grep08:20:313
2497899300,8cyclictest5233-21ssh09:33:253
2497899300,6cyclictest22734-21ssh10:48:513
2497899300,5cyclictest15074-21ssh11:41:003
2497899300,4cyclictest18554-21ssh10:15:123
2497899300,4cyclictest14791-21ssh11:10:433
2497899300,3cyclictest24266-21ssh09:50:313
2497899300,3cyclictest17483-21chrt10:14:033
2497899300,27cyclictest7881-21acpi10:35:183
2497799307,5cyclictest18043-21cut09:45:212
2497799301,9cyclictest4727-21sed07:35:302
2497799301,8cyclictest6824-21sort12:05:002
2497799301,8cyclictest12441-21tr09:40:202
24977993012,16cyclictest30508-21ntp_states10:55:312
2497799301,11cyclictest29129-21idleruntime-cro08:35:012
24977993011,16cyclictest11931-21diskmemload09:59:252
2497599309,17cyclictest20757-21sh09:17:180
24975993012,16cyclictest25111-21apt-get08:25:200
2497599300,3cyclictest8202chrt07:28:520
2497899298,18cyclictest26413-21grep07:10:323
2497899292,21cyclictest17595-21apt-get09:45:183
24978992912,14cyclictest6107-21sh11:03:273
24978992912,14cyclictest30577-21sh09:55:423
2497899290,5cyclictest31844-21chrt10:26:243
2497899290,5cyclictest22603-21runrttasks08:19:213
2497899290,4cyclictest26974-21cron08:30:003
2497899290,3cyclictest918-21taskset08:41:243
2497899290,3cyclictest8098-21/usr/sbin/munin12:35:323
2497899290,3cyclictest17325-21sed09:45:013
2497899290,3cyclictest14884-21taskset08:00:243
2497899290,2cyclictest5729-21ssh12:34:073
2497799298,5cyclictest32575-21systemctl07:25:342
2497799297,3cyclictest8445-21head12:35:352
2497799297,3cyclictest0-21swapper/212:35:012
2497799295,7cyclictest601-21dbus-daemon09:00:322
2497799295,6cyclictest27266-21cpuspeed_turbos07:15:212
2497799292,6cyclictest12998-21apt-get11:10:182
2497799291,6cyclictest27026-21ssh11:52:412
2497799291,12cyclictest29019-21grep10:55:012
24977992911,15cyclictest11931-21diskmemload10:33:382
2497799291,10cyclictest23644-21systemctl10:20:002
2497799291,10cyclictest20539-21apt08:15:002
2497899285,20cyclictest16580-21cat08:05:183
2497899281,24cyclictest8406-21gltestperf09:00:253
24978992812,13cyclictest10126-21cpuspeed_turbos09:05:193
24978992811,14cyclictest17702-21sh09:15:113
2497899280,4cyclictest6972-21sed08:55:333
2497899280,3cyclictest8092-21awk11:05:243
2497899280,3cyclictest5675-21latency_hist08:55:003
2497899280,3cyclictest26761-21ssh11:22:193
2497899280,3cyclictest11398-21date07:50:413
2497899280,2cyclictest18947-21ssh11:15:243
2497799287,4cyclictest18878-21ssh11:15:222
2497799286,4cyclictest0-21swapper/208:55:332
2497799281,8cyclictest17363-21munin-plugin-st09:14:592
2497799281,7cyclictest3461-21wc11:30:302
2497799281,7cyclictest3461-21wc11:30:302
2497799281,7cyclictest24635-21grep10:20:232
2497799281,7cyclictest11813-21perf09:40:002
2497799281,6cyclictest17305-21munin-run12:14:592
2497599281,10cyclictest29148-21idleruntime-cro08:35:010
2497599280,1cyclictest0-21swapper/007:45:320
2497899279,15cyclictest6123-21df07:40:193
2497899277,17cyclictest3396-21cat08:50:013
2497899270,3cyclictest0-21swapper/307:35:243
2497899270,2cyclictest25871-21ssh09:22:013
2497799272,7cyclictest22667-21turbostat.cron08:19:592
2497799272,10cyclictest23773-21/usr/sbin/munin11:20:332
2497799271,9cyclictest3066-21turbostat.cron07:34:592
2497799271,8cyclictest3900-21sed11:00:322
2497799271,7cyclictest29438-21fschecks_count11:55:192
2497799271,7cyclictest15796-21grep08:00:372
24977992712,13cyclictest14194-21ntpq10:10:272
2497799271,13cyclictest2160-21acpi10:00:182
2497899261,8cyclictest27628-21fschecks_count08:30:233
24978992615,8cyclictest1518-21df_inode07:30:233
2497899260,3cyclictest7917-21/usr/sbin/munin07:45:323
2497899260,3cyclictest20194-21ssh11:45:363
2497799263,6cyclictest27008-21basename08:30:002
2497799261,8cyclictest6294-21gltestperf07:40:212
2497799261,8cyclictest23325-21ssh09:50:232
2497799261,8cyclictest13785-21fschecks_time10:40:202
2497799261,7cyclictest13352-21awk07:55:322
2497799261,7cyclictest10481-21tune2fs09:05:222
2497799261,6cyclictest30325-21pmu-power07:20:332
42642250,1sleep10-21swapper/110:30:401
2497899257,16cyclictest12593-21tune2fs07:55:253
2497799251,8cyclictest5020-21sed08:50:352
2497799251,7cyclictest23043-21cpuspeed_turbos08:20:202
2497799251,7cyclictest23043-21cpuspeed_turbos08:20:192
2497799251,7cyclictest16486-21/usr/sbin/munin08:05:152
2497799251,7cyclictest1607-21tr08:45:212
176402250,3sleep10-21swapper/108:05:291
2497899241,7cyclictest31828-21uniq07:25:233
2497799242,6cyclictest0-21swapper/207:50:242
24976992415,6cyclictest14840-21fw_packets08:00:221
24976992314,7cyclictest22-21ksoftirqd/112:40:001
24976992313,7cyclictest24650-21grep12:20:311
2497699221,5cyclictest0-21swapper/109:14:131
24976992215,5cyclictest22-21ksoftirqd/109:54:591
24976992215,5cyclictest22-21ksoftirqd/109:50:001
24976992212,7cyclictest13556-21df_inode10:10:211
24976992210,8cyclictest29333-21apt-get07:20:221
202002220,2chrt0-21swapper/109:16:311
24976992113,6cyclictest18045-21seq12:15:211
24976992113,6cyclictest18045-21seq12:15:201
24976992112,6cyclictest3648-21df07:35:191
24976992112,6cyclictest12887-21forks12:10:221
24976992111,7cyclictest8186-21df_inode10:05:211
2497699206,3cyclictest141rcu_preempt08:50:361
2497699202,13cyclictest25111-21apt-get08:25:221
24976992013,5cyclictest14606-21cat11:40:321
24976992012,6cyclictest9551-21seq09:02:061
24976992012,6cyclictest9504-21seq07:46:061
24976992012,6cyclictest28756-21awk08:30:361
24976992012,6cyclictest25361-21entropy07:10:211
24976992012,6cyclictest23373-21cut08:20:221
24976992012,6cyclictest23373-21cut08:20:211
24976992012,6cyclictest20228-21seq08:11:171
24976992012,6cyclictest19933-21smartctl10:45:301
24976992012,6cyclictest13376-21entropy11:10:211
24976992011,7cyclictest25476-21users11:20:341
24976992011,7cyclictest23610-21tune2fs09:20:221
24976992011,6cyclictest10437-21grep07:50:221
24976991914,3cyclictest3349-21idleruntime-cro08:50:011
24976991914,3cyclictest1500-21sh10:59:291
24976991912,5cyclictest24196-21entropy11:50:181
24976991912,5cyclictest23894-21seq10:20:031
24976991911,6cyclictest681-21seq07:27:071
24976991911,6cyclictest3654-21cat10:00:331
24976991911,6cyclictest31618-21seq11:56:511
24976991911,6cyclictest21080-21cat08:15:231
24976991911,6cyclictest1921-21cut09:30:211
24976991911,6cyclictest1671-21expr12:30:221
24976991911,6cyclictest10043-21awk09:05:171
2497699181,8cyclictest8260-21df_inode10:35:211
2497699181,8cyclictest29105-21cat10:55:001
2497699181,4cyclictest0-21swapper/109:25:091
24976991813,3cyclictest9148-21sh12:05:481
24976991813,3cyclictest7236-21sh09:35:201
24976991813,3cyclictest30626-21timerandwakeup11:25:341
24976991813,3cyclictest22052-21sh10:17:541
24976991813,3cyclictest20920-21sh11:46:341
24976991813,3cyclictest14117-21sh10:40:241
24976991812,4cyclictest2756-21missed_timers12:00:291
24976991812,3cyclictest3360-21sh11:30:291
24976991812,3cyclictest3360-21sh11:30:281
24976991811,5cyclictest13772-21seq07:56:011
24976991810,6cyclictest28985-21cat09:55:231
2497699180,5cyclictest0-21swapper/110:25:031
2497699179,6cyclictest1724-21sed07:30:251
24976991712,3cyclictest28025-21netstat07:15:261
24976991712,3cyclictest22361-21sh11:18:371
24976991711,4cyclictest10967-21sh11:37:491
24976991710,5cyclictest2182-21sh11:00:051
24976991710,4cyclictest11931-21diskmemload09:42:211
2497699169,5cyclictest6067-21expr08:55:211
24976991611,3cyclictest5776-21/usr/sbin/munin07:40:161
24976991610,4cyclictest28492-21sh12:25:021
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional