You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2023-09-26 - 15:00

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #a, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackaslot8s.osadl.org (updated Tue Sep 26, 2023 00:45:10)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
76262220196,16sleep00-21swapper/019:05:350
78042176105,63sleep20-21swapper/219:07:532
79152173101,65sleep10-21swapper/119:09:211
79492162103,29sleep30-21swapper/319:09:473
796928158,16sleep10-21swapper/119:10:011
18922760,1sleep20-21swapper/222:17:332
129912690,2sleep00-21swapper/021:27:060
313372650,3sleep30-21swapper/323:15:373
312682630,8sleep122-21ksoftirqd/121:45:091
316072610,2sleep331603-21ssh21:45:203
139002570,1sleep30-21swapper/321:59:123
71182560,2sleep20-21swapper/222:22:102
313632540,8sleep334-21ksoftirqd/300:20:223
264952530,2sleep10-21swapper/122:40:311
212892520,2sleep00-21swapper/023:37:410
23522490,1sleep30-21swapper/322:49:173
108002470,3chrt34-21ksoftirqd/323:27:373
292992460,2sleep20-21swapper/220:00:292
15522390,1chrt0-21swapper/122:17:041
179372350,2chrt0-21swapper/323:03:533
80062340,2chrt0-21swapper/000:30:180
8243993119,7cyclictest51750irq/141-enp3s0-22:50:173
199852310,10sleep228-21ksoftirqd/223:35:462
31822300,2sleep30-21swapper/320:15:233
824199277,6cyclictest22-21ksoftirqd/121:35:221
8241992711,4cyclictest22-21ksoftirqd/122:39:341
59332270,7sleep012-21ksoftirqd/023:23:080
49492270,2sleep10-21swapper/122:20:251
142442270,3sleep30-21swapper/320:40:273
824399261,20cyclictest19184-21apt-get23:05:203
824199268,5cyclictest22-21ksoftirqd/100:16:401
8241992619,5cyclictest22-21ksoftirqd/122:10:001
8241992613,4cyclictest22-21ksoftirqd/122:51:021
8241992518,5cyclictest22-21ksoftirqd/123:00:001
7662250,2sleep20-21swapper/200:21:502
211832250,2sleep30-21swapper/321:35:243
824199247,9cyclictest22-21ksoftirqd/122:03:351
824199247,10cyclictest22-21ksoftirqd/122:34:371
824199246,10cyclictest22-21ksoftirqd/123:45:461
27842240,3sleep0131rcuc/022:49:530
95892230,2sleep20-21swapper/222:55:252
824399232,15cyclictest25963-21ssh22:10:183
824199239,3cyclictest141rcu_preempt00:10:531
824199238,5cyclictest22-21ksoftirqd/122:10:441
824199238,4cyclictest22-21ksoftirqd/123:50:131
8241992314,7cyclictest22-21ksoftirqd/123:10:001
8241992313,4cyclictest22-21ksoftirqd/100:30:311
824199231,12cyclictest22-21ksoftirqd/123:36:391
824199231,11cyclictest22-21ksoftirqd/122:45:241
8243992216,4cyclictest20886-21cron19:40:003
8243992210,3cyclictest3592latency_hist20:10:003
8243992210,3cyclictest1337-21/usr/sbin/munin23:50:313
824299227,5cyclictest0-21swapper/220:10:012
824299221,9cyclictest129922sh21:27:062
8242992213,7cyclictest8489-21ssh23:55:522
8242992213,7cyclictest16209-21interrupts21:30:252
8242992213,6cyclictest14175-21memory20:40:262
824199229,6cyclictest141rcu_preempt23:22:041
824199229,2cyclictest141rcu_preempt21:20:121
824199225,5cyclictest22-21ksoftirqd/121:12:561
824199224,3cyclictest141rcu_preempt21:50:311
824199223,3cyclictest22-21ksoftirqd/100:26:481
824199222,13cyclictest25636-21apt-get19:50:041
824199221,8cyclictest0-21swapper/100:21:201
8241992217,3cyclictest22-21ksoftirqd/122:29:591
8241992216,4cyclictest28174-21turbostat.cron20:00:001
8241992216,4cyclictest22-21ksoftirqd/123:20:001
824199221,14cyclictest10019-21apt-get21:25:031
8241992211,2cyclictest141rcu_preempt23:12:301
8241992211,2cyclictest141rcu_preempt00:35:021
824099223,16cyclictest19865-21grep19:35:250
824099222,17cyclictest23064-21turbostat19:45:000
8240992213,6cyclictest29462-21ls23:45:270
824399212,14cyclictest23332-21taskset19:45:213
824399211,5cyclictest14564-21grep22:29:593
8243992114,4cyclictest639-21runrttasks23:32:513
8243992113,6cyclictest29946-21awk20:00:353
8243992113,4cyclictest0-21swapper/321:50:233
8242992113,6cyclictest30395-21seq23:45:482
8242992113,6cyclictest22988-21seq19:44:182
8242992112,7cyclictest5975-21python321:20:302
824299211,12cyclictest26022-21diskmemload21:50:332
824299211,10cyclictest29640-21apt-get23:15:222
824299210,9cyclictest8163-21turbostat.cron22:55:002
824199218,4cyclictest22-21ksoftirqd/121:15:191
824199214,5cyclictest22-21ksoftirqd/100:00:281
824199211,5cyclictest9620-21apt-get21:55:191
824199211,5cyclictest22-21ksoftirqd/121:30:381
8241992113,6cyclictest22-21ksoftirqd/123:30:001
824099217,12cyclictest26471-21apt21:10:200
8240992113,6cyclictest6921-21seq20:25:030
8240992113,6cyclictest16270-21seq19:25:440
8240992112,7cyclictest9892-21seq19:11:360
8240992112,7cyclictest29305-21grep20:00:290
8240992111,7cyclictest661-21gdbus22:15:340
8240992110,8cyclictest26289-21apt-get19:55:150
8240992110,7cyclictest485-21apt-get20:10:200
824399209,7cyclictest601-21dbus-daemon21:35:003
824399207,11cyclictest9966-21find22:55:293
824399203,12cyclictest25402-21apt-get22:40:213
8243992013,4cyclictest27055-21sed21:10:233
8243992012,6cyclictest285-21systemd-journal19:17:003
8243992010,7cyclictest17993-21tr19:30:313
824299201,7cyclictest14553-21munin-run22:29:592
8242992014,4cyclictest25442-21sh23:10:282
8242992014,4cyclictest25267-21sendmail_mailtr21:05:302
8242992013,4cyclictest14356-21sh21:59:492
8242992012,6cyclictest8869-21sendmail_mailqu23:25:302
8242992012,6cyclictest25971-21sed22:40:262
8242992012,6cyclictest24604-21readlink23:40:322
8242992012,6cyclictest21991-21apt21:00:182
8242992012,6cyclictest17568-21seq22:01:342
8242992012,6cyclictest15218-21cat23:00:322
8242992012,5cyclictest24273-21open_inodes19:45:322
824299201,15cyclictest12857-21sh00:35:172
824199208,4cyclictest22-21ksoftirqd/119:40:341
824199207,3cyclictest141rcu_preempt23:56:581
824199205,4cyclictest22-21ksoftirqd/123:42:551
8241992014,4cyclictest22-21ksoftirqd/120:50:001
8241992014,4cyclictest14924-21sh23:31:001
824199201,2cyclictest141rcu_preempt00:10:001
8241992012,6cyclictest30101-21seq21:43:481
8241992011,6cyclictest19482-21fw_packets19:35:221
824099209,8cyclictest7641-21apt-get23:25:170
824099202,13cyclictest13888-21ssh23:00:190
824099202,12cyclictest4314-21apt-get22:20:190
8240992015,3cyclictest12-21ksoftirqd/021:40:000
8240992014,4cyclictest5740-21sh22:51:370
8240992014,4cyclictest20262-21sh00:09:000
8240992013,5cyclictest5139-21uniq21:20:210
8240992013,5cyclictest27670-21seq23:12:490
8240992013,5cyclictest2673-21smartctl23:50:300
8240992012,6cyclictest16454-21ntpq20:45:300
8240992012,5cyclictest20615-21timerandwakeup23:05:320
8240992011,7cyclictest11579-21df20:35:210
140062200,2sleep00-21swapper/022:29:150
824399199,7cyclictest16212-21grep19:25:333
824399199,7cyclictest0-21swapper/300:35:323
824399197,9cyclictest601-21dbus-daemon19:20:293
824399193,11cyclictest21181-21apt-get19:40:193
824399192,15cyclictest23872-21sh21:05:003
824399192,14cyclictest16639-21ssh22:00:343
824399192,13cyclictest24429-21ssh23:10:173
824399192,12cyclictest6966-21apt-get20:25:203
824399191,8cyclictest2553-21cpuspeed_turbos23:20:193
8243991914,4cyclictest34-21ksoftirqd/300:00:003
8243991912,5cyclictest26329-21expr21:40:203
8243991912,5cyclictest20463-21expr22:35:223
8243991911,6cyclictest8405-21expr19:10:233
8243991911,6cyclictest6695-21cron20:25:003
8243991911,6cyclictest3997-21cat00:25:293
8243991911,6cyclictest20805-21cut20:55:303
8243991911,6cyclictest15556-21perl22:30:253
8243991910,7cyclictest16544-21apt-get00:05:173
8243991910,6cyclictest51950irq/143-enp3s0-00:15:213
8243991910,6cyclictest27693-21sed19:55:273
8243991910,6cyclictest2081-21kworker/u8:222:20:043
824299192,12cyclictest8293-21apt-get19:10:212
8242991914,3cyclictest23805-21sh21:37:252
8242991913,4cyclictest3078-21sh21:18:252
8242991913,4cyclictest29766-21sh22:13:352
8242991913,4cyclictest23962-21sh22:07:512
8242991913,4cyclictest17540-21ntp_offset00:05:272
8242991913,4cyclictest11778-21timerwakeupswit19:15:332
8242991912,5cyclictest26387-21seq21:10:132
8242991912,5cyclictest26238-21cat00:15:232
8242991912,5cyclictest24638-21seq00:13:282
8242991912,5cyclictest15461-21tune2fs22:30:242
8242991911,6cyclictest3486-21cat23:20:292
8242991911,6cyclictest31828-21grep22:45:332
8242991911,6cyclictest2975-21cat20:15:202
8242991911,6cyclictest2750-21date23:50:302
8242991911,6cyclictest2468-21wc20:15:002
8242991911,6cyclictest14734-21apt19:25:182
8242991911,6cyclictest14569-21cat19:25:002
8242991911,6cyclictest13134-21cat23:30:222
8242991911,6cyclictest12095-21cat00:00:212
824299191,12cyclictest3194-21ssh00:25:212
824199195,2cyclictest141rcu_preempt23:03:281
8241991914,3cyclictest22-21ksoftirqd/120:25:001
8241991912,5cyclictest8558-21seq20:26:231
8241991912,5cyclictest30286-21seq20:04:321
8241991911,6cyclictest32025-21cat20:05:321
8241991911,6cyclictest24115-21cat19:45:291
8241991911,6cyclictest16471-21seq19:28:221
8241991911,6cyclictest11799-21cat19:15:331
8241991911,6cyclictest10988-21cron20:35:001
824099199,7cyclictest17213-21diskstats19:30:220
824099198,7cyclictest21289-21apt-get00:10:190
824099198,2cyclictest141rcu_preempt19:20:000
824099195,5cyclictest141rcu_preempt22:40:340
824099192,12cyclictest20481-21apt-get22:05:030
8240991914,3cyclictest27465-21sh23:44:060
8240991914,3cyclictest23303-21/usr/sbin/munin19:45:300
8240991914,3cyclictest2168-21sh21:17:080
8240991914,3cyclictest12553-21if_err_enp3s000:00:250
8240991913,4cyclictest6549-21sh21:51:220
8240991913,4cyclictest26022-21diskmemload22:10:240
8240991913,4cyclictest20335-21sh21:35:000
8240991913,4cyclictest15367-21sh00:36:570
8240991912,5cyclictest26608-21which00:15:260
8240991912,5cyclictest10095-21sh21:55:240
8240991911,6cyclictest31060-21expr00:20:180
8240991911,6cyclictest21578-21seq20:58:300
8240991910,7cyclictest32314-21users20:05:350
8240991910,7cyclictest11538-21date00:00:020
266132190,3sleep012-21ksoftirqd/021:40:230
824399182,13cyclictest18849-21python320:50:283
824399182,12cyclictest9117-21apt-get20:30:223
824399182,12cyclictest3704-21sort22:20:003
824399182,11cyclictest24138-21apt-get21:05:193
824399182,11cyclictest11873-21apt-get00:00:203
8243991812,3cyclictest14022-21sh21:28:333
8243991811,5cyclictest9355-21sed00:30:323
824399181,15cyclictest4224-21ssh21:20:003
8243991811,5cyclictest22804-21grep00:10:343
8243991811,5cyclictest11661-21grep20:35:233
8243991811,4cyclictest23330-21sh23:40:183
8243991811,4cyclictest21813-21processes22:05:293
8243991810,6cyclictest2083-21seq20:11:023
824299188,5cyclictest51650irq/140-enp3s0-23:05:192
8242991813,3cyclictest28568-21sh21:41:382
8242991812,4cyclictest22361-21sh22:36:162
8242991811,5cyclictest5119-21cat20:20:202
8242991811,5cyclictest27089-21cat19:55:212
8242991811,5cyclictest21225-21grep20:55:332
8242991810,6cyclictest9351-21cat20:30:222
8242991810,6cyclictest7304-21diskstats20:25:222
8242991810,6cyclictest17388-21seq20:49:382
824199189,7cyclictest1288-21memory20:10:281
824199182,11cyclictest19832-21apt-get20:55:201
824199182,11cyclictest12596-21apt-get19:20:191
8241991814,3cyclictest22-21ksoftirqd/121:10:001
8241991813,3cyclictest17743-21cpuspeed_turbos20:50:171
824199181,15cyclictest23866-21cron21:05:011
824099189,7cyclictest15791-21sed22:30:270
8240991813,3cyclictest32445-21sh21:45:280
8240991813,3cyclictest29467-21sh23:15:060
8240991813,3cyclictest23088-21sendmail_mailst21:00:300
8240991813,3cyclictest14643-21sh23:30:370
8240991812,4cyclictest5745-21sh00:27:090
8240991812,4cyclictest21071-21sh22:35:280
8240991812,4cyclictest17364-21sh22:01:170
8240991812,4cyclictest10242-21sendmail_mailtr20:30:320
8240991811,5cyclictest19712-21grep20:55:000
8240991810,6cyclictest9498-21cat22:55:250
8240991810,6cyclictest5412-21basename20:20:240
824399172,11cyclictest4690-21apt-get21:20:183
8243991712,3cyclictest28522-21/usr/sbin/munin23:45:253
824299179,6cyclictest17060-21cat19:30:212
824299177,7cyclictest18850-21smartctl20:50:282
824299172,10cyclictest31377-21apt-get21:45:172
8242991712,3cyclictest11231-21/usr/sbin/munin20:35:232
8242991712,3cyclictest10455-21sh00:31:502
8242991711,4cyclictest51650irq/140-enp3s0-19:35:182
824199173,4cyclictest211rcuc/119:30:181
8241991712,4cyclictest22-21ksoftirqd/120:40:001
8241991710,5cyclictest3511-21strings20:15:271
8241991710,5cyclictest13621-21cat20:40:201
824099172,9cyclictest25209-21apt-get19:50:040
8240991712,3cyclictest2639-21/usr/sbin/munin20:15:260
8240991712,3cyclictest24630-21grep21:05:220
8240991712,3cyclictest14035-21if_enp3s020:40:240
824399161,11cyclictest18161-21apt-get23:35:203
824099168,6cyclictest12969-21tune2fs19:20:210
824399158,5cyclictest26090-21sed19:55:003
824399156,6cyclictest16521-21pmu-power20:45:333
8242991510,3cyclictest639-21runrttasks19:50:202
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional