You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-13 - 07:54

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #c, slot #7

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rackcslot7s.osadl.org (updated Fri Sep 13, 2024 00:44:53)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
9571522980,7sleep1251rcuc/119:25:261
95003929458,29sleep30-21swapper/319:06:423
95011528955,28sleep00-21swapper/019:07:350
94998628766,14sleep20-21swapper/219:06:002
9600572860,4sleep00-21swapper/019:35:140
10052342820,8sleep31753-21gnome-shell21:40:233
95020827861,10sleep10-21swapper/119:08:441
9815132760,6sleep10-21swapper/120:35:141
10332012710,4sleep30-21swapper/323:00:213
950453997015,17cyclictest1753-21gnome-shell20:25:013
950453997015,17cyclictest1753-21gnome-shell20:25:003
10013992670,13sleep013-21ksoftirqd/021:30:140
950453996515,31cyclictest1753-21gnome-shell21:15:153
9748432630,6sleep1251rcuc/120:15:211
95045399637,18cyclictest1753-21gnome-shell19:25:133
950453996314,17cyclictest1753-21gnome-shell22:00:013
950453996213,30cyclictest1066682-21memory00:35:233
950453996213,29cyclictest1753-21gnome-shell21:45:223
95045399617,18cyclictest1753-21gnome-shell19:55:233
162399610,9rtkit-daemon1622-21rtkit-daemon20:57:473
95045399609,34cyclictest1753-21gnome-shell00:00:033
95045399608,17cyclictest1753-21gnome-shell22:40:013
95045399608,17cyclictest1753-21gnome-shell20:45:223
95045399608,17cyclictest1753-21gnome-shell20:45:213
950453996014,28cyclictest1753-21gnome-shell20:38:363
950453996014,27cyclictest1753-21gnome-shell21:25:183
950453996013,29cyclictest1753-21gnome-shell23:34:323
950453996013,28cyclictest1753-21gnome-shell22:40:283
950453996013,18cyclictest1753-21gnome-shell23:06:223
162399600,9rtkit-daemon1622-21rtkit-daemon21:49:022
162399600,9rtkit-daemon1622-21rtkit-daemon19:50:032
95045399597,17cyclictest1753-21gnome-shell20:10:023
95045399596,21cyclictest1003576-21ntp_states21:35:263
95045399596,21cyclictest1003576-21ntp_states21:35:253
950453995915,24cyclictest1753-21gnome-shell20:40:143
950453995914,27cyclictest1753-21gnome-shell20:51:173
950453995913,28cyclictest1753-21gnome-shell20:11:593
950453995913,28cyclictest1753-21gnome-shell19:51:093
950453995912,28cyclictest1753-21gnome-shell22:19:263
162399590,9rtkit-daemon1622-21rtkit-daemon22:35:042
162399590,9rtkit-daemon1622-21rtkit-daemon22:05:423
162399590,9rtkit-daemon1622-21rtkit-daemon19:16:072
162399590,10rtkit-daemon1622-21rtkit-daemon19:10:543
9510142580,5sleep0161rcuc/019:10:150
95045399587,16cyclictest1753-21gnome-shell23:25:023
950453995814,26cyclictest1753-21gnome-shell19:18:523
950453995813,27cyclictest1753-21gnome-shell22:29:253
950453995813,27cyclictest1753-21gnome-shell21:53:223
950453995813,27cyclictest1753-21gnome-shell21:02:163
950453995810,24cyclictest1029315-21taskset22:50:173
950453995810,17cyclictest998736-40lsb_release21:21:203
162399580,8rtkit-daemon1622-21rtkit-daemon00:25:252
9918522570,4sleep2331rcuc/221:05:122
950453995717,22cyclictest1753-21gnome-shell00:28:363
950453995714,26cyclictest1753-21gnome-shell00:13:343
950453995713,27cyclictest1753-21gnome-shell23:38:313
950453995713,20cyclictest1753-21gnome-shell21:34:043
950453995713,18cyclictest1015276-21expr22:10:163
950453995712,26cyclictest1753-21gnome-shell19:45:083
950453995710,17cyclictest1753-21gnome-shell20:15:303
950440995728,24cyclictest1033097-21diskstats23:00:170
95045399568,14cyclictest1753-21gnome-shell23:50:023
95045399567,18cyclictest958630-21tr19:30:203
95045399567,18cyclictest958630-21tr19:30:193
950453995615,23cyclictest1753-21gnome-shell19:20:213
950453995614,24cyclictest1753-21gnome-shell00:18:373
950453995612,26cyclictest1753-21gnome-shell19:41:193
950453995610,28cyclictest1753-21gnome-shell20:30:073
95045399559,19cyclictest1032213-40lsb_release22:57:243
95045399558,21cyclictest970044-21latency_hist20:05:023
95045399556,24cyclictest854-21avahi-daemon23:10:283
950453995514,20cyclictest1042006-21expr23:25:153
950453995513,26cyclictest1753-21gnome-shell00:30:243
950453995512,17cyclictest1061229-21expr00:20:153
950453995510,28cyclictest854-21avahi-daemon23:50:273
95045399549,17cyclictest1753-21gnome-shell20:25:043
95045399548,28cyclictest1753-21gnome-shell21:05:253
95045399548,26cyclictest854-21avahi-daemon00:03:353
950453995417,20cyclictest1753-21gnome-shell22:24:543
950453995417,20cyclictest1753-21gnome-shell22:24:533
950453995414,23cyclictest1753-21gnome-shell21:12:053
950453995412,24cyclictest1038579-21expr23:15:123
950453995412,18cyclictest1753-21gnome-shell00:05:093
9770842530,5chrt0-21swapper/120:20:321
9770842530,5chrt0-21swapper/120:20:321
95045399539,16cyclictest1753-21gnome-shell23:40:453
950453995315,17cyclictest1753-21gnome-shell22:48:423
950453995313,16cyclictest1753-21gnome-shell22:04:003
950453995313,16cyclictest1753-21gnome-shell22:03:593
950453995311,19cyclictest960751-21swap19:35:303
950453995212,16cyclictest1022403-21basename22:30:193
950448995225,22cyclictest982101-21ntp_states20:35:252
162399510,42rtkit-daemon994-21lldpd23:06:462
950440995012,6cyclictest1358-1kworker/u9:122:50:020
162399500,43rtkit-daemon1506-21systemd19:34:352
162399500,43rtkit-daemon1506-21systemd19:34:352
9553252480,5chrt0-21swapper/219:20:232
162399480,41rtkit-daemon1506-21systemd20:23:052
162399480,41rtkit-daemon1506-21systemd20:23:052
10048032480,5chrt0-21swapper/221:40:152
162399470,40rtkit-daemon1506-21systemd00:16:262
162399470,30rtkit-daemon994-21lldpd20:17:212
950448994619,14cyclictest1-21systemd00:00:022
95044599462,40cyclictest1753-21gnome-shell22:23:421
95044599462,40cyclictest1753-21gnome-shell22:23:421
950440994629,13cyclictest1030398-21cron22:55:000
950440994613,15cyclictest9566932expr19:25:130
9943962450,15sleep1251rcuc/121:10:221
950448994536,5cyclictest311irq_work/200:30:142
950448994523,17cyclictest987437-21python320:50:292
950448994513,5cyclictest141rcu_preempt22:25:262
950445994527,14cyclictest1033082-21df23:00:151
950440994530,10cyclictest1057810-21sort00:10:200
950440994511,9cyclictest13-21ksoftirqd/020:30:120
162399450,38rtkit-daemon1753-21gnome-shell21:59:172
950448994413,6cyclictest141rcu_preempt20:10:172
950445994412,6cyclictest1041883-21grep23:25:151
10279462440,16sleep234-21ksoftirqd/222:45:222
9761422430,8sleep01753-21gnome-shell20:20:150
9761422430,8sleep01753-21gnome-shell20:20:150
950448994315,7cyclictest141rcu_preempt22:45:002
950445994331,9cyclictest26-21ksoftirqd/119:18:501
950445994328,11cyclictest981091-21munin-run20:35:001
950445994319,20cyclictest950961-21expr19:10:141
95044599431,25cyclictest1048418-21cut23:45:001
950440994331,8cyclictest1753-21gnome-shell23:46:380
950440994324,14cyclictest438-20systemd-journal21:10:010
950440994319,6cyclictest141rcu_preempt23:31:160
950440994314,6cyclictest141rcu_preempt19:20:130
950440994313,8cyclictest13-21ksoftirqd/022:05:580
950448994224,6cyclictest34-21ksoftirqd/220:05:002
950448994213,5cyclictest141rcu_preempt21:21:202
950445994215,11cyclictest967309-21awk19:55:201
950440994225,6cyclictest13-21ksoftirqd/019:18:500
950440994211,9cyclictest13-21ksoftirqd/000:31:070
950440994210,6cyclictest141rcu_preempt22:15:160
9881252410,6sleep1251rcuc/120:55:011
95044899414,16cyclictest1753-21gnome-shell22:11:462
950448994124,13cyclictest34-21ksoftirqd/223:40:132
950448994114,6cyclictest141rcu_preempt00:20:122
950448994113,7cyclictest141rcu_preempt20:50:002
950448994113,7cyclictest141rcu_preempt20:50:002
95044899411,36cyclictest1039429-21sshd23:16:412
950448994110,9cyclictest1358-1kworker/u9:120:06:142
950445994126,11cyclictest989520-21seq20:57:011
950445994126,11cyclictest986284-21cron20:50:001
950445994126,11cyclictest986284-21cron20:50:001
950445994126,11cyclictest1056423-21awk00:05:281
950445994123,13cyclictest855-21dbus-daemon21:58:521
95044099415,11cyclictest141rcu_preempt22:39:250
950440994123,14cyclictest13-21ksoftirqd/019:55:010
95044099411,9cyclictest0-21swapper/022:00:110
95044099411,9cyclictest0-21swapper/022:00:110
950440994112,6cyclictest141rcu_preempt20:54:130
950440994111,12cyclictest141rcu_preempt20:40:220
950440994110,6cyclictest141rcu_preempt21:47:200
95044899407,9cyclictest311irq_work/223:20:282
95044899406,12cyclictest34-21ksoftirqd/220:25:252
950448994022,14cyclictest34-21ksoftirqd/221:50:272
950448994020,15cyclictest1031898-21wc22:55:302
950448994012,6cyclictest141rcu_preempt22:05:142
950448994011,6cyclictest141rcu_preempt22:15:112
950445994023,12cyclictest969218-21needreboot20:00:241
950445994013,23cyclictest1024435-21/usr/sbin/munin22:35:261
95044099403,11cyclictest141rcu_preempt21:15:180
950440994024,11cyclictest438-20systemd-journal23:58:280
950440994023,6cyclictest13-21ksoftirqd/019:45:300
950440994022,13cyclictest1054669-21ntp_states00:00:260
950440994016,6cyclictest141rcu_preempt00:35:370
95044099401,13cyclictest141rcu_preempt19:32:100
95044099401,13cyclictest141rcu_preempt19:32:100
95044099401,11cyclictest141rcu_preempt20:13:110
950440994010,6cyclictest141rcu_preempt21:40:240
950448993925,10cyclictest34-21ksoftirqd/200:37:392
95044899392,32cyclictest1023070-21users22:30:312
950448993921,14cyclictest34-21ksoftirqd/219:35:122
950448993920,6cyclictest34-21ksoftirqd/221:05:002
950448993913,6cyclictest141rcu_preempt19:30:012
950448993910,7cyclictest141rcu_preempt23:25:262
950448993910,11cyclictest141rcu_preempt22:50:262
95044599397,5cyclictest141rcu_preempt00:38:191
95044599393,4cyclictest141rcu_preempt22:10:211
950445993923,12cyclictest1039512-21sshd23:17:411
950445993922,13cyclictest1053924-21apt00:00:121
950445993922,12cyclictest1013761-21interrupts22:05:191
950445993922,12cyclictest1007368-21cat21:45:291
950445993921,13cyclictest1032229-21package-system-22:57:271
95044099399,8cyclictest13-21ksoftirqd/023:19:220
95044099398,6cyclictest141rcu_preempt22:33:090
95044099393,11cyclictest141rcu_preempt22:40:280
950440993924,11cyclictest974476-21cpuspeed_turbos20:15:130
950440993922,13cyclictest855-21dbus-daemon23:10:010
950440993922,12cyclictest1009876-21wc21:55:150
95044099392,16cyclictest1753-21gnome-shell21:26:300
95044099392,12cyclictest13-21ksoftirqd/000:26:350
95044099392,11cyclictest1358-1kworker/u9:122:13:220
950440993921,13cyclictest13-21ksoftirqd/023:41:510
950440993921,13cyclictest13-21ksoftirqd/023:39:540
950440993921,13cyclictest13-21ksoftirqd/020:46:110
950440993921,13cyclictest13-21ksoftirqd/020:46:100
95044099391,11cyclictest994523-21ntp_states21:10:250
950440993911,12cyclictest141rcu_preempt20:59:130
95044099391,10cyclictest141rcu_preempt19:40:280
95044899385,8cyclictest854-21avahi-daemon19:49:172
950448993827,7cyclictest1050571-21tr23:50:142
95044899382,32cyclictest1753-21gnome-shell19:41:022
950448993822,12cyclictest1011871-21expr22:00:152
950448993822,12cyclictest1011871-21expr22:00:152
950448993820,14cyclictest34-21ksoftirqd/221:40:012
950448993820,14cyclictest34-21ksoftirqd/221:40:012
950448993819,15cyclictest34-21ksoftirqd/222:23:162
950448993819,15cyclictest34-21ksoftirqd/222:23:162
950448993817,5cyclictest34-21ksoftirqd/221:25:152
950448993815,6cyclictest141rcu_preempt21:30:142
950448993814,6cyclictest141rcu_preempt21:15:132
950448993810,6cyclictest141rcu_preempt20:30:112
95044599384,30cyclictest1001705-21missed_timers21:30:221
95044599384,11cyclictest141rcu_preempt21:22:071
950445993828,6cyclictest961337-21cron19:39:591
950445993825,9cyclictest1753-21gnome-shell00:21:541
950445993825,10cyclictest26-21ksoftirqd/122:31:481
950445993824,10cyclictest1065701-21sshd00:33:391
950445993823,11cyclictest997020-21cron21:20:001
950445993823,10cyclictest26-21ksoftirqd/120:40:161
950445993822,12cyclictest990794-21ntp_states21:00:271
950445993822,12cyclictest973055-21ntp_kernel_err20:10:241
950445993822,12cyclictest963566-21cat19:45:151
950445993821,12cyclictest1025695-21expr22:40:131
950445993812,5cyclictest141rcu_preempt23:20:051
95044099389,6cyclictest141rcu_preempt23:13:280
95044099388,6cyclictest141rcu_preempt22:26:260
95044099388,6cyclictest141rcu_preempt00:23:050
95044099387,6cyclictest141rcu_preempt20:26:290
950440993825,8cyclictest999066-21cron21:25:000
950440993822,12cyclictest972091-21munin-run20:10:000
950440993821,12cyclictest1056472-21processes00:05:290
95044099382,10cyclictest0-21swapper/023:50:480
95044099381,11cyclictest141rcu_preempt23:25:220
950440993810,7cyclictest141rcu_preempt00:16:120
950448993726,7cyclictest1049199-21cat23:45:222
95044899372,30cyclictest1753-21gnome-shell23:14:032
950448993720,6cyclictest34-21ksoftirqd/219:15:012
950448993720,5cyclictest34-21ksoftirqd/223:01:302
950448993720,13cyclictest984098-21sed20:40:312
950448993715,5cyclictest141rcu_preempt20:59:172
950448993715,14cyclictest34-21ksoftirqd/219:55:102
950448993712,6cyclictest141rcu_preempt21:10:252
95044599376,27cyclictest1017862-21vmstat22:15:291
950445993723,10cyclictest966691-21cut19:55:031
950445993722,11cyclictest970989-21cat20:05:181
950445993722,11cyclictest1045329-21expr23:35:151
950445993721,11cyclictest1050375-21cut23:50:021
950445993720,13cyclictest1003832-21cat21:35:301
950445993720,13cyclictest1003832-21cat21:35:301
950445993719,13cyclictest1027565-21df22:45:131
950445993718,14cyclictest438-20systemd-journal22:55:001
95044599371,11cyclictest1012082-21ls22:00:221
95044599371,11cyclictest1012082-21ls22:00:221
95044099375,18cyclictest0-21swapper/019:55:280
950440993725,8cyclictest1753-21gnome-shell20:36:570
950440993722,11cyclictest1031814-21python322:55:290
95044099371,11cyclictest1358-1kworker/u9:121:50:390
950440993710,6cyclictest141rcu_preempt21:00:140
95044899368,6cyclictest141rcu_preempt00:05:242
95044899361,11cyclictest1753-21gnome-shell00:10:182
95044599368,23cyclictest993239-21cron21:10:001
95044599367,10cyclictest141rcu_preempt23:32:391
950445993622,10cyclictest1063821-21sshd00:26:531
950445993622,10cyclictest1034935-21ls23:05:151
950445993621,11cyclictest958492-21expr19:30:131
950445993621,11cyclictest958492-21expr19:30:131
950445993621,11cyclictest955564-21cat19:20:311
950445993621,11cyclictest1053375-21sshd23:57:041
950445993621,11cyclictest1020507-21expr22:25:141
950445993621,11cyclictest1009179-21seq21:51:031
950445993620,11cyclictest1059566-21cat00:15:201
950445993618,14cyclictest1037014-21cut23:10:201
95044099367,11cyclictest141rcu_preempt23:20:250
95044099362,6cyclictest0-21swapper/021:35:080
95044099362,6cyclictest0-21swapper/021:35:070
95044099362,10cyclictest1358-1kworker/u9:120:02:570
162399360,29rtkit-daemon0-21swapper/223:30:042
950448993521,10cyclictest1054352-21basename00:00:202
950445993525,6cyclictest962485-21python319:40:291
950445993521,10cyclictest1057562-21expr00:10:131
950445993521,10cyclictest1052002-21date23:55:021
950440993519,5cyclictest13-21ksoftirqd/022:20:300
950440993519,5cyclictest13-21ksoftirqd/022:20:290
162399350,28rtkit-daemon0-21swapper/223:35:082
95044599344,26cyclictest1000303-21timerandwakeup21:25:301
950445993419,11cyclictest978478-21cat20:25:221
95044599335,10cyclictest141rcu_preempt21:40:351
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional