You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-11-05 - 08:11
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 792 highest latencies:
System rack5slot2.osadl.org (updated Tue Nov 05, 2024 00:44:41)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
87972196179,12sleep00-21swapper/019:05:090
109432196179,12sleep10-21swapper/119:09:041
107622195179,11sleep30-21swapper/319:06:453
106942174157,12sleep20-21swapper/219:05:522
220402470,0sleep10-21swapper/123:42:541
71362460,0sleep30-21swapper/321:41:303
50082410,0sleep10-21swapper/121:05:131
58402400,0sleep20-21swapper/200:23:552
169612120,0sleep30-21swapper/322:35:163
111969992,3cyclictest29567-21perf20:50:000
24720270,0sleep1201rcuc/123:27:571
4668260,0sleep20-21swapper/221:24:492
111969966,0cyclictest3-21ksoftirqd/023:15:130
112079953,1cyclictest17349-21tr19:20:143
112079953,1cyclictest17349-21tr19:20:143
112079952,2cyclictest0-21swapper/323:10:113
112079950,5cyclictest7027-21diskmemload23:09:533
112079950,5cyclictest0-21swapper/323:59:383
112079950,5cyclictest0-21swapper/323:25:543
112079950,5cyclictest0-21swapper/323:16:243
112079950,5cyclictest0-21swapper/321:54:293
112079950,4cyclictest0-21swapper/321:00:093
112079950,0cyclictest0-21swapper/323:53:133
112079950,0cyclictest0-21swapper/300:36:343
112019955,0cyclictest0-21swapper/223:48:122
112019950,5cyclictest0-21swapper/222:56:152
111989950,5cyclictest0-21swapper/123:38:041
111989950,5cyclictest0-21swapper/122:25:101
111969955,0cyclictest0-21swapper/023:50:120
111969955,0cyclictest0-21swapper/021:32:070
111969954,1cyclictest3-21ksoftirqd/022:55:120
111969953,1cyclictest3-21ksoftirqd/021:55:220
111969952,3cyclictest30213-21ssh21:36:580
111969952,3cyclictest3-21ksoftirqd/023:25:120
111969950,5cyclictest0-21swapper/023:02:020
111969950,5cyclictest0-21swapper/022:09:220
111969950,5cyclictest0-21swapper/020:55:130
111969950,5cyclictest0-21swapper/000:05:280
111969950,4cyclictest29596-21apt-get19:45:020
111969950,4cyclictest0-21swapper/022:40:100
111969950,1cyclictest13614-21apt-get22:50:170
111969950,0cyclictest0-21swapper/023:20:310
111969950,0cyclictest0-21swapper/022:31:040
112079944,0cyclictest7027-21diskmemload22:48:543
112079944,0cyclictest0-21swapper/323:30:043
112079944,0cyclictest0-21swapper/322:20:143
112079944,0cyclictest0-21swapper/322:05:413
112079941,3cyclictest7027-21diskmemload22:15:203
112079941,2cyclictest0-21swapper/323:00:123
112079941,2cyclictest0-21swapper/320:55:103
112079941,2cyclictest0-21swapper/320:15:133
112079941,2cyclictest0-21swapper/319:25:113
112079941,1cyclictest0-21swapper/321:20:123
112079941,1cyclictest0-21swapper/319:40:113
112079941,1cyclictest0-21swapper/300:25:113
112079940,4cyclictest7027-21diskmemload00:20:113
112079940,4cyclictest30997-21snmpd19:15:113
112079940,4cyclictest292502sleep320:46:263
112079940,4cyclictest14123-21ssh00:11:173
112079940,4cyclictest0-21swapper/323:40:093
112079940,4cyclictest0-21swapper/322:53:523
112079940,4cyclictest0-21swapper/322:41:353
112079940,4cyclictest0-21swapper/322:31:343
112079940,4cyclictest0-21swapper/322:12:543
112079940,4cyclictest0-21swapper/322:00:123
112079940,4cyclictest0-21swapper/321:55:153
112079940,4cyclictest0-21swapper/321:35:243
112079940,4cyclictest0-21swapper/320:50:103
112079940,4cyclictest0-21swapper/300:15:063
112079940,4cyclictest0-21swapper/300:05:113
112079940,3cyclictest8139-21ssh23:20:013
112079940,3cyclictest7663-21users21:25:223
112079940,3cyclictest7408-21ssh21:10:073
112079940,3cyclictest5344-21df_inode20:00:143
112079940,3cyclictest28462-21ssh22:25:103
112079940,3cyclictest24779-21dpkg20:40:003
112079940,3cyclictest24025-21diskstats22:55:133
112079940,3cyclictest15053-21latency_hist21:30:013
112079940,3cyclictest0-21swapper/321:15:023
112079940,3cyclictest0-21swapper/319:10:123
112079940,3cyclictest0-21swapper/300:30:133
112079940,1cyclictest17207-21/usr/sbin/munin20:25:163
112019944,0cyclictest0-21swapper/222:30:392
112019944,0cyclictest0-21swapper/222:08:142
112019944,0cyclictest0-21swapper/222:03:002
112019944,0cyclictest0-21swapper/221:57:462
112019944,0cyclictest0-21swapper/221:37:412
112019944,0cyclictest0-21swapper/200:06:422
112019942,1cyclictest4819-21/usr/sbin/munin20:00:122
112019942,1cyclictest2314-21/usr/sbin/munin21:00:112
112019941,3cyclictest1315-21grep23:00:122
112019940,4cyclictest8426-21/usr/sbin/munin23:20:152
112019940,4cyclictest21074-21/usr/sbin/munin19:25:202
112019940,4cyclictest0-21swapper/223:57:012
112019940,4cyclictest0-21swapper/223:50:292
112019940,4cyclictest0-21swapper/223:25:012
112019940,4cyclictest0-21swapper/223:15:132
112019940,4cyclictest0-21swapper/223:10:112
112019940,4cyclictest0-21swapper/223:05:182
112019940,4cyclictest0-21swapper/222:48:092
112019940,4cyclictest0-21swapper/222:36:182
112019940,4cyclictest0-21swapper/222:20:112
112019940,4cyclictest0-21swapper/221:45:122
112019940,4cyclictest0-21swapper/221:15:122
112019940,4cyclictest0-21swapper/221:13:062
112019940,4cyclictest0-21swapper/219:36:022
112019940,4cyclictest0-21swapper/219:17:422
112019940,4cyclictest0-21swapper/200:15:122
112019940,4cyclictest0-21swapper/200:13:572
112019940,3cyclictest399-21rm22:26:482
112019940,3cyclictest19678-21apt-get20:30:102
112019940,3cyclictest18496-21expr00:30:162
112019940,3cyclictest14500-21munin-run20:20:002
112019940,3cyclictest13546-21latency_hist19:15:012
112019940,2cyclictest8119-21ssh21:25:392
112019940,1cyclictest8064-21/usr/sbin/munin00:25:132
112019940,1cyclictest32080-21/usr/sbin/munin19:50:162
112019940,1cyclictest24836-21/usr/sbin/munin20:40:212
112019940,1cyclictest24312-21irqstats00:00:172
112019940,1cyclictest2070-21munin-run21:00:002
112019940,1cyclictest15536-21/usr/sbin/munin21:30:112
112019940,1cyclictest13605-21/usr/sbin/munin22:50:232
111989944,0cyclictest0-21swapper/123:52:281
111989944,0cyclictest0-21swapper/122:33:451
111989944,0cyclictest0-21swapper/121:55:211
111989944,0cyclictest0-21swapper/121:43:251
111989944,0cyclictest0-21swapper/121:30:021
111989944,0cyclictest0-21swapper/121:13:441
111989941,2cyclictest0-21swapper/123:30:101
111989941,2cyclictest0-21swapper/123:00:101
111989941,2cyclictest0-21swapper/122:40:101
111989941,2cyclictest0-21swapper/122:00:121
111989941,2cyclictest0-21swapper/120:55:101
111989941,2cyclictest0-21swapper/119:55:111
111989941,2cyclictest0-21swapper/100:25:111
111989941,2cyclictest0-21swapper/100:20:111
111989940,4cyclictest0-21swapper/121:22:371
111989940,0cyclictest0-21swapper/123:23:311
111989940,0cyclictest0-21swapper/122:40:011
111989940,0cyclictest0-21swapper/122:13:321
111989940,0cyclictest0-21swapper/100:36:071
111969944,0cyclictest3-21ksoftirqd/023:05:110
111969944,0cyclictest3-21ksoftirqd/022:35:110
111969944,0cyclictest3-21ksoftirqd/021:00:090
111969944,0cyclictest3-21ksoftirqd/019:20:130
111969944,0cyclictest3-21ksoftirqd/019:20:130
111969944,0cyclictest0-21swapper/021:40:160
111969944,0cyclictest0-21swapper/000:12:450
111969942,1cyclictest16976-21latency_hist20:25:000
111969941,3cyclictest30997-21snmpd00:36:160
111969941,3cyclictest25720-21kworker/0:222:00:150
111969941,3cyclictest0-21swapper/000:20:120
111969940,4cyclictest13263-21apt-get21:45:120
111969940,4cyclictest0-21swapper/023:45:100
111969940,4cyclictest0-21swapper/023:40:120
111969940,4cyclictest0-21swapper/023:35:130
111969940,4cyclictest0-21swapper/023:30:100
111969940,4cyclictest0-21swapper/022:20:110
111969940,4cyclictest0-21swapper/022:11:240
111969940,4cyclictest0-21swapper/021:20:120
111969940,4cyclictest0-21swapper/021:15:120
111969940,4cyclictest0-21swapper/021:10:130
111969940,4cyclictest0-21swapper/020:25:110
111969940,4cyclictest0-21swapper/000:15:010
111969940,3cyclictest7818-21df_inode20:05:120
111969940,3cyclictest5780-21ssh22:45:290
111969940,3cyclictest5325-21/usr/sbin/munin21:05:140
111969940,3cyclictest532-21cat21:55:010
111969940,3cyclictest4778-21sed20:00:000
111969940,3cyclictest28884-21cstates22:25:110
111969940,3cyclictest23222-21ntpq19:30:180
111969940,3cyclictest21853-21wc19:30:010
111969940,3cyclictest14546-21/usr/sbin/munin19:15:170
111969940,3cyclictest12715-21expr20:15:140
111969940,3cyclictest0-21swapper/020:30:100
111969940,1cyclictest7054-21latency_hist20:05:000
111969940,1cyclictest0-21swapper/022:16:470
111969940,0cyclictest0-21swapper/023:59:490
17349230,0sleep1201rcuc/121:15:021
112079931,1cyclictest0-21swapper/320:20:113
112079930,3cyclictest7027-21diskmemload23:45:013
112079930,3cyclictest7027-21diskmemload23:35:003
112079930,3cyclictest23047-21ssh00:00:063
112079930,3cyclictest22921-21cat19:30:163
112079930,3cyclictest0-21swapper/321:45:053
112079930,3cyclictest0-21swapper/321:05:123
112079930,3cyclictest0-21swapper/320:35:123
112079930,3cyclictest0-21swapper/320:30:103
112079930,3cyclictest0-21swapper/320:10:103
112079930,3cyclictest0-21swapper/320:05:103
112079930,3cyclictest0-21swapper/319:55:003
112079930,3cyclictest0-21swapper/319:50:133
112079930,3cyclictest0-21swapper/319:45:093
112079930,3cyclictest0-21swapper/319:35:163
112019932,1cyclictest25769-21ssh22:40:042
112019931,1cyclictest0-21swapper/223:30:102
112019930,3cyclictest7522-21grep20:05:112
112019930,3cyclictest6038-21ssh23:35:142
112019930,3cyclictest29780-21apt-get20:50:102
112019930,3cyclictest27835-21df_inode20:45:132
112019930,3cyclictest1899-21apt-get19:55:012
112019930,3cyclictest17877-21perl20:25:132
112019930,3cyclictest16138-21cpuspeed23:40:132
112019930,3cyclictest15258-21df20:20:132
112019930,3cyclictest10287-21df_inode20:10:122
112019930,3cyclictest0-21swapper/221:50:122
112019930,3cyclictest0-21swapper/221:40:122
112019930,3cyclictest0-21swapper/221:05:122
112019930,3cyclictest0-21swapper/220:35:052
112019930,3cyclictest0-21swapper/219:45:012
112019930,3cyclictest0-21swapper/219:30:122
112019930,3cyclictest0-21swapper/219:20:142
112019930,3cyclictest0-21swapper/219:20:132
112019930,3cyclictest0-21swapper/200:35:122
112019930,2cyclictest0-21swapper/222:10:112
112019930,1cyclictest8213-21ssh22:15:012
112019930,0cyclictest0-21swapper/219:40:202
111989933,0cyclictest0-21swapper/123:17:521
111989933,0cyclictest0-21swapper/123:10:341
111989933,0cyclictest0-21swapper/121:50:061
111989933,0cyclictest0-21swapper/120:00:061
111989933,0cyclictest0-21swapper/100:16:461
111989931,1cyclictest0-21swapper/123:55:101
111989931,1cyclictest0-21swapper/123:45:111
111989931,1cyclictest0-21swapper/123:05:111
111989931,1cyclictest0-21swapper/122:55:111
111989931,1cyclictest0-21swapper/122:20:121
111989931,1cyclictest0-21swapper/122:15:101
111989931,1cyclictest0-21swapper/121:45:121
111989931,1cyclictest0-21swapper/120:50:101
111989931,1cyclictest0-21swapper/120:20:111
111989931,1cyclictest0-21swapper/120:15:121
111989931,1cyclictest0-21swapper/119:25:111
111989931,1cyclictest0-21swapper/119:10:121
111989931,1cyclictest0-21swapper/100:30:131
111989931,1cyclictest0-21swapper/100:10:101
111989930,2cyclictest29668-21chrt21:36:361
111989930,2cyclictest16714-21chrt19:19:161
111989930,2cyclictest0-21swapper/120:35:221
111989930,2cyclictest0-21swapper/120:25:111
111989930,2cyclictest0-21swapper/119:50:161
111989930,0cyclictest0-21swapper/119:37:061
111969933,0cyclictest3-21ksoftirqd/020:40:120
111969932,1cyclictest30997-21snmpd20:50:080
111969931,1cyclictest0-21swapper/021:25:130
111969931,1cyclictest0-21swapper/020:35:120
111969930,3cyclictest23119-21rm00:00:080
111969930,3cyclictest11309-21dpkg19:10:130
111969930,3cyclictest0-21swapper/023:10:000
111969930,3cyclictest0-21swapper/020:10:110
111969930,3cyclictest0-21swapper/019:50:130
111969930,3cyclictest0-21swapper/019:40:120
111969930,3cyclictest0-21swapper/019:35:120
111969930,3cyclictest0-21swapper/000:30:130
111969930,3cyclictest0-21swapper/000:25:110
111989921,1cyclictest0-21swapper/121:00:101
111989921,1cyclictest0-21swapper/120:05:101
111989921,1cyclictest0-21swapper/119:45:021
111989921,1cyclictest0-21swapper/100:00:121
111989921,0cyclictest0-21swapper/122:45:111
111989921,0cyclictest0-21swapper/100:05:111
111989920,1cyclictest19668-21chrt20:30:051
111989920,1cyclictest0-21swapper/122:50:181
111989920,1cyclictest0-21swapper/122:05:111
111989920,1cyclictest0-21swapper/121:25:131
111989920,1cyclictest0-21swapper/120:40:011
111989920,1cyclictest0-21swapper/120:10:101
111989920,1cyclictest0-21swapper/119:40:111
111989920,1cyclictest0-21swapper/119:30:131
111989920,1cyclictest0-21swapper/119:20:131
111989920,1cyclictest0-21swapper/119:20:131
111989910,1cyclictest0-21swapper/120:45:001
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional