You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-06-18 - 04:36

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #0, slot #8

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the 376 highest latencies:
System rack0slot8s.osadl.org (updated Tue Jun 18, 2024 00:44:51)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
2889911050,95ptp4l0-21swapper/119:09:481
188922890,0sleep00-21swapper/000:00:250
57427362,9sleep110-21swapper/1119:07:413
2989926252,8sleep50-21swapper/519:05:117
59026050,8sleep00-21swapper/019:07:540
2989126050,8sleep100-21swapper/1019:05:042
280932560,0sleep70-21swapper/722:50:129
55542550,0sleep10-21swapper/122:20:151
55542550,0sleep10-21swapper/122:20:151
2990225542,10sleep70-21swapper/719:05:139
58325444,8sleep60-21swapper/619:07:488
4102540,0sleep20-21swapper/220:05:254
41325341,9sleep40-21swapper/419:05:346
288991220,1ptp4l21-21ksoftirqd/121:40:111
118899211,7cyclictest0-21swapper/423:50:016
118899211,16cyclictest0-21swapper/400:35:006
118899211,15cyclictest0-21swapper/422:25:006
118899211,15cyclictest0-21swapper/422:25:006
288991200,0ptp4l0-21swapper/123:55:201
118899202,14cyclictest0-21swapper/400:10:026
117899204,10cyclictest0-21swapper/219:25:004
117899194,10cyclictest0-21swapper/220:25:004
288991180,1ptp4l21-21ksoftirqd/122:33:061
288991180,1ptp4l21-21ksoftirqd/122:33:061
288991180,0ptp4l0-21swapper/123:20:191
288991180,0ptp4l0-21swapper/120:35:301
121599181,2cyclictest0-21swapper/1000:25:182
118899180,3cyclictest0-21swapper/422:55:206
117899183,9cyclictest0-21swapper/223:15:004
117899183,9cyclictest0-21swapper/223:10:004
117899183,9cyclictest0-21swapper/221:10:344
117899183,9cyclictest0-21swapper/200:30:014
117899183,10cyclictest0-21swapper/221:50:004
6682178,7sleep80-21swapper/819:09:0210
288991170,0ptp4l0-21swapper/121:45:221
122099171,2cyclictest0-21swapper/1123:55:193
121199173,9cyclictest0-21swapper/922:45:0011
120699172,5cyclictest0-21swapper/820:50:0010
118499173,9cyclictest0-21swapper/323:30:015
1184991713,2cyclictest0-21swapper/300:25:185
117899173,9cyclictest0-21swapper/222:10:004
117899173,9cyclictest0-21swapper/219:50:004
117899173,8cyclictest0-21swapper/222:55:354
117899172,10cyclictest0-21swapper/221:20:334
117499175,7cyclictest0-21swapper/123:35:011
117499172,9cyclictest0-21swapper/122:00:011
288991160,1ptp4l21-21ksoftirqd/122:35:151
288991160,0ptp4l0-21swapper/123:15:151
288991160,0ptp4l0-21swapper/123:05:161
288991160,0ptp4l0-21swapper/120:30:281
1220991612,3cyclictest0-21swapper/1119:30:203
121599161,2cyclictest0-21swapper/1022:05:182
119399163,7cyclictest0-21swapper/523:45:017
118499163,8cyclictest0-21swapper/319:55:235
117899163,9cyclictest0-21swapper/200:25:004
117899160,10cyclictest0-21swapper/220:35:014
117499162,8cyclictest0-21swapper/122:00:001
116999163,8cyclictest0-21swapper/020:10:000
116999163,7cyclictest0-21swapper/023:20:010
46242150,0sleep90-21swapper/920:55:1411
288991150,6ptp4l21-21ksoftirqd/122:45:141
288991150,5ptp4l21-21ksoftirqd/121:20:141
288991150,1ptp4l21-21ksoftirqd/120:15:181
288991150,1ptp4l21-21ksoftirqd/100:20:291
288991150,0ptp4l0-21swapper/122:15:331
288991150,0ptp4l0-21swapper/122:15:331
288991150,0ptp4l0-21swapper/121:25:121
121199153,7cyclictest0-21swapper/923:45:3511
121199153,7cyclictest0-21swapper/921:25:0011
121199152,7cyclictest0-21swapper/923:20:0111
121199152,7cyclictest0-21swapper/919:40:0011
120699154,7cyclictest0-21swapper/820:05:0010
120299153,7cyclictest0-21swapper/721:35:009
120299152,8cyclictest0-21swapper/722:35:019
120299152,8cyclictest0-21swapper/721:20:009
120299150,3cyclictest0-21swapper/721:20:199
119799153,8cyclictest0-21swapper/622:50:008
119799152,8cyclictest0-21swapper/621:20:008
119399150,1cyclictest0-21swapper/520:13:457
119399150,14cyclictest16145-21fschecks_time19:30:167
119399150,14cyclictest12388-21cut21:45:257
119399150,14cyclictest10862-21processes22:25:267
119399150,14cyclictest10862-21processes22:25:267
119399150,14cyclictest0-21swapper/521:38:267
119399150,10cyclictest31777-21fschecks_count21:30:167
118499153,7cyclictest0-21swapper/320:15:345
118499152,8cyclictest0-21swapper/323:40:005
118499151,8cyclictest0-21swapper/300:05:015
117899152,9cyclictest0-21swapper/221:55:344
117899150,15cyclictest0-21swapper/222:15:364
117899150,15cyclictest0-21swapper/222:15:364
117899150,14cyclictest15071-21irqrtprio20:25:244
117499153,7cyclictest0-21swapper/123:15:001
116999153,7cyclictest0-21swapper/020:15:000
116999152,8cyclictest0-21swapper/021:55:000
116999152,7cyclictest0-21swapper/020:55:000
288991140,1ptp4l21-21ksoftirqd/123:25:151
288991140,1ptp4l21-21ksoftirqd/119:57:181
288991140,1ptp4l21-21ksoftirqd/119:42:221
288991140,1ptp4l21-21ksoftirqd/100:30:011
288991140,0ptp4l0-21swapper/123:40:311
288991140,0ptp4l0-21swapper/123:00:221
1211991412,1cyclictest0-21swapper/900:00:0111
120699143,7cyclictest0-21swapper/822:45:0010
120699143,6cyclictest0-21swapper/820:10:0010
120299143,6cyclictest0-21swapper/720:45:019
120299142,8cyclictest0-21swapper/722:30:009
120299142,8cyclictest0-21swapper/722:30:009
120299142,7cyclictest0-21swapper/719:45:009
120299141,8cyclictest0-21swapper/719:50:009
119399140,9cyclictest30607-21cpuspeed_turbos20:05:107
119399140,8cyclictest535-21latency_hist22:15:007
119399140,8cyclictest535-21latency_hist22:15:007
119399140,7cyclictest0-21swapper/521:10:127
119399140,6cyclictest32566-21cpuspeed_turbos00:20:117
119399140,4cyclictest6639-21fschecks_time20:15:187
119399140,4cyclictest2972-21grep22:15:367
119399140,4cyclictest2972-21grep22:15:357
119399140,4cyclictest20640-21irqrtprio19:35:207
119399140,3cyclictest1516-21if_enp1s0.120:50:187
119399140,3cyclictest10338-21perl23:50:197
119399140,2cyclictest30340-21cut20:45:227
119399140,1cyclictest0-21swapper/523:10:157
119399140,1cyclictest0-21swapper/520:55:237
119399140,13cyclictest5612-21netstat19:15:017
119399140,13cyclictest4575-21grep00:25:157
119399140,13cyclictest27792-21cat19:55:237
119399140,13cyclictest17844-21cut00:00:187
119399140,13cyclictest16875-21cat21:50:337
119399140,13cyclictest16874-21cat22:35:147
119399140,13cyclictest13582-21grep22:30:167
119399140,13cyclictest13582-21grep22:30:167
119399140,13cyclictest0-21swapper/523:35:297
119399140,13cyclictest0-21swapper/523:33:517
119399140,13cyclictest0-21swapper/522:50:147
119399140,13cyclictest0-21swapper/522:23:317
119399140,13cyclictest0-21swapper/522:23:317
118499142,7cyclictest0-21swapper/322:25:005
118499142,7cyclictest0-21swapper/322:25:005
118499142,7cyclictest0-21swapper/320:50:285
118499142,7cyclictest0-21swapper/320:35:005
118499141,7cyclictest0-21swapper/319:55:005
117499142,7cyclictest0-21swapper/100:15:011
116999143,7cyclictest0-21swapper/021:05:010
116999142,7cyclictest0-21swapper/000:15:340
288991130,1ptp4l21-21ksoftirqd/122:05:231
288991130,1ptp4l21-21ksoftirqd/120:51:311
288991130,1ptp4l21-21ksoftirqd/120:23:041
288991130,1ptp4l21-21ksoftirqd/100:39:011
288991130,0ptp4l0-21swapper/123:35:251
288991130,0ptp4l0-21swapper/120:41:131
122099133,6cyclictest0-21swapper/1123:05:013
122099132,7cyclictest0-21swapper/1100:20:013
122099132,6cyclictest0-21swapper/1120:55:003
121199132,7cyclictest0-21swapper/920:35:3611
120699133,7cyclictest0-21swapper/820:35:0010
120699133,6cyclictest0-21swapper/800:40:0110
120699132,7cyclictest0-21swapper/822:15:0010
120699132,7cyclictest0-21swapper/822:15:0010
120699130,12cyclictest11925-21awk22:30:0110
120699130,12cyclictest11925-21awk22:30:0010
119799132,7cyclictest0-21swapper/619:45:018
119799130,8cyclictest0-21swapper/600:10:018
119399130,3cyclictest23015-21cstates23:25:157
119399130,1cyclictest0-21swapper/523:55:067
119399130,1cyclictest0-21swapper/523:45:177
119399130,1cyclictest0-21swapper/523:20:077
119399130,1cyclictest0-21swapper/523:15:047
119399130,1cyclictest0-21swapper/523:05:037
119399130,1cyclictest0-21swapper/523:00:597
119399130,1cyclictest0-21swapper/522:45:237
119399130,1cyclictest0-21swapper/522:40:247
119399130,1cyclictest0-21swapper/522:06:357
119399130,1cyclictest0-21swapper/522:00:057
119399130,1cyclictest0-21swapper/521:55:127
119399130,1cyclictest0-21swapper/521:40:237
119399130,1cyclictest0-21swapper/521:25:427
119399130,1cyclictest0-21swapper/521:20:127
119399130,1cyclictest0-21swapper/521:15:217
119399130,1cyclictest0-21swapper/520:21:297
119399130,1cyclictest0-21swapper/520:00:087
119399130,1cyclictest0-21swapper/519:51:377
119399130,1cyclictest0-21swapper/519:45:197
119399130,1cyclictest0-21swapper/519:40:537
119399130,1cyclictest0-21swapper/519:25:157
119399130,1cyclictest0-21swapper/519:21:377
119399130,1cyclictest0-21swapper/519:13:007
119399130,1cyclictest0-21swapper/500:35:057
119399130,1cyclictest0-21swapper/500:10:077
119399130,1cyclictest0-21swapper/500:05:057
119399130,12cyclictest621-21irqrtprio22:55:227
119399130,12cyclictest15714-21smartctl20:25:317
119399130,12cyclictest10522-21grep21:00:307
119399130,12cyclictest0-21swapper/521:05:157
119399130,12cyclictest0-21swapper/520:40:217
119399130,12cyclictest0-21swapper/520:35:137
119399130,12cyclictest0-21swapper/520:30:107
119399130,12cyclictest0-21swapper/500:30:147
119399130,12cyclictest0-21swapper/500:15:167
118899130,13cyclictest0-21swapper/422:39:286
117899130,9cyclictest0-21swapper/221:15:354
117499131,7cyclictest0-21swapper/119:20:011
116999130,1cyclictest0-21swapper/021:06:290
288991120,5ptp4l21-21ksoftirqd/100:05:251
288991120,2ptp4l31289-21users00:15:351
288991120,2ptp4l15110-21irqstats20:25:241
288991120,1ptp4l21-21ksoftirqd/122:40:491
288991120,1ptp4l21-21ksoftirqd/119:38:361
288991120,1ptp4l0-21swapper/122:15:001
288991120,1ptp4l0-21swapper/122:15:001
288991120,1ptp4l0-21swapper/121:10:141
288991120,1ptp4l0-21swapper/121:00:141
288991120,0ptp4l0-21swapper/122:25:271
288991120,0ptp4l0-21swapper/122:25:271
288991120,0ptp4l0-21swapper/121:05:121
288991120,0ptp4l0-21swapper/119:30:251
288991120,0ptp4l0-21swapper/100:00:231
122099122,8cyclictest0-21swapper/1122:55:013
122099121,7cyclictest0-21swapper/1120:25:343
122099120,11cyclictest7502-21latency20:15:253
121599120,11cyclictest0-21swapper/1021:55:002
120699128,3cyclictest7799-21phc2sys20:15:2710
120699120,11cyclictest13811-21df23:55:1210
120299122,7cyclictest0-21swapper/721:55:009
120299122,6cyclictest0-21swapper/722:30:329
120299122,6cyclictest0-21swapper/722:30:329
120299122,6cyclictest0-21swapper/720:05:259
120299121,7cyclictest0-21swapper/720:30:009
120299121,6cyclictest0-21swapper/719:35:299
119799129,2cyclictest20374-21cat23:20:218
119799129,2cyclictest15027-21fschecks_time21:50:168
119799122,6cyclictest0-21swapper/622:40:028
1197991211,1cyclictest0-21swapper/620:27:298
118499122,6cyclictest0-21swapper/322:35:005
118499122,6cyclictest0-21swapper/322:35:005
117899120,11cyclictest0-21swapper/223:50:344
117499128,3cyclictest28723-21/usr/sbin/munin20:45:231
117499123,6cyclictest0-21swapper/121:15:001
116999120,1cyclictest0-21swapper/019:47:100
100842120,1sleep111033-21runrttasks20:20:123
4472116,4sleep90-21swapper/919:06:0311
288991110,6ptp4l3208-21sh21:35:141
288991110,4ptp4l21-21ksoftirqd/120:55:261
288991110,2ptp4l2126-21users22:55:371
288991110,2ptp4l12010-21irqrtprio19:20:231
288991110,1ptp4l21-21ksoftirqd/122:50:321
288991110,1ptp4l0-21swapper/121:30:331
288991110,0ptp4l0-21swapper/121:50:201
288991110,0ptp4l0-21swapper/120:10:301
288991110,0ptp4l0-21swapper/100:30:041
217622110,0sleep00-21swapper/020:35:150
122099118,2cyclictest1033-21runrttasks22:00:293
122099117,3cyclictest0-21swapper/1122:40:243
122099112,7cyclictest0-21swapper/1120:50:003
122099112,6cyclictest0-21swapper/1123:55:003
1220991110,1cyclictest0-21swapper/1121:50:203
121599112,6cyclictest0-21swapper/1019:15:002
121599112,3cyclictest131rcu_preempt20:45:002
121599111,6cyclictest0-21swapper/1021:40:282
121599110,4cyclictest0-21swapper/1000:25:002
121599110,2cyclictest0-21swapper/1021:20:182
121199112,5cyclictest0-21swapper/900:25:0111
121199111,6cyclictest0-21swapper/919:15:2611
121199110,5cyclictest0-21swapper/921:14:5511
121199110,1cyclictest0-21swapper/923:22:0311
120699112,5cyclictest0-21swapper/822:55:0010
120699112,5cyclictest0-21swapper/822:40:0010
120699112,5cyclictest0-21swapper/820:00:0010
120699110,1cyclictest0-21swapper/822:22:5710
120699110,1cyclictest0-21swapper/822:22:5710
120699110,11cyclictest0-21swapper/823:28:5110
120699110,11cyclictest0-21swapper/820:55:2410
120299110,1cyclictest0-21swapper/719:55:279
120299110,11cyclictest0-21swapper/700:35:409
120299110,11cyclictest0-21swapper/700:26:209
119799112,5cyclictest0-21swapper/621:35:008
119799111,10cyclictest0-21swapper/600:15:008
1197991110,1cyclictest0-21swapper/600:00:038
118899110,10cyclictest0-21swapper/423:10:196
118499114,6cyclictest1033-21runrttasks00:19:055
118499111,5cyclictest0-21swapper/321:05:325
118499110,1cyclictest0-21swapper/319:27:185
117899114,1cyclictest0-21swapper/221:43:314
117899110,1cyclictest0-21swapper/222:25:374
117899110,1cyclictest0-21swapper/222:25:374
117499110,10cyclictest1072-21basename20:10:001
116999119,1cyclictest0-21swapper/019:55:110
116999116,4cyclictest7788-21/usr/sbin/munin21:40:180
116999110,10cyclictest0-21swapper/000:40:010
94480100,1rtkit-daemon5192-21munin-node00:05:2611
288991100,5ptp4l21-21ksoftirqd/119:27:151
288991100,1ptp4l29893-21cat20:05:001
288991100,0ptp4l0-21swapper/123:50:141
288991100,0ptp4l0-21swapper/119:52:441
288991100,0ptp4l0-21swapper/119:10:191
122099100,1cyclictest0-21swapper/1119:47:193
121599107,2cyclictest1033-21runrttasks21:26:162
121599106,4cyclictest0-21swapper/1022:00:002
121599100,4cyclictest690-21systemd-journal19:57:542
121599100,1cyclictest0-21swapper/1019:39:532
121199109,1cyclictest0-21swapper/923:11:4011
121199107,2cyclictest15196-21fw_conntrack21:50:1711
121199101,6cyclictest0-21swapper/920:10:0011
121199100,9cyclictest31843-21timerandwakeup22:10:3411
121199100,9cyclictest31843-21timerandwakeup22:10:3411
121199100,7cyclictest1033-21runrttasks23:33:2211
121199100,6cyclictest0-21swapper/919:14:3711
121199100,3cyclictest4264-21cut23:00:1811
120699108,1cyclictest297892chrt00:15:2310
120699106,3cyclictest1696-21sshd21:32:3910
120699103,4cyclictest0-21swapper/820:40:0010
120699100,6cyclictest0-21swapper/823:45:4910
120699100,5cyclictest0-21swapper/822:07:4610
120699100,3cyclictest17599-21cat21:55:0010
120699100,10cyclictest0-21swapper/822:45:2410
120299109,1cyclictest0-21swapper/719:20:269
120299106,3cyclictest9734-21/usr/sbin/munin00:30:259
120299106,3cyclictest4569-21cut23:00:229
120299106,3cyclictest19951-21runrttasks00:02:019
120299106,3cyclictest1690-21/usr/sbin/munin00:20:229
120299105,4cyclictest3128-21irqrtprio23:40:249
120299103,2cyclictest6646-21cat21:40:119
120299100,9cyclictest905-21sh20:50:149
120299100,9cyclictest4567-21grep20:10:369
120299100,9cyclictest31312-21if_err_enp4s023:35:199
120299100,10cyclictest0-21swapper/722:04:559
119799109,1cyclictest0-21swapper/620:03:388
119799107,2cyclictest23085-21df_inode23:25:168
119799106,3cyclictest2747-21fschecks_time20:10:188
119799100,9cyclictest9758-21uname21:00:208
119799100,9cyclictest4487-21tr20:55:138
119799100,9cyclictest11051-21irqrtprio20:20:208
119799100,4cyclictest19051-21sed21:15:008
119799100,3cyclictest28746-21irqrtprio21:25:208
119799100,1cyclictest0-21swapper/620:45:138
119799100,10cyclictest0-21swapper/623:31:168
119799100,10cyclictest0-21swapper/622:14:458
119799100,10cyclictest0-21swapper/622:14:458
118899109,1cyclictest0-21swapper/419:55:376
118899109,1cyclictest0-21swapper/400:30:016
118899107,2cyclictest32591-21timerandwakeup123:35:296
118899103,7cyclictest0-21swapper/400:13:136
118899102,5cyclictest0-21swapper/420:12:156
118899101,7cyclictest0-21swapper/423:56:046
118899100,1cyclictest0-21swapper/423:29:216
118899100,1cyclictest0-21swapper/420:01:226
118499109,1cyclictest0-21swapper/322:51:085
118499106,3cyclictest31397-21chrt20:05:145
118499103,7cyclictest0-21swapper/323:03:145
118499103,2cyclictest20817-21latency19:35:215
118499100,2cyclictest0-21swapper/320:30:185
117899107,2cyclictest0-21swapper/221:30:164
117899107,2cyclictest0-21swapper/220:50:134
117899100,9cyclictest22578-21irqrtprio20:35:244
117899100,9cyclictest15064-21fschecks_time21:50:164
117899100,9cyclictest0-21swapper/220:45:234
117899100,9cyclictest0-21swapper/219:30:304
117899100,8cyclictest0-21swapper/220:40:304
117899100,4cyclictest16911-21/usr/sbin/munin23:15:294
117899100,10cyclictest0-21swapper/222:11:014
117899100,10cyclictest0-21swapper/222:11:014
116999105,4cyclictest25639-21smart_sda21:20:320
116999102,7cyclictest4274-21cut22:20:010
116999102,7cyclictest4274-21cut22:20:000
116999100,9cyclictest5068-21latency_hist23:45:010
116999100,9cyclictest4769-21find21:35:270
116999100,9cyclictest32652-21cut21:30:210
116999100,9cyclictest32612-21latency_hist22:15:010
116999100,9cyclictest32612-21latency_hist22:15:010
116999100,9cyclictest30001-21unixbench_multi21:25:300
116999100,9cyclictest29706-21basename20:05:000
116999100,9cyclictest26065-21idleruntime20:40:180
116999100,9cyclictest25279-21sed22:05:000
116999100,9cyclictest21295-21cstates00:05:120
116999100,9cyclictest17886-21timerwakeupswit19:30:300
116999100,9cyclictest17186-21/usr/sbin/munin20:30:240
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional