You are here: Home / Projects / OSADL QA Farm Real-time / Latency plots / 
2024-09-20 - 08:53

OSADL QA Farm on Real-time of Mainline Linux

About - Hardware - CPUs - Benchmarks - Graphics - Benchmarks - Kernels - Boards/Distros - Latency monitoring - Latency plots - System data - Profiles - Compare - Awards

Default latency plot of shadow in rack #5, slot #3

Rack #0/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #1/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #2/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #3/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #4/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #5/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #6/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #7/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #8/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #9/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #a/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #b/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #c/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #d/#0 #1 #2 #3 #4 #5 #6 #7 #8 -
Rack #e/#0 #1 #2 #3 #4 #5 #6 #7 #8 - Rack #f/#0 #1 #2 #3 #4 #5 #6 #7 #8 
Special  All - All RT - Optimization - Ethernet - Thumbnails - Next
  
Click here to display the system's profile data or here to proceed to next system.
Click on a legend element to toggle display of that core, ctrl-click inverts display, shift-click enables all.

Data to construct the above plot have been generated using the RT test utility cyclictest.
Unexpectedly long latencies may be caused by SMIs
Total number of samples: 100 million
Resolution of latency scale: normal
Duration: 5 hours, 33 minutes, lowest P state: performance
Compare latency of primary with shadow system
Characteristics of the highest latencies:
System rack5slot3s.osadl.org (updated Fri Sep 20, 2024 00:44:31)
Delayed (victim)Switcher (culprit)TimestampCPU
PIDPrioTotal
latency
(µs)
T*(,W**)
latency
(µs)
CmdPIDPrioCmd
29942239161,25sleep10-21swapper/119:09:471
27322238166,59sleep20-21swapper/219:06:252
27302234162,58sleep00-21swapper/019:06:230
29862202166,24sleep30-21swapper/319:09:403
1888121570,1sleep00-21swapper/020:55:220
3117998884,3cyclictest0-21swapper/223:34:492
3117995956,2cyclictest0-21swapper/220:04:452
174452510,1sleep10-21swapper/100:35:011
312299441,29cyclictest0-21swapper/322:50:113
3122994025,7cyclictest0-21swapper/323:01:323
312299391,29cyclictest0-21swapper/322:41:373
3122993826,7cyclictest0-21swapper/319:11:053
3122993826,4cyclictest0-21swapper/319:19:143
3122993817,8cyclictest0-21swapper/319:28:493
3122993726,3cyclictest0-21swapper/323:30:513
3122993726,3cyclictest0-21swapper/320:52:363
3122993717,9cyclictest0-21swapper/321:09:523
312299361,22cyclictest0-21swapper/320:28:273
3117993626,9cyclictest0-21swapper/219:34:072
3117993626,9cyclictest0-21swapper/219:34:072
312299351,31cyclictest0-21swapper/323:14:393
3122993414,8cyclictest0-21swapper/320:36:003
3122993414,8cyclictest0-21swapper/320:36:003
311399341,26cyclictest4748-21kworker/u8:021:48:351
310699346,13cyclictest25465-21munin-run23:40:020
3122993317,11cyclictest0-21swapper/300:04:233
312299331,26cyclictest0-21swapper/300:21:233
312299331,25cyclictest0-21swapper/321:41:373
312299331,14cyclictest0-21swapper/323:09:203
312299331,14cyclictest0-21swapper/321:27:553
312299330,12cyclictest0-21swapper/300:25:213
3122993220,3cyclictest0-21swapper/322:04:253
3122993217,3cyclictest0-21swapper/319:52:433
3122993216,3cyclictest0-21swapper/320:21:263
312299321,29cyclictest0-21swapper/321:45:453
312299321,20cyclictest0-21swapper/320:06:323
312299321,19cyclictest0-21swapper/300:33:213
3122993120,3cyclictest0-21swapper/320:42:223
3122993118,8cyclictest0-21swapper/300:38:423
3122993112,3cyclictest10685-21ntp_kernel_err21:50:213
312299310,28cyclictest0-21swapper/319:43:033
3122993016,8cyclictest0-21swapper/320:31:553
3122993013,8cyclictest0-21swapper/320:18:413
312299301,27cyclictest0-21swapper/320:58:243
312299301,20cyclictest0-21swapper/323:58:273
312299301,19cyclictest0-21swapper/321:03:123
312299301,14cyclictest0-21swapper/323:50:593
312299301,13cyclictest0-21swapper/321:24:553
312299301,13cyclictest0-21swapper/319:47:313
312299301,12cyclictest0-21swapper/322:17:433
312299301,12cyclictest0-21swapper/321:13:223
3122993011,2cyclictest0-21swapper/320:45:223
312299301,11cyclictest0-21swapper/300:05:253
312299300,14cyclictest0-21swapper/323:28:493
312299300,13cyclictest0-21swapper/322:34:553
311399307,7cyclictest6540-40dnf20:29:331
311399301,25cyclictest26401-21munin-run21:15:011
311399301,24cyclictest0-21swapper/122:42:141
3122992918,3cyclictest0-21swapper/323:38:593
3122992917,3cyclictest0-21swapper/319:21:393
3122992916,8cyclictest0-21swapper/319:33:373
3122992916,8cyclictest0-21swapper/319:33:373
311399291,12cyclictest0-21swapper/123:30:331
311399290,27cyclictest0-21swapper/121:00:191
311399290,17cyclictest0-21swapper/120:20:171
310699294,12cyclictest29227-21munin-run22:35:020
3122992815,9cyclictest0-21swapper/321:35:433
3122992814,9cyclictest0-21swapper/322:46:013
312299281,13cyclictest0-21swapper/323:45:193
312299281,12cyclictest0-21swapper/323:18:313
312299281,12cyclictest0-21swapper/320:00:563
3122992810,3cyclictest0-21swapper/322:23:293
312299280,14cyclictest0-21swapper/321:33:073
312299280,13cyclictest0-21swapper/323:20:413
312299280,12cyclictest0-21swapper/322:08:473
310699284,11cyclictest27961-21munin-run20:05:010
3122992714,8cyclictest0-21swapper/320:14:483
312299271,17cyclictest0-21swapper/321:56:373
312299271,14cyclictest0-21swapper/319:57:063
312299271,12cyclictest0-21swapper/323:40:353
312299271,12cyclictest0-21swapper/300:15:593
311399272,3cyclictest0-21swapper/100:37:421
311399272,22cyclictest1830-21diskstats22:45:141
311399272,12cyclictest7809-21needreboot20:30:181
311399271,6cyclictest0-21swapper/123:07:051
310699274,10cyclictest22006-21perf21:05:010
310699271,13cyclictest5660-21munin-run22:55:010
312299269,3cyclictest0-21swapper/322:35:133
312299261,13cyclictest6946-21chrt22:55:203
312299261,12cyclictest0-21swapper/300:13:073
312299261,11cyclictest0-21swapper/322:28:253
311399262,22cyclictest0-21swapper/100:14:581
311399262,10cyclictest842-21lldpd20:06:501
311399261,3cyclictest5652-21turbostat.cron22:55:011
311399261,3cyclictest4534-21grep21:35:261
311399261,22cyclictest30046-21cat23:50:011
311399261,22cyclictest13559-21ntp_states23:10:221
311399260,20cyclictest0-21swapper/120:15:191
312299258,8cyclictest0-21swapper/319:37:293
3122992511,3cyclictest0-21swapper/321:15:323
312299250,11cyclictest0-21swapper/322:11:113
311399254,19cyclictest23-21ksoftirqd/123:35:121
311399253,10cyclictest0-21swapper/100:26:171
311399251,5cyclictest2111-21kworker/u8:200:08:071
311399251,22cyclictest0-21swapper/119:10:241
311399251,21cyclictest25341-21expr22:25:131
311399251,21cyclictest19151-21latency_hist19:45:011
311399251,21cyclictest18223-21latency_hist22:10:011
311399251,21cyclictest17958-21ntp_states23:20:231
311399251,21cyclictest14757-21latency_hist19:35:011
311399251,21cyclictest14757-21latency_hist19:35:011
311399251,16cyclictest0-21swapper/122:19:061
3113992510,3cyclictest30732-21diskstats20:10:141
3113992510,3cyclictest28317-40dnf22:30:221
3113992510,3cyclictest14858-21irqstats22:00:171
311399249,4cyclictest0-21swapper/100:17:261
311399248,4cyclictest15330-21df_inode19:35:131
311399242,20cyclictest0-21swapper/119:26:581
311399242,15cyclictest0-21swapper/123:25:231
3113992418,3cyclictest16220-21grep23:15:271
311399241,2cyclictest86550irq/126-enp1s0-22:24:241
311399241,20cyclictest0-21swapper/122:35:241
311399241,15cyclictest0-21swapper/119:45:031
311399241,12cyclictest26581-21grep20:00:161
311399241,11cyclictest522-21latency_hist21:30:011
311399241,11cyclictest0-21swapper/122:56:441
311399241,11cyclictest0-21swapper/119:55:261
311399241,10cyclictest18987-21wc20:55:231
3113992410,3cyclictest86750irq/128-enp1s0-23:40:101
310699242,4cyclictest0-21swapper/021:51:090
310699241,5cyclictest0-21swapper/023:13:200
310699241,4cyclictest0-21swapper/021:40:020
310699240,12cyclictest8808-21perf20:35:020
311799232,12cyclictest18789-21df22:10:112
311799231,10cyclictest0-21swapper/223:50:082
311399238,3cyclictest12100-21expr21:55:171
311399232,11cyclictest86750irq/128-enp1s0-21:32:151
311399232,11cyclictest15394-21munin-run20:50:011
3113992318,3cyclictest0-21swapper/120:44:361
311399231,7cyclictest31251-21grep23:50:221
311399231,12cyclictest0-21swapper/100:20:021
311399231,11cyclictest26790-21ls21:15:131
311399230,3cyclictest86750irq/128-enp1s0-23:55:211
310699231,6cyclictest0-21swapper/023:29:570
310699230,11cyclictest32059-21awk23:55:020
311799222,5cyclictest0-21swapper/200:20:262
311799222,11cyclictest14634-21snmpd21:20:002
311399223,16cyclictest2487-21sort00:00:141
311399222,3cyclictest0-21swapper/121:24:041
3113992215,4cyclictest7714-21grep19:15:251
311399221,2cyclictest6479-21gzip21:40:201
311399221,20cyclictest0-21swapper/121:06:401
311399221,11cyclictest0-21swapper/122:10:241
311399220,16cyclictest0-21swapper/119:23:211
310699229,7cyclictest836-21runrttasks20:29:340
310699222,10cyclictest19039-21tail00:35:270
310699221,5cyclictest0-21swapper/021:27:220
311799211,7cyclictest8876-21cat21:45:262
311799211,6cyclictest20938-21grep19:45:272
311799211,12cyclictest0-21swapper/221:22:232
311799211,11cyclictest7052-21ps22:55:212
311799211,11cyclictest4888-21users19:10:262
311799211,11cyclictest0-21swapper/223:45:232
311399216,13cyclictest221ktimersoftd/120:35:241
311399216,13cyclictest221ktimersoftd/120:35:241
311399211,11cyclictest7171-21kworker/u8:121:52:011
3113992110,3cyclictest0-21swapper/119:53:461
310699212,11cyclictest29447-21/usr/sbin/munin22:35:140
310699211,12cyclictest15729-21sshd22:01:210
292502211,3sleep029285-21pluginstate20:05:210
248372210,4sleep0310699cyclictest19:55:240
311799206,3cyclictest18528-21idleruntime20:55:172
311799204,6cyclictest22055-21timerwakeupswit22:15:252
311799203,3cyclictest8531-21sleep00:15:012
311799202,6cyclictest30-21rcuc/200:35:272
311799202,16cyclictest16476-21grep00:30:222
311799201,7cyclictest0-21swapper/223:42:332
311799201,6cyclictest18445-21grep23:20:272
311799201,6cyclictest13473-21df00:25:122
311799201,5cyclictest24200-21diskmemload21:45:012
311799201,12cyclictest0-21swapper/222:29:292
311799201,10cyclictest20639-21unixbench_singl23:25:252
311799200,5cyclictest0-21swapper/221:05:172
311799200,14cyclictest0-21swapper/221:33:562
311399202,15cyclictest8428-21/usr/sbin/munin23:00:141
3113992013,4cyclictest16142-21grep20:50:141
310699209,3cyclictest21578-21/usr/sbin/munin19:50:180
310699207,4cyclictest0-21swapper/021:40:110
310699204,8cyclictest86750irq/128-enp1s0-21:20:170
310699201,6cyclictest0-21swapper/022:45:080
310699201,6cyclictest0-21swapper/022:12:160
310699201,11cyclictest8391-21/usr/sbin/munin19:20:160
310699201,10cyclictest9018-21/usr/sbin/munin20:35:120
310699201,10cyclictest9018-21/usr/sbin/munin20:35:120
310699201,10cyclictest19382-21/usr/sbin/munin19:45:130
3106992010,3cyclictest86750irq/128-enp1s0-23:07:480
106692200,4sleep232-21ksoftirqd/219:25:122
311799194,3cyclictest32473-21expr23:55:122
311799194,3cyclictest17816-21sed19:40:152
311799193,8cyclictest86550irq/126-enp1s0-20:40:172
311799191,9cyclictest16604-21diskstats22:05:142
311799191,7cyclictest0-21swapper/200:17:342
311799191,6cyclictest4239-21tune2fs22:50:152
311799191,6cyclictest24200-21diskmemload23:00:272
311799191,6cyclictest24200-21diskmemload21:37:252
311799191,6cyclictest0-21swapper/200:05:242
311799191,5cyclictest31994-21diskstats22:40:142
311799191,5cyclictest11109-21cat20:40:012
311799191,5cyclictest11109-21cat20:40:012
311799191,5cyclictest0-21swapper/222:23:012
3117991914,3cyclictest25456-21sadc23:40:002
311799191,12cyclictest0-21swapper/221:26:222
311799191,10cyclictest30621-21ps22:35:232
311799191,10cyclictest27252-21/usr/sbin/munin22:30:202
311799191,10cyclictest14634-21snmpd19:19:412
311799191,10cyclictest11853-21/usr/sbin/munin21:55:302
310699198,3cyclictest15336-21/usr/sbin/munin00:30:170
310699195,2cyclictest27678-21grep21:15:210
310699191,6cyclictest0-21swapper/023:41:060
310699191,5cyclictest22092-1kworker/u9:223:32:430
310699191,5cyclictest0-21swapper/021:59:170
310699191,4cyclictest0-21swapper/022:24:150
310699191,4cyclictest0-21swapper/022:06:120
310699191,3cyclictest0-21swapper/000:18:460
3106991912,4cyclictest741-21head20:15:150
310699191,10cyclictest13140-21/usr/sbin/munin00:25:190
3106991910,7cyclictest25053-21gltestperf21:10:140
311799184,3cyclictest86550irq/126-enp1s0-20:15:142
311799184,3cyclictest0-21swapper/221:04:562
311799183,4cyclictest86550irq/126-enp1s0-19:20:152
311799182,4cyclictest14485-21ntp_states20:45:212
311799181,7cyclictest0-21swapper/223:05:302
311799181,7cyclictest0-21swapper/222:49:522
311799181,7cyclictest0-21swapper/221:12:052
311799181,7cyclictest0-21swapper/200:00:252
311799181,6cyclictest0-21swapper/220:35:012
311799181,6cyclictest0-21swapper/219:37:252
311799181,5cyclictest31688-21/usr/sbin/munin20:10:252
3117991813,3cyclictest9653-21/usr/sbin/munin21:50:272
3117991813,3cyclictest15478-21timerwakeupswit22:00:252
311799181,11cyclictest0-21swapper/223:19:382
311799181,10cyclictest0-21swapper/220:51:592
311799180,12cyclictest1184-21kworker/u8:223:10:342
311799180,11cyclictest0-21swapper/220:05:112
310699187,8cyclictest3066-21gltestperf20:20:160
310699184,3cyclictest86550irq/126-enp1s0-21:30:150
310699183,12cyclictest10940-21/usr/sbin/munin00:20:140
310699181,8cyclictest842-21lldpd23:55:090
3106991812,3cyclictest7704-21expr21:45:120
3106991812,3cyclictest30617-21expr20:10:130
3106991812,3cyclictest2117-21/usr/sbin/munin00:00:230
3106991811,4cyclictest6073-21tune2fs19:15:050
3106991811,4cyclictest23461-21/usr/sbin/munin21:05:230
146132181,2sleep00-21swapper/019:33:190
146132181,2sleep00-21swapper/019:33:190
311799179,2cyclictest30-21rcuc/220:25:242
311799172,3cyclictest86850irq/129-enp1s0-19:55:232
311799171,14cyclictest0-21swapper/219:53:472
310699178,2cyclictest0-21swapper/019:40:110
310699171,8cyclictest0-21swapper/000:13:470
310699171,7cyclictest14634-21snmpd22:55:160
310699171,7cyclictest0-21swapper/023:00:590
310699171,6cyclictest16736-21sshd19:37:220
310699171,5cyclictest0-21swapper/023:45:490
310699171,5cyclictest0-21swapper/022:27:280
3106991712,3cyclictest32710-21ntp_states22:40:230
3106991711,3cyclictest3301-21/usr/sbin/munin19:10:140
3106991711,3cyclictest21327-21sh22:15:170
3106991711,3cyclictest17681-21cat23:20:190
310699171,10cyclictest0-21swapper/020:45:130
3106991710,4cyclictest5407-21sshd00:05:230
311799161,5cyclictest0-21swapper/220:20:192
310699161,5cyclictest0-21swapper/020:40:160
310699161,4cyclictest14634-21snmpd23:15:200
3106991610,3cyclictest16761-21awk20:50:210
3106991610,3cyclictest11028-21tune2fs19:25:160
*Timer  **Wakeup  (latency=timer+wakeup+contextswitch)

 

Valid XHTML 1.0 Transitional