Update haproxy config to include keystone request retry.

- Add keywork retry_on in haproxy::backend
- Add values retry_on in keystone.pp
- Modified keystone_http_connect_timeout 10 to 15 in api.pp, api_proxy.pp, certalarm.pp and certmon.pp

this workaround solves:
- DC Scale | RR Patch Orchestration fails as it cannot retrieve patches for subcloud after the apply
- DC Patch - Parallel patch orchestration fails to establish connection to MGMT interface of subclouds
- Patch orchestration fail due to transient keystone errors

Test plan:
1. (PASSED) Patch Creation:
    - Construct a "reboot required" RR patch that encompasses the specified changes.
    - Generate an "in-service test" NRR patch.

2. (PASSED) Initial Setup:
    - Commission a DC system with over 500 subclouds.
    - Assert that the patch encompassing the fix is applied successfully on the DC.

3. (PASSED) Strategy Creation and RR Patch Deployment (Max 250 Subclouds):
    - Created a RR patch strategy with max_parallel_subclouds set to 250
    - Checked that the RR patch strategy is applied to all subclouds successfully.
    - Repeat this process in more 250 subclouds
    - Checked that the patch strategy is applied to all subclouds successfully.

4. (PASSED) Strategy Alteration and NRR Patch Deployment (Max 500 Subclouds):
    - Eliminate the existing patch strategy.
    - Initiate a NRR patch strategy, adjusting the max_parallel_subclouds parameter to 500.
    - Checked that the "in-service test" NRR patch is successfully applied across all subclouds and that no linked issues arise.

Closes-Bug: #2025646
Change-Id: I95e9c8f3cd904d7f637da2ea69a83fd7fa5f03a1
Signed-off-by: Bezerra Filho, Moacir <Moacir.BezerraFilho@windriver.com>

modules/puppet-dcmanager/src/dcmanager/manifests/api.pp

@@ -95,7 +95,7 @@ class dcmanager::api (
   $keystone_identity_uri      = false,
   $keystone_user_domain       = 'Default',
   $keystone_project_domain    = 'Default',
-  $keystone_http_connect_timeout = '10',
+  $keystone_http_connect_timeout = '15',
   $keystone_http_request_max_retries = '3',
   $auth_type                  = 'password',
   $service_port               = '5000',

modules/puppet-dcorch/src/dcorch/manifests/api_proxy.pp

@@ -96,7 +96,7 @@ class dcorch::api_proxy (
   $keystone_identity_uri      = false,
   $keystone_user_domain       = 'Default',
   $keystone_project_domain    = 'Default',
-  $keystone_http_connect_timeout = '10',
+  $keystone_http_connect_timeout = '15',
   $dcmanager_keystone_user    = 'dcmanager',
   $auth_type                  = 'password',
   $service_port               = '5000',

modules/puppet-sysinv/src/sysinv/manifests/certalarm.pp

@@ -26,7 +26,7 @@ class sysinv::certalarm (
   $keystone_interface         = 'internal',
   $auth_type                  = 'password',
   $service_port               = '5000',
-  $keystone_http_connect_timeout = '10',
+  $keystone_http_connect_timeout = '15',
   $package_ensure             = 'latest',
   $bind_host                  = '::',
   $pxeboot_host               = undef,

modules/puppet-sysinv/src/sysinv/manifests/certmon.pp

@@ -34,7 +34,7 @@ class sysinv::certmon (
   $keystone_interface         = 'internal',
   $auth_type                  = 'password',
   $service_port               = '5000',
-  $keystone_http_connect_timeout = '10',
+  $keystone_http_connect_timeout = '15',
   $package_ensure             = 'latest',
   $bind_host                  = '::',
   $pxeboot_host               = undef,

puppet-manifests/src/modules/openstack/manifests/keystone.pp

@@ -169,6 +169,7 @@ class openstack::keystone::haproxy
       public_ip_address => $::platform::haproxy::params::private_dc_ip_address,
       public_port       => $api_port + 1,
       private_port      => $api_port,
+      retry_on          => 'conn-failure 0rtt-rejected',
     }
   }
 }

puppet-manifests/src/modules/platform/manifests/haproxy.pp

@@ -20,6 +20,7 @@ define platform::haproxy::proxy (
   $private_ip_address = undef,
   $server_timeout = undef,
   $client_timeout = undef,
+  $retry_on = undef,
   $x_forwarded_proto = true,
   $enable_https = undef,
   $https_ep_type = undef,
@@ -142,9 +143,10 @@ define platform::haproxy::proxy (
     collect_exported => false,
     name             => "${name}-internal",
     options          => {
-      'server'  => "${server_name} ${private_ip}:${private_port}",
-      'timeout' => $timeout_option,
-      'mode'    => $mode_option,
+      'server'   => "${server_name} ${private_ip}:${private_port}",
+      'timeout'  => $timeout_option,
+      'mode'     => $mode_option,
+      'retry-on' => $retry_on
     }
   }
 }