OpenShift Agent Based Installer Helper on VMware vSphere in a Disconnected Environment

Published: Mar 7, 2024 by

OpenShift Agent Based Installer Helper on VMware vSphere in a Disconnected Environment

Git Repository: https://github.com/Red-Hat-SE-RTO/openshift-agent-install

Prerequisites

  • A RHEL system to work from
  • OpenShift CLI Tools - run ./download-openshift-cli.sh then sudo cp ./bin/* /usr/local/bin/
  • NMState CLI dnf install nmstate
  • Ansible Core dnf install ansible-core
  • Ansible Collections for the automation: ansible-galaxy install -r openshift-agent-install/collections/requirements.yml
  • Red Hat OpenShift Pull Secret saved to a file: https://console.redhat.com/openshift/downloads#tool-pull-secret
  • Any other Pull Secret for a disconnected registry, joined with the Red Hat OpenShift Pull Secret

Enable Disk UUID on Virtual Machines
To enable disk UUID on virtual machines, you need to set the disk.EnableUUID parameter to TRUE in the VMware vSphere configuration. This ensures that the virtual machine’s disks are identified by their unique UUIDs, which is required for the OpenShift installer to work correctly.

Follow these steps to enable disk UUID:

  • Open the VMware vSphere client and navigate to the cluster or host where you want to enable disk UUID.
  • Right-click on the cluster or host and select “Edit Settings.”
  • In the “Advanced Settings” section, find the disk.EnableUUID parameter and set its value to TRUE.
  • Click “OK” to save the changes.

Usage - Declarative

In the examples directory you’ll find sample cluster configuration variables. By defining the cluster in its own folder with the cluster.yml and nodes.yml files, you can easily template and generate the ABI ISO in one shot with:

./hack/create-iso.sh $FOLDER_NAME

This script will take those defined files, generate the templates with Ansible, create the ISO, and present next step instructions.

Alternatively, you can perform those steps manually with the instructions below.

Usage - Manual

1. Templating Agent-Based Installer Manifests

You can quickly and easily template the ABI manifests with the provided create-manifests.yml Ansible Playbook.

# Make sure you're in the `playbooks` directory
cd playbooks/

# Execute the automation with your custom cluster configuration set in a YAML file
ansible-playbook -e "@your-cluster-vars.yml" create-manifests.yml

2. Creating the Agent Installer ISO

After running the automation to generate the manifests, you can create the ISO with the following:

# Create the ISO
openshift-install agent create image --dir ./generated_manifests/<cluster_name>

# Watch the Bootstrap process
openshift-install agent wait-for bootstrap-complete --dir ./generated_manifests/<cluster_name>

# Watch the installation process
openshift-install agent wait-for install-complete --dir ./generated_manifests/<cluster_name>

You’ll need to provide it some variables such as the following: Plase us the examples url for reference as it may be updated.

vmware-disconnected-example/cluster.yml

General Configuration Variables

# pull_secret path is the path to the pull-secret for the cluster
pull_secret_path: ~/ocp-install-pull-secret.json

# ssh_public_key_path is the path to the SSH public key to use for the cluster
# if this is not set then a new key pair will be generated
# ssh_public_key_path: ~/.ssh/id_rsa.pub

# Cluster metadata
base_domain: example.com
cluster_name: ocp4

# platform_type is the type of platform to use for the cluster (baremetal, vsphere, none)
# must be none for SNO
platform_type: vsphere
vcenter_host: "portal.example.com"
vcenter_username: "administrator@example.com"
vcenter_password: "example_password"
vcenter_datacenter: "Datacenter"
vcenter_default_datastore: "Datastore"
vcenter_cluster: "Cluster"
vcenter_network: "VM Network"
vcenter_folder_name: "ocp4"
vcenter_disk_type: thin

# VIPs - set as a list in case this is a dual-stack cluster
api_vips:
  -  192.168.180.4

app_vips:
  -  192.168.180.5

# Optional NTP Servers
ntp_servers:
  - 0.rhel.pool.ntp.org
  - 1.rhel.pool.ntp.org

# Optional DNS Server definitions
dns_servers:
  - 192.168.180.9
  - 192.168.180.10
dns_search_domains:
  - example.com
  - example.network

# cluster_network_cidr is the overall CIDR space for the Pods in the cluster
cluster_network_cidr: 10.128.0.0/14
# cluster_network_host_prefix is the number of bits in the cluster_network_cidr that are for each node
cluster_network_host_prefix: 23

# service_network_cidrs is the CIDR space for the Services in the cluster (ClusterIP/NodePort/LoadBalancer)
service_network_cidrs:
  - 172.30.0.0/16

# machine_network_cidr is the CIDR space for the Machines in the cluster
machine_network_cidrs:
  - 192.168.180.0/23

# networkType is the type of network to use for the cluster (OpenShiftSDN, OVNKubernetes)
network_type: OVNKubernetes

# rendezvous_ip is the IP address of the node that will be used for the bootstrap node
rendezvous_ip: 192.168.180.21

# Optional Outbound Proxy Configuration
# proxy:
#   http_proxy: http://192.168.42.31:3128
#   https_proxy: http://192.168.42.31:3128
#   no_proxy:
#     - .svc.cluster.local
#     - 192.168.0.0/16
#     - .example.network
#     - .example.labs

# Optional Additional CA Root Trust Bundle

create_ztp_manifests: false
#cluster_architecture: x86_64 # x86_64 | s390x | ppc64le | aarch64 | multi

# Optional Disconnected Registry Mirror configuration
disconnected_registries:
  # Must have a direct reference to the openshift-release-dev/ocp-release and openshift-release-dev/ocp-v4.0-art-dev paths
  - target: disconn-harbor.d70.kemo.labs/quay-ptc/openshift-release-dev/ocp-release
    source: quay.io/openshift-release-dev/ocp-release
  - target: disconn-harbor.d70.kemo.labs/quay-ptc/openshift-release-dev/ocp-v4.0-art-dev
    source: quay.io/openshift-release-dev/ocp-v4.0-art-dev

  - target: disconn-harbor.d70.kemo.labs/quay-ptc
    source: quay.io
  - target: disconn-harbor.d70.kemo.labs/registry-redhat-io-ptc
    source: registry.redhat.io
  - target: disconn-harbor.d70.kemo.labs/registry-connect-redhat-com-ptc
    source: registry.connect.redhat.com

# Optional Outbound Proxy Configuration
# proxy:
#   http_proxy: http://192.168.42.31:3128
#   https_proxy: http://192.168.42.31:3128
#   no_proxy:
#     - .svc.cluster.local
#     - 192.168.0.0/16
#     - .kemo.network
#     - .kemo.labs

# Optional Additional CA Root Trust Bundle
additional_trust_bundle_policy: Always
additional_trust_bundle: |
  -----BEGIN CERTIFICATE-----
  MIIG3TCCBMWgAwIBAgIUJSmf6Ooxg8uIwfFlHQYFQl5KMSYwDQYJKoZIhvcNAQEL
  BQAwgcMxIzAhBgkqhkiG9w0BCQEWFG5hLXNlLXJ0b0ByZWRoYXQuY29tMQswCQYD
  VQQGEwJVUzEXMBUGA1UECAwOTm9ydGggQ2Fyb2xpbmExEDAOBgNVBAcMB1JhbGVp
  Z2gxFDASBgNVBAoMC05vdCBSZWQgSGF0MRswGQYDVQQLDBJTRSBSVE8gTm90IElu
  Zm9TZWMxMTAvBgNVBAMMKFNvdXRoZWFzdCBSVE8gUm9vdCBDZXJ0aWZpY2F0ZSBB
  dXRob3JpdHkwHhcNMjIwMzA3MDAwNTA5WhcNNDIwOTE4MDAwNTA5WjCBwzEjMCEG
  CSqGSIb3DQEJARYUbmEtc2UtcnRvQHJlZGhhdC5jb20xCzAJBgNVBAYTAlVTMRcw
  FQYDVQQIDA5Ob3J0aCBDYXJvbGluYTEQMA4GA1UEBwwHUmFsZWlnaDEUMBIGA1UE
  CgwLTm90IFJlZCBIYXQxGzAZBgNVBAsMElNFIFJUTyBOb3QgSW5mb1NlYzExMC8G
  A1UEAwwoU291dGhlYXN0IFJUTyBSb290IENlcnRpZmljYXRlIEF1dGhvcml0eTCC
  AiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBANGozAIcO/PB4uIwI31kuiGW
  j+Nm+ZJruiOaG0P/Z99F/i7e9aOrQD8BHmlGOp9R0sdabrmidvowLE69g5z4+Q0E
  4+Uvt4GX/DYOBVR/xuV3E8LFJN1zXXbFtXJlSBz3PLWNaySAcg55a/Pwz68EWFA1
  H2RL5I/sPDpFiz0POnZ+MJ15BCQ2P5YCN7lsHSkmbRonz349WAhvE5OM6qIrBw/J
  Y6AJtAuEVnyiKoilqEvg0Gz6mSnog2yJY1CktMmP7S6/DPuJpTrw74027mp+g1Pm
  hRf8jVNsLNM7VPMo8AIodTCIc+Gv3EJ1bjMc/nF1F3K5jBQZrfe21QpgMKyeY/RV
  FvoHaCy2Miw2RFE9HOo0rwnOohiXlZM6ZSL5AUfDH2tSlJJNr08fE4op48UMIahz
  2My117CKFE2gRe5bhEEJAO9gOqsq1oOT4Oi3TP+lysjAVAIcnNFhQ1uRmJ93Y8HU
  qOCOgH+PV7N+kNtOwy8y32+Czh6njL09IbR8TNH2fOXYVt7JDZjnfU+FdzagNWc5
  C+aQCdpKIMig5OuU81Ac8k6+Aj0CBawOcBI63oxV/GWkUJPgQytmyo/2zswD9FcD
  yIVL1nvJOwVWNEyOLtDWmEzSda6CVLFFQnAw35qgS94Hc7IS3nQW6XFEGj7xzTmd
  b2xoEKhgx+dPw5h7AYPHAgMBAAGjgcYwgcMwHQYDVR0OBBYEFDzw4uwWVqsqJDNM
  2Rz+ztC/ZgUNMB8GA1UdIwQYMBaAFDzw4uwWVqsqJDNM2Rz+ztC/ZgUNMA8GA1Ud
  EwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMGAGA1UdHwRZMFcwVaBToFGGT2h0
  dHBzOi8vcmVkLWhhdC1zZS1ydG8uZ2l0aHViLmlvOjQ0My9jcmxzL3NlLXJ0by1y
  b290LWNlcnRpZmljYXRlLWF1dGhvcml0eS5jcmwwDQYJKoZIhvcNAQELBQADggIB
  AFu7g/6ghP0PaLsjjAPW+QWqv9tMk8w0MKbKgVeUOX5xz1I7Svc1ndi2dMcYwK8W
  pgF4bVR8T17NE3V0/xy6BGktN9BtErI9guk3zb3GBIx/1b3Mgce7134nGvhi4ik7
  ziNB2WYwOgwxEpSA1eS68WNMT6pWZvosEZu9AKMUQ8ULsfxiKwVT+Pj171JxIvDV
  blhilnOrBap7sP1XwS9OPcQhm0AMtFEj/zhadO1h2ynwKjd/VE2/nskfLvm1dXK5
  DtdHsCdtT/hJ0XQjLkwOkl87WHZsy4u6kxQzxKH+LDWfSOCOksYD86fBdfQC66gL
  7yJpX9BznEaGCKgFam3m42eH9msCIV/JTTLUbsrwzaEhxBLtpUeo6j1xF2khF8Ri
  45Sir0yotZE0i72S4TLllkgQx9AaOiRAWvtYkcP1TBJnzL5viac3pkTnPjLiQ9BO
  V8+9Y1O6wU0KTbLdMaz+Wfpti1lcnphQDsMJoGTe6wl3QpAK2jz32aFMoTkoyDK5
  MwQqiTMkyOkPCiY4Rq1RRnYGIU7Ob125IjaFqyLvG9KWuiFsH7yn2nVH5kwy7O75
  7yx0UiBuGVfG66I09YM1jR9nq7mKv30Sq1Fa/X76XyxDBGk0rLRCw02Ziq0rS8WG
  S5kIfhw8FM52x6RHCwRicArO8HSTCf4ueEkFL7yj5xSI
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  MIIDGzCCAgOgAwIBAgIQIiia9v4QpyL2ar7kW4K2cDANBgkqhkiG9w0BAQsFADAm
  MSQwIgYDVQQDExtLZW1vIExhYnMgU21hbGxTdGVwIFJvb3QgQ0EwHhcNMjIwNDAz
  MjAzNDA5WhcNMzIwMzMxMjAzNDA5WjAmMSQwIgYDVQQDExtLZW1vIExhYnMgU21h
  bGxTdGVwIFJvb3QgQ0EwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDF
  w9Y9/ScdF8bmgElmQQQFJku033PkOMtPOCakAaPCly1reqHm8r3Mjhjuqi8HiC94
  NQ+jWYRFzAGyMUZfR9PaCnN4EsVRjX1KAvttX4eRXgAo8SsIb9ExKHVSVDoBCm62
  /FpVYE24bgUcN5gqnp7lMdSlW69ArnpSnLOkQCGDXknSEBCGUdpz8jdehqAyXoFo
  yedC9oAxvisEQ3SjyMQDqKo7XNS2VEODozGp0bNcym5461VHeIVulo1/8/kPfEkv
  Zjr7ZzGjRiEn1a0wrbDtdTG5VSobGQW/I9VIgbXp0pTUUzurIAlOh+LzvnNwVKIv
  XFkJfutEGrAxFDYX6K9xAgMBAAGjRTBDMA4GA1UdDwEB/wQEAwIBBjASBgNVHRMB
  Af8ECDAGAQH/AgEBMB0GA1UdDgQWBBSuSpXt1rL5wa2CyYFfchCD3ZCzSjANBgkq
  hkiG9w0BAQsFAAOCAQEAe6Prprq70SLis/5LJVl8pC+MgPa/ICukJ1O3C6Jn4w6y
  GGCBwa26SAw49J54qizQPbQdks0px0fQCIzbKlVA44lfq6VwSpXrM/VgesGB4vez
  vfdDBvJwnc5/E93NxtxinbVvNps7xfa2kW22xu2GZoOueAJr3gcG8ZQMZc7oMY4a
  7OaqWH9OyUhax2Odv+37Its5PjBbr7vHabzw6F6849Lx1vDwlpg0dqCFVMSKvm4l
  KK4oFUbZw0+cLmAlOYHrw5QIHqAGT7p8Cew5zR/fuuKJx2yiKP/tsz6E1OMejN9q
  g/IdSjffp7OClDHFa3nuXzsKk87O3eTr4fzKALVpqQ==
  -----END CERTIFICATE-----
  -----BEGIN CERTIFICATE-----
  MIIGqzCCBJOgAwIBAgIUKMZCYZxHomZOUFLz8j0/ItBY/3cwDQYJKoZIhvcNAQEL
  BQAwgdwxKzApBgkqhkiG9w0BCQEWHGNlcnRtYXN0ZXJAcG9seWdsb3QudmVudHVy
  ZXMxCzAJBgNVBAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBDYXJvbGluYTEQMA4GA1UE
  BwwHUmFsZWlnaDEaMBgGA1UECgwRUG9seWdsb3QgVmVudHVyZXMxIjAgBgNVBAsM
  GVBvbHlnbG90IFZlbnR1cmVzIEluZm9TZWMxNTAzBgNVBAMMLFBvbHlnbG90IFZl
  bnR1cmVzIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5MB4XDTIxMTEyODIyMTE1
  MVoXDTQyMDYxMTIyMTE1MVowgdwxKzApBgkqhkiG9w0BCQEWHGNlcnRtYXN0ZXJA
  cG9seWdsb3QudmVudHVyZXMxCzAJBgNVBAYTAlVTMRcwFQYDVQQIDA5Ob3J0aCBD
  YXJvbGluYTEQMA4GA1UEBwwHUmFsZWlnaDEaMBgGA1UECgwRUG9seWdsb3QgVmVu
  dHVyZXMxIjAgBgNVBAsMGVBvbHlnbG90IFZlbnR1cmVzIEluZm9TZWMxNTAzBgNV
  BAMMLFBvbHlnbG90IFZlbnR1cmVzIFJvb3QgQ2VydGlmaWNhdGUgQXV0aG9yaXR5
  MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEAp6gp+cohBjNjCfQYaHyl
  3OChI32xFkU+S68cP27cG9GS6+GfOkmnxavVa8HfYKIYDYwLvev22DrF+ErkL+sq
  KO1c0cJd2jT0sf6snHhT3k0kS+uthscL4ZMHh6ux2ULhR0n9JPrAwGlSLZZpIph8
  clZU8JkLipVdhwAUpHEvcyxkk+e/tWiz1ir6uhI17WifFu+nDZiyjWtLqmRVWhnQ
  0ngTgcP2WwlRBrlSvaBfXYoFEZSwajGhoaKoJJ2pRcrzhz0ImmEykiqw+7cBXCw+
  9926rxqNtDVGC7P/f9pcks90JH+bg+PFPcgdHT5fGXHJct54bvvPLDMLPAjPUS0d
  NhCqF1EfLhI+1wbtuijMlfjyBdy9Mz6f5PmKvE2OcmvBm1EwoKJRJ5OY4d6U94L4
  ljLotCuLwCmRAO/wvIMxo0AM/yu86HuMgytE0SOhzXBL8GHjubfFpTOV5K6wFR/d
  YOOxKmGGftAH0rh/GyWRdQuGwy+XaFFMN3uU0v26dJw1qzFlObKD8D9h20Po8LxH
  dngQZvK0XwKb/rtG5ti7sMrm00kGeFilZqo4ZcJayCP9PRf7VTkBPa8lSaiq/Xgi
  aYBnnJ19wtWgupy+96yzf0Vv+lCpo1Q/ItND2mfvJ32Ec/WoytYE7ZGHiTzAQ7tB
  WFpwiqeC2u9cIbWYGPjY+JcCAwEAAaNjMGEwHQYDVR0OBBYEFAMrzv+9iNPU+QIl
  8QKZmWrXicOlMB8GA1UdIwQYMBaAFAMrzv+9iNPU+QIl8QKZmWrXicOlMA8GA1Ud
  EwEB/wQFMAMBAf8wDgYDVR0PAQH/BAQDAgGGMA0GCSqGSIb3DQEBCwUAA4ICAQAG
  cUG6p0q9ixjdfLwA75xfyrPbaTx3jAkVnCtre5GQV1j0IfGpfaGxzqb7zVHHfRkN
  LklrFQZRrnsraBGpCKLZCCJ93UTglTbRAcAwHr7C0vodikVe8VZUAcAonMS1VE0Q
  qC3tma56ddOog1nSRbqCQ8/Y7GJM6ZxbFs9TT7/ld4jCqBhXQeIRXKwlRZi2KVLm
  xdSVp5VWMrmFyvw2QGkj1evkheQREbDRaXhZ95NkhTQIVDFbKzSSVYaEV/Nh1ZUC
  7h7QiTWkULvsffjmYR/x2ifWBWb9jinM7Rl8hcJg4TvZz6qZuTWpPsA23huBbXor
  6mmpSL8sbeXKawfUESYYc37HJ2n2WQOJ0hFoDFpK2nZnn4T+OWgNsGvwtPA6tfrZ
  dXDCFKJDXlQIAZw+x9S5V1dr4AT1RShjrfD9gnimu0UxT9LvYmOLW1MjpycV5Zg9
  coRfdfOXmGZm7R22i47pXAju8byHodatBlhRgFiUpwTuUoueElosgXkui9HzkfXx
  tqUuc3dAaVH3vmeQpYb+LJ1wCn2mtnpJedui2N5dsqEQ1XsXjxp2bfWdhh52ZwHs
  LSMk1f3L54UjG+iMyolALyCvpibGD6g6PRMp8UTStZatPJDzT2/JbFu9mIhU5V4g
  zYML3t12ZU8JGpxxfUk2ObjKbixfSwSmTcWb+s8kgg==
  -----END CERTIFICATE-----

HA Cluster Deployment

vmware-disconnected-example/nodes.yml

# Node Counts the installer will expect
control_plane_replicas: 3
app_node_replicas: 3
octect: 192.168.180
node_one_mac: EC:F4:BB:C0:B9:C8
node_two_mac: EC:F4:BB:C0:B9:C9
node_three_mac: EC:F4:BB:C0:B9:CA
node_four_mac: EC:F4:BB:C0:B9:CB
node_five_mac: EC:F4:BB:C0:B9:CC
node_six_mac: EC:F4:BB:C0:B9:CD

# nodes defines the nodes to use for the cluster
nodes:
  - hostname: node-1
    role: master
    rootDeviceHints:
      deviceName: /dev/sda
      #deviceName: /dev/nvme0n1
    interfaces:
      - name: ens192
        mac_address: ''
    networkConfig:
      interfaces:
        - name: ens192
          type: ethernet
          state: up
          mac-address: ""
          ipv4:
            enabled: true
            address:
              - ip: ".21"
                prefix-length: 24
            dhcp: false
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: ".1"
            next-hop-interface: ens192
            table-id: 254
  - hostname: node-2
    role: master
    rootDeviceHints:
      deviceName: /dev/sda
      #deviceName: /dev/nvme0n1
    interfaces:
      - name: ens192
        mac_address: ""
    networkConfig:
      interfaces:
        - name: ens192
          type: ethernet
          state: up
          mac-address: ""
          ipv4:
            enabled: true
            address:
              - ip: ".22"
                prefix-length: 24
            dhcp: false
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: ".1"
            next-hop-interface: ens192
            table-id: 254
  - hostname: node-3
    role: master
    rootDeviceHints:
      deviceName: /dev/sda
      #deviceName: /dev/nvme0n1
    interfaces:
      - name: ens192
        mac_address: ""
    networkConfig:
      interfaces:
        - name: ens192
          type: ethernet
          state: up
          mac-address: ""
          ipv4:
            enabled: true
            address:
              - ip: ".23"
                prefix-length: 24
            dhcp: false
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: ".1"
            next-hop-interface: ens192
            table-id: 254
  - hostname: node-4
    role: worker
    rootDeviceHints:
      deviceName: /dev/sda
      #deviceName: /dev/nvme0n1
    interfaces:
      - name: ens192
        mac_address: ""
    networkConfig:
      interfaces:
        - name: ens192
          type: ethernet
          state: up
          mac-address: ""
          ipv4:
            enabled: true
            address:
              - ip: ".24"
                prefix-length: 24
            dhcp: false
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: ".1"
            next-hop-interface: ens192
            table-id: 254
  - hostname: node-5
    role: worker
    rootDeviceHints:
      deviceName: /dev/sda
      #deviceName: /dev/nvme0n1
    interfaces:
      - name: ens192
        mac_address: ""
    networkConfig:
      interfaces:
        - name: ens192
          type: ethernet
          state: up
          mac-address: ""
          ipv4:
            enabled: true
            address:
              - ip: ".25"
                prefix-length: 24
            dhcp: false
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: ".1"
            next-hop-interface: ens192
            table-id: 254
  - hostname: node-6
    role: worker
    rootDeviceHints:
      deviceName: /dev/sda
      #deviceName: /dev/nvme0n1
    interfaces:
      - name: ens192
        mac_address: ""
    networkConfig:
      interfaces:
        - name: ens192
          type: ethernet
          state: up
          mac-address: ""
          ipv4:
            enabled: true
            address:
              - ip: ".26"
                prefix-length: 24
            dhcp: false
      routes:
        config:
          - destination: 0.0.0.0/0
            next-hop-address: ".1"
            next-hop-interface: ens192
            table-id: 254

Share