Provision PAS Ops Manager & Bosh Director¶

In this lab you develop a new pipeline to provision a sandbox foundation. By the end of this lab you will have jobs to:

Create an Ops Manager VM, along with a BOSH Director
Periodically export the Ops Manager VM's installation
Periodically upgrade your Ops Manager

General Process¶

The process of developing this automation mirrors the steps one performs manually to provision a foundation.

For each step, there exists a corresponding platform automation task. Identify the task, review its usage, inputs, and outputs.

When a task requires a configuration file as input, author the file and commit and push it in the platform-config repository.

The configuration repository needs to be modeled as a Concourse resource.

If the configuration file relays credentials or secrets, those secrets need to be parameterized and the corresponding variable put into CredHub, using the prefix matching the Concourse team that the pipeline resides in: /concourse/sandbox.

The files fetched from the git repository need to be explicitly interpolated against CredHub with the credhub-interpolate task.

Finally, the products fetched from the Tanzu Network and placed into the blobstore are also inputs that must be fetched. Instead of modeling them as resources, we use the download-product task with a SOURCE of gcs to fetch them.

Each job we author will go something like this:

get platform automation image and task resources
get the git configuration repository (platform-config)
interpolate the config files from the repository against credhub (using credhub-interpolate)
download the products we need (using download-product)
invoke the platform automation task or tasks in question

Install Ops Manager¶

Begin with a job named install-opsman. The name is somewhat misleading, as this job needs to perform the following steps:

Create the Ops Manager VM
Configure authentication for Ops Manager
Configure a bosh director
Apply changes (provisions the bosh director VM)

These steps maps to the platform automation tasks create-vm, configure-authentication, configure-director, and apply-director-changes.

Attempt to write the job on your own, using the platform automation task reference as a guide.

We will build this job together in class, step by step.

If stuck, have a look at a draft of the job install-opsman, below.

Install OpsMan

---
resource_types:
- name: gcs
  type: docker-image
  source:
    repository: frodenas/gcs-resource

resources:
- name: platform-automation-tasks
  type: gcs
  source:
    bucket: ((bucket))
    regexp: platform-automation-tasks-(.*).zip
    json_key: ((json_key))

- name: platform-automation-image
  type: gcs
  source:
    bucket: ((bucket))
    regexp: platform-automation-image-(.*).tgz
    json_key: ((json_key))

- name: state
  type: gcs
  source:
    bucket: ((state-bucket))
    versioned_file: state.yml
    json_key: ((json_key))

- name: configuration
  type: git
  source:
    private_key: ((config-repo-key.private_key))
    uri: ((config-repo-uri))

jobs:
- name: install-opsman
  plan:
  - in_parallel:
    - get: platform-automation-tasks
      params: { unpack: true }
    - get: platform-automation-image
      params: { unpack: true }
    - get: configuration
    - get: state
  - task: interpolate-env-creds
    image: platform-automation-image
    file: platform-automation-tasks/tasks/credhub-interpolate.yml
    params:
      CREDHUB_CLIENT: ((credhub-client))
      CREDHUB_SECRET: ((credhub-secret))
      CREDHUB_SERVER: ((credhub-server))
      CREDHUB_CA_CERT: ((credhub-ca-cert.certificate))
      PREFIX: '/concourse/sandbox'
      SKIP_MISSING: false
    input_mapping:
      files: configuration
    output_mapping:
      interpolated-files: interpolated-configuration
  - task: download-opsman-product
    image: platform-automation-image
    file: platform-automation-tasks/tasks/download-product.yml
    params:
      CONFIG_FILE: download-product/opsman.yml
      SOURCE: gcs
    input_mapping:
      config: interpolated-configuration
    output_mapping:
      downloaded-product: opsman-image
  - task: create-vm
    image: platform-automation-image
    file: platform-automation-tasks/tasks/create-vm.yml
    input_mapping:
      image: opsman-image
      config: interpolated-configuration
    params:
      OPSMAN_CONFIG_FILE: sandbox/opsman.yml
    ensure:
      do:
      - put: state
        params: { file: generated-state/state.yml }
  - task: configure-authentication
    image: platform-automation-image
    file: platform-automation-tasks/tasks/configure-authentication.yml
    input_mapping:
      env: interpolated-configuration
      config: interpolated-configuration
    params:
      ENV_FILE: sandbox/env.yml
      AUTH_CONFIG_FILE: sandbox/auth.yml
  - task: configure-director
    image: platform-automation-image
    file: platform-automation-tasks/tasks/configure-director.yml
    input_mapping:
      env: interpolated-configuration
      config: interpolated-configuration
    params:
      ENV_FILE: sandbox/env.yml
      DIRECTOR_CONFIG_FILE: sandbox/director.yml
  - task: apply-changes
    image: platform-automation-image
    file: platform-automation-tasks/tasks/apply-director-changes.yml
    input_mapping:
      env: interpolated-configuration
    params:
      ENV_FILE: sandbox/env.yml

Set the pipeline, but leave it paused. Many of the inputs necessary for this job to function are not yet authored.

download-product config files¶

This time around, the download-product task is used to download products from the blobstore on gcs, not from the Tanzu Network.

Review the download-product config file documentation
Click on the GCS tab
Two gcs-specific properties must be added to the config files you drafted in the assemble the products lab: gcs-project-id, and gcs-service-account-json
Navigate to the ~/workspace/platform-config/download-product and edit opsman.yml to include these new fields. Be sure to reference the service account key from credhub. I.e. the value for the service account json should be set to ((json_key)). Otherwise we would compromise access to our blobstore the moment the file is pushed to github.

`create-vm`¶

The main inputs to the create-vm task are:

the Ops Manager image from the gcs blobstore, fetched in the prior step,
a configuration file named opsman.yml, yet to be written, documented here.

This configuration file will originate from the configuration repository and will need to be interpolated.
a state file (yaml).

Its output is a generated state file. That file captures the name of the VM created in GCP. It will serve as an input for a subsequent job, that of replacing the Ops Manager VM during upgrades.

`opsman.yml`¶

Terraform created a number of resources for this Ops Manager VM, including an external IP address, and an ingress firewall rule, which requires that we tag the VM with the network tag sandbox-ops-manager. Can you locate the firewall rule in question? What network tag is specified?

Navigate to ~/workspace/paving.
Study the contents of the template ci/configuration/gcp/ops-manager.yml.

Make a correction to the template, as shown by this diff output:

@@ -4,7 +4,7 @@ opsman-configuration:
    boot_disk_size: 100
    custom_cpu: 4
    custom_memory: 16
-    gcp_service_account: ((service_account_key))
+    gcp_service_account: ((ops_manager_service_account_key))
    project: ((project))
    public_ip: ((ops_manager_public_ip))
    region: ((region))

Figure out how to use om interpolate to generate a complete opsman.yml file from this template (hint: the variables referenced are all pas terraform outputs).

Problem: The generated file contains secrets. Those secrets must not be checked in to the git repository.

Navigate to ~/workspace/platform-config
Create a new subdirectory named sandbox.

All configuration files pertaining to the sandbox foundation will reside there.
Move the generated opsman.yml file to this subdirectory.
Grab the value of the json key gcp_service_account and put it into CredHub under the name /concourse/sandbox/opsman-service-account-key
Edit out the ops manager service account key value and replace it with a variable reference ((opsman-service-account-key))
Double-check that no sensitive credentials remain in opsman.yml and commit and push the new file to github.

The state file¶

One choice for a place to keep the state file is a versioned gcs bucket.

Create a bucket in GCP (single region is fine). Name it something like your-first-name-state-bucket

Enable versioning on the bucket

gsutil versioning set on gs://your-bucket-name-goes-here/

Author an empty yaml file named state.yml
```
---
# empty
```
Move state.yml to the bucket you just created (hint: use gsutil mv).
In your pipeline, locate the state resource, and set its bucket field to the name of the state bucket from step 1.

All inputs for the create-vm task are accounted for.

`configure-authentication`¶

After an Ops Manager VM is created, the next step is to configure authentication: selecting a username, password, and decryption passphrase.

All three come from a file canonically named auth.yml documented here.

Author your auth.yml file in platform-config alongside opsman.yml
Set the username to admin, but parameterize the other two values. Use the variable names ((opsman-password)) and ((opsman-decryption-passphrase))
Use the credhub generate command to generate a 12-character long password for the ops manager password, and the default 30-character long password for the passphrase.

The next input is the env.yml file, used for targeting the Ops Manager with the om cli. The syntax for env.yml is documented here.

Author your env.yml file in platform-config alongside auth.yml
Fill in the user name, and references to the ops manager password and decryption passphrase.
Set skip-ssl-validation to true
Set the target to your Ops Manager URL

One way to look up the url is to query gcp for the dns entries in your hosted zone:
```
gcloud dns record-sets list --zone=enter-your-hosted-zone-name-here
```
The hostname should match the pattern opsmanager.sandbox.your-environment-name.pal4pe.com.

Commit and push these new configuration files to github.

`configure-director`¶

The next file configures the bosh director.

Read about it here.

Generate this file from a template:

navigate to ~/workspace/paving
Edit the template ci/configuration/gcp.director.yml

Insert the two highlighted lines as shown here

properties-configuration:
director_configuration:
    ntp_servers_string: 169.254.169.254
    post_deploy_enabled: true
    resurrector_enabled: true
iaas_configuration:
    auth_json: ((opsman-service-account-key))
    default_deployment_tag: sandbox-vms

use the om interpolate command to interpolate the template against the variables in your pas terraform output
move the file to ~/workspace/platform-config/sandbox
remove the secret for the key auth_json and replace it with a reference to the variable that you placed in credhub.
double check that no secrets reside in director.yml
commit and push your changes to github

A few more secrets¶

Review your pipeline once more.

Certain pieces of information are missing from credhub: config-repo-key, config-repo-uri, json_key, bucket, and pivnet-token.

Technically they already exist in credhub under /concourse/main but they're needed by this pipeline which uses the prefix /concourse/sandbox.

Put these same secrets in credhub under /concourse/sandbox

One practical way to go about this is to:

Export the concourse main credentials to a file

credhub export --path /concourse/main --file credentials.yml

Edit the file and remove all other credentials (leave in the ones your need)
Replace all references to /main/ with /sandbox/
Import the modified file.

Run the job¶

Unpause the pipeline and trigger the job install-opsman.

Troubleshoot as necessary to make the job pass.

Two new VMs will be present in your gcp project: an Ops Manager and a bosh director.

Setup CLI access to the sandbox Ops Manager¶

Create a new directory ~/workspace/sandbox
Create a .envrc file similar to the one in ~/workspace/opsman-director
Edit the values of each environment variable. Set them to those of the sandbox Ops Manager.
Run direnv allow.

When in the ~/workspace/sandbox subdirectory, the om cli is configured to communicate with the sandbox ops manager.

Verify this.

Run:

om deployed-products

The output should list p-bosh (the bosh director) as the sole deployed product.

Export Installation¶

Add a second job to export the Ops Manager and Bosh Director installation. This step is analogous to manually clicking on the Export Installation Settings button from the Ops Manager's settings.

Use platform automation's export-installation task.

Add a trigger to run this job daily.

The exported installation zip file should be stored in a gcs bucket.

That is, after the task runs, add a put step to write the downloaded installation file to the gcs bucket.

Use either a versioned bucket (the same one used to store the state file), or a non-versioned bucket (the blobstore) together with a timestamp suffix to distinguish the multiple versions of the installation zip file that will be exported each time the job runs.

Solution

The yaml below shows only newly-added jobs and resources. It is not a complete pipeline.

resources:
- name: installation
  type: gcs
  source:
    bucket: ((state-bucket))
    versioned_file: installation.zip
    json_key: ((json_key))

- name: daily
  type: time
  source: { interval: 24h }

jobs:
- name: export-installation
  plan:
  - in_parallel:
    - get: platform-automation-tasks
      params: { unpack: true }
    - get: platform-automation-image
      params: { unpack: true }
    - get: configuration
    - get: daily
      trigger: true
  - task: interpolate-env-creds
    image: platform-automation-image
    file: platform-automation-tasks/tasks/credhub-interpolate.yml
    params:
      CREDHUB_CLIENT: ((credhub-client))
      CREDHUB_SECRET: ((credhub-secret))
      CREDHUB_SERVER: ((credhub-server))
      CREDHUB_CA_CERT: ((credhub-ca-cert.certificate))
      PREFIX: '/concourse/sandbox'
      SKIP_MISSING: false
    input_mapping:
      files: configuration
    output_mapping:
      interpolated-files: interpolated-configuration
  - task: export-installation
    image: platform-automation-image
    file: platform-automation-tasks/tasks/export-installation.yml
    input_mapping:
      env: interpolated-configuration
    params:
      ENV_FILE: sandbox/env.yml
  - put: installation
    params: { file: installation/installation-*.zip }

Set the pipeline, trigger the job and verify that the zip file is written to your bucket.

Upgrade Ops Manager¶

Given a running ops manager and an exported installation zip file, the steps to upgrade an Ops Manager are:

Delete the Ops Manager VM
Create an image for the new Ops Manager
Create a new Ops Manager VM instance from the new image
Import the installation zip file to restore state

The platform automation upgrade-opsman task first checks to see if a new version of ops manager is available, and then proceeds to perform the above steps.

Write a job named upgrade-opsman.

Similar to install-opsman, this step requires a reference to an opsmanager image, and so make sure to preface it with a download-product step.

The job should trigger each time a new installation is exported.

A new Ops Manager brings with a new bosh director.

Therefore the upgrade-opsman step must be followed by these:

configure-director
apply-director-changes
export installation (applying changes updates the state)

Upgrade Solution