The ability to emit group names instead of group IDs is a preview feature made available by Microsoft and is subject to their terms and conditions around preview features of their services.

Maximo Application Suite on ARO ( Azure Red Hat OpenShift )

IBM Maximo Application Suite (MAS) is a set of applications for asset monitoring, management, predictive maintenance and reliability planning. When combined with Azure Red Hat OpenShift ( ARO ), this frees up your Maximo and operations team to focus on what is important to them ( Maximo ) rather than having to worry about managing and building clusters.

This document outlines how to get quickly get started with ARO and installing Maximo all through automation.

Ansible Automation Platform (AAP) on ARO

Ansible Automation Platform (AAP) is a popular platform for centralizing and managing an organization’s automation content using Ansible as the engine for writing automation code. Prior to deployment, organizations are faced with the decision “where do I want to host this thing?”. In today’s landscape, there are several options between traditional Virtual Machines, running it on OpenShift, or even running it as a managed offering. This walkthrough covers a scenario when a customer wants to run AAP on top of a managed OpenShift offering like Azure Red Hat OpenShift (ARO).

Deploying Advanced Cluster Management and OpenShift Data Foundation for ARO Disaster Recovery

A guide to deploying Advanced Cluster Management (ACM) and OpenShift Data Foundation (ODF) for Azure Red hat OpenShift (ARO) Disaster Recovery

Overview

In today’s fast-paced and data-driven world, ensuring the resilience and availability of your applications and data has never been more critical. The unexpected can happen at any moment, and the ability to recover quickly and efficiently is paramount. That’s where OpenShift Advanced Cluster Management (ACM) and OpenShift Data Foundation (ODF) come into play. In this guide, we will explore the deployment of ACM and ODF for disaster recovery (DR) purposes, empowering you to safeguard your applications and data across multiple clusters.

ARO - Cross Tenant Provisioning

Summary

There may be situations where you want to create an ARO cluster where the organization has a policy which has a central entity that controls things such as encryption keys or networking components. This is desirable in large enterprises due to separation of concerns and limiting areas of control for groups to a small scope. This does present challenges, as those different groups must be able to integrate with one another. Often times, the integration is difficult, complex, and confusing. This document serves as a way to clear up some of the confusion by walking you through scenarios for cross-tenancy in ARO.

Use Azure Blob storage Container Storage Interface (CSI) driver on an ARO cluster

The Azure Blob Storage Container Storage Interface (CSI) is a CSI compliant driver that can be installed to an Azure Red Hat OpenShift (ARO) cluster to manage the lifecycle of Azure Blob storage.

When you use this CSI driver to mount an Azure Blob storage into a pod, it allows you to use blob storage to work with massive amounts of data.

You can refer also to the driver’s documentation here .

Configure an ARO cluster with Azure Files using a private endpoint

Effectively securing your Azure Storage Account requires more than just basic access controls. Azure Private Endpoints provide a powerful layer of protection by establishing a direct, private connection between your virtual network and storage resources—completely bypassing the public internet. This approach not only minimizes your attack surface and the risk of data exfiltration, but also enhances performance through reduced latency, simplifies network architecture, supports compliance efforts, and enables secure hybrid connectivity. It’s a comprehensive solution for protecting your critical cloud data.

Configure Red Hat SSO with Microsoft Entra ID as a Federated Identity Provider

This guide demonstrates how to install and configure Red Hat SSO (Keycloak) into an Azure Red Hat OpenShift (ARO) cluster. It will also also configure the ARO cluster to use the SSO server as a mechanism to login by way of the OIDC protocol. In addition, Red Hat SSO can federate user identities with other identity providers. We will use Azure AD as an additional identity provider to show how this could be done.

Using Azure Container Registry in Private ARO clusters

This guide describes how configure and deploy an Azure Container Registry, limiting the access to the registry and connecting privately from a Private ARO cluster, eliminating exposure from the public internet.

You can limit access to the ACR by assigning virtual network private IP addresses to the registry endpoints and using Azure Private Link .

What to consider when using Azure AD as IDP?

Author: Ricardo Macedo Martins

May 24, 2023

In this guide, we will discuss key considerations when using Azure Active Directory (AAD) as the Identity Provider (IDP) for your ARO or ROSA cluster. Below are some helpful references:

• Configure ARO to Use Azure AD
• Configuring IDP for ROSA, OSD, and ARO

Default Access for All Users in Azure Active Directory

Once you set up AAD as the IDP for your cluster, it’s important to note that by default, all users in your Azure Active Directory instance will have access to the cluster. They can log in using their AAD credentials through the OpenShift Web Console endpoint:

Configure Microsoft Entra ID as an OIDC identity provider for ARO with cli

The steps to add Azure AD as an identity provider for Azure Red Hat OpenShift (ARO) via cli are:

• Prerequisites
  • Have Azure cli installed
  • Login to Azure
• Azure
  • Define needed variables
  • Get oauthCallbackURL
  • Create manifest.json file to configure the Azure Active Directory application
  • Register/create app
  • Add Service Principal for the new app
  • Make Service Principal an Enterprise Application
  • Create the client secret
  • Update the Azure AD application scope permissions
  • Get Tenant ID
• OpenShift
  • Login to OpenShift as kubeadmin
  • Create an OpenShift secret###
  • Apply OpenShift OpenID authentication
  • Wait for authentication operator to roll out
  • Verify login through Azure Active Directory
  • Last steps

Prerequisites

Have Azure cli installed

Follow the Microsoft instuctions: https://docs.microsoft.com/en-us/cli/azure/install-azure-cli

Red Hat Cost Management for Cloud Services

Adopted from Official Documentation for Cost Management Service

Red Hat Cost Management is a software as a service (SaaS) offering available free of charge as part of your Red Hat subscriptions. Cost management helps you monitor and analyze your OpenShift Container Platform and Public cloud costs in order to improve the management of your business.

Some capabilities of cost management are :

• Visualize costs across hybrid cloud infrastructure
• Track cost trends
• Map charges to projects and organizations
• Normalize data and add markups with cost models
• Generate showback and chargeback information

In this document, I will show you how to connect your OpenShift and Cloud provider sources to Cost Management in order to collect cost and usage.

Azure Front Door with ARO ( Azure Red Hat OpenShift )

Securing exposing an Internet facing application with a private ARO Cluster.

When you create a cluster on ARO you have several options in making the cluster public or private. With a public cluster you are allowing Internet traffic to the api and *.apps endpoints. With a private cluster you can make either or both the api and .apps endpoints private.

How can you allow Internet access to an application running on your private cluster where the .apps endpoint is private? This document will guide you through using Azure Frontdoor to expose your applications to the Internet. There are several advantages of this approach, namely your cluster and all the resources in your Azure account can remain private, providing you an extra layer of security. Azure FrontDoor operates at the edge so we are controlling traffic before it even gets into your Azure account. On top of that, Azure FrontDoor also offers WAF and DDoS protection, certificate management and SSL Offloading just to name a few benefits.

Setup a VPN Connection into an ARO Cluster with OpenVPN

When you configure an Azure Red Hat OpenShift (ARO) cluster with a private only configuration, you will need connectivity to this private network in order to access your cluster. This guide will show you how to configute a point-to-site VPN connection so you won’t need to setup and configure Jump Boxes.

Prerequisites

• a private ARO Cluster
• git
• openssl

Create certificates to use for your VPN Connection

There are many ways and methods to create certificates for VPN, the guide below is one of the ways that works well. Note, that whatever method you use, make sure it supports “X509v3 Extended Key Usage”.

Using Cluster Logging Forwarder in ARO with Azure Monitor (<=4.12)

NOTE: These instructions are now only necessary for clusters on verions less than or equal to 4.12. The OpenShift Cluster Logging Operator supports a simplified configuration with Azure Monitor as of verison 5.9, which is available on clusters of version 4.13 or greater. Ideally, clusters should be ugpraded to keep them in support, so that’s a good first step to consider. If you ultimately still need the older procedure, see the setup document here .

Using Cluster Logging Forwarder in ARO with Azure Monitor (>=4.13)

NOTE: Starting from version 5.9, OpenShift Logging supports native forwarding to Azure Monitor and Azure Log Analytics, which is available on clusters running OpenShift 4.13 or higher. Please note that apiVersion was changed from logging.openshift.io/v1 to observability.openshift.io/v1 on OpenShift Logging 6.0, which is the version used on this guide. For clusters running OpenShift 4.12 or earlier, see the legacy setup document here for help with configuration.

If you’re running Azure Red Hat OpenShift (ARO), you may want to be able to view and query the logs the platform and your workloads generate in Azure Monitor. With the release of the Cluster Logging Operator version 5.9, this can be done in a single step with some YAML configuration.

Azure DevOps with Managed OpenShift

Author: Kevin Collins

Last edited: 03/14/2023

Adopted from Hosting an Azure Pipelines Build Agent in OpenShift and Kevin Chung Azure Pipelines OpenShift example

Azure DevOps is a very popular DevOps tool that has a host of features including the ability for developers to create CI/CD pipelines.

In this document, I will show you how to connect your Managed OpenShift Cluster to Azure DevOps end-to-end including running the pipeline build process in the cluster, setting up the OpenShift internal image registry to store the images, and then finally deploy a sample application. To demonstrate the flexibility of Azure DevOps, I will be deploying to a ROSA cluster, however the same procudures will apply to if you choose to deploy to any other OCP Cluster.

Upgrade a disconnected ARO cluster

Background

One of the great features of ARO is that you can create ‘disconnected’ clusters with no connectivity to the Internet. Out of the box, the ARO service mirrors all the code repositories to build OpenShift clusters to Azure Container Registry. This means ARO is built without having to reach out to the Internet as the images to build OpenShift are pulled via the Azure private network.

When you upgrade a cluster, OpenShift needs to call out to the Internet to get an upgrade graph to see what options you have to upgrade the cluster. This of course breaks the concept of having a disconnected cluster. This guide goes through how to upgrade ARO without having the cluster reach out to the Internet and maintaining the disconnected nature of an ARO cluster.

Deploying ARO using azurerm Terraform Provider

Overview

Infrastructure as Code has become one of the most prevalent ways in which to deploy and install code for good reason, especially on the cloud. This lab will use the popular tool Terraform in order to create a clear repeatable process in which to install an Azure Managed Openshift(ARO) cluster and all the required components.

Terraform

Terraform is an open-source IaC tool developed by HashiCorp. It provides a consistent and unified language to describe infrastructure across various cloud providers such as AWS, Azure, Google Cloud, and many others. With Terraform, you can define your infrastructure in code and store it inside of git. This makes it easy to version, share, and reproduce.

Helm Chart to set up extra MachineSets on ARO clusters

Please refer to the The Managed OpenShift Black Belt team maintained Helm chart at here .

Integrating Azure ARC with ARO

This document explain how to integrate ARO cluster with Azure Arc-enabled Kubernetes. When you connect a Kubernetes/OpenShift cluster with Azure Arc, it will:

• Be represented in Azure Resource Manager with a unique ID
• Be placed in an Azure subscription and resource group
• Receive tags just like any otherAzure resource

Azure Arc-enabled Kubernetes supports the following scenarios for connected clusters:

• Connect Kubernetes running outside of Azure for inventory, grouping, and tagging.
• Deploy applications and apply configuration using GitOps-based configuration management.
• View and monitor your clusters using Container Insights.
• Enforce threat protection using Microsoft Defender for Kubernetes.
• Apply policy definitions using Azure Policy for Kubernetes.
• Use Azure Active Directory for authentication and authorization checks on your cluster

Prerequisites

• a public ARO cluster
• azure cli
• oc cli
• An identity (user or service principal) which can be used to log in to Azure CLI and connect your cluster to Azure Arc.

Enable Extensions and Plugins

Install the connectedk8s Azure Cli extension of version >= 1.2.0

Shipping logs and metrics to Azure Blob storage

Azure Red Hat Openshift clusters have built in metrics and logs that can be viewed by both Administrators and Developers via the OpenShift Console. But there are many reasons you might want to store and view these metrics and logs from outside of the cluster.

The OpenShift developers have anticipated this needs and have provided ways to ship both metrics and logs outside of the cluster. In Azure we have the Azure Blob storage service which is perfect for storing the data.

Configure ARO to use Microsoft Entra ID

This guide demonstrates how to configure Azure AD as the cluster identity provider in Azure Red Hat OpenShift. This guide will walk through the creation of an Azure Active Directory (Azure AD) application and configure Azure Red Hat OpenShift (ARO) to authenticate using Azure AD.

This guide will walk through the following steps:

1. Register a new application in Azure AD for authentication.
2. Configure the application registration in Azure AD to include optional claims in tokens.
3. Configure the Azure Red Hat OpenShift (ARO) cluster to use Azure AD as the identity provider.
4. Grant additional permissions to individual users.

Before you Begin

If you are using zsh as your shell (which is the default shell on macOS) you may need to run set -k to get the below commands to run without errors. This is because zsh disables comments in interactive shells from being used .

Configure Microsoft Entra ID as an OIDC identity provider for ROSA/OSD

This guide demonstrates how to configure Azure AD as the cluster identity provider in Red Hat OpenShift Service on AWS (ROSA). This guide will walk through the creation of an Azure Active Directory (Azure AD) application and configure Red Hat OpenShift Service on AWS (ROSA) to authenticate using Azure AD.

This guide will walk through the following steps:

1. Register a new application in Azure AD for authentication.
2. Configure the application registration in Azure AD to include optional and group claims in tokens.
3. Configure the OpenShift cluster to use Azure AD as the identity provider.
4. Grant additional permissions to individual groups.

Before you Begin

Create a set of security groups and assign users by following the Microsoft documentation .

Azure Service Operator V1 in ARO

The Azure Service Operator (ASO) provides Custom Resource Definitions (CRDs) for Azure resources that can be used to create, update, and delete Azure services from an OpenShift cluster.

This example uses ASO V1, which has now been replaced by ASO V2. ASO V2 does not (as of 5/19/2022) yet have an entry in the OCP OperatorHub, but is functional and should be preferred for use, especially if V1 isn’t already installed on a cluster. MOBB has documented the [install of ASO V2 on ROSA]. MOBB has not tested running the two in parallel.

Azure Service Operator V2 in ARO

The Azure Service Operator (ASO) provides Custom Resource Definitions (CRDs) for Azure resources that can be used to create, update, and delete Azure services from an OpenShift cluster.

This example uses ASO V2, which is a replacement for ASO V1. Equivalent documentation for ASO V1 can be found here . For new installs, V2 is recommended. MOBB has not tested running them in parallel.

Prerequisites

• Azure CLI
• An Azure Red Hat OpenShift (ARO) cluster
• The helm CLI tool

Prepare your Azure Account and ARO Cluster

1. Install cert-manager:

   Setting up Quay on an ARO cluster via Console

   

   Red Hat Quay setup on ARO (Azure Openshift)

   A guide to deploying an Azure Red Hat OpenShift Cluster with Red Hat Quay.

   Author: [Kristopher White x Connor Wooley]

   Video Walkthrough

   If you prefer a more visual medium, you can watch [Kristopher White] walk through Quay Registry Storage Setup on YouTube .

   Red Hat Quay Setup

   Backend Storage Setup

   1. Login to Azure

      Adding infrastructure nodes to an ARO cluster

      This document shows how to set up infrastructure nodes in an ARO cluster and move infrastructure related workloads to them. This can help with larger clusters that have resource contention between user workloads and infrastructure workloads such as Prometheus.

      Important note: Infrastructure nodes are billed at the same rates as your existing ARO worker nodes.

      You can find the original (and more detailed) document describing the process for a self-managed OpenShift Container Platform cluster here

      Apply Azure Policy to Azure Red Hat Openshift ( ARO )

      Azure Policy helps to enforce organizational standards and to assess compliance at-scale. Azure Policy supports arc enabled kubernetes cluster with both build-in and custom policies to ensure kubernetes resources are compliant. This article demonstrates how to make Azure Redhat Openshift cluster compliant with azure policy.

      Setting up Quay on an ARO cluster via CLI

      Pre Requisites

      • An ARO cluster
      • oc cli
      • azure cli

      Steps

      Create Azure Resources

      1. Create Storage Account

      2. Create Storage Container

         Note: this command returns a json by default with your keyName and Values, command above specifies yaml

         Accessing the Internal Registry from ARO

         Kevin Collins

         06/28/2022

         One of the advantages of using OpenShift is the internal registry that comes with OpenShfit to build, deploy and manage container images locally. By default, access to the registry is limited to the cluster ( by design ) but can be extended to usage outside of the cluster. This guide will go through the steps required to access the OpenShift Registry on an ARO cluster outside of the cluster.

         Configure ARO with OpenShift Data Foundation

         NOTE: This guide demonstrates how to setup and configure self-managed OpenShift Data Foundation in Internal Mode on an ARO Cluster and test it out.

         Prerequisites

         • An Azure Red Hat OpenShift cluster ( verion 4.10+ )
         • kubectl cli
         • oc cli
         • moreutils (sponge)
         • jq

         Install compute nodes for ODF

         A best practice for optimal performance is to run ODF on dedicated nodes with a minimum of one per zone. In this guide, we will be provisioning 3 additional compute nodes, one per zone. Run the following script to create the additional nodes:

         ARO with Nvidia GPU Workloads

         ARO guide to running Nvidia GPU workloads.

         Prerequisites

         • oc cli
         • Helm
         • jq, moreutils, and gettext package
         • An ARO 4.14 cluster

         Note: If you need to install an ARO cluster, please read our ARO Terraform Install Guide . Please be sure if you’re installing or using an existing ARO cluster that it is 4.14.x or higher.

         Note: Please ensure your ARO cluster was created with a valid pull secret (to verify make sure you can see the Operator Hub in the cluster’s console). If not, you can follow these instructions.

         ARO Custom domain with cert-manager and LetsEncrypt

         ARO guide to deploying an ARO cluster with custom domain and automating certificate management with cert-manager and letsencrypt certificates to manage the *.apps and api endpoints.

         Prerequisites

         • az cli (already installed in Azure Cloud Shell)
         • oc cli
         • jq (already installed in Azure Cloud Shell)
         • OpenShift 4.10+
         • domain name to use (we will create zones for this domain name during this guide)

         We’ll go through this setup using the bash terminal on the Azure Cloud Shell. Be sure to always use the same terminal/session for all commands since we’ll reference environment variables set or created through the steps.

         Trident NetApp operator setup for Azure NetApp files

         Note: This guide a simple “happy path” to show the path of least friction to showcasing how to use NetApp files with Azure Red Hat OpenShift. This may not be the best behavior for any system beyond demonstration purposes.

         Prerequisites

         • An Azure Red Hat OpenShift cluster installed with Service Principal role/credentials.
         • kubectl cli
         • oc cli
         • helm 3 cli
         • Review official trident documentation

         In this guide, you will need service principal and region details. Please have these handy.

         Enable the Managed Upgrade Operator in ARO and schedule Upgrades

         THIS DOCUMENT IS OUTDATED, please reference the official MUO documentation here

         Prerequisites

         • an Azure Red Hat OpenShift cluster

         Get Started

         1. Run this oc command to enable the Managed Upgrade Operator (MUO)

         2. Wait a few moments to ensure the Management Upgrade Operator is ready

            Adding an additional ingress controller to an ARO cluster

            Prerequisites

            • an Azure Red Hat OpenShift cluster
            • a DNS zone that you can easily modify

            Get Started

            1. Create some environment variables

            2. Create a certificate for the ingress controller

            3. Create a secret for the certificate

            4. Create an ingress controller

               Using Group Sync Operator with Azure Active Directory and ROSA

               This guide focuses on how to synchronize Identity Provider (IDP) groups and users after configuring authentication in OpenShift Cluster Manager (OCM). For an IDP configuration example, please reference the Configure Azure AD as an OIDC identity provider for ROSA/OSD guide.

               To set up group synchronization from Azure Active Directory (AD) to ROSA/OSD you must:

               1. Define groups and assign users in Azure AD
               2. Add the required API permissions to the app registration in Azure AD
               3. Install the Group Sync Operator from the OpenShift Operator Hub
               4. Create and configure a new Group Sync instance
               5. Set a synchronization schedule
               6. Test the synchronization process

               Define groups and assign users in Azure AD

               To synchronize groups and users with ROSA/OSD they must exist in Azure AD

               Configuring IDP for ROSA, OSD and ARO

               Red Hat OpenShift on AWS (ROSA) and OpenShift Dedicated (OSD) provide a simple way for the cluster administrator to configure one or more identity providers for their cluster[s] via the OpenShift Cluster Manager (OCM) , while Azure Red Hat OpenShift relies on the internal cluster authentication operator .

               The identity providers available for use are:

               Registering an ARO cluster to OpenShift Cluster Manager

               Registering an ARO cluster to OpenShift Cluster Manager

               ARO clusters do not come connected to OpenShift Cluster Manager by default, because Azure would like customers to specifically opt-in to connections / data sent outside of Azure. This is the case with registering to OpenShift cluster manager, which enables a telemetry service in ARO.

               Prerequisites

               • An Red Hat account. If you have any subscriptions with Red Hat, you will have a Red Hat account. If not, then you can create an account easily at https://cloud.redhat.com .

               Steps

               1. Login to https://console.redhat.com with you Red Hat account.

                  Azure Key Vault CSI on Azure Red Hat OpenShift

                  This document is adapted from the Azure Key Vault CSI Walkthrough specifically to run with Azure Red Hat OpenShift (ARO).

                  Prerequisites

                  1. An ARO cluster
                  2. The AZ CLI (logged in)
                  3. The OC CLI (logged in)
                  4. Helm 3.x CLI

                  Environment Variables

                  1. Run this command to set some environment variables to use throughout

                     Note if you created the cluster from the instructions linked above these will re-use the same environment variables, or default them to openshift and eastus.

                     Shipping logs to Azure Log Analytics

                     This document follows the steps outlined by Microsoft in their documentation

                     Follow docs.

                     Step 4, needs additional command of:

                     to capture resource ID of ARO cluster as well, needed for export in step 6

                     bash enable-monitoring.sh --resource-id $azureAroV4ClusterResourceId --workspace-id $logAnalyticsWorkspaceResourceId works successfully

                     can verify pods starting

                     ARO - Considerations for Disaster Recovery

                     This is a high level overview of disaster recovery options for Azure Red Hat OpenShift. It is not a detailed design, but rather a starting point for a more detailed design.

                     What is Disaster Recovery (DR)

                     Disaster Recovery is an umbrella term that includes the following:

                     1. Backup (and restore!)
                     2. Failover (and failback!)
                     3. High Availability
                     4. Disaster Avoidence

                     The most important part of Disaster Recovery is the “Recovery”. Whatever your DR plan it must be tested and ideally performed on a semi-regular basis.

                     Private ARO Cluster with access via JumpHost

                     A Quickstart guide to deploying a Private Azure Red Hat OpenShift cluster.

                     Once the cluster is running you will need a way to access the private network that ARO is deployed into.

                     Authors: Paul Czarkowski , Ricardo Macedo Martins

                     Using the Egressip Ipam Operator with a Private ARO Cluster

                     This guide is only valid for ARO clusters created on version 4.10 or earlier.

                     Clusters created on version 4.11 and later use OVNKubernetes as their Container Network Interface, and egressip-ipam-operator does not support OVNKubernetes.

                     In addition, please refer here to create a private ARO cluster without using public IP address. This way, you will be using UserDefinedRouting for egress .

                     Federating System and User metrics to Azure Blob storage in Azure Red Hat OpenShift

                     By default Azure Red Hat OpenShift (ARO) stores metrics in Ephemeral volumes, and its advised that users do not change this setting. However its not unreasonable to expect that metrics should be persisted for a set amount of time.

                     This guide shows how to set up Thanos to federate both System and User Workload Metrics to a Thanos gateway that stores the metrics in Azure Blob Container and makes them available via a Grafana instance (managed by the Grafana Operator).

                     Installing Astronomer on a private ARO cluster

                     see here for public clusters.

                     This assumes you’ve already got a private ARO cluster installed. You could also follow the same instructions to create a public Astronomer, just use a regular DNS zone and skip the private parts.

                     A default 3-node cluster is a bit small for Astronomer, If you have a three node cluster you can increase it by updating the replicas count machinesets in the openshift-machine-api namespace.