# Azure Golden Images Deep Dive: VM Image Builder, Compute Gallery, Trusted Launch, and Version Rollout

Build and validate a patched Windows Server 2025 golden-image pipeline with Azure VM Image Builder, Compute Gallery replication, Trusted Launch, immutable version rollout, and rollback.

![Neon Azure Workshopz architecture showing Windows Server flowing through Image Builder patching and validation into gallery versions, regional replicas, a Trusted Launch VM, and rollback.](azure-golden-images-deep-dive-cover.png)

## Introduction

A golden image is useful only when its lifecycle is repeatable. Building a VM manually, taking a snapshot and calling it a baseline leaves too many unanswered questions: Which patches were present? Did the service actually start? Can the artifact be reproduced? How is a failed release stopped? What does rollback mean after a VM is already running?

This workshop builds the complete lifecycle. Azure VM Image Builder starts from the Gen2 Windows Server 2025 Azure Edition marketplace image, installs IIS, writes a release manifest, installs current non-preview updates, performs controlled restarts and runs an in-VM validator. A failed validator stops distribution. Successful releases become exact `1.0.0` and `2.0.0` versions in Azure Compute Gallery, each replicated with Standard_ZRS storage to West Europe and North Europe.

A private validation VM proves the artifact without a public IP or inbound RDP. It uses Trusted Launch, Secure Boot and vTPM. The lab replaces that VM first with exact version `2.0.0`, then excludes v2 from `latest` and replaces it again from exact `1.0.0`.

The word *replace* matters: standalone Azure VMs are not downgraded in place. Rollback is an immutable deployment operation using a known-good image version.

PowerShell 7 orchestrates Azure CLI and `az rest`; Azure Portal and sanitized command-line evidence make each boundary visible.

## Timing and level

- Level: intermediate to advanced.
- Full run: approximately 2.5–4 hours.
- Each Windows Update build: usually 45–90 minutes.
- Cross-region gallery replication: commonly another 10–30 minutes per release.
- Validation, screenshots and cleanup: about 30–45 minutes.

The long-running stages poll Azure with bounded timeouts. You can rerun completed stages safely from the private runtime state.

## Cost warning

This lab creates Image Builder compute and temporary storage, container resources in two staging groups, two cross-region gallery versions, Standard_ZRS replicas, a managed identity, networking resources and a `Standard_D2s_v5` validation VM. Image building, storage, cross-region transfer and the validation VM can generate charges even during a short workshop.

Capture the evidence, verify the download and live page, then run cleanup. Deallocating the validation VM stops compute charges but does not remove gallery, disk or replica storage charges.

## What you will prove

By the end of the lab, you will have evidence that:

- preflight checked authentication, permissions, API support, image capability, Image Builder regions, quota, provider state and collisions without mutating Azure;
- `Microsoft.VirtualMachineImages` was registered only by the Foundation stage;
- the image definition is generalized, Gen2 and `TrustedLaunchSupported`;
- Image Builder completed both in-VM validators before distribution;
- versions `1.0.0` and `2.0.0` replicated successfully to West Europe and North Europe;
- the validation VM has no public IP or inbound RDP;
- Trusted Launch, Secure Boot and vTPM are enabled;
- v1 starts IIS and reports release `1.0.0`;
- a replacement VM proves the v2-only configuration marker and updated health content;
- v2 is excluded from `latest` before rollback;
- the final replacement VM is pinned to exact version `1.0.0`, and the v2-only marker is absent.

## Architecture

```text
Windows Server 2025 marketplace image (Gen2)
                    |
                    v
         Azure VM Image Builder
  patch -> customize -> restart -> validate
       |                              |
       +-- failure stops distribution-+
                    |
                    v
 Azure Compute Gallery: win2025Golden
       |                         |
   version 1.0.0             version 2.0.0
       |                         |
       +---- West Europe + North Europe replicas
                    |
                    v
 private Trusted Launch VM, no public IP / no RDP
        1.0.0 -> replace with 2.0.0 -> replace with 1.0.0
```

Image Builder uses its managed temporary networking in the two staging resource groups. Custom builder networking is deliberately omitted so the lab stays focused on image lifecycle. The validation VM is separate and explicitly private.

## Bundle layout

```text
Start-AzureGoldenImageWorkshop.ps1
Get-AzureGoldenImageEvidence.ps1
Remove-AzureGoldenImageWorkshop.ps1
GoldenImageWorkshop.Common.ps1
windows/Configure-GoldenImage.ps1
windows/Finalize-GoldenImage.ps1
windows/Validate-GoldenImage.ps1
windows/Test-GoldenImage.ps1
templates/image-template.schema.json
evidence/
screenshots/
```

Raw identifiers, passwords, deployment output and Run Command responses remain under `work/azure-golden-images-deep-dive` and are never packaged.

## 1. Run the read-only preflight

```powershell
$subscriptionId = az account show --query id --output tsv

./Start-AzureGoldenImageWorkshop.ps1 `
  -SubscriptionId $subscriptionId `
  -Location westeurope `
  -ReplicaLocation northeurope `
  -BuildVmSize Standard_D2s_v5 `
  -ValidationVmSize Standard_D2s_v5 `
  -Stage Preflight
```

Preflight does not register providers or create resources. It reads the provider baseline so cleanup can restore it later, and fails closed if one of the fixed workshop resource groups already exists without matching private state.

The tested subscription exposed Windows Server 2025 Azure Edition as a Gen2, Trusted Launch-capable image and had sufficient regional and D-family capacity. `Microsoft.VirtualMachineImages` was initially unregistered; the check recorded that fact without changing it.

![Sanitized Windows Terminal preflight showing the successful read-only checks and zero Azure mutations.](screenshots/01-terminal-read-only-preflight.png)

## 2. Create the scoped foundation

```powershell
./Start-AzureGoldenImageWorkshop.ps1 `
  -SubscriptionId $subscriptionId `
  -Stage Foundation
```

Foundation registers Image Builder, creates the three tagged resource groups and a user-assigned identity. Contributor is granted only at those three resource-group scopes. The identity does not receive subscription-wide Contributor.

The stage also creates the private validation VNet, NSG and subnet. `defaultOutboundAccess=false` prevents accidental implicit egress for the validation VM, while Run Command remains the management and proof channel.

## 3. Create the gallery contract

```powershell
./Start-AzureGoldenImageWorkshop.ps1 -SubscriptionId $subscriptionId -Stage Gallery
```

The gallery definition is the compatibility contract for every version:

```powershell
az sig image-definition create `
  --resource-group rg-golden-images-deepdive-we `
  --gallery-name galWorkshopzGoldenWe `
  --gallery-image-definition win2025Golden `
  --publisher BeyondCloudWithChriz `
  --offer WindowsServer2025 `
  --sku golden-trustedlaunch `
  --os-type Windows `
  --os-state Generalized `
  --hyper-v-generation V2 `
  --features SecurityType=TrustedLaunchSupported
```

![Azure Portal image definition showing Gen2 and Trusted Launch support.](screenshots/04-gallery-trusted-launch-definition.png)

## 4. Build and replicate version 1.0.0

```powershell
./Start-AzureGoldenImageWorkshop.ps1 -SubscriptionId $subscriptionId -Stage BuildV1
```

The template uses the Windows Server 2025 Azure Edition marketplace source and a `Standard_D2s_v5` builder. Its ordered customizers install IIS, create the release marker, restart, run Windows Update while excluding previews, restart again, write the patch/build manifest and prepare the generalized image.

The validation section is a release gate:

```powershell
$actual = (Get-Content C:\ProgramData\Workshopz\image-version.txt -Raw).Trim()
if ($actual -ne '1.0.0') { throw 'Unexpected release marker.' }
if (-not (Get-WindowsFeature Web-Server).Installed) { throw 'IIS is missing.' }
$response = Invoke-WebRequest http://localhost/ -UseBasicParsing
if ($response.StatusCode -ne 200) { throw 'IIS health validation failed.' }
```

`continueDistributeOnFailure=false` means a validation failure prevents the gallery version from being published. A passing build sends exact version `1.0.0` to West Europe and North Europe with one Standard_ZRS replica in each region.

![Azure Portal Image Builder v1 template showing its source, customizers and validator.](screenshots/02-image-builder-v1-template.png)

![Azure Portal showing the successful v1 Image Builder run.](screenshots/03-image-builder-v1-success.png)

![Azure Portal version 1.0.0 replication status for West Europe and North Europe.](screenshots/05-v1-regional-replication.png)

## 5. Deploy and prove version 1.0.0

```powershell
./Start-AzureGoldenImageWorkshop.ps1 -SubscriptionId $subscriptionId -Stage DeployV1
```

The VM is created from the exact gallery version ID, not from `latest`. It has a private NIC, Standard SSD OS disk and this security profile:

```powershell
az vm create `
  --name vm-golden-current `
  --image '<gallery-definition>/versions/1.0.0' `
  --security-type TrustedLaunch `
  --enable-secure-boot true `
  --enable-vtpm true
```

Run Command verifies the release marker, IIS feature, local HTTP 200, versioned content, patch inventory and baseline revision. It then deallocates the VM while the next image builds.

![Azure Portal validation VM configuration showing Trusted Launch, Secure Boot, vTPM and no public IP.](screenshots/06-v1-vm-trusted-launch.png)

![Sanitized command-line proof for version 1.0.0, IIS and patch manifest.](screenshots/07-v1-command-proof.png)

## 6. Build version 2.0.0 and roll forward

```powershell
./Start-AzureGoldenImageWorkshop.ps1 -SubscriptionId $subscriptionId -Stage BuildV2
./Start-AzureGoldenImageWorkshop.ps1 -SubscriptionId $subscriptionId -Stage RolloutV2
```

The v2 template retains the patched baseline and adds `baseline-v2.marker`, baseline revision `2`, and health content containing release `2.0.0`. It passes the same validator and replicates to the same two regions.

Rollout deletes only the tagged validation VM and its disposable NIC/disk, then recreates `vm-golden-current` from exact version `2.0.0`. If the v2 VM fails runtime validation, the script automatically recovers the validation machine from v1 before returning the failure.

![Azure Portal showing successful version 2.0.0 build and regional replication.](screenshots/08-v2-build-replication.png)

![Sanitized command-line proof showing the version 2.0.0 rollout and baseline revision 2.](screenshots/09-v2-rollout-proof.png)

## 7. Perform the immutable rollback

```powershell
./Start-AzureGoldenImageWorkshop.ps1 -SubscriptionId $subscriptionId -Stage RollbackV1
```

Rollback first marks v2 `excludeFromLatest=true` and explicitly keeps v1 eligible. It then replaces the validation VM from the exact v1 version. Acceptance requires version `1.0.0`, baseline revision `1`, HTTP 200, and no v2-only marker.

This is not an in-place downgrade. The old operating-system disk is disposable and the VM is reconstructed from a known image artifact. In production, application data must be externalized and rollout usually belongs in VM Scale Sets, deployment rings or another immutable orchestration layer.

![Azure Portal gallery versions showing v2 excluded from latest while v1 remains eligible.](screenshots/10-gallery-latest-exclusion.png)

![Sanitized command-line rollback proof showing exact version 1.0.0 restored and the v2 marker absent.](screenshots/11-rollback-v1-proof.png)

## 8. Validate and export sanitized evidence

```powershell
./Start-AzureGoldenImageWorkshop.ps1 -SubscriptionId $subscriptionId -Stage Validate
./Get-AzureGoldenImageEvidence.ps1 -SubscriptionId $subscriptionId
```

Final validation rereads the gallery definition, both versions, VM, NIC and security profile. It requires two successful versions, v1 eligible for `latest`, v2 excluded, final exact version `1.0.0`, Trusted Launch, Secure Boot, vTPM, no public IP and no inbound RDP rule.

The evidence exporter publishes only safe booleans, version labels, regions, counts and size names. It scans the result for GUIDs, Azure IDs, IP addresses and secret-like text before accepting it.

## Troubleshooting map

| Symptom | Check | Likely fix |
|---|---|---|
| Template cannot be created | Provider state and identity scopes | Run Foundation; confirm Contributor exists on all three workshop groups |
| Build waits in `Building` | Staging group resources | Allow Windows Update to finish within the 180-minute timeout |
| Validation prevents distribution | Build logs and in-VM validator | Fix the release marker, IIS health, or manifest; rerun the same version stage |
| Replication never completes | Target region status | Confirm the gallery version has West Europe and North Europe in its publishing profile |
| VM creation rejects the image | Security contract | Use a Gen2 definition with `TrustedLaunchSupported` and create the VM with Trusted Launch |
| Windows VM name is rejected | Guest computer name length | Keep the Azure resource name `vm-golden-current`, but pass a separate guest computer name of 15 characters or fewer |
| Run Command cannot find files | Finalizer order | Ensure the build manifest is written before Image Builder generalizes the image |
| Rollout proof shows v1 | Exact version ID | Recreate the VM from the `/versions/2.0.0` resource ID, not `latest` |
| Rollback still shows v2 marker | VM was reused | Delete and replace the VM and OS disk; do not attempt an in-place downgrade |

## Production hardening

- Pin every rollout to an exact image version and promote artifacts between environments.
- Treat image templates, customization scripts and validation rules as reviewed source code.
- Use private Image Builder networking when the build must reach private package sources or when egress inspection is required.
- Put secrets in managed identities and Key Vault; never bake them into image layers.
- Add vulnerability scanning, Defender for Cloud and organization-specific compliance validators before distribution.
- Use deployment rings and health gates for VM Scale Sets or orchestrated replacement fleets.
- Define version retention, replica storage, regional availability and end-of-life policy.
- Monitor Image Builder failures, gallery replication health and consumption of excluded images.
- Keep application state outside disposable OS disks so rollback remains safe.
- Test restore and rollback regularly; an untested image version is not a recovery strategy.

## Evidence and cleanup

The ZIP includes PowerShell orchestration, image-template source, Windows customization and validation scripts, sanitized evidence, screenshots, the article, manifest and publishing brief. It excludes the private runtime directory.

After the post, ZIP and Workshops tile are verified:

```powershell
./Remove-AzureGoldenImageWorkshop.ps1 `
  -SubscriptionId $subscriptionId `
  -Force
```

Cleanup verifies the exact three resource-group names and both ownership tags before deletion. It removes Image Builder templates and their temporary resources with the groups. The Image Builder provider is returned to its original unregistered baseline only when there are no non-workshop templates.

## Final result

You now have a reproducible Windows Server 2025 image pipeline with two immutable versions, validation before distribution, cross-region replicas, a private Trusted Launch consumer, a proven forward replacement and a proven rollback replacement.

The most important result is the release contract: build, validate, publish, deploy exact version, observe, and replace. A golden image is no longer a manual snapshot; it is a tested artifact with a lifecycle.

## Microsoft references

- [Azure VM Image Builder overview](https://learn.microsoft.com/en-us/azure/virtual-machines/image-builder-overview)
- [VM Image Builder JSON template reference](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/image-builder-json)
- [Azure Compute Gallery](https://learn.microsoft.com/en-us/azure/virtual-machines/azure-compute-gallery)
- [Trusted Launch for Azure virtual machines](https://learn.microsoft.com/en-us/azure/virtual-machines/trusted-launch)
- [VM Image Builder best practices](https://learn.microsoft.com/en-us/azure/virtual-machines/image-builder-best-practices)
- [VM Image Builder networking options](https://learn.microsoft.com/en-us/azure/virtual-machines/linux/image-builder-networking)
