[Verify the ZIP checksum]({{RECEIPT_URL}}). Use the ZIP to run the lab: it preserves the complete folder structure, pinned dependencies and executable files. Keep the extracted source separate from your private execution directory.
**Audience:** Intermediate Windows/PowerShell learners. **Duration:** 3–4 hours.
A green pipeline is useful. Knowing which identity ran it, which plan it applied and which state it changed is better. In this workshop you build a small private delivery path, establish a healthy baseline and then break three different parts of that path: federation, state locking and infrastructure consistency.
You diagnose each fault using a healthy control and evidence from Azure, Terraform and GitHub Actions. You repair only the owned fault, repeat the repair safely and remove the complete disposable environment.
**Observed workshop status: {{LIVE_STATUS}}.** {{VALIDATION_SUMMARY}} Expected results below describe acceptance conditions. A result becomes observed evidence only when it has a genuine capture and a matching sanitized report. The cover is conceptual artwork; it is separate from the twenty environment screenshots.
## Block 1 — Scenario and architecture
**Objective.** Follow one change from a Windows operator to a private runner, an Azure identity and a private state blob. Distinguish the operator's bootstrap permissions from the pipeline's deliberately smaller permissions.
The Windows controller creates two disposable resource groups. The Foundation group contains a private Ubuntu runner, outbound NAT, a private state account, monitoring and two user-assigned managed identities. The Workload group contains a small VNet, a subnet and two NSGs. The exercise NSG has a synthetic marker tag; the canary NSG gives you an independent object that must remain unchanged.
```text
Windows operator ── authenticated dispatch ──> private GitHub repository
│ workflow_dispatch, main
▼
private Ubuntu runner
│ │
GitHub OIDC token │ private DNS / HTTPS
▼ ▼
deployment UAMI Blob state backend
│ tfstate/workload.tfstate
▼
Workload RG
exercise NSG + canary
Independent control: separate UAMI, Reader on the Workload RG only
Operator Foundation/Federation state: private local directory, not the blob above
```
The runner has no managed identity, VM public IP or inbound administration route. Bootstrap uses Azure managed Run Command. NAT provides outbound connectivity; it does not make the VM an inbound endpoint. The state account disables public network access and Shared Key authorization. Its Blob private endpoint and private DNS link give the runner an internal route to the backend.
| Boundary | Workshop default | Evidence to inspect |
|---|---|---|
| Foundation network | `10.176.0.0/22`; runner `10.176.0.0/24`; private endpoint `10.176.1.0/24` | Private runner NIC, DNS resolution and state networking |
| Workload network | `10.177.0.0/24`; subnet `10.177.0.0/26` | Reviewed plan and final Azure inventory |
| Deployment identity | Contributor on the new Workload RG; Storage Blob Data Contributor on the exact new state container | Two explicit role scopes, no subscription role |
| Control identity | Reader on the new Workload RG | Fresh read-only login succeeds during the deployment-identity fault |
| Workflow scope | Private repository, `main`, manual dispatch only | Actual claims, workflow ref and run attempt |
| State ownership | Operator Foundation state and runner Workload state remain separate | Exact state paths, backend key and ownership ledger |
**Expected evidence.** A baseline validates a fresh control login, private state access, a no-change workload plan and the correct run marker. It does not infer health from an old successful workflow.
**Recovery.** If an address range overlaps an existing VNet, stop before deployment and adjust the reviewed design consistently. Never attach this workshop to a production state account, runner or resource group.
## Block 2 — Preflight and operating limits
**Objective.** Establish the exact Azure/GitHub context, usable screenshot access and a conservative cost forecast before creating paid resources.
Use Windows with PowerShell **7.6.9**, Azure CLI **2.88.0** and Terraform **1.15.8**. AzureRM is pinned to **5.8.0** with its dependency lockfile. Git, GitHub CLI and OpenSSH client tools are also required. The Ubuntu image and GitHub runner release are resolved and recorded during preflight, including their immutable version or checksum. Do not replace a pinned dependency with `latest` while reproducing a report.
The bootstrap operator needs permission to create the two groups, their resources, scoped role assignments and federated credentials. GitHub authentication needs permission to create and manage the disposable private repository and its workflows/runners. OIDC removes an Azure client secret from pipeline login; bootstrap GitHub authentication and short-lived runner registration tokens still require private handling.
```powershell
# Run from the extracted workshop folder in PowerShell 7.
az login
az account set --subscription ''
gh auth status
$subscriptionId = (az account show --query id -o tsv).Trim()
# Keep state, keys, raw logs and captures outside this source tree and Git.
$private = Join-Path $env:LOCALAPPDATA (
'AzureWorkshopz\TerraformDelivery\' + [guid]::NewGuid().ToString('N'))
$state = Join-Path $private 'run.json'
$lab = Join-Path (Get-Location) 'Invoke-TerraformDeliveryWorkshop.ps1'
& $lab -Stage Preflight -StatePath $state `
-SubscriptionId $subscriptionId -CaptureAccessVerified
```
Set `-CaptureAccessVerified` only after opening the authenticated Azure Portal and GitHub repository/workflow surfaces and proving that a readable CLI capture can be saved. A later display of retained command output is acceptable when its label preserves the original command, execution UTC and source checksum. It must not be described as fresh execution.
Preflight checks the selected tenant/subscription, existing resources, policies, required providers, operator permissions, regional B2s availability/quota, image availability, address overlap and current West Europe rates. Existing resource identifiers and raw results stay in the private execution directory. The initial review must resolve any restrictive policy before deploying; an assigned policy is not proof that its effects are harmless.
**Cost boundary.** The total operational limit is **€20**, including a **€5 cleanup reserve**. Deployment requires a conservative six-hour active estimate of at most **€15** using current regional prices. Begin cleanup by **4½ hours** and target verified absence before **six hours**. This is an operator/controller limit, not an Azure-enforced spending cap. Azure billing can arrive later, and a forecast is not an invoice. If any stage cannot finish within the remaining window, proceed to the exact-owned-resource cleanup path.
**Portal checkpoint.** Confirm that the selected subscription is correct and all three capture surfaces are usable. Read the printed forecast before approving the Foundation plan.
**Expected evidence.** A timestamped passing preflight and a reviewable plan limited to the new disposable scopes. No paid resources exist at the end of Preflight.
**Recovery.** Fix authentication, quota, policy or capture access before continuing. Preserve the same private ledger for this run; use a new private directory for a new run. If an operation is interrupted later, reconcile its recorded Azure operation or GitHub run before using `-Resume` on that operation. Never start a competing controller against the same state.
## Block 3 — Deploy Foundation, observe claims and add federation
**Objective.** Build the private execution path, then create trust from claims actually issued to this repository.
```powershell
& $lab -Stage Plan -Phase Foundation -StatePath $state
# Review the complete printed Foundation plan and its saved checksum.
& $lab -Stage Deploy -Phase Foundation -StatePath $state `
-Execute -ReviewedPlan -AcknowledgeCost
& $lab -Stage Bootstrap -StatePath $state -Execute
```
Bootstrap installs verified tool/runner downloads and creates the private execution repository. Every dispatched job receives a fresh ephemeral runner registration on the same disposable VM. Registration disappears after one job; the VM is **not** reimaged between jobs. This lab runs only its reviewed `main` workflows, with no untrusted pull-request job. Ephemeral registration alone does not provide full host isolation. See [GitHub's self-hosted runner security guidance](https://docs.github.com/en/actions/reference/security/secure-use).
The claims workflow requests an OIDC token and retains only an allowlisted claim report, never the token. Check its issuer, audience, repository identity and `main` ref. The exact `sub` is an input to the next phase. GitHub subject formats can differ for newer repositories, so copying an old `repo:owner/name:ref:...` example is not a valid observation. `id-token: write` permits token issuance; Azure role assignments still determine authorization. See [GitHub OIDC claims](https://docs.github.com/en/actions/reference/security/oidc) and [Azure federation setup](https://learn.microsoft.com/en-us/azure/developer/github/connect-from-azure-openid-connect).
```powershell
& $lab -Stage Plan -Phase Federation -StatePath $state
# Review the two exact federated credentials, including observed issuer/sub/aud.
& $lab -Stage Deploy -Phase Federation -StatePath $state `
-Execute -ReviewedPlan
```
The Federation phase adds exact trust to the deployment and control identities. It does not add a wildcard subject or GitHub Environment. Allow a bounded interval for federation and role propagation, then prove fresh logins. A stale successful token does not validate a newly changed trust relationship.
**Portal checkpoint.** Inspect the deployment identity's resource-group and container scopes, the control identity's Reader scope and the exact federated credential. Inspect state networking and the runner's lack of public/inbound access.
**Expected evidence.** Claims match the new private repository and its `main` workflow; fresh control and deployment logins work; the runner resolves and reaches the private blob endpoint.
**Recovery.** If claims or scopes differ, stop and correct the new owned configuration through a fresh reviewed plan. Do not broaden the subject or assign subscription Contributor to make a test pass. Reconcile any failed initial apply against actual Terraform state before retrying.
## Block 4 — Review a saved plan and apply that exact binary
**Objective.** Establish a healthy delivery baseline and prove that approval identifies one plan artifact, one run attempt and one hash.
```powershell
& $lab -Stage Pipeline -Action Plan -StatePath $state -Execute
```
Open the exact `plan.yml` run. Review its complete human-readable workload diff and safe metadata. The raw binary and full JSON remain in the private artifact/private execution directory: Terraform plan files and `terraform show -json` can contain sensitive values even when terminal output redacts them. Treat repository artifact access accordingly. See [saved Terraform plans](https://developer.hashicorp.com/terraform/cli/commands/plan) and [JSON output sensitivity](https://developer.hashicorp.com/terraform/cli/commands/show).
Copy the exact run ID, run attempt, artifact ID and reviewed SHA256 from this run. The values below are learner inputs, not example evidence.
```powershell
$review = @{
PlanRunId = ''
PlanRunAttempt = 1 # Replace if the reviewed attempt is different.
ArtifactId = ''
PlanSha256 = ''
}
& $lab -Stage Pipeline -Action Apply -StatePath $state `
-Execute -ReviewedPlan @review
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
# Preserve this exact ORIGINAL successful approval for the refusal tests.
$baselineReview = @{} + $review
```
The approval lifetime is **15 minutes**. The private artifact retention is **one day**. These are different controls: an artifact can still exist after it is too old to apply. Wrong hashes, wrong run attempts, mismatched provenance, expired plans and consumed approvals must be rejected. Review the article first; generate a new plan when ready to review and apply promptly.
Apply checks the source/ref/commit, artifact metadata, backend and state bindings, then makes a fresh semantic comparison before applying the **original reviewed binary**. The fresh comparison is a consistency check, not a replacement unreviewed apply plan. These checks do not atomically freeze Azure against other writers. Backend locking protects state operations; an external ARM change can still race with plan/apply. Keep the lab scopes exclusive and validate the resulting state.
There is no protected GitHub Environment approval in this workshop. `-ReviewedPlan` and the exact run/hash inputs record the operator's explicit approval; they do not impersonate an independent GitHub reviewer.
**GitHub checkpoint.** The successful Apply run names the reviewed plan identity. The subsequent Validate run performs fresh identity, state and no-drift checks. Portal workload objects carry this run's marker; the canary is present.
### Prove three distinct approval refusals
Run these immediately after the successful baseline Apply, while the original plan is younger than 15 minutes. Keep the execution repository's `main` commit unchanged throughout all four guard tests. Each `-ExpectedFailure` dispatch must produce the specific recorded refusal **before Terraform starts**. A missing runner or generic failed job does not count.
```powershell
# 1. Same plan, wrong independently supplied review hash.
$wrongHash = @{} + $baselineReview
$firstHex = if ($baselineReview.PlanSha256[0] -eq '0') { '1' } else { '0' }
$wrongHash.PlanSha256 = $firstHex + $baselineReview.PlanSha256.Substring(1)
& $lab -Stage Pipeline -Action Apply -StatePath $state `
-Execute -ReviewedPlan -ExpectedFailure @wrongHash
# Required refusal: REVIEWED_HASH_MISMATCH; terraform_started=false.
# 2. Exact successful plan approval has already been consumed.
& $lab -Stage Pipeline -Action Apply -StatePath $state `
-Execute -ReviewedPlan -ExpectedFailure @baselineReview
# Required refusal: PLAN_ALREADY_CONSUMED; terraform_started=false.
# 3. Use the actual evidence artifact from the SAME plan run, not its plan artifact.
$repo = (Get-Content -LiteralPath $state -Raw | ConvertFrom-Json).repository
$artifactList = gh api "repos/$repo/actions/runs/$($baselineReview.PlanRunId)/artifacts" |
ConvertFrom-Json
$evidenceName = 'evidence-{0}-{1}' -f $baselineReview.PlanRunId, $baselineReview.PlanRunAttempt
$evidenceArtifact = @($artifactList.artifacts | Where-Object name -CEQ $evidenceName)
if ($evidenceArtifact.Count -ne 1) { throw 'Exact evidence artifact not found; reconcile this run.' }
$wrongArtifact = @{} + $baselineReview
$wrongArtifact.ArtifactId = [string]$evidenceArtifact[0].id
& $lab -Stage Pipeline -Action Apply -StatePath $state `
-Execute -ReviewedPlan -ExpectedFailure @wrongArtifact
# Required refusal: ARTIFACT_PROVENANCE_MISMATCH; terraform_started=false.
```
The fourth refusal uses this original plan after genuine expiry in Block 8. Do not edit its timestamps or consumption receipt. Preserve the actual failed-run results and fresh healthy validation; the final acceptance audit binds each refusal to its original source.
**Expected evidence.** Original-binary apply succeeds, fresh control/deployment access is healthy and a fresh workload plan has no changes. A successful historical run or a merely downloadable artifact does not satisfy this baseline.
**Recovery.** If approval expires or anything changes, discard that approval and create/review a new plan. Do not edit metadata, extend a timestamp or regenerate a binary under an old hash. Diagnose an access error before attempting broader permissions.
## Block 5 — Troubleshoot an exact OIDC trust failure
**Objective.** Distinguish a federation failure from runner networking, state access and Azure authorization failures.
Start with a fresh healthy validation. The fault changes only the deployment identity's exact federated subject to a known wrong value. The independent control identity retains its exact trust and Reader scope.
```powershell
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
& $lab -Stage Fault -Scenario Oidc -StatePath $state -Execute
```
Inspect the actual `drill-oidc.yml` job: deployment token exchange must fail for the intended subject mismatch while a fresh control login/read succeeds. This is an expected fault result, not a generic successful workflow claim. A timeout, missing runner, DNS error or expired bootstrap token is not the intended failure.
**Propagation checkpoint.** A successful subject-update PUT does not prove that token exchange sees the change. Microsoft documents cached older FIC data and recommends retries. During rehearsal, the first three fresh attempts did not establish the intended failure; the first, 48 seconds after the PUT, returned `EXPECTED_OIDC_FAILURE_DID_NOT_OCCUR` with control healthy and zero Terraform commands. The fourth fresh attempt produced the intended `AADSTS700213` rejection at **19:09:26 UTC**, after the original **18:47:55 UTC** injection. Fresh repairs then passed at **19:11:43** and **19:14:03 UTC**. Cache delay remains a plausible inference rather than a proved cause. Reconcile an attempt before bounded fresh retries of the unchanged configuration; use `Fault -Resume` only for its recorded pending operation. Elapsed time alone is not proof. [Microsoft FIC propagation guidance](https://learn.microsoft.com/en-us/entra/workload-id/workload-identity-federation-considerations#time-for-federated-credential-changes-to-propagate).
```powershell
& $lab -Stage Repair -Scenario Oidc -StatePath $state -Execute
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
# Repeat repair to prove that restoring the intended subject is safe.
& $lab -Stage Repair -Scenario Oidc -StatePath $state -Execute
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
```
**Checkpoint.** Compare the failing exchange with the allowlisted observed claims and the actual deployment FIC. Preserve the error category and UTC, not the JWT. A failed token exchange occurs before a backend request, so a missing storage log row is not proof of missing federation evidence.
**Expected evidence.** Fresh control succeeds during the fault; deployment exchange fails for the subject mismatch; both recover after exact restoration; repeated repair and validation pass.
**Recovery.** Restore the recorded owned subject. Never use a wildcard or alter the control identity. If there is an interrupted operation, reconcile that exact run before repeating repair.
## Block 6 — Diagnose an owned state-lock conflict
**Objective.** Observe contention without corrupting state or interfering with an unknown owner's lease.
The AzureRM backend uses Blob-native locking. This exercise acquires a **60-second finite lease** on the exact owned state blob, runs Terraform with a short lock timeout, verifies the lock conflict and releases only the recorded lease ID. A fresh control read remains healthy. See [AzureRM backend locking](https://developer.hashicorp.com/terraform/language/backend/azurerm) and [Terraform lock recovery](https://developer.hashicorp.com/terraform/language/state/locking).
```powershell
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
& $lab -Stage Exercise -Scenario StateLock -StatePath $state -Execute
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
```
One bounded drill includes the post-release HEAD availability check used as repeat-safety evidence. It does not need a second lease acquisition for acceptance. **Optional:** repeat the full `Exercise -Scenario StateLock` only for additional rehearsal, after a fresh healthy validation and within the cleanup window; that separate drill acquires its own new finite lease.
**Checkpoint.** Inspect the `drill-lease.yml` report and corroborating `StorageBlobLogs`. The report must show the intended Terraform lock conflict, owned-lease release and successful post-release operation. Compare the state content hash and serial before/after: the drill must not change state. Logs are a corroborating data-plane record, not a substitute for the Terraform diagnostic or hash check.
**Read the pinned backend behavior.** Terraform 1.15.8 reads blob properties and checks the locked lease status **before** calling AcquireLease. It can therefore reject a locked state locally without sending a conflicting lease-acquisition request. Do not require HTTP 409 or a `LeaseAlreadyPresent` response as the only acceptable symptom. [Terraform 1.15.8 Azure backend source](https://raw.githubusercontent.com/hashicorp/terraform/v1.15.8/internal/backend/remote-state/azure/client.go).
**Accepted live evidence.** A fresh drill retained native Terraform exit 1 at **19:57:04 UTC**, with a five-second lock timeout and these verified diagnostic excerpts:
```text
Error: Error acquiring the state lock
Error message: state blob is already locked
blob metadata "terraformlockid" was empty
```
The metadata message accompanies the workshop's externally injected lease; it is not a separate fault to repair. The owned lease was released, the recovery plan returned 0, HEAD confirmed availability and state SHA256/serial remained unchanged at **3 → 3**. Eleven matching Blob log rows corroborated acquisition (201), held-lease property reads (200) and release (200), with no required HTTP 409. Original journal/stderr bytes were retrieved privately and their hashes matched the workflow's sanitized diagnostic record.
**Evidence-retention history.** The earlier **19:17:24 UTC** workflow reported serial **2 → 2** and recovery, but the pinned runner cleared `RUNNER_TEMP` before the outer collector retrieved its journals. Those bytes are unrecoverable and were not used as the accepted native diagnostic. The correction moved **only private journals and command streams** to a protected job directory under `RUNNER_WORKSPACE`; authentication, Terraform data and plans remained in `RUNNER_TEMP`. The fresh Validate and StateLock rehearsal then supplied the verified replacement proof above. The four real approval refusals completed before that source change and retain their original per-run artifact/commit provenance; an old approval cannot authorize a new commit.
**Expected evidence.** The short-timeout operation fails because the owned lease is held; the next operation succeeds after its release; state hash/serial remain unchanged. The post-release HEAD reports availability without releasing an unknown lease. A second full exercise is optional.
**Recovery.** Let the finite lease expire if the controller loses its response, then reconcile the recorded operation. Do not use `-lock=false`, break an unidentified lease or force-unlock a lock you do not own. A resource-group cleanup is safer than improvising against another backend.
## Block 7 — Detect and repair one out-of-band tag drift
**Objective.** Prove that drift belongs to one owned object and repair it through a newly reviewed saved plan.
```powershell
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
& $lab -Stage Fault -Scenario Drift -StatePath $state -Execute
```
The Fault stage changes only the exercise NSG's synthetic marker tag from `baseline-` to `fault-` through Azure and dispatches the corrective Plan. The canary NSG must remain unchanged. That plan must refresh the remote object and show the intended tag correction without a replacement, deletion or unrelated network change. Its NSG security rule is unchanged; the tag is the exercise marker. A no-change plan against stale state is not adequate diagnosis.
```powershell
# Review the NEW repair plan, then replace every review value.
$repairReview = @{
PlanRunId = ''
PlanRunAttempt = 1
ArtifactId = ''
PlanSha256 = ''
}
& $lab -Stage Repair -Scenario Drift -StatePath $state `
-Execute -ReviewedPlan @repairReview
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
# Repeated Repair must confirm the restored marker without introducing drift.
& $lab -Stage Repair -Scenario Drift -StatePath $state -Execute
& $lab -Stage Pipeline -Action Validate -StatePath $state -Execute
```
**Checkpoint.** Compare the reviewed repair diff, successful saved-binary Apply and fresh no-change validation. The canary's recorded properties must match its baseline throughout.
**Observed recovery.** The one-NSG marker correction was applied through the reviewed binary. First and repeated repair checks passed, and the independent evidence report records repeated recovery at **19:33:31 UTC** with the canary preserved. The final workload and Foundation validations also reported zero drift.
**Expected evidence.** Only the exercise tag drifts, only that tag is corrected and a new validation reports no drift. Repeated repair is safe. Do not use a direct Azure tag write as the learner's repair: the point is to review and apply the Terraform correction.
**Recovery.** If the diff includes another change, stop and explain it before approving a plan. A changed source commit, consumed approval or expired plan requires a new plan/review cycle.
## Block 8 — Validate, correlate evidence and clean up twice
**Objective.** Finish with fresh checks, remove both layers in dependency order and prove that unrelated resources survived.
First prove the fourth guard using the original baseline plan after genuine expiry. Blocks 5–7 normally provide enough elapsed time. If necessary, the loop below waits in bounded 30-second steps, stops at the exercise deadline and never changes metadata or the clock.
```powershell
$metadataPath = Join-Path $private (
'evidence/run-{0}-{1}/plans-{0}-{1}/plan-metadata.json' -f
$baselineReview.PlanRunId, $baselineReview.PlanRunAttempt)
$baselineMetadata = Get-Content -LiteralPath $metadataPath -Raw | ConvertFrom-Json
$expiredAfter = [DateTimeOffset]::Parse($baselineMetadata.created_utc).AddMinutes(15).AddSeconds(5)
$runLedger = Get-Content -LiteralPath $state -Raw | ConvertFrom-Json
$exerciseStop = [DateTimeOffset]::Parse($runLedger.exercise_stop_utc)
if ($expiredAfter -gt [DateTimeOffset]::UtcNow.AddMinutes(16) -or $expiredAfter -ge $exerciseStop) {
throw 'No safe expiry-test window; reconcile time metadata or begin cleanup.'
}
while ([DateTimeOffset]::UtcNow -lt $expiredAfter) {
if ([DateTimeOffset]::UtcNow -ge $exerciseStop) { throw 'Begin cleanup now.' }
Start-Sleep -Seconds 30
}
& $lab -Stage Pipeline -Action Apply -StatePath $state `
-Execute -ReviewedPlan -ExpectedFailure @baselineReview
# Required refusal: PLAN_EXPIRED_OR_FUTURE from genuine AGE, not a future timestamp.
# terraform_started=false; original plan metadata remains unchanged.
```
```powershell
& $lab -Stage Validate -StatePath $state -Execute -CollectLogs
```
The final report must reconcile identity checks, every intended fault, first and repeated recoveries, owned-lease invariants, canary preservation and a fresh no-change workload plan. Corroborate lease operations/statuses by the exact owned blob, OAuth principal and execution window; retain the actual Storage correlation IDs. A Storage correlation ID by itself does not prove a shared runner client-request ID. The optional request-ID query is stronger only when an original runner report retains that same identifier. Corroborate the successful drift-repair NSG write by its exact resource, principal and bounded repair window. Do not label every `403` as the OIDC fault or assume that a denied token exchange reaches Storage. Queries in [the KQL folder](kql/blob-operations.kql) project only documented fields; the final report must state which matching rows were actually observed.
**Observed final validation.** All 23 required live checks were satisfied across 39 recorded check events, and all nine independent evidence observations passed. The four approval refusals were genuine failures before Terraform started. Final workload validation was observed at **20:00:50 UTC** and the Foundation no-drift check passed at **20:01:59 UTC**. Eight sanitized retained CLI reports are included under `docs/evidence/retained-cli/`; they are original-execution records displayed later, not commands rerun during capture.
Create and review a **destroy plan** while the private runner, backend and federation still exist. Copy its own four approval values. A normal create/update plan cannot authorize destroy.
```powershell
& $lab -Stage Pipeline -Action Plan -StatePath $state -Execute -DestroyPlan
$destroyReview = @{
PlanRunId = ''
PlanRunAttempt = 1
ArtifactId = ''
PlanSha256 = ''
}
& $lab -Stage Pipeline -Action Destroy -StatePath $state `
-Execute -ReviewedPlan @destroyReview
# Preview exact owned Foundation cleanup, then approve that preview.
& $lab -Stage Cleanup -StatePath $state
& $lab -Stage Cleanup -StatePath $state -Execute -ReviewedPlan
# Repeated cleanup must verify absence and be a safe no-op.
& $lab -Stage Cleanup -StatePath $state -Execute -ReviewedPlan
& $lab -Stage Export -StatePath $state
```
The controller retains its exact ownership ledger. Cleanup removes the owned workload before deleting the private backend/runner Foundation. It disables workflows and removes ephemeral runner registrations and owned runtime artifacts. The **repository and its workflow history remain private**; Azure trust and paid Azure resources are removed. Public downloadable source files are separate from that repository. Never delete an existing repository or resource group merely because its name resembles the lab prefix.
**Observed cleanup.** The reviewed workload Destroy succeeded. Exact owned-group absence passed at **20:26:29 UTC** and repeated cleanup at **20:33:15 UTC**; all **27** unrelated baseline resources were preserved. Both checks finished less than 2½hours after paid deployment began at **18:09:29 UTC**, inside the original six-hour operating window. The retained private repository has seven disabled workflows and zero runner registrations or runtime artifacts. Its history remains private; Azure trust and paid workshop resources were removed.
**Portal/GitHub checkpoint.** The exact two resource groups are absent; owned artifacts and runner registrations are removed; repository workflows are disabled. The repository and its workflow history remain private. Independent repeated Azure absence checks agree. Compare the recorded unrelated-resource baseline by identifiers privately and publish only sanitized counts/results.
**Expected evidence.** Final acceptance passes, both owned scopes are absent twice, no unrelated baseline resource is missing and cleanup finishes inside the operational window. The export excludes raw plans/state, JWTs, registration tokens, keys, raw logs and unredacted images.
**Recovery.** If pipeline destroy fails or the runner/backend is unavailable, use the controller's reviewed exact-owned-resource fallback to prevent cost overrun. Reconcile actual remaining resources before another attempt. A failed cleanup is outstanding work; a command returning success without independent absence evidence is not completion.
## Block 9 — Learning pack
Try the [independent challenge](#challenge) before opening the [hints](#hints) or [worked solution](#solution). The [eight knowledge questions](#knowledge-check) include explanations, not just answer keys. The full learning pack is embedded below and also included as editable files in the download.
Use the troubleshooting table to choose the next discriminating check. Diagnose the layer that failed before changing a permission, retrying an apply or releasing a lock.
| Symptom | Discriminating evidence | Safe next step |
|---|---|---|
| Deployment login fails; fresh control login succeeds | Actual issuer/sub/aud versus deployment FIC; token-exchange diagnostic | Restore only the exact owned subject |
| Deployment exchange still succeeds soon after subject corruption | Confirm the recorded PUT, actual unchanged fault subject and fresh control; inspect propagation timing | Reconcile the attempt and retry within bounds; the intended fault remains unproved |
| Both identities fail before an Azure request | Runner registration, workflow ref, outbound HTTPS and actual GitHub run | Reconcile runner/workflow prerequisites |
| Login succeeds; backend access fails | Private DNS, Blob reachability, container role and Shared Key/public-network settings | Fix only the owned route or scope |
| Plan fails with a lock conflict | Finite owned lease, exact state key, Terraform lock report | Release only your recorded lease or await its expiry |
| Apply refuses an available artifact | Run/attempt/artifact/hash, age, consumed marker and source/state bindings | Create and review a new plan |
| Plan corrects an unexpected object | Fresh remote diff and canary baseline | Stop approval and investigate the extra change |
| Final logs have no matching request | Workflow UTC/observed request ID and log-ingestion delay | Poll within bounds; report missing correlation honestly |
| Cleanup reports success but a group exists | Fresh exact-name absence query and owned inventory | Reconcile and repeat precise cleanup |
### Source notes and further reading
The public [source map](docs/source-map.md) links the official references for OIDC, saved plans, private backends, runners and `StorageBlobLogs`. [Capture guidance](docs/screenshot-guide.md) separates retained real command output from fresh execution. [Validation notes](docs/validation.md) separate local tests from observed Azure/GitHub results. The source is editable in [article.md](article.md); executable entry point: [Invoke-TerraformDeliveryWorkshop.ps1](Invoke-TerraformDeliveryWorkshop.ps1).