DevOps Specialist
Domain knowledge for CI/CD pipelines, Docker builds, Helm/K8s deployments, and environment promotion.
Scope: Project-specific DevOps patterns only. For general DevOps workflow and investigation → DevOps Expert agent.
Azure DevOps Organization
| Project |
Purpose |
| Main |
Main development — Backend services, Pipeline-Templates |
| ReleaseManagement |
Change/Test/Release Management — wikis, release process docs, Developer-Handbook |
| IaC |
Infrastructure as Code — Terraform modules, Azure resource provisioning |
Key repos in the Main project:
- My-Backend — main monorepo for all domain services
- Pipeline-Templates — shared pipeline templates (
Deployment/k8s-deployment.yaml, variable templates)
Branching Strategy
GitHub Flow:
master is always deployable
- Feature branches from
master, PRs trigger validation (test-pr + sonar-pr)
- Tags on
master trigger release pipelines
- Preview tags on feature branches deploy to A environment
Pipeline Architecture
Pipeline Types
| Pipeline |
Trigger |
Purpose |
| test-pr |
PR to master |
Unit/integration tests, build validation |
| sonar-pr |
PR to master |
SonarQube code quality analysis |
| release |
Tag on master |
Build Docker images, push to ACR, deploy |
Repository Pipeline Structure
.devops/
├── azure-pipelines.release-<Service>.yml
├── azure-pipelines.sonar-pr.yml
├── azure-pipelines.test-pr.yml
└── templates/
├── template.detect-changes.yml
├── template.release.yml
├── template.sonar-pr.yml
└── template.test-pr.yml
Shared Templates (Pipeline-Templates)
K8s Deployment Template (Deployment/k8s-deployment.yaml) key parameters:
| Parameter |
Purpose |
DockerImageName |
Docker image name |
HelmReleaseName |
Helm release name in cluster |
HelmChartName |
Defaults to deployment-ng () |
HelmChartVersion |
Chart version (e.g., 12.12.0) |
Environment |
Target: A, A2, UAT, UAT2, PAV, PAV2, DEV001 |
Namespace |
Kubernetes namespace |
KubernetesFilePath |
Path to K8s values (default: app/k8s) |
Agent pool: scm-vmss-agentpool-001 (fallback: ubuntu-latest)
Environment Variable Templates
Per-environment variables from k8s-vars-{env}.yaml define: ACR, ConfigEnvironment, ConfigVault, ConfigDecryptionKey, AspNetCoreEnvironment, KubernetesServiceConnection, ConfixDecryptServiceConnectionName, OTEL_EXPORTER_OTLP_ENDPOINT, ELASTIC_APM_URL, ELASTIC_APM_TOKEN, TeamsHookUri.
Environments & Promotion
| Environment |
Purpose |
Deployment |
Approval |
| A / A2 |
Development |
Auto on preview tags / Manual |
None |
| UAT / UAT2 |
Business testing |
Tag-triggered / Manual |
May require approval |
| PAV / PAV2 |
Production |
Tag-triggered / Manual |
Approval gate required |
| DEV001 |
Isolated sandbox |
Manual |
None |
Promotion flow: A → UAT → PAV
Docker & Container Build
Multi-Stage Dockerfile Pattern
FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base
WORKDIR /app
EXPOSE 8080
FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
WORKDIR /src
COPY ["src/<Service>/Host/Host.csproj", "src/<Service>/Host/"]
RUN dotnet restore "src/<Service>/Host/Host.csproj"
COPY . .
WORKDIR "/src/src/<Service>/Host"
RUN dotnet build "Host.csproj" -c Release -o /app/build
FROM build AS publish
RUN dotnet publish "Host.csproj" -c Release -o /app/publish
FROM base AS final
WORKDIR /app
COPY --from=publish /app/publish .
ENTRYPOINT ["dotnet", "Host.dll"]
Image Structure per Service
docker/<service-name>/
├── Dockerfile
└── k8s/
├── A/
│ ├── values.yaml
│ └── appsettings.json # Confix-encrypted
├── UAT/
└── PAV/
ACR
- Images via
$(ACR) variable, tag = $(Build.SourceBranchName) (git tag, SemVer)
- Helm chart registry:
$(HelmRepoName).azurecr.io (OCI-based)
Helm Deployment Flow
- Download Docker artifact from build stage
- Copy K8s manifests + values.yaml
- Confix Decrypt —
dotnet confix decrypt via Azure CLI service connection
- Split
BffContainer from appsettings.json → appsettings.bff.json
- Helm login:
helm registry login $(HelmRepoName).azurecr.io
- Helm pull:
oci://$(HelmRepoName).azurecr.io/deployment-ng
- Helm upgrade
--atomic --timeout=600s --create-namespace with environment values
- Teams notification
Helm Values Set During Deployment
image.name, image.repository=$(ACR), image.tag=$(Build.SourceBranchName)
env.DEPLOYMENT_ENVIRONMENT, env.DEPLOYMENT_VAULT, env.ASPNETCORE_ENVIRONMENT
env.OTEL_EXPORTER_OTLP_ENDPOINT, env.REMOTE_CONFIGURATION_URL
envSecrets.DEPLOYMENT_DECRYPTIONKEY, envSecrets.DEPLOYMENT_SHAREDSECRET
envSecrets.REMOTE_CONFIGURATION_TOKEN
Health Probes
All services: /_health/live (liveness), /_health/ready (readiness)
Versioning & Release
- SemVer:
MAJOR.MINOR.PATCH, preview: MAJOR.MINOR.PATCH-preview.N
- Release tags on
master → full deployment pipeline
- Flow: feature branch → PR → merge → release tag → A (auto) → UAT (approval) → PAV (approval)
Configuration Management
Confix
Encrypts appsettings.json at rest. Pipeline decrypts using $(ConfixDecryptServiceConnectionName).
Azure Key Vault
Secrets via DEPLOYMENT_VAULT + DEPLOYMENT_DECRYPTIONKEY. Managed Identity in production.
Infrastructure as Code
- Terraform for Azure resources (I_IaC project)
- Helm for K8s packages
- ARM Templates for legacy resources
- CCOE manages foundational infra, quarterly compliance reviews
Observability
| Component |
Purpose |
| OpenTelemetry |
Distributed tracing, metrics |
| Elastic APM |
Performance monitoring |
| Structured logging |
ILogger + Serilog → Elasticsearch |
| Health checks |
K8s liveness/readiness probes |
Pipeline Troubleshooting
Build Failures
| Symptom |
Likely Cause |
Resolution |
dotnet restore fails |
NuGet feed auth, version mismatch |
Check nuget.config, Directory.Packages.props |
| Docker build fails |
Missing COPY files, SDK mismatch |
Check paths, global.json |
| Test failures |
Flaky tests, missing infra |
Run locally, check Squadron containers |
Deployment Failures
| Symptom |
Likely Cause |
Resolution |
| Helm upgrade fails |
Chart not found, values.yaml error |
Verify HelmChartVersion, validate YAML |
| Confix decrypt fails |
Service connection permissions |
Check ConfixDecryptServiceConnectionName |
| CrashLoopBackOff |
Config error, missing secrets |
kubectl logs, verify appsettings |
| ImagePullBackOff |
ACR auth, tag not found |
Verify ACR creds, confirm image push |
| Timeout (600s) |
Pod not ready, low resources |
Increase resources in values.yaml |
Pipeline Permission Issues
| Symptom |
Likely Cause |
Resolution |
Authorization failed |
Service connection expired |
Renew ADO service connections |
| Pipeline not triggered |
Tag trigger mismatch |
Verify trigger in pipeline YAML |
| Template not found |
Repo resource ref wrong |
Check resources.repositories branch |
Diagnostic Commands
kubectl get pods -n <ns> -l app=<svc>
kubectl logs -n <ns> -l app=<svc> --tail=100
kubectl describe pod <pod> -n <ns>
kubectl rollout status deployment/<svc> -n <ns>
kubectl rollout undo deployment/<svc> -n <ns>
helm list -n <ns>
helm history <release> -n <ns>
Key File Locations
| File |
Purpose |
.devops/azure-pipelines.release-*.yml |
Release pipelines per service |
.devops/azure-pipelines.test-pr.yml |
PR test pipeline |
.devops/templates/template.*.yml |
Shared pipeline templates |
docker/<service>/Dockerfile |
Docker build |
docker/<service>/k8s/{A,UAT,PAV}/values.yaml |
Helm values per environment |
Directory.Build.props |
Central MSBuild properties |
Directory.Packages.props |
Central NuGet package versions |
global.json |
.NET SDK version pinning |
Key Pipeline Variables
| Variable |
Source |
Purpose |
$(ACR) |
Variable template |
Azure Container Registry URL |
$(ConfigEnvironment) |
Variable template |
Target environment name |
$(ConfigVault) |
Variable template |
Key Vault name |
$(KubernetesServiceConnection) |
Variable template |
K8s service connection |
$(ConfixDecryptServiceConnectionName) |
Variable template |
Confix decrypt service connection |
$(HelmRepoName) |
Variable template |
Helm OCI registry name |
$(TeamsHookUri) |
Variable template |
Teams webhook |
$(Build.SourceBranchName) |
Built-in |
Git tag / branch name (image tag) |
1---2name: devops-specialist3description: Deep domain knowledge for DevOps — Azure DevOps pipeline templates, Docker multi-stage builds, Helm/K8s deployment patterns, environment promotion flow, Confix configuration management, and pipeline troubleshooting. Triggers on: azure-pipelines YAML, Dockerfile, Helm values.yaml, k8s-deployment, Confix decrypt, ACR push, environment promotion A/UAT/PAV, release pipeline, pipeline template, HelmChartVersion, k8s-vars, service connection, pod CrashLoopBackOff, ImagePullBackOff.4---5
6# DevOps Specialist
7
8Domain knowledge for CI/CD pipelines, Docker builds, Helm/K8s deployments, and environment promotion.
9
10> **Scope**: Project-specific DevOps patterns only. For general DevOps workflow and investigation → `DevOps Expert` agent.
11
12## Azure DevOps Organization
13
14<!-- TODO: Replace with your actual Azure DevOps project names and purposes -->
15| Project | Purpose |
16|---|---|
17| **Main** | Main development — Backend services, Pipeline-Templates |
18| **ReleaseManagement** | Change/Test/Release Management — wikis, release process docs, Developer-Handbook |
19| **IaC** | Infrastructure as Code — Terraform modules, Azure resource provisioning |
20
21Key repos in the **Main** project:
22- **My-Backend** — main monorepo for all domain services
23- **Pipeline-Templates** — shared pipeline templates (`Deployment/k8s-deployment.yaml`, variable templates)
24
25## Branching Strategy
26
27GitHub Flow:
28- `master` is always deployable
29- Feature branches from `master`, PRs trigger validation (`test-pr` + `sonar-pr`)
30- Tags on `master` trigger release pipelines
31- Preview tags on feature branches deploy to A environment
32
33## Pipeline Architecture
34
35### Pipeline Types
36
37| Pipeline | Trigger | Purpose |
38|---|---|---|
39| **test-pr** | PR to master | Unit/integration tests, build validation |
40| **sonar-pr** | PR to master | SonarQube code quality analysis |
41| **release** | Tag on master | Build Docker images, push to ACR, deploy |
42
43### Repository Pipeline Structure
44
45```
46.devops/
47├── azure-pipelines.release-<Service>.yml
48├── azure-pipelines.sonar-pr.yml
49├── azure-pipelines.test-pr.yml
50└── templates/
51 ├── template.detect-changes.yml
52 ├── template.release.yml
53 ├── template.sonar-pr.yml
54 └── template.test-pr.yml
55```
56
57### Shared Templates (Pipeline-Templates)
58
59K8s Deployment Template (`Deployment/k8s-deployment.yaml`) key parameters:
60
61| Parameter | Purpose |
62|---|---|
63| `DockerImageName` | Docker image name |
64| `HelmReleaseName` | Helm release name in cluster |
65| `HelmChartName` | Defaults to `deployment-ng` (<!-- TODO: replace with your chart name -->) |
66| `HelmChartVersion` | Chart version (e.g., `12.12.0`) |
67| `Environment` | Target: A, A2, UAT, UAT2, PAV, PAV2, DEV001 |
68| `Namespace` | Kubernetes namespace |
69| `KubernetesFilePath` | Path to K8s values (default: `app/k8s`) |
70
71Agent pool: `scm-vmss-agentpool-001` (fallback: `ubuntu-latest`)
72
73### Environment Variable Templates
74
75Per-environment variables from `k8s-vars-{env}.yaml` define: `ACR`, `ConfigEnvironment`, `ConfigVault`, `ConfigDecryptionKey`, `AspNetCoreEnvironment`, `KubernetesServiceConnection`, `ConfixDecryptServiceConnectionName`, `OTEL_EXPORTER_OTLP_ENDPOINT`, `ELASTIC_APM_URL`, `ELASTIC_APM_TOKEN`, `TeamsHookUri`.
76
77## Environments & Promotion
78
79| Environment | Purpose | Deployment | Approval |
80|---|---|---|---|
81| **A / A2** | Development | Auto on preview tags / Manual | None |
82| **UAT / UAT2** | Business testing | Tag-triggered / Manual | May require approval |
83| **PAV / PAV2** | Production | Tag-triggered / Manual | Approval gate required |
84| **DEV001** | Isolated sandbox | Manual | None |
85
86Promotion flow: `A` → `UAT` → `PAV`
87
88## Docker & Container Build
89
90### Multi-Stage Dockerfile Pattern
91
92```dockerfile
93FROM mcr.microsoft.com/dotnet/aspnet:8.0 AS base
94WORKDIR /app
95EXPOSE 8080
96
97FROM mcr.microsoft.com/dotnet/sdk:8.0 AS build
98WORKDIR /src
99COPY ["src/<Service>/Host/Host.csproj", "src/<Service>/Host/"]
100RUN dotnet restore "src/<Service>/Host/Host.csproj"
101COPY . .
102WORKDIR "/src/src/<Service>/Host"
103RUN dotnet build "Host.csproj" -c Release -o /app/build
104
105FROM build AS publish
106RUN dotnet publish "Host.csproj" -c Release -o /app/publish
107
108FROM base AS final
109WORKDIR /app
110COPY --from=publish /app/publish .
111ENTRYPOINT ["dotnet", "Host.dll"]
112```
113
114### Image Structure per Service
115
116```
117docker/<service-name>/
118├── Dockerfile
119└── k8s/
120 ├── A/
121 │ ├── values.yaml
122 │ └── appsettings.json # Confix-encrypted
123 ├── UAT/
124 └── PAV/
125```
126
127### ACR
128
129- Images via `$(ACR)` variable, tag = `$(Build.SourceBranchName)` (git tag, SemVer)
130- Helm chart registry: `$(HelmRepoName).azurecr.io` (OCI-based)
131
132## Helm Deployment Flow
133
1341. Download Docker artifact from build stage
1352. Copy K8s manifests + values.yaml
1363. **Confix Decrypt** — `dotnet confix decrypt` via Azure CLI service connection
1374. Split `BffContainer` from `appsettings.json` → `appsettings.bff.json`
1385. Helm login: `helm registry login $(HelmRepoName).azurecr.io`
1396. Helm pull: `oci://$(HelmRepoName).azurecr.io/deployment-ng`
1407. **Helm upgrade** `--atomic --timeout=600s --create-namespace` with environment values
1418. Teams notification
142
143### Helm Values Set During Deployment
144
145```
146image.name, image.repository=$(ACR), image.tag=$(Build.SourceBranchName)
147env.DEPLOYMENT_ENVIRONMENT, env.DEPLOYMENT_VAULT, env.ASPNETCORE_ENVIRONMENT
148env.OTEL_EXPORTER_OTLP_ENDPOINT, env.REMOTE_CONFIGURATION_URL
149envSecrets.DEPLOYMENT_DECRYPTIONKEY, envSecrets.DEPLOYMENT_SHAREDSECRET
150envSecrets.REMOTE_CONFIGURATION_TOKEN
151```
152
153### Health Probes
154
155All services: `/_health/live` (liveness), `/_health/ready` (readiness)
156
157## Versioning & Release
158
159- SemVer: `MAJOR.MINOR.PATCH`, preview: `MAJOR.MINOR.PATCH-preview.N`
160- Release tags on `master` → full deployment pipeline
161- Flow: feature branch → PR → merge → release tag → A (auto) → UAT (approval) → PAV (approval)
162
163## Configuration Management
164
165### Confix
166
167Encrypts `appsettings.json` at rest. Pipeline decrypts using `$(ConfixDecryptServiceConnectionName)`.
168
169### Azure Key Vault
170
171Secrets via `DEPLOYMENT_VAULT` + `DEPLOYMENT_DECRYPTIONKEY`. Managed Identity in production.
172
173## Infrastructure as Code
174
175- **Terraform** for Azure resources (I_IaC project)
176- **Helm** for K8s packages
177- **ARM Templates** for legacy resources
178- CCOE manages foundational infra, quarterly compliance reviews
179
180## Observability
181
182| Component | Purpose |
183|---|---|
184| **OpenTelemetry** | Distributed tracing, metrics |
185| **Elastic APM** | Performance monitoring |
186| **Structured logging** | ILogger + Serilog → Elasticsearch |
187| **Health checks** | K8s liveness/readiness probes |
188
189## Pipeline Troubleshooting
190
191### Build Failures
192
193| Symptom | Likely Cause | Resolution |
194|---|---|---|
195| `dotnet restore` fails | NuGet feed auth, version mismatch | Check `nuget.config`, `Directory.Packages.props` |
196| Docker build fails | Missing COPY files, SDK mismatch | Check paths, `global.json` |
197| Test failures | Flaky tests, missing infra | Run locally, check Squadron containers |
198
199### Deployment Failures
200
201| Symptom | Likely Cause | Resolution |
202|---|---|---|
203| Helm upgrade fails | Chart not found, values.yaml error | Verify `HelmChartVersion`, validate YAML |
204| Confix decrypt fails | Service connection permissions | Check `ConfixDecryptServiceConnectionName` |
205| CrashLoopBackOff | Config error, missing secrets | `kubectl logs`, verify appsettings |
206| ImagePullBackOff | ACR auth, tag not found | Verify ACR creds, confirm image push |
207| Timeout (600s) | Pod not ready, low resources | Increase resources in values.yaml |
208
209### Pipeline Permission Issues
210
211| Symptom | Likely Cause | Resolution |
212|---|---|---|
213| `Authorization failed` | Service connection expired | Renew ADO service connections |
214| Pipeline not triggered | Tag trigger mismatch | Verify trigger in pipeline YAML |
215| Template not found | Repo resource ref wrong | Check `resources.repositories` branch |
216
217### Diagnostic Commands
218
219```bash
220kubectl get pods -n <ns> -l app=<svc>
221kubectl logs -n <ns> -l app=<svc> --tail=100
222kubectl describe pod <pod> -n <ns>
223kubectl rollout status deployment/<svc> -n <ns>
224kubectl rollout undo deployment/<svc> -n <ns>
225helm list -n <ns>
226helm history <release> -n <ns>
227```
228
229## Key File Locations
230
231| File | Purpose |
232|---|---|
233| `.devops/azure-pipelines.release-*.yml` | Release pipelines per service |
234| `.devops/azure-pipelines.test-pr.yml` | PR test pipeline |
235| `.devops/templates/template.*.yml` | Shared pipeline templates |
236| `docker/<service>/Dockerfile` | Docker build |
237| `docker/<service>/k8s/{A,UAT,PAV}/values.yaml` | Helm values per environment |
238| `Directory.Build.props` | Central MSBuild properties |
239| `Directory.Packages.props` | Central NuGet package versions |
240| `global.json` | .NET SDK version pinning |
241
242## Key Pipeline Variables
243
244| Variable | Source | Purpose |
245|---|---|---|
246| `$(ACR)` | Variable template | Azure Container Registry URL |
247| `$(ConfigEnvironment)` | Variable template | Target environment name |
248| `$(ConfigVault)` | Variable template | Key Vault name |
249| `$(KubernetesServiceConnection)` | Variable template | K8s service connection |
250| `$(ConfixDecryptServiceConnectionName)` | Variable template | Confix decrypt service connection |
251| `$(HelmRepoName)` | Variable template | Helm OCI registry name |
252| `$(TeamsHookUri)` | Variable template | Teams webhook |
253| `$(Build.SourceBranchName)` | Built-in | Git tag / branch name (image tag) |