Artifact Signing in CI/CD¶
Artifact signing is a crucial practice in the CI/CD process that ensures the integrity and authenticity of software artifacts. By signing artifacts, developers can provide assurance that the artifacts have not been tampered with and originate from a trusted source.
How Artifact Signing Works¶
Artifact signing uses cryptographic techniques to create a verifiable link between the software artifact and its creator or publisher. The process involves generating a digital signature that can be verified by recipients.
sequenceDiagram
participant Publisher as Publisher
participant Artifact as Software Artifact
participant Consumer as Consumer/Verifier
Note over Publisher: Private Key
Note over Consumer: Public Key
Publisher->>Publisher: Generate hash of artifact
Publisher->>Publisher: Encrypt hash with private key (sign)
Publisher->>Consumer: Distribute artifact + signature
Consumer->>Consumer: Generate hash of received artifact
Consumer->>Consumer: Decrypt signature with publisher's public key
Consumer->>Consumer: Compare hashes
Note over Consumer: If hashes match,<br/>artifact is authentic and unmodified
Cryptographic Foundations¶
Artifact signing relies on asymmetric cryptography:
- The private key is used by the publisher to create the signature
- The public key is distributed to users to verify the signature
- The cryptographic hash provides a unique fingerprint of the artifact
- The signature is the encrypted hash that proves authenticity
Importance of Artifact Signing¶
-
Integrity: Signing artifacts helps verify that the code has not been altered after it was built. This is essential for maintaining the trustworthiness of the software.
-
Authenticity: A signature confirms the identity of the publisher, allowing users to verify that the artifact comes from a legitimate source.
-
Compliance: Many regulatory frameworks require the use of signing to ensure that software components meet security standards.
-
Non-Repudiation: Signed artifacts cannot be disavowed by the signer, providing accountability in the software supply chain.
-
Trust Establishment: Signatures create verifiable trust chains from developer to end user across the entire software distribution process.
SolarWinds Attack Implications
The 2020 SolarWinds attack demonstrated the critical importance of artifact signing. Attackers were able to inject malicious code into the build process, which was then signed with SolarWinds' legitimate code-signing certificate. While the attack still succeeded, proper verification of build provenance and more granular signing could have helped detect the compromise.
Types of Artifacts to Sign¶
Organizations should implement signing for all artifact types across the development and deployment pipeline:
Artifact Type | Examples | Signing Method | Verification Point |
---|---|---|---|
Source Code | Git commits, tags | GPG/SSH signatures | Code review, CI pipeline |
Packages | NPM, PyPI, Maven | Package-specific, sigstore | Installation, CI pipeline |
Containers | Docker images, OCI | Cosign, Notary | Before deployment |
Executables | .exe, .msi, .appx | Authenticode, codesign | Installation time |
Firmware | BIOS, device firmware | Hardware-backed signing | Boot time |
Infrastructure | Terraform, CloudFormation | Checksums, OIDC | Before application |
Key Signing Technologies¶
Code Signing Certificates¶
X.509 certificates issued by Certificate Authorities (CAs) that verify the identity of the publisher:
# Windows Authenticode signing example
signtool sign /a /tr http://timestamp.digicert.com /td sha256 /fd sha256 app.exe
# macOS code signing example
codesign --sign "Developer ID Application: Your Company (ABCDE12345)" app.dmg
Sigstore/Cosign¶
Sigstore is an open-source project aimed at improving software supply chain integrity by making code signing more accessible:
# Sign a container with key-based signing
cosign sign --key cosign.key myregistry.io/myimage:v1.0
# Sign with keyless method using OIDC identity
cosign sign myregistry.io/myimage:v1.0
# Verify a signed container image
cosign verify myregistry.io/myimage:v1.0
GPG/PGP Signing¶
Used primarily for signing Git commits, tags, and some package formats:
# Configure Git to use GPG signing
git config --global user.signingkey YOUR_GPG_KEY_ID
git config --global commit.gpgsign true
# Sign a tag
git tag -s v1.0.0 -m "Release v1.0.0"
# Verify a tag
git verify-tag v1.0.0
Best Practices for Artifact Signing¶
Key Management¶
- Use Hardware Security Modules (HSMs) for storing production signing keys
- Implement key rotation schedules to limit the impact of key compromise
- Establish clear custody chains for access to signing keys
- Create separate signing keys for different environments (development, staging, production)
- Implement multi-party signing ceremonies for critical releases
Process Integration¶
- Automate the Signing Process: Integrate artifact signing into the CI/CD pipeline to ensure that all artifacts are signed consistently and automatically.
# Example GitHub Action for signing a container
name: Sign Container
on:
push:
tags: ['v*']
jobs:
sign:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- name: Set up Cosign
uses: sigstore/cosign-installer@main
- name: Log in to registry
run: echo "${{ secrets.REGISTRY_PASSWORD }}" | docker login -u ${{ secrets.REGISTRY_USERNAME }} --password-stdin
- name: Build and push image
run: |
docker build -t registry.example.com/app:${{ github.ref_name }} .
docker push registry.example.com/app:${{ github.ref_name }}
- name: Sign the container
run: |
cosign sign --key cosign.key \
registry.example.com/app:${{ github.ref_name }}
env:
COSIGN_PASSWORD: ${{ secrets.COSIGN_PASSWORD }}
- Generate and verify SBOMs during the signing process to document artifact contents
- Implement signing policies that define which artifacts require signatures
- Use attestations to document build environment and process characteristics
Verification and Enforcement¶
flowchart TD
classDef verify fill:#6a89cc, stroke:#333, stroke-width:1px, color:white
classDef gate fill:#ff6b6b, stroke:#333, stroke-width:1px, color:white
classDef action fill:#42b983, stroke:#333, stroke-width:1px
A[Artifact Arrives] --> B{Signature Present?}:::verify
B -->|No| C[Reject & Alert]:::gate
B -->|Yes| D{Valid Signature?}:::verify
D -->|No| C
D -->|Yes| E{Trusted Publisher?}:::verify
E -->|No| C
E -->|Yes| F{Matches SBOM?}:::verify
F -->|No| C
F -->|Yes| G[Deploy Artifact]:::action
classDef gate stroke-dasharray: 5 5
- Verify Signatures: Implement checks to verify signatures before deploying artifacts to production environments.
- Enforce signature verification at runtime for critical components
- Implement trust policies that define which signatures are trusted
- Use binary transparency frameworks to log and verify artifact publication
- Monitor and alert on signature verification failures
Implementation Approaches¶
Centralized Signing Service¶
Creating a dedicated signing service that handles all artifact signing requests:
flowchart TD
subgraph "CI/CD Pipeline"
A[Build Job] --> B[Create Artifact]
B --> C[Request Signing]
end
subgraph "Signing Service"
D[Authentication] --> E[Policy Check]
E --> F[Sign Artifact]
F --> G[Log Operation]
end
subgraph "Key Management"
H[(HSM)] --> F
end
C --> D
F --> I[Return Signed Artifact]
I --> J[Deploy]
- Provides centralized logging and audit trail
- Enforces consistent signing policies
- Allows better key protection
- Enables multi-party authorization
Keyless Signing with OIDC¶
Modern approaches like Sigstore's keyless signing remove the need to manage signing keys:
- The developer authenticates with an identity provider (e.g., GitHub, Google)
- A short-lived signing key is generated
- The artifact is signed with this ephemeral key
- The signature, certificate, and artifact hash are recorded in a transparent log
- The ephemeral private key is discarded
This approach provides: - Verification tied to developer identity - Elimination of long-term key management issues - Public transparency logs of all signed artifacts
Regulatory and Compliance Considerations¶
Many industries and governments now mandate code signing:
- US Executive Order 14028 emphasizes software supply chain security including signed releases
- EU Cyber Resilience Act proposes requirements for signed software
- SLSA Framework (Supply chain Levels for Software Artifacts) includes signing as a key component
- PCI DSS requires code signing for payment application development
Conclusion¶
Artifact signing is no longer optional but a fundamental control for ensuring software supply chain integrity. By implementing comprehensive signing across all artifact types and enforcing verification, organizations can dramatically reduce the risk of supply chain compromise.
Implementation Checklist
- ✅ Identify all artifact types in your environment
- ✅ Select appropriate signing technologies for each
- ✅ Implement secure key management practices
- ✅ Integrate signing into CI/CD pipelines
- ✅ Configure verification at deployment and runtime
- ✅ Monitor for signing or verification failures
- ✅ Document procedures for key rotation and recovery
- ✅ Train developers and operators on signing importance
Advanced Signing Architectures¶
As organizations scale their signing operations, more sophisticated architectures become necessary to maintain security while enabling efficient processes.
Enterprise Signing Service Architecture¶
flowchart TB
classDef blue fill:#3498db, stroke:#333, stroke-width:1px, color:white
classDef green fill:#2ecc71, stroke:#333, stroke-width:1px
classDef yellow fill:#f1c40f, stroke:#333, stroke-width:1px
classDef red fill:#e74c3c, stroke:#333, stroke-width:1px, color:white
subgraph "Authentication Layer"
A[OIDC Provider]:::blue
B[MFA Service]:::blue
end
subgraph "Signing Service"
C[API Gateway]:::green
D[Authorization Service]:::green
E[Signing Orchestrator]:::green
F[Signing Worker]:::green
G[Audit Logger]:::green
end
subgraph "Key Management"
H[HSM Cluster]:::red
I[Key Management Service]:::red
J[Certificate Authority]:::red
end
subgraph "Monitoring & Compliance"
K[Log Analytics]:::yellow
L[Metrics Dashboard]:::yellow
M[Alerting System]:::yellow
N[Compliance Reporter]:::yellow
end
A --> C
B --> C
C --> D
D --> E
E --> F
F --> H
F --> G
H --> I
I --> J
G --> K
K --> L
K --> M
K --> N
Key components of an enterprise signing service:
- Authentication & Authorization
- Multi-factor authentication for signing requests
- Fine-grained authorization policies
-
Just-in-time access for critical operations
-
Signing Infrastructure
- Redundant, geographically distributed signing capabilities
- Automated key rotation
-
Support for multiple signing technologies and formats
-
Monitoring & Compliance
- Real-time monitoring of signing operations
- Anomaly detection for unusual signing patterns
-
Compliance reporting and attestation
-
Hardware Security
- FIPS 140-⅔ validated HSMs
- Physical security controls
- Tamper-evident seals and monitoring
Multi-Party Signing for Critical Releases¶
For high-value software releases, multi-party signing provides additional security through consensus requirements:
sequenceDiagram
participant Dev as Developer
participant Build as Build System
participant Sign1 as Signer #1 (Engineering)
participant Sign2 as Signer #2 (Security)
participant Sign3 as Signer #3 (Release Manager)
participant Repo as Artifact Repository
Dev->>Build: Initiate release build
Build->>Build: Create release artifacts
Build->>Sign1: Request signature #1
Note over Sign1: Verify build provenance<br/>Check automated test results
Sign1->>Build: Provide signature #1
Build->>Sign2: Request signature #2
Note over Sign2: Verify security scan results<br/>Check for vulnerabilities
Sign2->>Build: Provide signature #2
Build->>Sign3: Request signature #3
Note over Sign3: Verify release documentation<br/>Ensure compliance requirements met
Sign3->>Build: Provide signature #3
Build->>Build: Combine signatures
Build->>Repo: Publish fully signed artifact
Note over Repo: All 3 signatures required<br/>for artifact to be valid
Implementation considerations:
- Quorum Requirements: Define the minimum number of signatures required (e.g., 2 of 3, 3 of 5)
- Role Separation: Ensure signers represent different organizational roles
- Timeout Handling: Define procedures for when signatures aren't provided in time
- Emergency Procedures: Create break-glass procedures for urgent situations
- Audit Trail: Maintain comprehensive records of signature requests and approvals
Signing Implementation Patterns¶
Pattern 1: Developer and CI/CD Signing¶
This pattern combines developer-level signing (for code commits) with CI/CD-level signing (for artifacts):
graph TD
classDef dev fill:#3498db, stroke:#333, stroke-width:1px, color:white
classDef cicd fill:#2ecc71, stroke:#333, stroke-width:1px
classDef prod fill:#e74c3c, stroke:#333, stroke-width:1px, color:white
A[Developer Signing]:::dev --> A1[Git Commit Signing]
A --> A2[Developer Certificates]
B[CI/CD Signing]:::cicd --> B1[Build Attestation]
B --> B2[Artifact Signing]
B --> B3[Container Signing]
C[Production Release Signing]:::prod --> C1[Code Signing Certificate]
C --> C2[Multi-Person Ceremony]
C --> C3[Hardware Security Module]
A1 --> D[Verified Code Source]
B1 --> E[Verified Build Process]
B2 --> F[Verified Artifacts]
C1 --> G[Verified Production Release]
D --> E
E --> F
F --> G
Pattern 2: Keyless Signing with OIDC¶
Modern approaches like Sigstore's keyless signing remove the need to manage signing keys:
# Example GitHub Action for keyless signing with Sigstore
name: Keyless Signing
on:
push:
tags:
- 'v*'
jobs:
sign:
runs-on: ubuntu-latest
permissions:
contents: read
id-token: write # Required for keyless signing
steps:
- uses: actions/checkout@v3
- name: Build artifact
run: make build
- name: Install cosign
uses: sigstore/cosign-installer@main
- name: Sign the artifact with GitHub OIDC
run: |
# Keyless signing with GitHub identity
cosign sign-blob \
--output-signature artifact.sig \
--output-certificate artifact.crt \
artifact.bin
- name: Verify signature
run: |
cosign verify-blob \
--signature artifact.sig \
--certificate artifact.crt \
artifact.bin
- name: Upload signed artifact
uses: actions/upload-artifact@v3
with:
name: signed-artifact
path: |
artifact.bin
artifact.sig
artifact.crt
This approach provides: - Verification tied to developer identity - Elimination of long-term key management issues - Public transparency logs of all signed artifacts - Short-lived certificates that reduce key compromise risk
Pattern 3: Threshold Signing with Multiple Parties¶
For high-security environments, threshold cryptography allows signing to occur only when multiple parties agree:
# Conceptual implementation of threshold signing (t-of-n)
from shamir import split_secret, combine_shares
import cryptography.signatures as sig
# Setup: Split signing key into n shares, t required to reconstruct
def setup_threshold_signing(key, t, n):
"""Split a signing key into n shares, requiring t shares to reconstruct."""
return split_secret(key, t, n)
# Signing: Requires t participants to contribute their shares
def threshold_sign(message, shares, t):
"""Sign a message using at least t shares of the key."""
# Collect at least t shares from participants
collected_shares = [] # In practice, these would come from different people
# Reconstruct the private key using t shares
if len(collected_shares) >= t:
private_key = combine_shares(collected_shares)
# Sign the message with the reconstructed key
return sig.sign(message, private_key)
else:
raise ValueError(f"Need at least {t} shares, only {len(collected_shares)} provided")
Implementation by Artifact Type¶
Source Code Signing¶
# Set up Git for commit signing
git config --global user.signingkey <YOUR_GPG_KEY_ID>
git config --global commit.gpgsign true
# Sign a commit
git commit -S -m "Add important feature"
# Sign a tag
git tag -s v1.0.0 -m "Release version 1.0.0"
# Verify a signed tag
git verify-tag v1.0.0
# Configure GitHub to require signed commits for protected branches
# (This would be done through GitHub repository settings)
Package Signing¶
NPM Packages:
# Generate keys for npm package signing
npm login
npm profile enable-2fa auth-and-writes
# Publish a signed npm package
npm publish --provenance
# Verify npm package
npm audit signatures <package-name>
Python Packages:
# Generate GPG key for package signing
gpg --gen-key
# Configure PyPI token
poetry config pypi-token.pypi <your-token>
# Build and sign package
poetry build
gpg --detach-sign -a dist/*.tar.gz
gpg --detach-sign -a dist/*.whl
# Upload signed package
poetry publish
Container Signing with Cosign¶
# Generate a keypair
cosign generate-key-pair
# Sign a container image
cosign sign --key cosign.key \
registry.example.com/myapp:1.0.0
# Verify a container image
cosign verify --key cosign.pub \
registry.example.com/myapp:1.0.0
# Add attestations (SBOM)
cosign attest --key cosign.key \
--type cyclonedx \
--predicate sbom.json \
registry.example.com/myapp:1.0.0
Binary/Executable Signing¶
Windows Authenticode:
# Sign an executable with a certificate from the Windows certificate store
$cert = Get-ChildItem -Path Cert:\CurrentUser\My -CodeSigningCert
Set-AuthenticodeSignature -FilePath .\MyApp.exe -Certificate $cert
# Sign with timestamp
Set-AuthenticodeSignature -FilePath .\MyApp.exe -Certificate $cert -TimestampServer "http://timestamp.digicert.com"
# Verify signature
Get-AuthenticodeSignature -FilePath .\MyApp.exe | Format-List
macOS Signing:
# Sign a macOS application
codesign --sign "Developer ID Application: Your Company (ABCDE12345)" \
--options runtime \
--timestamp \
./MyApp.app
# Verify macOS signature
codesign --verify --verbose ./MyApp.app
Verification Workflows¶
Runtime Verification¶
flowchart TD
classDef start fill:#3498db, stroke:#333, stroke-width:1px, color:white
classDef decision fill:#f1c40f, stroke:#333, stroke-width:1px
classDef action fill:#2ecc71, stroke:#333, stroke-width:1px
classDef stop fill:#e74c3c, stroke:#333, stroke-width:1px, color:white
A[Start Deployment Process]:::start --> B{Is Artifact Signed?}:::decision
B -->|No| C[Reject Artifact]:::stop
B -->|Yes| D{Signature Valid?}:::decision
D -->|No| C
D -->|Yes| E{Signed by Trusted Entity?}:::decision
E -->|No| C
E -->|Yes| F{SBOM Available?}:::decision
F -->|No| G[Generate Warning]:::action
F -->|Yes| H{SBOM Verified?}:::decision
H -->|No| G
G --> I[Deploy with Warning]:::action
H -->|Yes| J[Deploy Artifact]:::action
CI/CD Integration for Verification¶
GitHub Actions Example:
name: Verify Artifact Signatures
on:
workflow_dispatch:
inputs:
image:
description: 'Image to verify'
required: true
type: string
jobs:
verify:
runs-on: ubuntu-latest
steps:
- name: Install cosign
uses: sigstore/cosign-installer@main
- name: Verify container signature
run: |
# Verify using public key
cosign verify \
--key cosign.pub \
${{ github.event.inputs.image }}
- name: Verify attestations
run: |
# Verify SBOM attestation
cosign verify-attestation \
--key cosign.pub \
--type cyclonedx \
${{ github.event.inputs.image }}
# Extract SBOM for vulnerability scanning
cosign download attestation \
--key cosign.pub \
${{ github.event.inputs.image }} > attestation.json
- name: Check SBOM for vulnerabilities
run: |
# Extract SBOM from attestation
cat attestation.json | jq -r '.payload' | base64 -d | jq -r '.predicate' > sbom.json
# Use Grype to scan SBOM for vulnerabilities
grype sbom:sbom.json --fail-on high
Custom Signature Verification Tools¶
Organizations often need to build custom tools to verify signatures across their supply chain:
# Example Python script for batch signature verification
#!/usr/bin/env python3
import os
import sys
import subprocess
import json
def verify_container_signature(image_name, public_key=None):
"""Verify a container signature using cosign."""
cmd = ["cosign", "verify"]
if public_key:
cmd.extend(["--key", public_key])
else:
# Use keyless verification
cmd.append("--certificate-identity-regexp=.*")
cmd.append("--certificate-oidc-issuer=https://token.actions.githubusercontent.com")
cmd.append(image_name)
try:
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
return True, result.stdout
except subprocess.CalledProcessError as e:
return False, e.stderr
def verify_artifact_signature(artifact_path, signature_path, public_key):
"""Verify a file signature using OpenSSL."""
cmd = [
"openssl", "dgst", "-sha256",
"-verify", public_key,
"-signature", signature_path,
artifact_path
]
try:
result = subprocess.run(cmd, capture_output=True, text=True, check=True)
return True, result.stdout
except subprocess.CalledProcessError as e:
return False, e.stderr
def batch_verify(manifest_path):
"""Verify all artifacts listed in a manifest file."""
with open(manifest_path, 'r') as f:
manifest = json.load(f)
results = {
"verified": [],
"failed": []
}
for item in manifest["artifacts"]:
artifact_type = item["type"]
if artifact_type == "container":
success, output = verify_container_signature(
item["image"],
item.get("public_key")
)
elif artifact_type == "file":
success, output = verify_artifact_signature(
item["path"],
item["signature"],
item["public_key"]
)
else:
success = False
output = f"Unsupported artifact type: {artifact_type}"
if success:
results["verified"].append({
"artifact": item,
"message": "Verified successfully"
})
else:
results["failed"].append({
"artifact": item,
"error": output
})
return results
if __name__ == "__main__":
if len(sys.argv) < 2:
print("Usage: verify_signatures.py <manifest_path>")
sys.exit(1)
results = batch_verify(sys.argv[1])
# Print results
print(f"✅ Verified {len(results['verified'])} artifacts")
if results["failed"]:
print(f"❌ Failed to verify {len(results['failed'])} artifacts:")
for fail in results["failed"]:
print(f" - {fail['artifact'].get('path', fail['artifact'].get('image'))}: {fail['error']}")
sys.exit(1)
sys.exit(0)
Signing Maturity Model¶
Organizations can assess their artifact signing maturity using this framework:
Level | Description | Key Characteristics | Example Practices |
---|---|---|---|
Level 1: Ad-hoc | Basic, inconsistent signing | Manual signing processes, limited verification | Occasional code signing, no enforcement |
Level 2: Repeatable | Standardized signing for critical components | Documented processes, basic key management | CI/CD signing for releases, manual key management |
Level 3: Defined | Comprehensive signing across artifacts | Automated signing in pipelines, formal key management | All artifacts signed, HSM for key storage, verification gates |
Level 4: Managed | Measured, optimized signing operations | Metrics-driven, integrated with risk management | Multi-party signing, anomaly detection, signing SLAs |
Level 5: Optimizing | Advanced signing infrastructure | Continuous improvement, innovative approaches | Threshold cryptography, automated key rotation, transparency logs |
Common Challenges and Solutions¶
Key Management Challenges¶
Challenge | Solution | Implementation Example |
---|---|---|
Key Protection | Use Hardware Security Modules | Configure YubiHSM for code signing with restricted access |
Key Distribution | Implement secure key distribution protocols | Use Hashicorp Vault for dynamic key issuance with short TTL |
Key Rotation | Automate key rotation procedures | Script to generate, distribute, and update keys quarterly |
Recovery Planning | Establish backup and recovery processes | Air-gapped backup HSMs with m-of-n recovery scheme |
Access Control | Implement robust access policies | Require MFA and quorum approval for signing operations |
Scaling Challenges¶
Challenge | Solution | Implementation Example |
---|---|---|
High Volume Signing | Deploy distributed signing infrastructure | Regional signing services with load balancing |
Verification Performance | Implement caching and parallel verification | Signature verification proxy with result caching |
Developer Experience | Create user-friendly tools and processes | CLI tools and IDE plugins for seamless signing |
Cross-Platform Support | Standardize on portable signature formats | Use cosign, which works across container registries |
Integration Complexity | Create reusable integration components | Shared libraries and CI templates for signing |
Future Trends in Artifact Signing¶
The field of artifact signing continues to evolve with several emerging technologies and approaches:
- Transparent Supply Chains
- Public transparency logs (like Certificate Transparency for signing)
- Verifiable build systems with cryptographic attestations
-
Distributed verification networks
-
Post-Quantum Cryptography
- Transition to quantum-resistant algorithms
- Hybrid signature schemes during transition
-
New key management approaches for PQC algorithms
-
Identity-Based Signing
- Further development of ephemeral certificate approaches
- Integration with organizational identity systems
-
Attribute-based verification policies
-
Policy-Driven Verification
- Dynamic verification requirements based on risk
- Context-aware signature validation
- Integration with software supply chain risk management
Conclusion¶
Artifact signing is a fundamental control for software supply chain security, providing a cryptographic guarantee of integrity and authenticity. By implementing comprehensive signing across all artifact types and enforcing verification, organizations can dramatically reduce the risk of supply chain compromise.
A robust artifact signing strategy should include: - End-to-end signing from source code to deployment - Strong key management practices - Automated verification at critical points - Clear policies and procedures for signing operations - Regular assessment and improvement of signing practices
As supply chain attacks continue to increase in sophistication, artifact signing serves as a critical defense by ensuring that only authorized, unmodified software enters your environment.
Implementation Checklist
- ✅ Identify all artifact types in your environment
- ✅ Select appropriate signing technologies for each
- ✅ Implement secure key management practices
- ✅ Integrate signing into CI/CD pipelines
- ✅ Configure verification at deployment and runtime
- ✅ Monitor for signing or verification failures
- ✅ Document procedures for key rotation and recovery
- ✅ Train developers and operators on signing importance