OffsiteBackupStack — Backblaze B2 Backup

Background

Problem

FoodFight’s production data (S3 assets, RDS MySQL databases) lives entirely within AWS. A catastrophic account-level event — accidental deletion, ransomware, account compromise, or AWS-side failure — would leave no independent recovery path. The goal is a fully automated daily backup to an off-AWS provider (Backblaze B2) to satisfy disaster recovery requirements.

Scope

Source What Method
S3 buckets (assets, analytics-events, cognito-backups) Object-level mirror rclone copy --checksum via Orchestrator Lambda
RDS MySQL (Users DB) Snapshot export → Parquet/CSV → B2 rds:StartExportTask → staging bucket → RdsExportSync Lambda

Lightsail is out of scope for this stack. The Lightsail managed MySQL database (timesheet-db) and instance (timesheet-instance) belong to the foodfight-root account (951627545505) and are managed in the separate repo at /Users/tracecarrasco/Documents/Projects/FoodFight/HR. Offsite backup for those resources should be implemented there.

Backblaze B2 buckets

One bucket exists per AWS account. The bucket name is always foodfight-aws-backup-{aws_account_id}, resolved dynamically via self.account at synth time.

AWS Account Environment B2 Bucket Name B2 Bucket ID Object Lock RDS Export Lifecycle
420002417245 prod foodfight-aws-backup-420002417245 0819a23a69ec819c93d5041f Compliance, 45 days Auto-delete after 90 days
654654219312 dev foodfight-aws-backup-654654219312 0889527a69ec819c93d5041f Compliance, 3 days Auto-delete after 7 days
951627545505 root foodfight-aws-backup-951627545505 a899524a69fc819c93d5041f Compliance, 45 days Auto-delete after 90 days

The B2 bucket ID is required when creating scoped application keys in the Backblaze console. Object Lock must be enabled on each bucket (see below).

Where it lives

OffsiteBackupStack is a new Python CDK stack added to the existing infrastructure/ app — the same app that contains BetAuditorStack, UserServiceStack, etc.

infrastructure/userservice/offsite_backup_stack.py

Registered in infrastructure/app.py alongside other stacks. No new CDK app or separate repo is needed.

Key architectural decisions

Two Lambdas — Orchestrator + RdsExportSync

A single 15-minute Lambda cannot handle the RDS snapshot export path. rds:StartExportTask is asynchronous and exports can take 30+ minutes. The solution uses two Lambdas:

  • BackupOrchestratorLambda (02:00 UTC, CfnSchedule): Discovers and syncs S3 buckets to B2 in parallel, then initiates the RDS snapshot export to the staging bucket. Posts a start/completion status to Mattermost.
  • RdsExportSyncLambda (S3-triggered): Fires when the RDS export lands in the staging bucket. Rclone-syncs the exported files to B2. Posts success/failure to Mattermost.

Dynamic resource discovery at runtime

The Orchestrator Lambda discovers what to back up at invocation time — it does not rely on a hardcoded list baked into the function code or environment:

  • S3 buckets: Calls s3:ListBuckets + s3:GetBucketTagging at runtime. All buckets are synced by default. Add tag OffsiteBackup=false to any bucket that should be excluded (e.g. analytics-events, which can be very large and is not required for disaster recovery).
  • RDS instance: Calls rds:DescribeDBInstances filtering by tag OffsiteBackup=true, then rds:DescribeDBSnapshots to find the most recent automated snapshot for export. Automated snapshots are already enabled on the RDS instance (backup_retention = 10 days — validated).

Graceful self-reinvocation for large bucket sets

If the Orchestrator Lambda is approaching its 15-minute ceiling, it gracefully hands off remaining work rather than timing out hard. At the 10-minute mark, handler.py checks context.get_remaining_time_in_millis() and, if under 5 minutes remain, it:

  1. Stops processing the current bucket list
  2. Invokes itself asynchronously (lambda:InvokeAsync) with the remaining bucket names in the payload
  3. Returns {"status": "continued", "remaining": [...]} cleanly

The reinvoked Lambda picks up exactly where the previous one stopped. Because rclone is idempotent, any bucket that was mid-sync just resumes — no duplicate transfers, no data loss.

Fresh invocations (from the scheduler) have an empty payload and discover all buckets from scratch. Continuation invocations receive {"remaining_buckets": ["bucket-a", ...]} and skip discovery entirely.

This pattern also means the IAM role needs lambda:InvokeFunction on itself.

BackupVerificationLambda

A third Lambda runs at 06:00 UTC (4 hours after the backup starts) to verify the previous night’s backup:

  • Uses rclone to list B2 contents per prefix
  • Checks that each expected source has files with a recent modification time
  • Verifies file count is non-zero for each backed-up bucket
  • Posts a structured report to Mattermost — always posts, whether pass or fail

This is the only mechanism that catches silent failures (rclone succeeds but writes nothing, export never completes, etc.).

Go handlers — single static binary, no layer, near-zero dependencies

Lambda handlers are written in Go and compiled to a single static bootstrap binary for linux/arm64 (Graviton2 — cheaper and faster than x86 on Lambda). The runtime is provided.al2023. No Lambda layer, no Python runtime, no dependency management.

The only external dependency is github.com/aws/aws-lambda-go for the Lambda runtime loop. Everything else is Go standard library:

Concern stdlib package
AWS API calls (S3, RDS, Secrets Manager) net/http + SigV4 via crypto/hmac, crypto/sha256
JSON payloads encoding/json
rclone subprocess os/exec
Mattermost HTTP net/http
Concurrency sync, context

The rclone binary is bundled directly inside the Go zip alongside bootstrap. The Go handler shells out to rclone for sync and encryption — this keeps the rclone crypt file format intact so the recovery runbook works with standard rclone locally without any custom tooling.

Goroutines handle genuine parallelism for per-bucket S3 sync — no GIL, no thread pool limits.

go.mod has exactly one require entry:

require github.com/aws/aws-lambda-go v1.x.x

Build per Lambda:

GOOS=linux GOARCH=arm64 go build -o bootstrap ./cmd/orchestrator
zip orchestrator.zip bootstrap bin/rclone
aws s3 cp orchestrator.zip s3://<lambda-s3-bucket>/OffsiteBackup/orchestrator.zip

B2 Object Lock — Compliance mode, 14-day retention

Ransomware scenario: AWS is compromised, S3 objects are destroyed, and the nightly rclone sync faithfully overwrites the good B2 backup. Object Lock prevents this by making objects in B2 immutable for the retention period.

Configuration (Backblaze console — cannot be set via CDK):

  • Enable Object Lock on each B2 bucket in the Backblaze console
  • Mode: Compliance — no one can delete or modify a locked object before the retention period expires, including Backblaze and account admins. Governance mode was considered but rejected: it requires bypass keys, special deletion procedures, and adds operational complexity with no meaningful benefit over Compliance for this use case.
  • Default retention: 3 days (dev) / 45 days (prod + root) — long enough to outlast a ransomware incident discovery period. After the retention period, files unlock automatically and can be deleted normally with no special keys.
  • RDS export lifecycle rule on rds-export/ prefix: 7 days (dev) / 90 days (prod + root) — always longer than the Object Lock retention so B2 can auto-delete once the compliance window has passed.

rclone copy uses --checksum and writes each object with the bucket’s default retention applied automatically.

rclone copy –checksum — how file change detection works

rclone copy (not rclone sync) is used for all S3→B2 transfers. The distinction matters:

Behaviour rclone copy rclone sync
Upload new files ✓ ✓
Re-upload changed files ✓ ✓
Delete destination files that no longer exist at source ✗ ✓

rclone sync would try to delete B2 objects when S3 objects are removed — Object Lock would block those deletes and cause errors. rclone copy never deletes at the destination, which is exactly the right behaviour for a backup: files deleted from S3 are preserved in B2 for the Object Lock retention window.

How --checksum detects changes

By default rclone compares modification time + size. This can miss changes when a file is overwritten in-place with the same size, or when metadata is copied without updating the mtime. --checksum compares the ETag (MD5 hash) of each object instead:

  1. rclone lists all objects in the S3 bucket with their ETags
  2. rclone lists all objects in B2 with their ETags
  3. Any object whose ETag differs (or is absent in B2) is re-uploaded in full
  4. Objects with matching ETags are skipped — no transfer, no cost

Mutable files (appended or overwritten)

S3 does not support true append — “appending” to a file means replacing the entire object with a PUT. When this happens:

  • The object’s ETag changes (new MD5 of the full content)
  • rclone detects the ETag mismatch on the next nightly run
  • The full file is re-uploaded to B2
  • B2 versioning preserves the previous copy as a locked version

The full file is always re-uploaded — rclone does not do delta/diff transfers. For files that grow incrementally (audit logs, exports), this means uploading the entire file each night. This is acceptable for a nightly DR backup.

Versioning behaviour in B2

With B2 versioning enabled (required for Object Lock):

Day 0:  images/test.png  uploaded  → B2 version v1 (locked 14 days)
Day 3:  images/test.png  deleted from S3
        rclone copy: no action (copy never deletes)
        B2 still holds v1
Day 5:  images/test.png  re-added to S3 (new content)
        rclone copy: ETag differs from v1 → re-uploads
        B2 creates version v2 (locked 14 days from day 5)
        B2 still holds v1 (locked until day 14)

Normal access (restore, download) always returns the latest version. To access a specific historical version, use the B2 API or console to list versions by file name and download by version ID (see Recovery Runbook).

Restoring historical versions

# List all versions of a specific file
b2 ls --versions --long "b2-encrypted:s3/your-bucket/images/test.png"

# Download a specific version by ID
b2 download-file-by-id <version-id> ./restored-test.png

Encryption — rclone crypt

All data is encrypted client-side before reaching B2. Backblaze never receives plaintext. rclone crypt uses XSalsa20-Poly1305 (256-bit) for file content and AES-256-CTR for file names.

The rclone config is constructed at runtime from Secrets Manager values:

[b2]
type = b2
account = <application_key_id>
key = <application_key>

[b2-encrypted]
type = crypt
remote = b2:foodfight-aws-backup-<account>
password = <encryption_password>   # rclone-obscured at runtime
password2 = <encryption_salt>      # second factor

The Go handler calls rclone obscure at startup to convert plaintext values from Secrets Manager into rclone’s obscured format before writing the config to /tmp/rclone.conf. No plaintext config persists after the invocation.

Encryption key storage — Bitwarden + multi-region Secrets Manager

  • Bitwarden (shared vault) — authoritative out-of-band copy. Survives a total AWS account loss. This is the recovery key.
  • AWS Secrets Manager (us-east-1, primary) — Lambda runtime access.
  • AWS Secrets Manager (us-west-2, replica — prod only) — regional fallback for manual recovery if us-east-1 is unavailable. No resources need to exist in us-west-2 for this to work.

Mattermost notifications

All three Lambda handlers post directly to Mattermost on success and failure, using the same bot token pattern as BetAuditorStack (alerts/{cdk_env}/mattermost-bot-token). No SNS or CloudWatch alarm infrastructure is needed — the verification Lambda posting every day at 06:00 UTC is the heartbeat; silence means something is broken.

Environment Channel ID
prod tf9xx1krfpg7fq4ufmyf4ujrmw
dev / QA 6a8b7q7fsjd8jmzw3nku9b9b5c

Each Lambda posts on both success and failure. The verification Lambda always posts, making it the daily heartbeat for backup health.

Handler deployment via CI/CD

Source lives in backend/projects/offsite_backup/. On push to main, .github/workflows/offsite_backup_qa_deploy.yml cross-compiles all three Go binaries, zips each with the rclone binary, uploads to S3, and triggers a CodeDeploy deployment per function (same deploy_lambda_with_codedeploy.sh script used by all other services). The prod workflow mirrors this on push to release. For the bootstrap case (first deploy, no S3 key yet), upload placeholder binaries manually — see Step 3.

CodeDeploy intentionally omitted

create_lambda_deployment (CodeDeploy canary) is used by BetAuditorStack because traffic shifting matters for a frequently-invoked function. The backup Lambdas run once daily in a non-user-facing context — a canary deployment would add complexity with no benefit. CodeDeploy is deliberately not used here.

Prerequisites before first deploy

  1. Go toolchain — go 1.22+ with GOOS=linux GOARCH=arm64 cross-compilation
  2. rclone binary — download the linux/arm64 build from rclone.org once and commit it to infrastructure/userservice/lambda/offsite_backup/bin/rclone (gitignored via .gitignore; CI/CD fetches it during build)
  3. Backblaze B2 console — buckets already exist (see table above).
    • Confirm Object Lock is enabled on each bucket (must be set at bucket creation; cannot be added retroactively)
    • Create an application key scoped to the target bucket and save application_key_id and application_key in Bitwarden
  4. Encryption password — generate a password and salt: 
    openssl rand -base64 32  # encryption_password
openssl rand -base64 32  # encryption_salt

    Save both to Bitwarden before deploying. Without these, the B2 backup cannot be decrypted even if all files are intact.

  5. Tag RDS instance — add tag OffsiteBackup=true to the RDS instance in UserServiceStack so the Orchestrator Lambda can discover it
  6. AWS SSO login — aws sso login before running CDK commands

RDS automated snapshots — validated

backup_retention=Duration.days(10) is set in userservice_stack.py:245. Automated daily snapshots are already enabled. The Orchestrator Lambda will call rds:DescribeDBSnapshots at runtime to find the most recent automated snapshot and pass its ARN to rds:StartExportTask. No changes to userservice_stack.py are required.

Current status

Implementation not yet started. This document is the working plan. The CDK stack and all three Lambda handlers need to be written.

Implementation Plan

Files to create or modify

File Action
infrastructure/userservice/offsite_backup_stack.py Create — Python CDK stack
infrastructure/app.py Modify — register OffsiteBackupStack
backend/projects/offsite_backup/cmd/orchestrator/main.go Create — Orchestrator handler
backend/projects/offsite_backup/cmd/rds/main.go Create — RdsExportSync handler
backend/projects/offsite_backup/cmd/verification/main.go Create — Verification handler
backend/projects/offsite_backup/go.mod Create — Go module (module foodfight/offsite_backup)
backend/projects/offsite_backup/internal/ Create — shared stdlib packages: awssig (SigV4), awsapi (S3/RDS/Secrets Manager HTTP clients), rclone (exec wrapper), mattermost (HTTP client)
backend/projects/offsite_backup/bin/rclone Download — linux/arm64 rclone binary (gitignored via .gitignore)
.github/workflows/offsite_backup_qa_deploy.yml Create — CI/CD deploy to dev on push to main
.github/workflows/offsite_backup_prod_deploy.yml Create — CI/CD deploy to prod on push to release

The Go source lives in backend/projects/offsite_backup/ alongside all other backend services. The bin/rclone binary is gitignored; CI/CD downloads it at build time.

Step 1 — offsite_backup_stack.py

1a. Imports 

import os
import aws_cdk as cdk
from aws_cdk import (
    Duration,
    RemovalPolicy,
    Stack,
    aws_iam as iam,
    aws_lambda as lambda_,
    aws_lambda_event_sources as lambda_events,
    aws_logs as logs,
    aws_s3 as s3,
    aws_scheduler as scheduler,
    aws_secretsmanager as secretsmanager,
    aws_ssm as ssm,
)
from constructs import Construct

1b. Class definition 

class OffsiteBackupStack(Stack):
    """
    Daily offsite backup to Backblaze B2.

    Backs up:
      - S3 buckets (rclone copy --checksum)
      - RDS MySQL snapshot export → staging S3 → B2

    Three Lambdas:
      1. BackupOrchestratorLambda  — 02:00 UTC, discovers resources,
                                     syncs S3, initiates RDS export
      2. RdsExportSyncLambda       — S3-triggered when RDS export lands
      3. BackupVerificationLambda  — 06:00 UTC, checks B2 and reports

    All Lambdas post to Mattermost on success and failure.
    CodeDeploy intentionally omitted — once-daily, non-user-facing.
    """

    def __init__(self, scope: Construct, construct_id: str, **kwargs) -> None:
        super().__init__(scope, construct_id, **kwargs)

        cdk_env = os.environ.get("CDK_ENV", "dev")

        mattermost_channel_id = (
            "tf9xx1krfpg7fq4ufmyf4ujrmw" if cdk_env == "prod"
            else "6a8b7q7fsjd8jmzw3nku9b9b5c"
        )

1c. Secrets Manager secrets (with prod multi-region replication) 

        replica_regions = (
            [secretsmanager.ReplicaRegion(region="us-west-2")]
            if cdk_env == "prod" else []
        )

        # Secret shape (populate after deploy — store in Bitwarden first):
        # {
        #   "application_key_id": "...",
        #   "application_key":    "...",
        #   "encryption_password": "...",
        #   "encryption_salt":    "..."
        # }
        b2_secret = secretsmanager.Secret(
            self, "B2Credentials",
            secret_name=f"foodfight-offsite-backup/{cdk_env}/b2-credentials",
            description="Backblaze B2 app key + rclone encryption password/salt",
            removal_policy=RemovalPolicy.RETAIN,
            replica_regions=replica_regions,
        )

        # Secret shape (populate after deploy — store in Bitwarden first):
        # Same replica policy — needed for manual recovery from us-west-2
        # (prod only; left empty for dev)

1d. RDS export staging bucket 

        export_staging_bucket = s3.Bucket(
            self, "RdsExportStaging",
            bucket_name=f"foodfight-rds-export-staging-{self.account}",
            removal_policy=RemovalPolicy.RETAIN,
            lifecycle_rules=[
                s3.LifecycleRule(
                    id="expire-rds-exports",
                    expiration=Duration.days(7),
                )
            ],
            block_public_access=s3.BlockPublicAccess.BLOCK_ALL,
            encryption=s3.BucketEncryption.S3_MANAGED,
        )

1e. RDS export IAM role (assumed by the RDS service, not Lambda) 

        rds_export_role = iam.Role(
            self, "RdsExportRole",
            assumed_by=iam.ServicePrincipal("export.rds.amazonaws.com"),
            description="Allows RDS to export snapshots to the staging S3 bucket",
        )
        export_staging_bucket.grant_read_write(rds_export_role)
        # KMS grant for the export role (needed if staging bucket uses KMS)
        rds_export_role.add_to_policy(iam.PolicyStatement(
            actions=["kms:GenerateDataKey", "kms:Decrypt", "kms:DescribeKey"],
            resources=["*"],
            conditions={
                "StringLike": {
                    "kms:ViaService": f"s3.{self.region}.amazonaws.com"
                }
            },
        ))

1f. Lambda S3 bucket reference 

        s3_bucket_name = ssm.StringParameter.value_for_string_parameter(
            self, f"/foodfight/{cdk_env}/lambda-s3-bucket",
        )
        lambda_bucket = s3.Bucket.from_bucket_name(
            self, "LambdaZipBucket", s3_bucket_name
        )

No Lambda layer — each Go zip bundles bootstrap + rclone directly.

1g. Shared Lambda IAM role 

        shared_role = iam.Role(
            self, "OffsiteBackupRole",
            assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"),
        )
        shared_role.add_managed_policy(
            iam.ManagedPolicy.from_aws_managed_policy_name(
                "service-role/AWSLambdaBasicExecutionRole"
            )
        )

        # B2 credentials + encryption keys
        b2_secret.grant_read(shared_role)

        # Staging bucket
        export_staging_bucket.grant_read_write(shared_role)

        # S3 — list all buckets, check opt-out tag, read objects
        shared_role.add_to_policy(iam.PolicyStatement(
            actions=[
                "s3:ListAllMyBuckets",
                "s3:GetBucketTagging",   # for OffsiteBackup=false opt-out
                "s3:GetObject",
                "s3:ListBucket",
                "s3:GetBucketLocation",
            ],
            resources=["*"],
        ))

        # RDS — describe instances, snapshots, and start exports
        shared_role.add_to_policy(iam.PolicyStatement(
            actions=[
                "rds:DescribeDBInstances",
                "rds:DescribeDBSnapshots",
                "rds:StartExportTask",
                "rds:DescribeExportTasks",
                "tag:GetResources",
            ],
            resources=["*"],
        ))

        # KMS (for RDS snapshot export)
        shared_role.add_to_policy(iam.PolicyStatement(
            actions=["kms:GenerateDataKey", "kms:Decrypt", "kms:DescribeKey"],
            resources=["*"],
            conditions={
                "StringLike": {
                    "kms:ViaService": [
                        f"s3.{self.region}.amazonaws.com",
                        f"rds.{self.region}.amazonaws.com",
                    ]
                }
            },
        ))

        # PassRole — Lambda hands the RDS export role to the RDS service
        shared_role.add_to_policy(iam.PolicyStatement(
            actions=["iam:PassRole"],
            resources=[rds_export_role.role_arn],
            conditions={
                "StringEquals": {
                    "iam:PassedToService": "export.rds.amazonaws.com"
                }
            },
        ))

        # Self-invocation — Orchestrator reinvokes itself when approaching timeout
        # (resource ARN is set after orchestrator_fn is created; grant added below)

        # Secrets Manager — read Mattermost bot token
        shared_role.add_to_policy(iam.PolicyStatement(
            actions=["secretsmanager:GetSecretValue"],
            resources=["*"],
        ))

1h. Shared Lambda environment 

        shared_env = {
            "CDK_ENV":                cdk_env,
            "RDS_EXPORT_ROLE_ARN":    rds_export_role.role_arn,
            "RDS_EXPORT_BUCKET":      export_staging_bucket.bucket_name,
            "B2_BUCKET":              f"foodfight-aws-backup-{self.account}",
            "B2_SECRET_NAME":         b2_secret.secret_name,
            "MATTERMOST_ENABLED":     "true",
            "MATTERMOST_BASE_URL":    "https://chat.foodfight-hr.com",
            "MATTERMOST_CHANNEL_ID":  mattermost_channel_id,
            "MATTERMOST_BOT_TOKEN_SECRET_ARN": f"alerts/{cdk_env}/mattermost-bot-token",
            "DRY_RUN":                "false",
        }

1i. Lambda 1 — BackupOrchestratorLambda 

        orchestrator_fn = lambda_.Function(
            self, "BackupOrchestratorFunction",
            function_name=f"foodfight-backup-orchestrator-{cdk_env}",
            runtime=lambda_.Runtime.PROVIDED_AL2023,
            handler="bootstrap",
            architecture=lambda_.Architecture.ARM_64,
            code=lambda_.Code.from_bucket(
                bucket=lambda_bucket,
                key="OffsiteBackup/orchestrator.zip",
            ),
            timeout=Duration.seconds(900),
            memory_size=3008,
            role=shared_role,
            environment=shared_env,
            log_retention=logs.RetentionDays.THREE_MONTHS,
            description="Discovers S3 buckets and RDS instance via tags, syncs S3→B2, initiates RDS export",
        )

        # Grant Orchestrator permission to reinvoke itself
        orchestrator_fn.grant_invoke(shared_role)

1j. Lambda 2 — RdsExportSyncLambda 

        rds_sync_fn = lambda_.Function(
            self, "RdsExportSyncFunction",
            function_name=f"foodfight-rds-export-sync-{cdk_env}",
            runtime=lambda_.Runtime.PROVIDED_AL2023,
            handler="bootstrap",
            architecture=lambda_.Architecture.ARM_64,
            code=lambda_.Code.from_bucket(
                bucket=lambda_bucket,
                key="OffsiteBackup/rds.zip",
            ),
            timeout=Duration.seconds(900),
            memory_size=3008,
            role=shared_role,
            environment=shared_env,
            log_retention=logs.RetentionDays.THREE_MONTHS,
            description="Triggered by S3 when RDS export lands in staging bucket; syncs to B2",
        )

        # S3 event trigger — fires when RDS export writes objects to staging bucket
        rds_sync_fn.add_event_source(
            lambda_events.S3EventSource(
                export_staging_bucket,
                events=[s3.EventType.OBJECT_CREATED],
                filters=[s3.NotificationKeyFilter(prefix="rds-export/")],
            )
        )

1k. Lambda 3 — BackupVerificationLambda 

        verification_fn = lambda_.Function(
            self, "BackupVerificationFunction",
            function_name=f"foodfight-backup-verification-{cdk_env}",
            runtime=lambda_.Runtime.PROVIDED_AL2023,
            handler="bootstrap",
            architecture=lambda_.Architecture.ARM_64,
            code=lambda_.Code.from_bucket(
                bucket=lambda_bucket,
                key="OffsiteBackup/verification.zip",
            ),
            timeout=Duration.seconds(300),
            memory_size=3008,
            role=shared_role,
            environment=shared_env,
            log_retention=logs.RetentionDays.THREE_MONTHS,
            description="Runs at 06:00 UTC; checks B2 for recent files and posts report to Mattermost",
        )

1l. EventBridge schedules 

        def make_scheduler_role(construct_id: str, fn: lambda_.Function) -> iam.Role:
            r = iam.Role(
                self, construct_id,
                assumed_by=iam.ServicePrincipal("scheduler.amazonaws.com"),
            )
            r.add_to_policy(iam.PolicyStatement(
                actions=["lambda:InvokeFunction"],
                resources=[fn.function_arn],
            ))
            return r

        orchestrator_sched_role = make_scheduler_role(
            "OrchestratorSchedulerRole", orchestrator_fn
        )
        scheduler.CfnSchedule(
            self, "DailyBackupSchedule",
            description="Trigger FoodFight offsite backup daily at 02:00 UTC",
            schedule_expression="cron(0 2 * * ? *)",
            flexible_time_window=scheduler.CfnSchedule.FlexibleTimeWindowProperty(
                mode="OFF",
            ),
            target=scheduler.CfnSchedule.TargetProperty(
                arn=orchestrator_fn.function_arn,
                role_arn=orchestrator_sched_role.role_arn,
            ),
        )

        verification_sched_role = make_scheduler_role(
            "VerificationSchedulerRole", verification_fn
        )
        scheduler.CfnSchedule(
            self, "DailyVerificationSchedule",
            description="Trigger FoodFight backup verification daily at 06:00 UTC",
            schedule_expression="cron(0 6 * * ? *)",
            flexible_time_window=scheduler.CfnSchedule.FlexibleTimeWindowProperty(
                mode="OFF",
            ),
            target=scheduler.CfnSchedule.TargetProperty(
                arn=verification_fn.function_arn,
                role_arn=verification_sched_role.role_arn,
            ),
        )

1m. CfnOutputs 

        cdk.CfnOutput(self, "OrchestratorFunctionName",
            value=orchestrator_fn.function_name,
        )
        cdk.CfnOutput(self, "RdsExportSyncFunctionName",
            value=rds_sync_fn.function_name,
        )
        cdk.CfnOutput(self, "VerificationFunctionName",
            value=verification_fn.function_name,
        )
        cdk.CfnOutput(self, "B2SecretArn",
            value=b2_secret.secret_arn,
            description="Populate after deploy. Store values in Bitwarden first.",
        )
        cdk.CfnOutput(self, "RdsExportRoleArn",
            value=rds_export_role.role_arn,
        )

Step 2 — Register in app.py 

from userservice.offsite_backup_stack import OffsiteBackupStack

# ...existing stack instantiations...

offsiteBackupStack = OffsiteBackupStack(app, "OffsiteBackupStack", env=env_USA)

No add_dependency calls needed — the stack reads from Secrets Manager and SSM directly with no runtime dependency on other stacks.

Step 3 — Bootstrap placeholder artifacts (first deploy only)

The CDK stack references three S3 keys that must exist before cdk deploy can create the Lambda functions. Build and upload minimal placeholder binaries once, by hand. CI/CD replaces them on the first push.

mkdir -p /tmp/placeholder
cat > /tmp/placeholder/main.go << 'EOF'
package main

import (
    "context"
    "log"
    "github.com/aws/aws-lambda-go/lambda"
)

func handler(ctx context.Context) error {
    log.Println("placeholder — not yet deployed via CI/CD")
    return nil
}

func main() { lambda.Start(handler) }
EOF

cd /tmp/placeholder
go mod init placeholder
go get github.com/aws/aws-lambda-go/lambda
GOOS=linux GOARCH=arm64 go build -o bootstrap .

for cmd in orchestrator rds verification; do
  zip ${cmd}.zip bootstrap
  aws s3 cp ${cmd}.zip s3://lambda.qa.getfoodfight.link/OffsiteBackup/${cmd}.zip
done

Step 4 — Deploy CDK stack 

cd infrastructure

CDK_ENV=dev cdk synth OffsiteBackupStack
CDK_ENV=dev cdk deploy OffsiteBackupStack

Step 5 — GitHub Actions CI/CD

Two workflow files handle ongoing deployments. On every push to main the QA workflow builds all three Go binaries, zips each with the rclone binary, and triggers a CodeDeploy deployment per function. The prod workflow is identical but targets the release branch and the prod AWS account.

offsite_backup_qa_deploy.yml 

name: Deploy Offsite Backup Lambdas (QA)
permissions:
  id-token: write
  contents: read

on:
  push:
    branches:
      - main
    paths:
      - 'backend/projects/offsite_backup/**'
      - '.github/workflows/offsite_backup_qa_deploy.yml'
      - '.github/scripts/deploy_lambda_with_codedeploy.sh'

jobs:
  build:
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: backend/projects/offsite_backup

    steps:
      - name: Checkout code
        uses: actions/checkout@v2

      - name: Set up Go
        uses: actions/setup-go@v5
        with:
          go-version: '1.22'

      - name: Download rclone (linux/arm64)
        run: |
          curl -Lo rclone.zip https://downloads.rclone.org/rclone-current-linux-arm64.zip
          unzip -j rclone.zip rclone-*-linux-arm64/rclone -d bin/
          chmod +x bin/rclone
          rm rclone.zip

      - name: Build and package handlers
        run: |
          for cmd in orchestrator rds verification; do
            GOOS=linux GOARCH=arm64 go build -o bootstrap ./cmd/${cmd}
            zip ${cmd}.zip bootstrap bin/rclone
            rm bootstrap
          done

      - name: Configure AWS Credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          role-to-assume: arn:aws:iam::654654219312:role/GitHub-Actions-Auth-QA
          aws-region: us-east-1

      - name: Upload to S3
        run: |
          for cmd in orchestrator rds verification; do
            aws s3 cp ${cmd}.zip s3://lambda.qa.getfoodfight.link/OffsiteBackup/${cmd}.zip
          done

      - name: Deploy orchestrator (CodeDeploy)
        run: |
          bash "$GITHUB_WORKSPACE/.github/scripts/deploy_lambda_with_codedeploy.sh" \
            --s3-bucket lambda.qa.getfoodfight.link \
            --s3-key OffsiteBackup/orchestrator.zip \
            --function-name-pattern "^OffsiteBackupStack-BackupOrchestratorFunction" \
            --alias Live \
            --application-name backup-orchestrator-dev \
            --deployment-group-name backup-orchestrator-dg-dev

      - name: Deploy rds-export-sync (CodeDeploy)
        run: |
          bash "$GITHUB_WORKSPACE/.github/scripts/deploy_lambda_with_codedeploy.sh" \
            --s3-bucket lambda.qa.getfoodfight.link \
            --s3-key OffsiteBackup/rds.zip \
            --function-name-pattern "^OffsiteBackupStack-RdsExportSyncFunction" \
            --alias Live \
            --application-name rds-export-sync-dev \
            --deployment-group-name rds-export-sync-dg-dev

      - name: Deploy verification (CodeDeploy)
        run: |
          bash "$GITHUB_WORKSPACE/.github/scripts/deploy_lambda_with_codedeploy.sh" \
            --s3-bucket lambda.qa.getfoodfight.link \
            --s3-key OffsiteBackup/verification.zip \
            --function-name-pattern "^OffsiteBackupStack-BackupVerificationFunction" \
            --alias Live \
            --application-name backup-verification-dev \
            --deployment-group-name backup-verification-dg-dev

offsite_backup_prod_deploy.yml

Identical to the QA workflow with three changes:

Field QA Prod
branches main release
role-to-assume arn:aws:iam::654654219312:role/GitHub-Actions-Auth-QA arn:aws:iam::420002417245:role/GitHub-Actions-Auth-Dev
S3 bucket lambda.qa.getfoodfight.link lambda.dev.getfoodfight.link
application/group names *-dev *-prod

Step 6 — Test invoke 

# Test orchestrator (use DRY_RUN to avoid writing to B2)
aws lambda invoke \
  --function-name foodfight-backup-orchestrator-dev \
  --payload '{"dry_run": true}' \
  --cli-binary-format raw-in-base64-out \
  /tmp/result.json && cat /tmp/result.json

# Test verification
aws lambda invoke \
  --function-name foodfight-backup-verification-dev \
  --payload '{}' \
  --cli-binary-format raw-in-base64-out \
  /tmp/result.json && cat /tmp/result.json

Repeating for Prod and Root Account

Prod (420002417245)

  • Backblaze console — create app key scoped to foodfight-aws-backup-420002417245 (bucket ID 0819a23a69ec819c93d5041f), Read & Write, no list-all-buckets
  • Generate new encryption password and salt — openssl rand -base64 32 twice; save both to Bitwarden under a prod entry (do not reuse dev values)
  • CDK deploy 
    CDK_ENV=prod cdk synth OffsiteBackupStack
CDK_ENV=prod cdk deploy OffsiteBackupStack
  • Populate secret 
    aws secretsmanager put-secret-value \
  --secret-id "foodfight-offsite-backup/prod/b2-credentials" \
  --secret-string '{
    "application_key_id": "YOUR_PROD_KEY_ID",
    "application_key":    "YOUR_PROD_APP_KEY",
    "encryption_password": "YOUR_PROD_PASSWORD",
    "encryption_salt":    "YOUR_PROD_SALT"
  }'
  • S3 opt-out tags — tag any large/non-critical buckets with OffsiteBackup=false in the prod account
  • Test invoke (dry run against prod functions) 
    aws lambda invoke \
  --function-name foodfight-backup-orchestrator-prod \
  --payload '{"dry_run": true}' \
  --cli-binary-format raw-in-base64-out \
  /tmp/result.json && cat /tmp/result.json

The offsite_backup_prod_deploy.yml GitHub Action handles all future code deployments automatically on push to release. No manual S3 uploads needed after the first CDK deploy.

Root account (951627545505) — HR repo

This account is out of scope for this stack. The Lightsail MySQL database (timesheet-db) and instance (timesheet-instance) are managed in the separate repo at /Users/tracecarrasco/Documents/Projects/FoodFight/HR. Offsite backup for those resources should be implemented there, reusing the same patterns: Go handlers, rclone crypt, B2 bucket foodfight-aws-backup-951627545505 (ID a899524a69fc819c93d5041f), and a matching Secrets Manager secret.

Recovery Runbook

Use this when AWS is unavailable and you need to restore from Backblaze B2.

What you need

  • rclone installed locally (brew install rclone or download from rclone.org)
  • Bitwarden access — retrieve application_key_id, application_key, encryption_password, encryption_salt from the shared vault

Step 1 — Configure rclone locally 

# Create a local rclone config (never commit this file)
mkdir -p ~/.config/rclone

cat > ~/.config/rclone/rclone.conf << 'EOF'
[b2]
type = b2
account = YOUR_APPLICATION_KEY_ID
key = YOUR_APPLICATION_KEY

[b2-encrypted]
type = crypt
remote = b2:foodfight-aws-backup-ACCOUNT_ID
password = RCLONE_OBSCURED_PASSWORD
password2 = RCLONE_OBSCURED_SALT
EOF

# Obscure the passwords (rclone requires its own encoding)
rclone obscure YOUR_ENCRYPTION_PASSWORD   # paste output as `password`
rclone obscure YOUR_ENCRYPTION_SALT       # paste output as `password2`

Step 2 — List available backups 

# List top-level prefixes (one per backed-up S3 bucket + rds-export/)
rclone lsd b2-encrypted:

# List RDS export files
rclone ls b2-encrypted:rds-export/

# List a specific S3 mirror
rclone ls b2-encrypted:s3/your-bucket-name/

Step 3 — Download and restore S3 data 

# Download a full bucket mirror to a local directory
rclone copy b2-encrypted:s3/your-bucket-name/ ./restore/your-bucket-name/

# Re-upload to a new (or recovered) S3 bucket
aws s3 sync ./restore/your-bucket-name/ s3://your-new-bucket/

Step 4 — Restore RDS from exported snapshot

RDS snapshot exports produce Parquet files, not a SQL dump. Restoration requires importing via AWS Data Migration Service or reading the Parquet files directly into a new RDS instance.

# Download the RDS export
rclone copy b2-encrypted:rds-export/ ./restore/rds/

# The export is organized by table:
# restore/rds/<export-task-id>/<database>/<table>/1.parquet

Refer to the AWS documentation on restoring from snapshot exports for the full import procedure. This step requires a running AWS account (or a local MySQL instance for partial data recovery).

Step 5 — Verify decryption worked 

# Spot-check: list a few files and confirm they have content
rclone ls b2-encrypted:s3/your-bucket-name/ | head -20

# If you see file names and sizes, decryption is working.
# If you see errors, double-check your encryption_password and salt from Bitwarden.

Notes

  • RemovalPolicy.RETAIN on the staging bucket and secrets. Never change to DESTROY.
  • B2 Object Lock (Compliance mode, 14-day retention) must be configured in the Backblaze console — it cannot be disabled once enabled. After 14 days files unlock automatically and can be deleted normally with no special keys.
  • DRY_RUN=true can be passed in the Lambda payload for safe test runs.
  • The verification Lambda is the daily heartbeat — if it stops posting to Mattermost, something is wrong with the backup pipeline.
  • CodeDeploy is intentionally omitted. These Lambdas run once daily in a non-user-facing context; canary deployments add complexity with no benefit.