Install on AWS with CloudFormation

Provision a Workforce Intelligence box with the CloudFormation template: parameters, access over Session Manager, outputs, generated secrets, and data protection.

For: Administrators deploying to AWS · Last updated

The template deploy/cloudformation/ec2.yaml provisions the EC2 instance, its role, its security group, the data protections described below, and installs Docker. It does everything that does not need a secret that only you know. You then connect to the box and run install.sh with your registry credentials.

Read Deployment requirements first.

What the template creates#

  • An EC2 instance with an instance role that grants SSM Session Manager access (you do not need an SSH key) and, by default, the minimum Amazon Bedrock permissions described in Instance role, IMDS hop limit and AWS access.
  • A security group allowing ports 80 and 443, plus port 22 only if you set SshCidr. Nothing else is opened (see "What the box exposes" below).
  • Docker and the Compose plugin, installed on first boot. The Compose plugin binary is pinned to an exact release and checksum-verified.
  • install.sh and find-setup-token.sh written to /opt/wi/. You never copy these by hand on an AWS box.
  • /opt/wi/.env, generated once on first boot (see below).
  • Data protection for the root volume: termination protection, a root volume that survives termination, and daily AWS Backup snapshots. See Backups and recovery on AWS.
  • Optionally an Elastic IP and a Route 53 record.

The instance requires IMDSv2, with a hop limit of 2 by default so containers can use the instance role.

Parameters#

Parameter Default Meaning
VpcId required VPC to launch into.
SubnetId required Subnet to launch into. It must belong to VpcId and must be a public subnet (see below).
AssociatePublicIp true true assigns a public IP on the instance's network interface. false is for a private-subnet deploy reachable only through Session Manager port forwarding or an internal load balancer you build yourself.
AllocateElasticIp false true also allocates an Elastic IP that survives stop and start. Elastic IPs are an account-wide quota (5 per region by default), so check headroom first.
InstanceType t3.large Instance size. The box runs Postgres, the app, the web server, Dokku, and every company's Coworker workbench container.
RootVolumeSizeGiB 40 (minimum 20) Root volume size. Docker images and workbench containers live here.
HttpCidr 0.0.0.0/0 CIDR allowed to reach port 80.
HttpsCidr 0.0.0.0/0 CIDR allowed to reach port 443.
SshCidr blank CIDR allowed to reach port 22. Blank disables SSH entirely and relies on Session Manager.
KeyName blank EC2 key pair name, used only when SshCidr is set.
NamePrefix wi Prefix for resource Name tags, for example wi-acme.
LatestAmiId a pinned Amazon Linux 2023 x86_64 AMI for us-east-1 A literal AMI id that is never looked up automatically. In another region, pass that region's AMI id. Changing it replaces the instance.
DomainName blank Creates a Route 53 A record. Requires HostedZoneId and AllocateElasticIp=true. Does nothing else.
HostedZoneId blank Route 53 hosted zone for DomainName. Both must be set together.
SnapshotId blank Recovery only. Builds the root volume from this EBS snapshot. See Backups and recovery on AWS.
BackupVaultName blank Blank on the first deployment in an account and region; the shared vault name on every later one.
BackupRetentionDays 14 (minimum 1) Used only by the primary backup stack.
ImdsHopLimit 2 (allowed: 1 or 2) IMDSv2 hop limit. 2 lets the app and Coworker workbench containers obtain the instance role's credentials; 1 restricts them to the host. See the section below.
EnableBedrockAccess true true adds an inline policy to the instance role that allows Amazon Bedrock model invocation and listing. false adds nothing.
AdditionalInstanceRolePolicyArns blank Comma-separated IAM managed policy ARNs attached to the instance role in addition to the SSM policy.

To find the current AMI id for another region, run:

aws ssm get-parameter \
  --name /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64 \
  --query Parameter.Value --output text

Use a genuinely public subnet#

SubnetId must be a subnet whose route table sends 0.0.0.0/0 to an Internet Gateway, not to a NAT gateway. An Elastic IP attached to an instance in a NAT-routed subnet is unreachable from the internet whatever the security group allows. Check with:

aws ec2 describe-route-tables \
  --filters "Name=association.subnet-id,Values=<SubnetId>" \
  --query 'RouteTables[0].Routes'

You want to see a route with GatewayId: igw-..., not NatGatewayId.

Backup vault on the first and later deployments#

All Workforce Intelligence deployments in one AWS account and region share one backup vault and one daily plan. Check which case you are in with aws backup list-backup-vaults.

  • First deployment: leave BackupVaultName blank. The stack creates the shared wi-backup-vault, the plan, the selection, and the service role.
  • Every later deployment: pass BackupVaultName=wi-backup-vault. The stack creates no backup resources of its own. Its root volume carries the tag wi-backup=true, and the shared plan picks it up.

Do not delete the first (primary) stack while others still exist. That removes the shared plan and stops backups for all of them. The vault and the existing backups are kept.

Deploy the stack#

aws cloudformation deploy \
  --template-file deploy/cloudformation/ec2.yaml \
  --stack-name wi-<customer-name> \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
    VpcId=vpc-xxxxxxxx \
    SubnetId=subnet-xxxxxxxx \
    AssociatePublicIp=true \
    NamePrefix=wi-<customer-name> \
    BackupVaultName=wi-backup-vault

Omit BackupVaultName on the first deployment in the region.

Outputs#

Output Meaning
InstanceId The EC2 instance id.
PublicIp The Elastic IP if you allocated one, otherwise the instance's own public IP. Without an Elastic IP this address changes if the instance is stopped and started again (a reboot alone does not change it).
SsmConnectCommand The aws ssm start-session command for this instance.
DnsName The Route 53 record name. Present only when DomainName, HostedZoneId, and AllocateElasticIp=true are all set.
BackupVault The shared vault this deployment's root volume is backed up into.

Connect with Session Manager#

You need the Session Manager plugin for the AWS CLI.

aws ssm start-session --target "$(aws cloudformation describe-stacks \
  --stack-name wi-<customer-name> \
  --query 'Stacks[0].Outputs[?OutputKey==`InstanceId`].OutputValue' --output text)"

Then cd /opt/wi. You find install.sh, find-setup-token.sh, and .env there.

The generated .env#

On first boot the template writes /opt/wi/.env, readable by root only (mode 0600), with two values:

  • WI_MASTER_KEY protects every credential your company stores. See Master key custody.
  • WI_SESSION_SECRET signs sign-in sessions.

The values are generated on the box with openssl rand -base64 32. They never appear in the template, in a stack parameter, or in any AWS API call. Later boots never regenerate them.

The file contains nothing else. install.sh adds the values that describe your deployment (the next section).

Run install.sh#

  1. Decide your domain, for example wi.example.com. Without one, use WI_ALLOWED_ORIGIN=http://<PublicIp> (the stack's PublicIp output) in its place, but browsers do not keep a sign-in made over plain HTTP, so use a domain with HTTPS for real use.

  2. Run the installer with your registry credentials and the domain:

    cd /opt/wi
WI_DOMAIN=wi.example.com \
GHCR_USER=<your-github-username> \
GHCR_TOKEN=<your-token> \
  ./install.sh

    install.sh appends WI_DOMAIN to the existing .env if it has no such line, and never changes a line that is there. That is the only address setting: the web origin, the base URLs for emailed links, webhooks and deployed Apps, and the host Builders use for git push are all derived from it. What each value means is in Configuration reference.

    The script logs in to ghcr.io with sudo docker login, pulls the wi-server image, extracts docker-compose.prod.yml from it into /opt/wi, and starts the stack under the Compose project name wi-prod. The app container applies database migrations and never seeds data, so the database starts empty.

  3. Open your address in a browser and continue with First-run setup token.

Instance role, IMDS hop limit and AWS access#

The instance role's temporary credentials are served by the instance metadata service (IMDS) at 169.254.169.254. The template requires IMDSv2, a token handshake whose reply is limited by a hop limit. The ImdsHopLimit parameter sets it, and the default is 2.

At a hop limit of 1 only processes on the host itself can read the role's credentials. The app and every Coworker workbench run in Docker containers on a bridge network, one hop further away, so they cannot. At 2 they can. That has these consequences:

  • The app container and every Coworker workbench container on the instance can obtain the instance role's credentials.
  • This is what makes the Use the server's IAM role option work for an Amazon Bedrock provider. See Use AWS Bedrock.
  • It also lets a custom connector's script call any AWS service the role is granted (for example S3), with no stored keys.
  • Anything granted to the role is available to code that Coworkers run. It is also reachable by a request-forgery bug in a container. Grant least privilege, and grant nothing to the role that you would not give to the app itself.

What the role grants#

By default the role has the AWS-managed AmazonSSMManagedInstanceCore policy and, because EnableBedrockAccess is true, an inline policy that allows:

  • bedrock:InvokeModel and bedrock:InvokeModelWithResponseStream on foundation models in any region and on your account's inference profiles. A cross-region inference profile (an id starting us., eu. or global.) is also authorized in each destination region, which is why the regions are a wildcard.
  • bedrock:ListFoundationModels and bedrock:ListInferenceProfiles on all resources (AWS does not support narrowing list actions), used to fill the model list.

Set EnableBedrockAccess=false if you use only access keys or a Bedrock API key, or want to manage Bedrock permissions in your own policy.

Grant the role more access#

Create an IAM managed policy for exactly what a connector needs, then pass its ARN in AdditionalInstanceRolePolicyArns (comma-separated for several). The ARNs are attached to the instance role alongside the SSM policy. You do not edit the template.

aws cloudformation deploy \
  --template-file deploy/cloudformation/ec2.yaml \
  --stack-name wi-<customer-name> \
  --capabilities CAPABILITY_NAMED_IAM \
  --parameter-overrides \
    AdditionalInstanceRolePolicyArns=arn:aws:iam::<account-id>:policy/wi-connector-s3-read

Pass every parameter that you set on the original deploy again, or use aws cloudformation update-stack --use-previous-template with UsePreviousValue=true for the rest. Otherwise the stack reverts them to their defaults.

Set the hop limit back to 1#

Set ImdsHopLimit=1 in a stack update. The instance then serves role credentials only to processes on the host. What stops working:

  • The Use the server's IAM role Bedrock option, and any other use of the instance role from inside the app or a workbench container.
  • Connector scripts that rely on the role for AWS access.

What keeps working: Bedrock providers configured with Access keys or a Bedrock API key, every other provider, and Session Manager access.

An existing stack adopts it with a stack update#

Update the CloudFormation stack with the current template. The hop limit and the role policy change in place: the instance is not replaced and not restarted, and the containers need no restart. Read the change set first as described below. The new parameters default to the values above, so an update with no parameter changes turns the hop limit to 2 and adds the Bedrock policy. To keep an existing stack as it was, pass ImdsHopLimit=1 and EnableBedrockAccess=false in the same update.

A box that is not on EC2 has no instance role. Put AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_REGION in /opt/wi/.env and run docker compose up -d in /opt/wi, or configure the provider with access keys. An in-app update does not re-read .env. See What an update changes.

What the box exposes#

The security group allows inbound TCP 80 and 443, and TCP 22 only when you set SshCidr. No other inbound port is open.

Builders push Apps with git push to the Dokku container, which publishes the box's port WI_DOKKU_SSH_PORT (default 22). For a push to reach it from outside the VPC:

  • set SshCidr to a CIDR that contains the Builders' network, and
  • make sure port 22 on the host is free for Dokku. Amazon Linux's own SSH server normally uses it. If docker compose reports that port 22 is already allocated, set WI_DOKKU_SSH_PORT in .env to a free port and add a matching inbound rule to the security group yourself, because the template opens only 22.

Builders push to deployed Apps over port 2222, which the template opens to DokkuSshCidr (default 0.0.0.0/0; narrow it to restrict who can connect). The application's own calls to Dokku stay on the internal Docker network and need no inbound port.

DomainName and the instance#

The DomainName and HostedZoneId parameters create the DNS record and nothing else. They never change the instance or /opt/wi/.env, so updating them can never interrupt or replace the box. Whenever you set or change the domain, update WI_DOMAIN in .env yourself, then run sudo docker compose -f docker-compose.prod.yml up -d in /opt/wi. See Domains and HTTPS.

Upgrading an existing box and the template#

The template writes install.sh only when an instance first boots. To move an installed box to a release that changes the stack, copy the current install.sh to /opt/wi and re-run it (see Upgrade the stack by re-running install.sh). Updating the CloudFormation stack is not needed for that. Updating it with a newer template changes the instance's startup script, which CloudFormation applies as a stop and start of the instance (not a replacement; termination protection stays). Without an Elastic IP that stop and start changes the public IP, so set AllocateElasticIp=true first or update WI_DOMAIN (or WI_ALLOWED_ORIGIN) afterwards.

Always read the change set before updating the stack#

Before you update the stack, create a change set and read it:

aws cloudformation create-change-set ...
aws cloudformation describe-change-set ...

If Instance shows Replacement: True, do not execute the change set unless you intend to rebuild the box. Changes to LatestAmiId, SnapshotId, RootVolumeSizeGiB, and the root volume tags each replace the instance.

Termination protection and DeletionPolicy: Retain keep the old instance in place after a replacement, so a replacement leaves a new, empty box next to the old one with your data. The Elastic IP and DNS move to the new box, so the site looks like a fresh install. If that happens, stop and follow Backups and recovery on AWS instead of starting onboarding again.