By Sebastian Stadil —

Terraform Remote-Exec: A Concise Guide

Introduction

Terraform's remote-exec provisioner enables infrastructure teams to execute commands directly on newly provisioned resources, bridging the gap between infrastructure creation and configuration. While HashiCorp recommends using provisioners as a "last resort," there are legitimate scenarios where remote-exec provides the flexibility needed for specific configuration tasks.

This guide provides a comprehensive overview of remote-exec provisioners, including practical examples, security considerations, and best practices for managing them at enterprise scale. We'll also explore how modern infrastructure management platforms can help teams implement these patterns more effectively.

Understanding Remote-Exec Provisioners

The remote-exec provisioner invokes scripts or commands on a remote resource after it has been created. Unlike local-exec (which runs commands on the machine executing Terraform), remote-exec connects to and executes commands directly on the target resource.

When to Use Remote-Exec

Appropriate use cases:

  • Simple post-deployment configuration tasks
  • Software installation on newly created servers
  • Service initialization and bootstrapping
  • Testing and validation of deployed resources
  • One-time setup operations that can't be handled by cloud-init

When to avoid remote-exec:

  • Complex configuration management (use specialized tools)
  • Ongoing maintenance tasks (not tracked in Terraform state)
  • When cloud-init or user data scripts are sufficient
  • For changes that should be detected by future Terraform runs

The key limitation is that provisioners aren't declarative, aren't tracked in state, can't detect drift, and aren't idempotent by default.

Basic Configuration and Syntax

Connection Block Structure

Every remote-exec provisioner requires a connection block that defines how Terraform connects to the remote resource:

resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
  key_name      = aws_key_pair.deployer.key_name

  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = file("${path.module}/private_key.pem")
    host        = self.public_ip
    timeout     = "5m"
  }

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y nginx",
      "sudo systemctl start nginx",
      "sudo systemctl enable nginx"
    ]
  }
}

Execution Methods

Remote-exec supports three execution methods:

1. Inline commands:

provisioner "remote-exec" {
  inline = [
    "command1",
    "command2",
    "command3"
  ]
}

2. Single script file:

provisioner "remote-exec" {
  script = "path/to/setup.sh"
}

3. Multiple script files:

provisioner "remote-exec" {
  scripts = [
    "path/to/first_script.sh",
    "path/to/second_script.sh"
  ]
}

Failure Handling

Control how Terraform handles provisioner failures:

provisioner "remote-exec" {
  inline = [
    "command1",
    "command2"
  ]
  on_failure = "continue"  # or "fail" (default)
  when = "create"          # or "destroy"
}

Connection Types: SSH and WinRM

SSH for Linux/Unix Systems

Basic SSH configuration:

connection {
  type        = "ssh"
  user        = "ec2-user"
  private_key = file("~/.ssh/id_rsa")
  host        = self.public_ip
  port        = 22
  timeout     = "2m"
}

SSH with bastion host:

connection {
  type                = "ssh"
  user                = "ubuntu"
  private_key         = file("~/.ssh/id_rsa")
  host                = self.private_ip
  bastion_host        = "bastion.example.com"
  bastion_user        = "bastion-user"
  bastion_private_key = file("~/.ssh/bastion_key")
}

WinRM for Windows Systems

Basic WinRM configuration:

connection {
  type     = "winrm"
  user     = "Administrator"
  password = var.admin_password
  host     = self.public_ip
  port     = 5985
  timeout  = "10m"
}

Windows instance with WinRM setup:

resource "aws_instance" "windows" {
  ami           = "ami-windows-server-2019"
  instance_type = "t2.medium"
  
  user_data = <<EOF
<powershell>
Enable-PSRemoting -Force
winrm quickconfig -q
winrm set winrm/config/service '@{AllowUnencrypted="true"}'
winrm set winrm/config/service/auth '@{Basic="true"}'
netsh advfirewall firewall add rule name="WinRM-HTTP" dir=in localport=5985 protocol=TCP action=allow
</powershell>
EOF

  connection {
    type     = "winrm"
    user     = "Administrator"
    password = var.admin_password
    host     = self.public_ip
    port     = 5985
    timeout  = "10m"
  }

  provisioner "remote-exec" {
    inline = [
      "powershell.exe -Command \"Install-WindowsFeature -Name Web-Server -IncludeManagementTools\"",
      "powershell.exe -Command \"Add-Content -Path C:\\inetpub\\wwwroot\\index.html -Value '<h1>Hello from Terraform!</h1>'\""
    ]
  }
}

Practical Use Cases with Code Examples

Application Deployment

resource "aws_instance" "app_server" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
  key_name      = aws_key_pair.deployer.key_name

  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = file("~/.ssh/id_rsa")
    host        = self.public_ip
  }

  # Copy application files
  provisioner "file" {
    source      = "app/"
    destination = "/tmp/app"
  }

  # Install dependencies and deploy
  provisioner "remote-exec" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y nodejs npm nginx",
      "cd /tmp/app && npm install",
      "sudo cp -r /tmp/app /var/www/",
      "sudo chown -R www-data:www-data /var/www/app",
      "sudo systemctl start nginx",
      "sudo systemctl enable nginx"
    ]
  }
}

Database Configuration

resource "aws_instance" "database" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.small"

  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = file("~/.ssh/id_rsa")
    host        = self.public_ip
  }

  provisioner "remote-exec" {
    inline = [
      "sudo apt-get update",
      "sudo apt-get install -y postgresql postgresql-contrib",
      "sudo systemctl start postgresql",
      "sudo systemctl enable postgresql",
      "sudo -u postgres createdb ${var.database_name}",
      "sudo -u postgres psql -c \"CREATE USER ${var.db_user} WITH PASSWORD '${var.db_password}';\"",
      "sudo -u postgres psql -c \"GRANT ALL PRIVILEGES ON DATABASE ${var.database_name} TO ${var.db_user};\""
    ]
  }
}

Cluster Node Configuration

resource "aws_instance" "k8s_worker" {
  count         = var.worker_count
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.medium"

  connection {
    type        = "ssh"
    user        = "ubuntu"
    private_key = file("~/.ssh/id_rsa")
    host        = self.public_ip
  }

  provisioner "remote-exec" {
    inline = [
      "curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -",
      "echo 'deb https://apt.kubernetes.io/ kubernetes-xenial main' | sudo tee /etc/apt/sources.list.d/kubernetes.list",
      "sudo apt-get update",
      "sudo apt-get install -y kubelet kubeadm kubectl",
      "sudo kubeadm join ${var.master_ip}:6443 --token ${var.join_token} --discovery-token-ca-cert-hash ${var.cert_hash}"
    ]
  }

  depends_on = [aws_instance.k8s_master]
}

Security Best Practices

Credential Management

Use variables for sensitive data:

variable "admin_password" {
  type        = string
  sensitive   = true
  description = "Administrator password for Windows instances"
}

connection {
  type     = "winrm"
  user     = "Administrator"
  password = var.admin_password
  host     = self.public_ip
}

Leverage external secret management:

data "aws_secretsmanager_secret_version" "db_creds" {
  secret_id = "prod/database/credentials"
}

locals {
  db_creds = jsondecode(data.aws_secretsmanager_secret_version.db_creds.secret_string)
}

provisioner "remote-exec" {
  inline = [
    "export DB_USER='${local.db_creds.username}'",
    "export DB_PASS='${local.db_creds.password}'",
    "bash /tmp/configure_db.sh"
  ]
}

Network Security

Restrict access with security groups:

resource "aws_security_group" "provisioning" {
  name        = "terraform-provisioning"
  description = "Temporary access for Terraform provisioning"

  ingress {
    from_port   = 22
    to_port     = 22
    protocol    = "tcp"
    cidr_blocks = ["${data.external.my_ip.result.ip}/32"]
  }

  egress {
    from_port   = 0
    to_port     = 0
    protocol    = "-1"
    cidr_blocks = ["0.0.0.0/0"]
  }

  tags = {
    Name = "terraform-provisioning"
  }
}

Script Security

Make scripts idempotent and secure:

provisioner "remote-exec" {
  inline = [
    "set -euo pipefail",  # Exit on error, undefined vars, pipe failures
    "export DEBIAN_FRONTEND=noninteractive",
    "if ! command -v nginx &> /dev/null; then",
    "  sudo apt-get update",
    "  sudo apt-get install -y nginx",
    "fi",
    "sudo systemctl enable nginx",
    "sudo systemctl start nginx"
  ]
}

Troubleshooting Common Issues

Connection Problems

SSH timeout errors:

connection {
  type        = "ssh"
  user        = "ubuntu"
  private_key = file("~/.ssh/id_rsa")
  host        = self.public_ip
  timeout     = "10m"  # Increase timeout
  agent       = false  # Disable SSH agent
}

Host key verification:

connection {
  type        = "ssh"
  user        = "ubuntu"
  private_key = file("~/.ssh/id_rsa")
  host        = self.public_ip
  host_key    = null  # Skip host key verification (use carefully)
}

Script Execution Issues

Handle missing dependencies:

provisioner "remote-exec" {
  inline = [
    "which curl || sudo apt-get install -y curl",
    "which jq || sudo apt-get install -y jq",
    "curl -s https://api.example.com/status | jq '.health'"
  ]
}

Wait for system readiness:

provisioner "remote-exec" {
  inline = [
    "until [ -f /var/lib/cloud/instance/boot-finished ]; do sleep 1; done",
    "sudo apt-get update",
    "sudo apt-get install -y nginx"
  ]
}

Alternatives and When to Use Them

Cloud-Init for Initial Configuration

Instead of remote-exec for basic setup:

resource "aws_instance" "web" {
  ami           = "ami-0c55b159cbfafe1f0"
  instance_type = "t2.micro"
  
  user_data = <<-EOF
    #!/bin/bash
    apt-get update
    apt-get install -y nginx
    systemctl enable nginx
    systemctl start nginx
    echo "<h1>Hello World</h1>" > /var/www/html/index.html
  EOF
}

Configuration Management Tools

Ansible integration:

resource "null_resource" "ansible_playbook" {
  provisioner "local-exec" {
    command = "ansible-playbook -i '${aws_instance.web.public_ip},' --private-key=${var.private_key_path} playbook.yml"
  }
  
  depends_on = [aws_instance.web]
}

Provider-Specific Solutions

AWS Systems Manager:

resource "aws_ssm_document" "install_nginx" {
  name          = "install-nginx"
  document_type = "Command"
  
  content = jsonencode({
    schemaVersion = "2.2"
    description   = "Install nginx"
    mainSteps = [{
      action = "aws:runShellScript"
      name   = "installNginx"
      inputs = {
        runCommand = [
          "apt-get update",
          "apt-get install -y nginx",
          "systemctl start nginx"
        ]
      }
    }]
  })
}

resource "aws_ssm_association" "install_nginx" {
  name = aws_ssm_document.install_nginx.name
  targets {
    key    = "InstanceIds"
    values = [aws_instance.web.id]
  }
}

Managing Remote-Exec at Scale

Challenges with Large Deployments

When managing infrastructure at enterprise scale, several challenges emerge with remote-exec provisioners:

  1. Concurrent connection limits - Most systems limit simultaneous SSH/WinRM connections
  2. Network reliability - Provisioners can fail due to transient network issues
  3. Credential management - Securely distributing and rotating access credentials
  4. Audit and compliance - Tracking what commands were executed where and when
  5. State management - Provisioners don't integrate with Terraform state for drift detection

Enterprise Solutions

Modern infrastructure management platforms address these challenges by providing:

  • Centralized execution - Commands are executed from a secure, managed environment
  • Credential injection - Secrets are securely injected at runtime without storage in configuration
  • Execution logging - All provisioner activities are logged for audit purposes
  • Policy enforcement - Governance policies can prevent risky provisioner configurations
  • Scalable orchestration - Handle hundreds of concurrent provisioning operations

For organizations managing complex Terraform deployments, platforms like Scalr provide these capabilities while maintaining the flexibility of remote-exec when needed.

Summary and Recommendations

Aspect Recommendation Notes
Use Cases Last resort for simple, one-time configuration Prefer cloud-init, user data, or configuration management tools
Connection Type SSH for Linux, WinRM for Windows Use key-based authentication over passwords
Script Organization Use external scripts for complex logic Inline commands for simple, short operations
Error Handling Always include set -e in bash scripts Use on_failure = "continue" judiciously
Security Store credentials in external secret management Never hardcode passwords or keys
Network Access Restrict to specific IPs via security groups Use bastion hosts for private resources
Timeout Settings Set realistic timeouts (5-10 minutes) Consider long-running operations carefully
Idempotency Make scripts idempotent with proper checks Avoid destructive operations without guards
Scalability Use orchestration platforms for large deployments Consider centralized execution for enterprise use
Alternatives Evaluate cloud-init, SSM, or config management first Reserve remote-exec for specific requirements

Key Takeaways

  1. Remote-exec is powerful but limited - It provides flexibility but lacks state management and drift detection
  2. Security is paramount - Always use secure authentication and restrict network access
  3. Idempotency matters - Scripts should be safe to run multiple times
  4. Consider alternatives first - Cloud-init, user data, and configuration management tools are often better choices
  5. Scale requires orchestration - Enterprise deployments benefit from centralized management platforms

For teams managing Terraform at scale, combining remote-exec with modern infrastructure management platforms provides the best of both worlds: the flexibility to handle edge cases while maintaining security, compliance, and operational excellence.

The remote-exec provisioner remains a valuable tool in the Terraform ecosystem when used appropriately. By following the best practices outlined in this guide and considering the broader context of your infrastructure management strategy, you can leverage remote-exec effectively while avoiding common pitfalls.