Skip to content

Cosmian VM

Cosmian VM allows you to deploy an application on a cloud provider instance, running in a confidential context with verifiability at any time.

  • No code modification: the application doesn’t need any third party library or any specific adaptation
  • Simple: reduce at its minimum the number of manual actions the user has to do to spawn a confidential VM
  • Confidentiality: the application runs in a Trusted Execution Environment with encrypted memory
  • Generic: same solution to support any kind of Trusted Execution Environment (Intel, AMD, etc.)
  • Verifiability: a user is able to verify the integrity of the system (Operating System and applications) at any time

Threat Model

Cosmian VM is designed to secure your application against passive (honest-but-curious) and active (malicious) cloud provider staff member

Cosmian offers a Linux-based system image preconfigured with all the tools to verify the VM at anytime. It can be used like a regular Linux distribution with Confidential VM instances on most cloud providers.

The system image contains:

  • cosmian_vm_agent serves on demand, through HTTPS, the collaterals used to verify the trustworthiness of the Cosmian VM, including IMA measurement log with TPM attestation and the TEE attestation
  • cosmian_certtool to generate a certificate signed by Let’s Encrypt. This tool is an add-on and is not required in most cases
  • cosmian_fstool to generate a luks container and enroll the TPM to be automatically mounted on reboot

Our open source cosmian_vm CLI client can be used by system administrators to query cosmian_vm_agent and verify the trustworthiness of a specific instance running Cosmian VM.

setup flow

Audit

Cosmian VM code and image generation process can be found in the following GitHub repository: https://github.com/Cosmian/cosmian_vm

Usage flow

usage flow

There are at least 2 actors:

  • [SYS ADMIN] The system administrator spawning the VM
  • [DEV] The application developer installing the application inside the VM

The application developer doesn’t need any skill or knowledge about trusted environments: Cosmian VM is used as a regular VM.

Prerequisites

Download & Install the CLI as follow:

$ # Feel free to download the proper version of the CLI. For example for 1.0.0:
$ wget https://github.com/Cosmian/cosmian_vm/releases/download/1.0.0/cosmian_vm && chmod +x ./cosmian_vm

This CLI has been compiled for Linux only. On MacOS/Windows, run it inside ubuntu docker for instance.

Setup

The setup stages can be sum up as follow:

  • [SYS ADMIN] instanciates a SEV/TDX VM based on Cosmian VM image through GCP/AWS
  • [SYS ADMIN] configures the VM on a regular basis (users, networking, etc.)
  • [SYS ADMIN] delivers this VM to the developer
  • [DEV] installs its own application & deps
  • [SYS ADMIN] snapshots (see below for more details) the VM using the CLI as following:
$ cosmian_vm --url https://cosmian-vm.company.com snapshot

Verification steps

Verification process of the Cosmian VM is performed by the system administrator using cosmian_vm which will check:

  • IMA (Integrity Measurement Architecture) measurement log with the list of executable and configuration file’s hash digest, to be compared against the snapshot
  • TPM (Trusted Platform Module) attestation of the IMA measurement log
  • TEE (Trusted Execution Environment) trustworthiness to provide assurance that the code is running on secure hardware using encrypted memory

simple verification flow

detailed verification flow

The verification is done using the CLI as follow which can easily be automated using cron or any monitoring system:

$ cosmian_vm --url https://cosmian-vm.company.com verify --snapshot cosmian_vm.snapshot  

When verifying a Cosmian VM you can also check that the TLS certificate of your services installed inside this VM are the one used when querying the Cosmian VM Agent during the verification. The goal is to verify your services are actually running inside this Cosmian VM. To do so use --application (as many times as you want) as follow:

$ cosmian_vm --url https://cosmian-vm.company.com verify --snapshot cosmian_vm.snapshot \
                                                         --application service1.company.com:3655 \
                                                         --application service2.company.com 

Coverage

Cosmian VM supports these kinds of TEE:

  • Intel SGX (contact us for more practical details)
  • Intel TDX
  • AMD SEV

Cosmian VM is available on these cloud providers & distributions:

GCP AWS
TDX Ubuntu 22.04 Not supported
SEV Ubuntu 22.04 & RHEL 9 Amazon Linux

Production usage

We strongly recommend to use the RHEL image. This image enables SELinux which is used to detect any file modification. For example, on RHEL, the configuration files from applications running inside the VM are monitored and checked. On Ubuntu, only the executable binaries are monitored.

What does the snapshot contain?

The snapshot is performed by the Cosmian VM Agent by querying the filesystem, the TPM and the TEE to produce a single file standing for the trusted state of the machine at the current time. This file is returned to the Cosmian CLI when using the subcommand snapshot.

The snapshot file is a JSON file containing:

  • The IMA measurement log at the current time
  • The list of all measured files and their corresponding hash (sha256 by default)
  • The list of trusted TEE parameters when the snapshot occurs
  • The list of trusted TPM parameters when the snapshot occurs

Here an sample:

{
  "tee_policy":{
    "Sev":{
      "measurement":"73797518025d1d20e09efdf10e383cd0115e00562109b04ec577b5bd5d2ddab12f1f9ee22758f50a121355cf5aac6507",
      "report_data":"00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
      "family_id":"00000000000000000000000000000000",
      "image_id":"00000000000000000000000000000000",
      "guest_svn":0,
      "id_key_digest":"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
      "author_key_digest":"000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000",
      "policy":196608,
      "report_id":"b6db1e43f868a4823f708df92942a0448750fc5c37f7a724bc49292ffd111e1a",
      "report_id_ma":"ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff"
    }
  },
  "tpm_policy":{
    "reset_count":15,
    "restart_count":0
  },
  "filehashes":[
    ["/snap/google-cloud-cli/187/platform/gsutil/third_party/mock/mock/__pycache__/__init__.cpython-39.pyc", "f708df92942a044875[...]abc"],
    ...
  ]
}

How fast is the whole process?

  • Snapshotting the Cosmian VM is a long process depending in the disk usage of the VM
  • Verifying the Cosmian VM is pretty instant

Why does the verification fail after a rebooting?

Due to the IMA design, the VM can’t be trusted after a reboot. Indeed the IMA is reset at each restart. Cosmian VM uses the TPM to detect a VM reboot. If the reset counter differs from the one captured during the snapshot, the verification fails.

If this reboot is trusted (for instance it has been made by the system administrator), you just need to create a new snapshot. If the reboot occurs without the system administrator consent, the VM should be investigated to detect any malicious modification.

How the Ubuntu/RHEL base image is modified by Cosmian?

The modifications are related to the installation and the configuration of Cosmian VM components. The whole modifications are proceed using Packer and can be found in Github page.

This image:

  • contains the fully configured IMA
  • contains the fully configured SELinux (RHEL only)
  • disables the auto-update (to avoid any modification of the Cosmian VM after having snapshoted it)
  • contains the fully configured cosmian_vm_agent

This is an abstract of the updated file tree:

.
├── etc
│   ├── apt
│   │    └── apt.conf.d
│   │       └── 10periodic
│   ├── cosmian_vm
│   │   └── agent.toml
│   ├── default
│   │   └── grub
│   ├── ima
│   │   └── ima-policy
│   └── supervisor
│       ├── supervisord.conf
│       └── conf.d
│           └── cosmian_vm_agent.conf
├── usr
│   └── sbin
│       ├── cosmian_certtool
│       ├── cosmian_fstool
│       └── cosmian_vm_agent
└── var
    ├── lib
    │   └── cosmian_vm
    │       ├── container
    │       ├── tmp
    │       └── data
    │           ├── cert.pem
    │           └── cert.key
    └── log
        └── cosmian_vm
            ├── agent.err.log
            └── agent.out.log

Why is the auto-update disabled?

The verification performed by the Cosmian VM relies on the fact that once the snapshot has been made, the VM content shouldn’t be altered. If any modifcation is detected, the VM is considered compromised.

An auto-update processing alters the VM and makes the comparison with the snapshot impossible.

You shall update the Cosmain VM manually and create a new snapshot afterwards.

How to configure the HTTPS through the Cosmian VM Agent?

On a fresh installation, the cosmian_vm_agent uses a self-signed certificate generated at the start of the service and set the CommonName of the certificate to the value of the machine hostname.

Therefore when using the CLI you should add the --allow-insecure-tls to ignore SSL errors. This is not a good practice in production.

You can change that at will:

  • Edit your DNS registry to point to that VM
  • Create a trusted certificate using the method of your choice (Let’s encrypt for instance) or using cosmian_certtool
  • Edit the cosmian_vm_agent configuration file to point to the location of the TLS certificate and private key

How to configure the Cosmian VM Agent?

The Cosmian VM Agent relies on a configuration file located at /etc/cosmian_vm/agent.toml. Feel free to edit it. A minimal required configuration file is:

[agent]
host = "127.0.0.1"
port = 5355
ssl_certificate = "data/cert.pem"
ssl_private_key = "data/key.pem"
tpm_device = "/dev/tpmrm0"

By default, cosmian_vm_agent uses the port 5355 on localhost.

Here an example of configuration to use the certificate from another application running inside the Cosmian VM:

[agent]
host = "0.0.0.0"
port = 5355
ssl_certificate = "/etc/letsencrypt/live/app.company.com/fullchain.pem" 
ssl_private_key = "/etc/letsencrypt/live/app.company.com/privkey.pem"
tpm_device = "/dev/tpmrm0"

Could I use a proxy in front of the Cosmian VM Agent?

Although it’s technically possible to configure an HTTPS proxy in front of the Cosmian VM Agent, it will prevent you from proceeding the verification through the CLI if you configure the proxy as an SSL forward.

Indeed, the TLS certificate configured in the Cosmian VM Agent is a part of the exchange with the TEE. To avoid a malicious program to send fake valid collaterals, the TLS certificate used to get these collaterals should be the same as the one configured in the Cosmian VM Agent. This certificate should stay inside the Cosmian VM.

If you use a proxy, the TLS tunnel, from the CLI to the agent, uses the certificate of the proxy and not the certificate of the agent. The verification will therefore fail, being not able to determine if it’s a malicious or healthy behaviour.

However, if you use a proxy as an SSL passthrough, it will works as a charm. Here an example with HAProxy:

[...]

defaults
    mode                 tcp
    option               tcplog
    [...]

# Entrypoint of the ha_proxy listen on 443
frontend https-in
    # Do not decrypt ssl yet
    bind *:443

    tcp-request inspect-delay 60s
    tcp-request content accept if { req_ssl_hello_type 1 }
    use_backend 8d14ff4ac2d452c3.cosmian.io if { req_ssl_sni -i 8d14ff4ac2d452c3.cosmian.io }
    # No default backend: return an error as fallback

backend 8d14ff4ac2d452c3.cosmian.io
    server haproxy-app 162.19.91.151:27283 check 

Other resources:

How to start/stop the Cosmian VM Agent?

You can start/restart/stop the Cosmian VM Agent as follow:

$ # If the surpervisor configuration file has been edited, reload it first
$ supervisorctl reload cosmian_vm_agent
$ supervisorctl start cosmian_vm_agent
$ # Or
$ supervisorctl restart cosmian_vm_agent
$ # Or
$ supervisorctl stop cosmian_vm_agent

Where the Cosmian VM logs are located?

You can find the logs in /var/log/cosmian_vm/.

Feel free to contact us for any assistance.

How the Cosmian VM filesystem is protected?

Out of the box, Cosmian VM filesystem is not entirely protected against the cloud provider. However, you get two ways to store securely your sensitive data:

  • A luks container
  • A RAMFS

Using /var/lib/cosmian_vm/data

At the first start of the cosmian_vm_agent a luks container is created at: /var/lib/cosmian_vm/container (size=500MB). This container is mounted into /var/lib/cosmian_vm/data. This directory can be used to store any sensitive data for your application. For example, the TLS certificate should be stored inside it. The password of this default container can be retrieved in the logs of cosmian_vm_agent. As a matter of fact, this container shouldn’t be used in production. Instead, regenerate your own container as follow:

$ cosmian_fstool --size 100GB \
                 --location /data/container \

Define your own container size and save the container to an additional disk for back-up (/data/container in that example).

A prompt will invite you to set the password. Save this password. It could be useful if you want to migrate the container to another VM. For this Cosmian VM the luks is enrolled on the current TPM therefore the password won’t be asked again, even at reboot.

The field ssl_certificate and ssl_private_key could be relative paths. In that case, this is relative to /var/lib/cosmian_vm/.

If you need to manage the container (extend its size, revoke the password, etc.), you can use cryptsetup.

Using /var/lib/cosmian_vm/tmp

The Cosmian VM also contains an encrypted RAMFS at: /var/lib/cosmian_vm/tmp. The data stored inside it are volatile: they will be deleted when rebooting the VM. The size of this directory is 500MB.

How to use Cosmian VM on SGX?

Feel free to contact us.

Cosmian VM on SGX offers you new usage scenarios against more extended threat models.

How to remotely control the application?

The cosmian_vm CLI also contains two subcommands designed to drive your application running inside the Cosmian VM.

It could be relevant if the personnel in charge of the application doesn’t have the rights to connect to the Cosmian VM through SSH for instance.

Security

If your Cosmian VM is reachable over Internet, be aware that anyone can control your application. Out of the box the access to the cosmian_vm_agent endpoints is not authenticated.

Before going any further, you need to add a paragraph app inside the Cosmian VM configuration file, as follow:

[agent]
host = "0.0.0.0"
port = 5355
ssl_certificate = "data/cosmianvm.cosmian.dev/cert.pem"
ssl_private_key = "data/cosmianvm.cosmian.dev/key.pem"
tpm_device = "/dev/tpmrm0"

[app]
service_type = "supervisor"
service_name = "cosmian_helloworld"
app_storage = "data/app"

The service_type could be standalone, systemd or supervisor and defines the way to start/stop the application. The service_name field defines the name of the service or the binary to start/stop. The app_storage field defines the root path where the application will store its data. If it’s a relative path, the root path will be located inside /var/lib/cosmian_vm/. If you put it in the subfolder data, this directory is therefore encrypted and protected from the cloud provider.

Let’s imagine, the application is installed in the Cosmian VM but not configured and started. Then, you can provide a configuration file (containing secrets for instance) and start it using:

$ cosmian_vm --url https://cosmian-vm.company.com app init --conf app.json

The configuration file can be anything the application expects. Here, a json file. It will be sent to the cosmian_vm_agent and stored in the luks container in /var/lib/cosmian_vm/data/app/app.conf.

If you call again init the previous configuration file is overwritten.

The restart subcommand can restart the application identified in service_name field.

© Copyright 2018-2024 Cosmian. All rights reserved.