Configuring and running a Kubernetes agent#

In this guide, you'll configure and run a Kubernetes agent. Kubernetes agents are used to launch your code in Kubernetes Jobs and Services.

Prerequisites#

To complete the steps in this guide, you'll need:

• Permissions in Dagster+ that allow you to manage agent tokens. Refer to the User permissions documentation for more info.

• A Kubernetes cluster into which you can deploy the agent. This can be a self-hosted Kubernetes cluster or a managed offering like Amazon EKS, Azure AKS, or Google GKE.

• Access to a container registry to which you can push images and from which pods in the Kubernetes cluster can pull images. This can be:

  • A self-hosted registry,
  • A public registry such as DockerHub, or
  • A managed offering such as Amazon ECR, Azure ACR, or Google GCR

• To have Helm installed. Refer to the Helm installation documentation for more info.

Step 1: Generate a Dagster+ agent token#

In this step, you'll generate a token for the Dagster+ agent. The Dagster+ agent will use this to authenticate to the agent API.

1. Sign in to your Dagster+ instance.
2. Click the user menu (your icon) > Organization Settings.
3. In the Organization Settings page, click the Tokens tab.
4. Click the + Create agent token button.
5. After the token has been created, click Reveal token.

Keep the token somewhere handy - you'll need it to complete the setup.

Step 2: Configure the agent#

In this step, you'll create a Kubernetes namespace for your Dagster+ resources. You'll also add the agent token to the Kubernetes cluster.

1. Create a Kubernetes namespace for your Dagster+ resources:

   kubectl create namespace <namespace>
   

   For example, if the namespace is dagster-plus:

   kubectl create namespace dagster-plus
   

2. Add the agent token you created in Step 1 as a secret in the Kubernetes cluster:

   kubectl create secret generic dagster-cloud-agent-token \
       --from-literal=DAGSTER_CLOUD_AGENT_TOKEN=<your_agent_token> \
       --namespace dagster-plus
   

Step 3: Start the agent#

In this step, you'll spin up the agent with Helm.

1. Add the agent chart repository:

   helm repo add dagster-cloud https://dagster-io.github.io/helm-user-cloud
   helm repo update
   

2. Create a values.yaml file, indicating the Dagster+ deployment the agent will be responsible for. Note: When your Dagster+ organization is first created, the default deployment is prod:

   dagsterCloud:
     deployment: prod
   

3. Install the agent chart, specifying:

   • The namespace into which to install the chart
   • The values.yaml file created in the previous step

   helm upgrade \
      --install user-cloud dagster-cloud/dagster-cloud-agent \
      --namespace dagster-plus \
      --values ./values.yaml
   

After the Helm chart is installed into the Kubernetes cluster, you can view the agent in Dagster+.

In Dagster+, navigate to the Status page and click the Agents tab. You should see the agent running in the Agent statuses section:

Next steps#

Now that you've got your agent running, what's next?

• If you're getting Dagster+ set up, the next step is to add a code location using the agent.

• If you're ready to load your Dagster code, refer to the Adding Code to Dagster+ guide for more info.

You can also further configure the jobs the agent runs, including adding secrets, mounting volumes, and setting resource limits.