Dagster¶

Dagster Overview¶

Dagster, an orchestration platform for managing data pipelines (DAGs), automates Tailscale and MDM workflows in this project, ensuring efficient and reliable task execution.

Integrating Dagster with Django allows access to Django’s ORM, settings, and management commands within pipelines. This exploratory feature aims to simplify tasks like database interactions by leveraging Django for schema management and maintaining consistency between application and data workflows.

For production, Dagster is deployed in a Kubernetes environment, but it can also be run locally for development and testing.

Running the Development Server¶

Set the following environment variables in your shell before running the Dagster development server:

# sample .envrc file
# Android Enterprise MDM server-wide service account file
export ANDROID_ENTERPRISE_SERVICE_ACCOUNT_FILE=

To run the Dagster development server locally, just run the following command:

dagster dev

This command starts the Dagster web server, which provides a user interface for monitoring and managing the pipelines. You can access the web server at http://localhost:3000 in your web browser.

Tailscale¶

Tailscale integration maintains a list of devices and metadata, including device names, IP addresses, and last seen timestamps. Tailscale OAuth clients allow fine-grained control on the access granted to the client using scopes, unlike a fully-permitted access token which grants full access to the Tailscale API.

To configure Tailscale OAuth, follow the Setting up an OAuth client guide to create a new client with the devices:core:read scope, and then configure the following environment variables:

export TAILSCALE_OAUTH_CLIENT_ID=
export TAILSCALE_OAUTH_CLIENT_SECRET=
export TAILSCALE_TAILNET=

Now you can run the Dagster development server and sync Tailscale device snapshots from your tailnet.

OAuth client for Deletion of Inactive Tailscale Devices¶

Deletion of tailscale devices requires an OAuth client with write access for the devices:core scope. To create this, do as follows:

Login to the tailscale admin console UI and navigate to the Settings tab.
Go to the OAuth clients.
Click Generate OAuth client
Select the devices:core scope and check the write checkbox.
Select a tag as it is required for the write scope. If a tag doesn’t exist, go to the the Access Controls tab in the UI console and update policy file. For instance to create a tag named app, add to the tagOwners code block and nest it to a parent tag as shown below:
```
"tagOwners": {
        "tag:admin": ["autogroup:admin"],
        "tag:app":   ["tag:admin"],
}
```
Click Save and copy the generated Client ID and Client Secret.