workflow

Note

big-config.workflow has replaced big-config.step.

big-config.workflow

The goal of the BigConfig Workflow is to enable independent development of automation units while providing a structured way to compose them into complex pipelines.

Workflow Types

• tool-workflow: The fundamental unit. It renders templates and executes CLI tools (e.g., Terraform/OpenTofu, Ansible).
• comp-workflow: A high-level orchestrator that sequences multiple tool-workflows to create a unified lifecycle (e.g., create, delete).
• system-workflow: A lifecycle management engine (see big-config.system) that coordinates starting and stopping system components as an alternative to Integrant.

Usage Syntax

BigConfig Workflow can be used as a library or as a CLI using Babashka. The engine is powered by pluggable steps, allowing for seamless extension via multimethods.

# Execute a tool workflow directly
bb <tool-workflow> <step|cmd>+ [-- <raw-command>]

# Execute a composite workflow
bb <comp-workflow> <step>+

Examples

# Individual development/testing
bb tool-wf-a render tofu:init -- tofu apply -auto-approve
bb tool-wf-b render ansible-playbook:main.yml

# Orchestrated execution
bb comp-wf-c create

In this example, comp-wf-c composes tool-wf-a and tool-wf-b. Development and debugging happen within the individual tool workflows, while the composite workflow manages the sequence.

Available Steps

tool-workflow

Step	Description
render	Generate the configuration files.
git-check	Verifies the working directory is clean and synced with origin.
git-push	Pushes local commits to the remote repository.
lock	Acquires an execution lock.
unlock-any	Force-releases the lock, regardless of the current owner.
exec	Executes commands provided.

comp-workflow

Step	Description
create	Invokes one or more tool-workflows to create a resource.
delete	Invokes one or more tool-workflows to delete a resource.
git-check	Verifies the working directory is clean and synced with origin.
git-push	Pushes local commits to the remote repository.
lock	Acquires an execution lock.
unlock-any	Force-releases the lock, regardless of the current owner.

Extending Workflows with Custom Steps

The workflow engine is powered by pluggable steps. You can extend or override any step by defining a method for the big-config.pluggable/handle-step multimethod. This allows you to add custom logic or completely change the behavior of built-in steps like render or lock.

(require '[big-config.pluggable :as pluggable])
(require '[big-config.core :as core])

(defmethod pluggable/handle-step ::my-custom-step
  [step step-fns opts]
  ;; Your custom logic here
  (println "Executing custom step!")
  (core/ok opts))

;; Usage in a workflow
(run-steps step-fns {::workflow/steps [:my-custom-step]})

When using parse-args (e.g., in Babashka tasks), you might need to register new step names so they are recognized as steps rather than shell commands. This is done by rebinding the dynamic var *parse-args-steps*.

(binding [workflow/*parse-args-steps* (conj workflow/*parse-args-steps* "my-custom-step")]
  (workflow/parse-args ["my-custom-step" "render"]))

Core Logic & Functions

• run-steps: The engine for dynamic workflow execution. It’s a pluggable workflow.
• ->workflow*: Creates a workflow of workflows.
• prepare: Shared logic for rendering templates and initializing execution environments.
• parse-args: Utility function to normalize string or vector-based arguments.
• select-globals: Utility function to copy the global options across workflows.
• merge-params: Utility function to merge the package params with the tool params. tools is a sequence of qualified keywords.
• read-bc-pars: Utility function to override the package params with environment variable.

Options for wf*-opts.

• :first-step (required): First step of the workflow.
• :last-step (optional): Optional last step.
• :pipeline (required): A vector containing the repetition of:
  • The qualified keyword of the tool workflow (e.g., ::tools/tofu).
  • A vector containing:
    • Arguments for the tool workflow (e.g., ["render ..."]).
    • An optional opts-fn to adapt/merge outputs from previous steps into the current params.

Options for opts and step render

• ::name (required): The unique identifier for the workflow instance.
• ::path-fn (optional): Logic for resolving file paths.
• ::params (optional): The input data for the workflow. The conventions is to use unqualified keywords with prefixes like: :oci-config-file-profile or :hcloud-server-type.

Options for prepare-opts and step render

• ::name (required): The unique identifier for the workflow instance.

Options for prepare-overrides and step render

• ::path-fn (optional): Logic for resolving file paths.
• ::params (optional): The input data for the workflow.

Options for opts and step create or delete

• ::create-fn (required): The workflow to create the resource.
• ::delete-fn (required): The workflow to delete the resource.
• ::create-opts (optional): The override opts for create.
• ::delete-opts (optional): The override opts for delete.

Naming Conventions

To distinguish between the library core and the Babashka CLI implementation:

• [workflow name]: The library-level function. Requires step-fns and opts.
• [workflow name]*: The Babashka-ready task. Accepts args and optional opts.

(ns wf)

(defn tofu
  [step-fns opts]
  (let [opts (prepare {::name ::tofu
                       ::render/templates [{:template "tofu"
                                            :overwrite true
                                            :transform [["tofu"
                                                         :raw]]}]}
                      opts)]
    (run-steps step-fns opts)))

(ns wf)

(defn tofu*
  [args & [opts]]
  (let [opts (merge (parse-args args)
                    opts)]
    (tofu [] opts)))

; bb.edn
{:deps {group/artifact {:local/root "."}}
 :tasks
 {:requires ([wf :as wf])
  tofu {:doc "bb tofu render tofu:init tofu:apply:-auto-approve"
        :task (wf/tofu* *command-line-args*)}
  ansible {:doc "bb ansible render -- ansible-playbook main.yml"
           :task (wf/ansible* *command-line-args*)}
  resource {:doc "bb resource create"
            :task (wf/resource* *command-line-args*)}}}

Decoupled Data Sharing

Standard Terraform/HCL patterns often lead to tight coupling, where downstream resources must know the exact structure of upstream providers (e.g., the specific IP output format of AWS vs. Hetzner).

BigConfig Workflow solves this through Parameter Adaptation:

1. Isolation: tool-wf-b (Ansible) never talks directly to tool-wf-a (Tofu).
2. Orchestration: The comp-workflow acts as a glue layer. It uses path to discover outputs from the previous workflows (e.g., via tofu output --json) and maps them to the ::params required by the next.
3. Interchangeability: You can swap a Hetzner workflow for an AWS workflow without modifying the downstream Ansible code. Only the mapping logic in the comp-workflow needs to be updated.

Note: Resource naming and state booking are outside the scope of BigConfig Workflow.

Composite workflow

This example demonstrates a composite workflow orchestrating one or more tool workflows. To maintain modularity, tool workflows must be isolated using distinct opts maps. They may operate within the same directory—for instance, when pairing tofu apply and tofu destroy—or function in separate environments.

• resource-create: The composite workflow to create the resource.
• resource-delete: The composite workflow to delete the resource.
• resource: The composite workflow to expose the steps interface.

Note: resource-create and resource-delete share the same :first-step to ensure they utilize the same directory when Tofu is using local state.

(defn opts-fn
  [opts]
  (let [ip (-> (p/shell {:dir (workflow/path opts ::tool/tofu)
                         :out :string} "tofu show --json")
               :out
               (json/parse-string keyword)
               (->> (s/select-one [:values :root_module :resources s/FIRST :values :ipv4_address])))]
    (merge-with merge opts {::workflow/params {:ip ip}})))

(def resource-create
  (workflow/->workflow* {:first-step ::start-create-or-delete
                         :last-step :end-create-or-delete
                         :pipeline [::tool/tofu ["render tofu:init tofu:apply:-auto-approve"]
                                    ::tool/ansible ["render ansible-playbook:main.yml" opts-fn]
                                    ::tool/ansible-local ["render ansible-playbook:main.yml" opts-fn]]}))

(def resource-delete
  (workflow/->workflow* {:first-step ::start-delete-or-delete
                         :last-step ::end-delete-or-delete
                         :pipeline [::tool/tofu ["render tofu:destroy:-auto-approve"]]}))

(defn resource
  [step-fns opts]
  (let [opts (merge {::workflow/create-fn resource-create
                     ::workflow/delete-fn resource-delete}
                    opts)
        wf (core/->workflow {:first-step ::start
                             :wire-fn (fn [step step-fns]
                                        (case step
                                          ::start [(partial workflow/run-steps step-fns) ::end]
                                          ::end [identity]))})]
    (wf step-fns opts)))

*parse-args-steps*

_Source

->workflow*

(->workflow* wf*-opts)

Function.

Creates a workflow of workflows. See the namespace big-config.workflow.

_Source

merge-params

(merge-params tools params opts)

Function.

Merge the package params with the tool params. Tools is a seq of qualified keywords. See the namespace big-config.workflow.

_Source

parse-args

(parse-args str-or-args)

Function.

Utility functions to normalize string or vector-based arguments. See the namespace big-config.workflow.

_Source

path

("path[{:keys [::prefix]} name]")

Function.

Find the path of a previous workflow to extract the outputs. See the namespace big-config.workflow.

_Source

prepare

(prepare opts overrides)

Function.

Prepare opts. See the namespace big-config.workflow.

_Source

print-step-fn

(print-step-fn step opts)

Function.

Print all steps of the workflow. See the namespace big-config.workflow.

_Source

read-bc-pars

(read-bc-pars opts)

Function.

Function to override any params with an environment variable. If the param is cloudflare-zone-id then the environment variable is export BC_PAR_CLOUDFLARE_ZONE_ID="your-zone-id" See the namespace big-config.workflow.

_Source

run-steps

(run-steps step-fns opts)

Function.

A dynamic workflow that takes a list of steps, a create function, and a delete function. See the namespace big-config.workflow

_Source

select-globals

(select-globals {:keys [globals], :as opts})

Function.

_Source