Skip to content

Version 2.1.0

Support for Git repositories hosted on GitLab, Azure DevOps and BitBucket

We now support public and private repositories hosted on GitLab, Azure DevOps or BitBucket, as well as GitHub!

Working with private repositories

We have renamed the secrets that need to be created for working with private Git repositories - please refer to the User Guide for more information.

Git commit hash exposed as an environment variable in every Bodywork Container

The Git commit hash of a project can be accessed from within any Bodywork container, via the GIT_COMMIT_HASH environment variable. This allows you to tag any artefacts produced by your pipelines, such as datasets and trained models, with the precise version of the pipeline used to create them. For example,

import os

git_hash = os.getenv('GIT_COMMIT_HASH')
model_filename = f'my-classifier--pipeline={git_hash}.pkl'
save_model(model, model_filename)

Run a batch stage upon failure to trigger Notifications

It is now possible to specify a batch stage that is only executed after a workflow fails to complete successfully - i.e. when one of its stages fails. This special stage can now be configured using the optional project.run_on_failure parameter within bodywork.yaml, as follows,

version: "1.0"
project:
  name: bodywork-ml-pipeline-project
  docker_image: bodyworkml/bodywork-core:latest
  DAG: stage_1_train_model >> stage_2_scoring_service
  run_on_failure: send_notifications
stages:
  stage_1_train_model:
    ...
  stage_2_scoring_service:
    ...
  send_notifications:
    executable_module_path: raise_alerts.py
    requirements:
      - requests==2.22.0
    cpu_request: 0.5
    memory_request_mb: 100
    batch:
      max_completion_time_seconds: 30
      retries: 1

Within the raise_alerts.py module you could trigger a webhook for posting to Slack, use the Python standard library to send an e-mail, or use the Python client for your company's chosen incident response platform (e.g. PagerDuty).

Other Improvements and Bug Fixes

  • We've upgraded to the latest version of the Kubernetes Python client, which supports Kubernetes v1.17.17.
  • We've introduced an alias for the bodywork workflow command. For consistency, bodywork deployment create ... --local-workflow-cotroller will run a deployment with the workflow-controller running locally for testing.
  • We've introduced the bodywork deployment delete-job command, for cleaning-up completed workflow-controller jobs. Not all clusters are configured to clean-up these jobs up automatically, in which case you may have to delete them manually.