Version 2.1.0¶
Support for Git repositories hosted on GitLab, Azure DevOps and BitBucket¶
We now support public and private repositories hosted on GitLab, Azure DevOps or BitBucket, as well as GitHub!
Working with private repositories
We have renamed the secrets that need to be created for working with private Git repositories - please refer to the User Guide for more information.
Git commit hash exposed as an environment variable in every Bodywork Container¶
The Git commit hash of a project can be accessed from within any Bodywork container, via the GIT_COMMIT_HASH
environment variable. This allows you to tag any artefacts produced by your pipelines, such as datasets and trained models, with the precise version of the pipeline used to create them. For example,
import os
git_hash = os.getenv('GIT_COMMIT_HASH')
model_filename = f'my-classifier--pipeline={git_hash}.pkl'
save_model(model, model_filename)
Run a batch stage upon failure to trigger Notifications¶
It is now possible to specify a batch stage that is only executed after a workflow fails to complete successfully - i.e. when one of its stages fails. This special stage can now be configured using the optional project.run_on_failure
parameter within bodywork.yaml
, as follows,
version: "1.0"
project:
name: bodywork-ml-pipeline-project
docker_image: bodyworkml/bodywork-core:latest
DAG: stage_1_train_model >> stage_2_scoring_service
run_on_failure: send_notifications
stages:
stage_1_train_model:
...
stage_2_scoring_service:
...
send_notifications:
executable_module_path: raise_alerts.py
requirements:
- requests==2.22.0
cpu_request: 0.5
memory_request_mb: 100
batch:
max_completion_time_seconds: 30
retries: 1
Within the raise_alerts.py
module you could trigger a webhook for posting to Slack, use the Python standard library to send an e-mail, or use the Python client for your company's chosen incident response platform (e.g. PagerDuty).
Other Improvements and Bug Fixes¶
- We've upgraded to the latest version of the Kubernetes Python client, which supports Kubernetes v1.17.17.
- We've introduced an alias for the
bodywork workflow
command. For consistency,bodywork deployment create ... --local-workflow-cotroller
will run a deployment with the workflow-controller running locally for testing. - We've introduced the
bodywork deployment delete-job
command, for cleaning-up completed workflow-controller jobs. Not all clusters are configured to clean-up these jobs up automatically, in which case you may have to delete them manually.