Component Redesign

Aim: Redesign the components to enable a more modern and consistent workflow. The new structure allows:

• A CLI and an API (importable from other python packages)
• YAML based human configuration interface
• Wildcard branching
• Consolidated execution of gitdh and multi-instance gitdh
  • Both: Multi-repository configuration file and (recursive) directory reading
• A daemon / server mode
  • Arbitrary webhook support (could be provided externally, e.g. through adnanh/webhook)
  • GitHub Post-Push webhook integration
  • GitHub Deployment integration
  • GitLab Post-Push webhook integration
• Initiators
  • Extend concept of different "run modes" to the configuration file
  • Allow different partial configuration in different modes
    • Overwrite arbitrary parts of the default pipeline?
    • Replace the source of new data: Webhook when run as a service; Pull when run in one off / cron job mode.
• Extended installation commands
• Integration with generalised deployment server

Similar projects:

• https://github.com/kubernetes/git-sync

Configuration

So far gitdh only had one source of truth for configuration: The configuration file.

To offer a converged experience executing gitdh via cli / service and using gitdh as an imported module a more hybrid approach is necessary.

Status

So far gitdh has used simple strings for storing / indicating the status.

The status is now a compound structure holding statuses for different modules. Every module specifies the format of its status.

Modules

Deployment

The deployment module has configurable deployers.

Git Deployer (Default)

The Git Deployer atomically updates the deployed repository.

https://unix.stackexchange.com/questions/5093/how-does-one-atomically-change-a-symlink-to-a-directory-in-busybox

An efficient way to extract a copy of a git revision to a path is the git archive command.

$ git archive --format tar $COMMIT_REF | tar -x -C /path/to/destination/directory

To integrate with an existing tool-chain, additional plugins can be called before and after the checkout. For example, hooks are available for pre- and post-checkout commands. These commands will be executed in the temporary directory, i.e. before the directory is moved over to the final destination.

Files in the original directory may be preserved. These files are copied (hardlinked) over before the atomic move takes place. Directories can be symlinked over. For this it would be simpler to keep such files and directories in a separate, persistent directory.

There are several available modes for deploying: Atomic symlink, atomic directory (not possible), non atomic directory (double rename), clean and recreate.

Because internal git files may be deleted, the Git Deployer can optionally hold a remote tracking repository in another location. This avoids having to re-download very big repositories on every deploy. Otherwise a (shallow?) clone takes place.

Instead of implementing own locking mechanics, use de-facto standard python modules.

TODO: Reference locking module

Instead of carefully managing the .ssh/config files to use specific private keys, the git command will be called with an option substituting the default ssh command.

https://superuser.com/questions/232373/how-to-tell-git-which-private-key-to-use

Command Deployer

The Command Deployer performs a custom command, this is used to wrap external deployment tools, e.g. r10k.

Webhook Deployer

The Webhook Deployer performs an HTTP request during the deployment step. The destination and the information to be included are configurable, as is the processing of the response. See adnanh/webhook for inspiration.

Hooks

So far, gitdh was equipped with a configuration option for Pre- and Postdeploy hooks.

Hooks are now integrated more closely with gitdh. Information about the deployed commit as well as the reason for the call and further lifecycle information are included in the environment of the process started.

Notification

So far custom scripts had to be deployed to enable notifications about deployments.

Enables notifications via email, webhook, slack, ... about finished or failed deployments.

• Email (SMTP)
• Webhook (With custom payload)
• Slack