Secrets
Sometimes projects require the use of secure information, like passwords or api keys, that you wouldn't want to store in GitHub as text. This document will offer some recommendations for working with secret files.
Keep secrets in the environment
Secrets should never be hardcoded. Instead they should be stored as environment variables.
direnv can take much of the hard work out of maintaining environment variable - as it automates the loading of environment variables when you cd into a directory containing a .envrc file. To get direnv using homebrew brew install direnv.
.envrc files are blocked by default (to prevent them running automatically if you've cloned a repo) so enable them with direnv allow.
Example
This example uses a separate secrets file, as some programs do not support the export var=secret syntax.
.envrc:
#!bash
# `set -a` will export all variables that are sourced
set -a
# Call the secrets file
. ./secrets
# Return set the the default: `set +a`
set +a
secrets:
VERY_SECRET_ENV_VAR=12345
ANOTHER_SERET_ENV_VAR="password"
prefs.py:
import os
VERY_SECRET_ENV_VAR = os.environ['VERY_SECRET_ENV_VAR']
Secret files should never be committed
Secret files should never be committed to GitHub in a readable form.
To ensure this is the case maintain a .gitignore file that will prevent any files being committed. One idea is to end all secret files with a particular extension, then include that in the .gitignore file:
**/secrets
**/*.key
Perform checks before pushing
To check for files that contain secrets in a repo use trufflehog. This looks for strings with high entropy (that are probably passwords or keys) in a git repo.
# To install
pip3 install trufflehog
# Look for keys in the current repo
trufflehog .
If you find keys that have been committed, then clean them using bfg.
Encrypting secrets for GitHub with GPG
One problem with this setup is that secrets are not backed up in any way and cannot be shared with others in the team easily. A solution to this problem is to encrypt files using public key encryption - which allows you to give permission to others to decrypt the file.
Tools
To do this we will use keybase, gpg, and git-crypt.
Using keybase is optional, but acts as a backup of your gpg key in the event of device loss, and allows you to easily import other peoples public keys. Before you start, make sure you've got all these installed
▶ brew cask install keybase && brew install gpg git-crypt
Keybase
You'll probably already have a Keybase account, if not, create one. It's a good idea to add a number of proofs to you Keybase account, so you can recover it in the event of a lost device. The most foolproof option is to write down a paper key (generate one in the app or run keybase paperkey). Keybase will act as a backup of your gpg key, which will unencrypt your files.
If you already have a PGP key
If you already have a gpg key in Keybase, you'll be able to see it by running:
▶ keybase pgp list
Keybase Key ID: ***************************************************
PGP Fingerprint: **** **** **** **** **** **** **** **** **** ****
PGP Identities:
Will Bowditch <will@bowdit.ch>
You'll also want to check if the key is available in your local keychain:
▶ gpg --list-secret-keys
/Users/willbowditch/.gnupg/pubring.kbx
--------------------------------------
sec rsa4096 2018-04-17 [SC] [expires: 2034-04-13]
****************************************
uid [ultimate] Will Bowditch <will@bowdit.ch>
ssb rsa4096 2018-04-17 [E] [expires: 2034-04-13]
If it isn't available, export it to the local keychain with (note that if you have multiple keys then use the key ID to specify the one you want to export):
$ keybase pgp export
# ▶ WARNING Found several matches:
# user: Patrick Stadler <patrick.stadler@gmail.com>
# 4096-bit RSA key, ID CB86A866E870EE00, created 2016-04-06
# user: keybase.io/ps <ps@keybase.io>
# 4096-bit RSA key, ID 31DBBB1F6949DA68, created 2014-03-26
$ keybase pgp export -q CB86A866E870EE00 | gpg --import
$ keybase pgp export -q CB86A866E870EE00 --secret | gpg --allow-secret-key-import --import
If you don't have a PGP key
If not, you'll want to generate one:
▶ keybase pgp gen --multi
# Enter your real name, which will be publicly visible in your new key: Patrick Stadler
# Enter a public email address for your key: patrick.stadler@gmail.com
# Enter another email address (or <enter> when done):
# Push an encrypted copy of your new secret key to the Keybase.io server? [Y/n] Y
# When exporting to the GnuPG keychain, encrypt private keys with a passphrase? [Y/n] Y
# ▶ INFO PGP User ID: Patrick Stadler <patrick.stadler@gmail.com> [primary]
# ▶ INFO Generating primary key (4096 bits)
# ▶ INFO Generating encryption subkey (4096 bits)
# ▶ INFO Generated new PGP key:
# ▶ INFO user: Patrick Stadler <patrick.stadler@gmail.com>
# ▶ INFO 4096-bit RSA key, ID CB86A866E870EE00, created 2016-04-06
# ▶ INFO Exported new key to the local GPG keychain
git-crypt
git-crypt lets you encrypt files that can be unlocked by a list of authorised peoples using their public gpg keys.
Encrypting files using git-crypt
Files are automatically encrypted by git-crypt when you git push this is useful as they will exist in an unencrypted state on your machine, but will be encrypted in the remote repository.
Setup the repository to use git-crypt:
▶ cd super_secret_repo
▶ git-crypt init
Generating key...
You specify which files are encrypted in a .gitattributes file. For example:
<yoursecretfile> filter=git-crypt diff=git-crypt
<*.key> filter=git-crypt diff=git-crypt
Then add the gpg keys of the user you want to give permissions to deencrypt the files:
▶ USER_ID=will@bowdit.ch
▶ git-crypt add-gpg-user $USER_ID
[master (root-commit) ca13fbf] Add 1 git-crypt collaborator
2 files changed, 4 insertions(+)
create mode 100644 .git-crypt/.gitattributes
create mode 100644 .git-crypt/keys/default/0/4923366018AC7D9D21EC2E93BB09223257667F62.gpg
That's it. When you git push the files in .gitattributes will be encrypted and when you git pull they'll be decrypted. This automation is a double edge sword: any secrets that are not in .gitattributes will not be encrypted. You can check using:
▶ git-crypt status -e
encrypted: secret_file
Note: $USER_ID can be a number of things, but the most straightforward is the users gpg email.
Decrypting a file using git-crypt
If you pull a repository that is encrypted it may not decrypt automatically, if this doesn't happen run:
▶ git-crypt unlock
Importing others gpg keys from keybase
It's very easy to get gpg keys from keybase.
# pgp pull will only pull gpg keys for tracked users
# So if you haven't tracked a user already do so
# Here, joesmith, is the keybase username
▶ keybase track joesmith
# This will pull all pgp keys from tracked keybase users
# into your local gpg keyring
▶ keybase pgp pull
INFO Imported key for Joe Smith.
# You can check the available keys in gpg
# Note that the key identifer can be the email of the key
# In this case: joe@smith.com or the key ID
▶ gpg -k
/Users/willbowditch/.gnupg/pubring.kbx
--------------------------------------
pub rsa2048 2017-07-18 [SC] [expires: 2018-07-18]
****************************************
uid [ unknown] Joe Smith <joe@smith.com>
sub rsa2048 2017-07-18 [E] [expires: 2018-07-18]
Troubleshooting
If you get the error gpg: There is no assurance this key belongs to the named user it's probably because gpg doesn't have a trust indicator for the imported gpg key, as it is from keybase, not created locally. To fix this:
gpg --edit-key <KEY_ID>
gpg> trust
# You'll be given some options, set it to 5
Deploying encrypted secrets without gpg
Using gpg for encryption works great on your local machine where you can store your private key - but you don't want to be setting up your private key on deployment servers that anyone could access. git-crypt offers a workaround for this, where you can export a symmetric key for use without gpg:
# Generate the symmetric key
git-crypt export-key /tmp/key
# Move it to the server
cp /tmp/key /remote/server/key
# From the repo issue the unencrypt command using the key
cd /remote/repo
git-crypt unlock /remote/server/key
Be careful with the symmetric keys and don't commit them to the repo!
Appendix
Verified commits with GitHub
Now you've got a GPG key you can use this to verify your commits on GitHub - this is a precaution, but it effectively signs your commits, verifying that they came from your computer (or at least one with your private GPG key on it). This is very much optional, but is straightforward to setup now you have a GPG key. See GitHub's guide.
Passphrase is required every time I use GPG
If you are prompted for a passphrase every time you use gpg, you can use pinentry-mac to avoid this. But it does require some configuration.