1 Overview

Strac is conducting workshops for Turing School on the week of Nov 14. This post provides instructions on how to set up workshop #1 on API keys. Please see Collecting Sensitive Data and Building a Document Vault for workshops 2 & 3.

In this workshop, we will build a Ruby on Rails app that integrates with multiple third-party services using API keys for authentication. We will show you common pitfalls and techniques you can use to keep API keys secure.

Prerequisites

1. We recommend you have a laptop ready for this workshop so you can follow along. Otherwise, please try to follow the instructions & screenshots during the session.
2. Join our AllThingsPII Slack community and join the #turing-school-qa Slack channel

2 Problem

Nearly every SaaS company now integrates with other SaaS companies in the cloud. According to statista.com, the average number of SaaS integrations used by organizations increased from 16 to 110 in just the last five years.

A common way to integrate with these SaaS applications is through API keys, a secret to identify the calling application. For example, a Stripe API key for handling payments looks like sk_test_9hqlyjwdarjtt1zdp7dc and an AWS access key for deploying workloads on the cloud resembles 2PDZxrVb3VLkZKopeuoYbZyExQbHymG060Hr1fyZFu.

Properly managing API keys is challenging for several reasons:

1. Secure storage and usage of it: Exposing API keys allow unauthorized users to access and change your application's data. It's a surefire way to lose the trust of your customers. It can happen for various reasons like accidental leakage through sharing on communication channels like Slack or technical reasons like storing it directly in the source code or an open network location.
2. Fine-grain access control: SaaS vendors may provide applications with a simple access control model where an API Key grants access to all data. Sensitive applications, especially those dealing with health or financial data, may require additional access controls like IP address allowlist, API based access per application, which can be challenging to implement.
3. Key rotation: Periodically changing access credentials is good security practice because it cuts off access to unauthorized users in case of an issue with #1. It also helps the team practice the procedure in the event of a security incident. But this is a time-consuming task that some teams deprioritize, leading to security & compliance risks.

3 Scenario

You are building an identity verification application that sends customer information to three 3rd party partners for verification. Each partner uses API keys to authenticate to their API. You noticed the API keys were checked into source control and want to improve its security.

4 Setting up the app

We will build a Ruby on Rails website to implement the scenario.

4.1 Install Ruby on Rails with SQLite (✍️ hands-on)

If you already have Ruby on Rails and SQLite installed, you can skip 4.1. To find out, you can run the following commands to test:

1. Test ruby and verify you have version 3.0+: ruby -v
2. Test rails and verify you have version 6.0+: rails -v
3. Test SQLite and verify you have version 3.0+: sqlite3 —version

Otherwise, download from gorails website, the instructions there are fairly good. I will paste the steps for Mac Monterey here for you to follow. If you're not using this OS, please follow the instructions directly from the Rails website

4.1.1 Install Ruby 3+:

1. Install homebrew if you don't already have it (check with brew version): /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"
2. Install rbenv: brew install rbenv ruby-build
3. Add rbenv to zshrc: echo 'if which rbenv > /dev/null; then eval "$(rbenv init -)"; fi' >> ~/.zshrc && source ~/.zshrc
4. Install ruby and set default version to 3.1.2 rbenv install 3.1.2 && rbenv global 3.1.2
5. Confirm you have ruby ruby -v you should see something like ruby 3.1.2p20 (2022-04-12 revision 4491bb740a) [arm64-darwin21]
6. 💁 If the version is not 3.1.2 try restarting your terminal again
7. 💁 If that doesn't work, try changing from bash to zsh chsh -s /bin/zsh

4.1.2 Install Rails 6+

1. Install rails: gem install rails -v 7.0.4
2. Tell rbenv to see rails executable: rbenv rehash
3. Confirm you have Rails rails -v you should see something like Rails 7.0.4

4.1.3 Install SQLite 3

1. Install sqlite: brew install sqlite3
2. Confirm you have sqlite sqlite3 -version you should see something like 3.37.0 2021-12-09 01:34:53 9ff244ce0739f8ee52a3e9671adb4ee54c83c640b02e3f9d185fd2f9a179aapl

4.2 Install Strac Project (hands-on)

1. Download the workshop from GitHub called securewithstrac/build-secure-app-workshop From the command line, run git clone https://github.com/securewithstrac/build-secure-app-workshop.git
2. Install from project root (e.g., cd build-secure-app-workshop) using: bundle install && rake db:migrate
3. Start the local server: rails s
4. Navigate to the page on localhost at http://127.0.0.1:3000

You should see a home page like this:

4.3 Play with Strac Project (hands-on)

On the home page you will see two sections. Links in the first section is used to set up the app for calling 3rd party APIs and links in the second section is used to call 3rd party APIs.

4.3.1 Create an Address

1. Click on Create an address
2. Enter a valid address and give it a name. For example:
3. Alias: Work
4. Line 1: 1400 112th Avenue South East
5. City: Bellevue
6. State: Washington
7. Zip code: 98004

4.3.2 Get info for address

1. Click on API 1: Get city / zip code info
2. Select the address you created and hit get info

4.3.2 Calculate address distance

1. Click on API 2: Calculate address distance
2. Enter a valid zip code. For example 98126

You can repeat the same exercise for the Create personal identity (SSN) and API 3: Validate SSN API to try the last API. Note: Please don't enter your real SSN, this is a demo app.

Now that you're familiar with the app, we can look at how API keys are handled.

5 Handling API keys

5.1 Security Design Goals (📚 theory)

The example application is integrated with three APIs, each requiring API keys to access. To minimize the attack surface for API keys, we want to prioritize removing access for API keys in the following order:

1. Source code: public repositories are accessible worldwide and have the largest attack surface. Private repositories are also shared with other parts of the organization, including contractors and temporary employees with a large attack surface. SCM also uses version control to track every change, making removing API keys from its commit history complicated. As a best practice, API keys should be moved to Secret Managers such as AWS Secrets Manager, HashiCorp Vault or Strac Tokens.
2. Developers: a successful application requires a large development team to maintain, including application engineers in different teams and engineers in other disciplines like IT, DevOps, data and security. Breaking down monolithic architectures and ensuring proper access controls within Secret Managers is essential for limiting access to developers.
3. Application: an application needs a secret to identify itself, so there's no way to remove access completely. But there are ways to reduce access to API keys within the application, such as (1) moving sensitive API calls from the front end to the back end, (2) using proxy services such as Strac Interceptors to reduce the number of API keys from many to one (3) restrict API keys to access only the necessary resources

5.2 High-Level Architecture (📚 theory)

The demo project will show how to achieve all 3 using Strac APIs. Before digging into the code, let's take a high-level view of the system architecture with versus without Strac.

In the insecure design below, the developer directly puts API keys into the source code, violating security design goal #1. This is the most basic design goal and will significantly increase the attack surface. Not a good design!

In the second design, we address all three security design goals listed above:

1. By tokenizing the API keys using Strac, the source code only need to see safe references (tokens), not the actual API keys.
2. Organizations can use access policies to control who can upload API keys and which applications can use API keys giving app developers fine grain control over their application security. An exciting outcome of the proxy design is its ability to improve application security for 3rd party APIs also. Consider a SaaS offering with 10 APIs controlled using a single API Key. Using Strac's proxy design, it is now possible to restrict access to only some of the APIs, which was impossible before.
3. The application only needs to maintain a single identity to authenticate to Strac. It is a smaller attack surface than maintaining N identity secrets, one for each API integration.

5.3 Storing API keys using Strac Tokenization API (✍️ hands-on)

The app you played within section 4.3 uses an insecure design. You will be able to find three API keys directly in the source code by navigating to the project root and then opening the file open ./config/application.rb using your favorite editor.

Before switching over to our Secret Manager implementation, we will first want to rotate our existing API keys. You can download a new set of API keys here, it will be in the following format:

Strac API key: <STRAC API KEY>

New third-party API keys:
* API 1 get city / zip code info: <NEW_API_KEY_1>
* API 2 calculate address distance: <NEW_API_KEY_2>
* API 3 validate SSN: <NEW_API_KEY_3>

The application developer will use the Strac API key to tokenize third-party API keys into Strac Vault and the Rails server will use it to call Strac Proxy to communicate with the 3rd party APIs.

To tokenize third-party API keys, you will need to make an HTTP call using curl via the command line (download link), or the Postman app (download link).

The curl command is as follows, replace <STRAC API KEY> with the Strac API key from above and <THIRD PARTY API KEY> with a single, new third-party API Key from above. Repeat the steps for all partner keys to get three tokens back:

curl -X POST 'https://api.test.tokenidvault.com/tokens' \
-H 'x-api-key: <STRAC API KEY>' \
-H 'content-type: application/json' \
-d '{"data": "<THIRD PARTY API KEY>", "type":"api-key"}'

You should see a response like the following:

{"id":"tkn_p0Zi7v3Vw2sP1ua0qG5bRYMb","type":"api-key"}

The id contains the token reference for the third-party API key, for example tkn_p0Zi7v3Vw2sP1ua0qG5bRYMb

If you're having trouble with curl or Postman you can create the token using the helpful tools link Create token for API key in the demo app.

5.4 Using Strac Proxy for 3rd party API calls (✍️ hands-on)

Now, you will need to change the code to call the three partner endpoints via Strac Proxy and API key tokens. The steps are as follows:

1. Stop the Rails server using Ctrl + C
2. Use environment variables to specify API keys instead of source code and start the server again.

USE_STRAC_PROXY=true \
STRAC_API_KEY=<STRAC API KEY> \
ADDRESS_GET_INFO_API_KEY=<TOKEN_1> \
ADDRESS_CALCULATE_DISTANCE_API_KEY=<TOKEN_2> \
SSN_VALIDATE_API_KEY=<TOKEN_3> \
rails s

1. 💁  environment variables allow API keys to be specified dynamically during runtime, which can be more easily secured using runtime environments like AWS System Manager or Secret Managers like Hashicorp Vault.
2. Go to 127.0.0.1:3000. The app should be up again at work in the same way as before. Except, your API keys are now secure and not accessible via the source code, working as per the secure design diagram above.
3. The last step to clean up the API key leak is to complete the key rotation. This is typically done by working with the third party to remove the old API key from their allow list. For this demo, we'll pretend the key rotation is done and you can remove the old API keys from the source code. This is a crucial step to prevent compromised API keys from being used after the code has been fixed.

6 Wrapping Up

In this session, we learned about security considerations around API keys. We resolved a security issue with API Key leaks in a simple Ruby on Rails project by moving API Keys to a Secret Manager and rotating the old API keys.

We also discussed some advanced security controls for API keys like IP whitelisting, and fine-grain access control policies but did not implement them using Strac in today's session.

I hope you all had fun and learned something useful for work. We will move from securing API Keys to collecting sensitive customer data in the next session. Hope to see you all there!

7 FAQ

Q: How is Strac different from existing Secret Manager solutions like AWS Secrets Manager?

A: Let's start with the similarities:

1. Both Strac & AWS allow API keys to be substituted with an alias. Strac calls it a token, AWS calls it a name. They are both aimed at keeping API keys out of the source code as we discussed in Section 5.1.
2. Both Strac and AWS require the application to show a secret to authenticate in order to use the API key. Strac uses API keys, AWS uses IAM credentials. They both allow enterprises to narrow down access to individual API keys to internal teams as needed.

Where they differ is:

1. Strac removes the API keys from the application's memory through the proxy pattern. Whereas AWS (and other secret managers) requires the application to pull API keys in memory which exposes them to issues like the Heartbleed bug.
2. Strac allows additional access controls beyond what the 3rd party APIs provide through the proxy pattern. For example, IP whitelisting & fine-grain role-based access into 3rd party APIs using the same API key. Whereas AWS (and other secret managers) treat API keys as just another secret without additional controls.
3. Strac is platform agnostic allowing it to add additional value beyond the AWS ecosystem. For example, it can redact API keys in communication channels like Slack, Gmail, Zendesk and we're also working on making this happen for other platforms like Github.

Q: Why not just specify API Keys using environment variables?

A: Environment variables are specified during runtime, allowing API Keys to be moved from application source code to configuration. Environment variables work in conjunction with runtime systems to keep API Keys work, so its security depends on how the application is deployed. We will review two implementations and look at common pitfalls with these approaches:

1. Custom deployment system using a series of scripts, services and pages to facilitate the deployment process
2. Kubernetes, which uses its own Secrets Store to inject the API Keys.

Common pitfalls:

1. Checking into source code: YAML is a common way to store application configuration because of its ease of use. A common pitfall with a custom deployment approach is checking in the YAML file containing API keys directly into a separate source code.
2. Security at rest: The right way to do this is to store the API Key in a separate service and apply proper encryption. The issue is that general-purpose deployment systems tend to be highly configurable to work with different applications. For example, you must manually configure an EncryptionConfiguration in Kubernetes which is specified in YAML format as shown below. An inexperienced developer may check this into version control. They could leave copies of this file before uploading it to Kubernetes. They could choose the wrong cipher suite. They could deploy it on an unpatched machine with unpatched bugs in their crypto modules. And many more.

1. Security in transit: The API keys must transit from the 3rd party website to the application servers. We have seen in the previous section how copies of the API Key may be left behind on disk while it moves to the secret servers. Another risk is leaving it on the network in transit. For example, if the developer moves the secret file using an unprotected network like airport WiFi, sent via unencrypted channels like plain FTP then there's a risk of the file being sniffed by a side-channel attacker. Encrypting the file helps, but you have to keep the private keys secure. Care must also be taken when moving API keys from the secrets server to the application server. TLS is the typical way to do this but you will need to manage your TLS certificates properly (that's a topic for another day)
2. Access controls: Allowing unauthorized users access to secret servers or the application servers will defeat all the security processes you may have in place. Common pitfalls include misconfiguring your secret server store (e.g., setting S3 buckets to public), forgetting to remove access from a former employee and failing to patch your machines.
3. Incidence response time: Security control failures are inevitable, it is only with a layered defense that a system can become truly secure. Consider an average application with 15 dependencies. If even one server gets compromised, all 15 API Keys must be rotated across all servers. If it takes only 3 hours to rotate a single key (I've seen it take days to weeks to complete in large code bases), it can take days or weeks before the application is secured again.

With Strac, the 3rd party API Keys can transit safely to Strac Vault. From there, the application never has access to the API Keys preventing it from being leaked.  The Strac API Key will be the only secret the application needs to maintain, reducing operational burden. In the event that Strac API Key is compromised, attackers still do not have access to the 3rd party API Keys. Mitigating the issue requires only a single key rotation instead of N.