Valkeyrie: a key-value store cache

Some constraints and use cases:

The main catch is that this is actually a key-value cache and stored values will only stick around from anywhere from about 10 minutes to at most an hour if they are never accessed. If they are frequently accessed (say once a minute) they should stick around for much longer (potentially indefinitely), but there are no persistence guarantees.

Also, the largest value you can store is about 44k.

Even with these limitations I think here are still some valuable use cases. I'm personally using this as a signaling channel to connect WebRTC clients (just using a URL) without any other specialized server infrastructure.

Also, if you happen to be using this from within Fastly's network (say from a Compute@Edge service), there is a POP-local version (at pop-local.valkeyrie.com) that is equivalent except that it stores the keys only in the POP that the request lands on (if another request for the same key hits a different POP it wouldn't see that key). The advantage of having local state is that if the request comes from within the POP it should be blazing fast, since the request won't have to leave the POP.

$ curl https://kv.valkeyrie.com/some/random/key/cfOn3VPYMhEVKr6V -d 'A Value!'
A Value!

A successful POST will return a 200 OK, along with a body containing the updated value.

By default, the post body has a size limit of 2k. The new contents of the key are echoed back in the body of the response:

$ curl https://kv.valkeyrie.com/some/key -d 'A Value!'
A Value!

To set or update a larger value (up to about 44k), base64 encode it and put it into an HTTP header named do-post-base64. The do-post-base64 can support about 60k of encoded data. You do not need to decode the data on future GET's

$ curl -X POST https://kv.valkeyrie.com/some/key -H "do-post-base64: `echo Some Value | base64`"
Some Value
$ curl https://kv.valkeyrie.com/some/key
Some Value

Since this is actually a key-value cache, stored values will only stick around from anywhere from about 10 minutes to at most an hour if they are never accessed. If they are frequently accessed (say once a minute) they should stick around for much longer (potentially indefinitely), but there are no persistance guarantees.

Keys:

You can use any valid URL as a key, except for a few reserved keys. For example:

curl 'https://kv.valkeyrie.com/this/is/a/crazy.but?totally=valid&key'

The key in this case is: /this/is/a/crazy.but?totally=valid&key

But I recommend using randomly generated keys with at least 128 bits of entropy (about 22 random alpha numeric characters). Simple randomly generated keys are best for your sanity.

Reserved Keys

There are a few keys reserved for safety reasons:

/favicon.ico
/robots.txt
/index.html
/

These will always return either a 404 Not Found or a hardcoded value (such as this page!)

Content-Type:

If you include a Content-Type header when you set a value:

$ curl https://kv.valkeyrie.com/some/random/key/cfOn3VPYMhEVKr6V -H 'Content-Type: application/json' -d '{ "a": "json value" }'
'{ "a": "json value" }'

a Content-Type header will be included in future responses containing that value:

$ curl -s -D - https://kv.valkeyrie.com/some/random/key/cfOn3VPYMhEVKr6V | grep content-type
content-type: application/json

Cross-Origin Resource Sharing (CORS) Headers:

I won't go into the details here, but Valkeyrie properly sets CORS headers (and supports preflight OPTION requests) so that requests to Valkeyrie can be made from javascript in a browser from any origin.

Conditional Update:

Conditional updates are supported via If-Match.

With a successful GET or POST, an ETag header will be provided in the response:

$ curl -s -D - https://kv.valkeyrie.com/some/key | grep etag
etag: "9133e4c9b29a03c17d1e3dc7f6629704d3fe4f170f7d8fbc12dcaade9e744daa"

You can use this ETag in a follow up POST along with an If-Match header to ensure that you are only replacing the value that you are expecting to. If the value in your If-Match header does not match the value's ETag then it won't be replaced and you'll get a 412 Precondition Failed response instead of a 200 OK.

For example, given some old value:

$ curl https://kv.valkeyrie.com/some/key -d 'Old Value'
Old Value
$ curl -D - https://kv.valkeyrie.com/some/key
HTTP/2 200
server: Varnish
retry-after: 0
etag: "cbe4ed30c2dc071cd88ffedfbe39a165065dc634177958d646435bd57284aceb"
via: 1.1 varnish, 1.1 varnish, 1.1 varnish
accept-ranges: bytes
date: Fri, 13 Nov 2020 18:16:25 GMT
age: 0
x-served-by: cache-chi21173-CHI, cache-chi21162-CHI, cache-chi21163-CHI
x-cache: HIT, MISS, MISS
x-cache-hits: 0, 0, 0
x-timer: S1605291386.653519,VS0,VE2
content-length: 9

Old Value

If we try to replace it with a different ETag for our If-Match the value won't be replaced, and we'll get a 412 Precondition Failed back:

$ curl https://kv.valkeyrie.com/some/key -H 'If-Match: asdf' -d 'My new value'
Precondition Failed

But using the ETag given in the response for the old value will let us replace it (watch out for the quotes!):

$ curl https://kv.valkeyrie.com/some/key -H 'If-Match: "cbe4ed30c2dc071cd88ffedfbe39a165065dc634177958d646435bd57284aceb"' -d 'My new value'
My new value

This allows multiple clients to simultaneously attempt to update the same value without accidentally interfering with each other.

You shouldn't need to calculate the ETag manually, (you can get it from the response), but if you want to for some reason, it's just the sha256 of the stored value (put in quotes when in the ETag or If-Match headers).

$ curl https://kv.valkeyrie.com/some/random/key/cfOn3VPYMhEVKr6V
A Value!

Non-existent keys

A GET for a non-existent key will return a 404 Not Found:

$ curl https://kv.valkeyrie.com/asdfasdf
Not Found

$ curl -X DELETE https://kv.valkeyrie.com/some/random/key/cfOn3VPYMhEVKr6V

A successful DELETE for an existing key will return a 204 No Content.

A DELETE for a non-existent key will return a 404 Not Found:

$ curl -X DELETE https://kv.valkeyrie.com/asdfasdf
Not Found

Conditional Delete:

Conditional deletes are also supported via If-Match.

With a successful GET or POST, an ETag header will be provided in the response:

$ curl -s -D - https://kv.valkeyrie.com/some/key | grep etag
etag: "9133e4c9b29a03c17d1e3dc7f6629704d3fe4f170f7d8fbc12dcaade9e744daa"

You can use this ETag in a follow up DELETE along with an If-Match header to ensure that you are only deleting the value that you are expecting to. If the value in your If-Match header does not match the value's ETag then it won't be deleted and you'll get a 412 Precondition Failed response instead of a 204 OK.

For example, given some old value:

$ curl https://kv.valkeyrie.com/some/key -d 'Old Value'
Old Value
$ curl -D - https://kv.valkeyrie.com/some/key
HTTP/2 200
server: Varnish
retry-after: 0
etag: "cbe4ed30c2dc071cd88ffedfbe39a165065dc634177958d646435bd57284aceb"
via: 1.1 varnish, 1.1 varnish, 1.1 varnish
accept-ranges: bytes
date: Fri, 13 Nov 2020 18:16:25 GMT
age: 0
x-served-by: cache-chi21173-CHI, cache-chi21162-CHI, cache-chi21163-CHI
x-cache: HIT, MISS, MISS
x-cache-hits: 0, 0, 0
x-timer: S1605291386.653519,VS0,VE2
content-length: 9

Old Value

If we try to delete it with a different ETag for our If-Match the value won't be deleted, and we'll get a 412 Precondition Failed back:

$ curl -X DELETE https://kv.valkeyrie.com/some/key -H 'If-Match: asdf' -d 'My new value'
Precondition Failed

But using the ETag given in the response for the old value will let us delete it (watch out for the quotes!):

$ curl -X DELETE https://kv.valkeyrie.com/some/key -H 'If-Match: "cbe4ed30c2dc071cd88ffedfbe39a165065dc634177958d646435bd57284aceb"'

This allows multiple clients to simultaneously attempt to delete/update the same value without accidentally interfering with each other.

You shouldn't need to calculate the ETag manually, (you can get it from the response), but if you want to for some reason, it's just the sha256 of the stored value (put in quotes when in the ETag or If-Match headers).