From Secrets to Signals: My Spotify Data Engine

Built with Terraform, secured with Vault, scaled with DataStax

© Image generated with DALL·E 3 via ChatGPT by OpenAI for illustrative and editorial purposes only.

Most people build dashboards.
I wanted a data engine.

This isn’t a tutorial. It’s a system I built, learned from, and rebuilt — one that ties Terraform, Vault, DataStax, and Nuxt into a self-driven data engine.

A data engine that could pull my Spotify world into a secure pipeline, shape it for the questions I care about, and give me the freedom to add new “views” on the fly — without hand-editing schemas every time a new idea hits.

• Vault guards the secrets.
• DataStax (Cassandra) stores and serves.
• Nuxt 3 makes it feel alive.

This is how I built it — and what I learned along the way.

1. Why I built this

It started as a small itch: “Could I move or inspect my playlists more intelligently?”
Then the scope evolved — the way my work often does:

• Import me, playlists, tracks, albums, artists
• Enrich artists and albums from Spotify’s APIs
• Add embeddings to power “similar artists”
• Make search actually usable (exact, prefix, vectors)
• And then the fun bit: add new sorted views dynamically from the frontend by creating derived tables safely on the backend

I’ve worked a lot with Couchbase. Moving to Cassandra forced different thinking:
query-first design, partitions, clustering columns, and avoiding hot spots.
Instead of fighting that, I embraced it — and turned schema generation into a feature.

2. High-level architecture

Spotify API ──▶ Express backend ──▶ Astra DB (Cassandra)
                    ▲    │
                    │    └─▶ HuggingFace embeddings
                    │
                 Vault (secrets + tokens)

Nuxt 3 frontend  ◀── Express APIs (auth, sync, search, stats, vectors, Alterations)

• Vault delivers dynamic credentials and keeps .env files empty.
• Express orchestrates data collection, enrichment, and dynamic table generation.
• Astra DB handles persistence, scalability, and query-optimized schema.
• Nuxt 3 makes it visual, reactive, and human.

Source code

• Terraform · Astra DB + Vault setup
  https://github.com/raymonepping/IBM-HashiCorp-Datastax
• Spotify Data Engine (backend + Nuxt 3 frontend)
  https://github.com/raymonepping/music_library

Try it locally (quick start)

Prereqs: Docker, Node 20+, a Vault dev or cluster, and an Astra DB token (or use your existing Astra DB).

# 1) Clone both repos
git clone https://github.com/raymonepping/IBM-HashiCorp-Datastax
git clone https://github.com/raymonepping/music_library

# 2) (Optional) Provision Astra DB with Terraform
cd IBM-HashiCorp-Datastax/terraform

# export TF_VAR_astra_token (or pull from Vault if you’ve wired that)
terraform init && terraform apply -auto-approve

# 3) Configure Vault secrets (example paths)
# VAULT_ADDR and VAULT_TOKEN should be set in your shell
vault kv put secret/spotify \
  client_id=... client_secret=... refresh_token=...
vault kv put secret/astra \
  token=... database_id=... keyspace=planetary
vault kv put secret/app \
  jwt_signing_key=$(openssl rand -hex 32)

# 4) Start the app stack
cd ../../music_library

# If using docker compose:
docker compose build
docker compose up -d

# Or run locally:
# (backend)
cd backend && npm ci && npm start

# (frontend)
cd ../frontend && npm ci && npm run dev

# 5) Sanity checks
# Backend health:
curl -s http://localhost:3002/health

# Stats snapshot:
curl -s http://localhost:3002/api/stats | jq

# Frontend: open http://localhost:8075

3. Deploying Astra DB with Terraform

Before writing a single line of backend code, I deployed DataStax Astra DB through Terraform.

It’s declarative, predictable, and repeatable — exactly how infrastructure should be.

# main.tf
provider "datastax-astra" {
  token = var.astra_token
}

resource "datastax-astra_database" "spotify_engine" {
  name           = "spotify_engine"
  keyspace       = "planetary"
  cloud_provider = "gcp"
  region         = "us-east1"
  tier           = "serverless"
}

Sensitive values like astra_token live in Vault, not in terraform.tfvars.
Terraform fetches them dynamically at runtime — no leaks, no drift, no manual copy-paste.

Outputs connect directly to my Vault configuration:

# outputs.tf
output "astra_db_id" {
  value = datastax-astra_database.spotify_engine.id
}

output "astra_keyspace" {
  value = datastax-astra_database.spotify_engine.keyspace
}

Result: a production-ready Astra DB instance, provisioned by Terraform, configured by Vault, consumed by the backend.

Zero clicks. Zero human error. Total reproducibility.

4. Spotify ingestion (and how I made it scale)

I use two Spotify flows:

• App token (client credentials) for bulk enrichment
• User refresh token for personal playlists and followed artists

Requests are chunked and retried automatically on 429s.

Example log from a real run:

[collect] playlists: found 248
➡ processing playlist 6/248: "Spanish"
   ... 200 tracks processed
[collect] unique artistIds=14293, albumIds=25425
[artists] upserted 850/14293

Each artist gets an embedding built from a sentence like:

“Metallica. Genres: heavy metal, thrash metal. Popularity: 92. Followers: 21,000,000.”

Those vectors power similar artist search later on.

5. Cassandra data model

Base tables

• artists(artist_id, name, name_lc, genres, followers, popularity, images, embedding, updated_at)
• albums(album_id, name, album_type, release_date, total_tracks, images, artists, updated_at)
• albums_by_artist(artist_id, album_id, name, release_date)
• artists_by_prefix(prefix, name_lc, artist_id, name)

Cassandra sorts within partitions — not globally.
So every query that needs ordering gets its own table.
When I want a new sort order, I build a new table.

6. The Alterations page — dynamic table creation

My favorite part.

Get Raymon Epping’s stories in your inbox

Join Medium for free to get updates from this writer.

Remember me for faster sign in

The frontend offers a menu:

“Create a derived table of artists by popularity or followers.”

Once triggered, it calls the backend:

1. Validates the key (e.g. popularity, followers, name)
2. Generates the table dynamically
3. Scans artists, buckets them by hash, and writes them asynchronously
4. Updates job progress every few seconds

Example generated table:

CREATE TABLE IF NOT EXISTS artists_by_popularity (
  bucket int,
  popularity int,
  name_lc text,
  artist_id text,
  name text,
  images text,
  PRIMARY KEY ((bucket), popularity, name_lc, artist_id)
) WITH CLUSTERING ORDER BY (popularity DESC, name_lc ASC, artist_id ASC);

Every build runs as a job with progress tracking:

POST /iterate/build           # start
GET  /iterate/status/:job_id  # poll
GET  /iterate/top?by=popularity  # preview

Jobs update every few seconds — the UI shows a shimmering progress bar until 100%.

Handy endpoints

• Search (exact): GET /search/artists/exact?q=andre hazes
• Search (prefix, SAI): GET /search/artists/prefix-sai?q=rammstein&limit=20
• Similar artists (vectors): GET /vectors/artists/:id/similar?limit=6
• Alterations (build derived table): POST /iterate/build { "by": "popularity" }
• Alteration status: GET /iterate/status/:job_id
• Top preview: GET /iterate/top?by=popularity&limit=12

7. Search that behaves

Exact

SELECT artist_id, name FROM artists WHERE name_lc = ?;

Prefix

No LIKE queries. I precompute prefixes in artists_by_prefix(prefix, name_lc, artist_id, name):

SELECT artist_id, name
FROM artists_by_prefix
WHERE prefix = 'ram'
LIMIT 25;

Vectors

Stored as JSON arrays on each artist row.
For now, I compute cosine similarity server-side. Later, Astra’s vector index will handle it natively.

8. Vault everywhere

Every secret comes from Vault — not from .env.
Spotify tokens, Astra credentials, JWT signing keys — all ephemeral, all injected dynamically.

const vault = require("node-vault")({
  endpoint: process.env.VAULT_ADDR,
  token: process.env.VAULT_TOKEN
});

async function fetchDynamicCredentials(path) {
  const { data } = await vault.read(path);
  return data;
}

• No hardcoded tokens.
• No stale credentials.
• No excuses.

9. Stats, health, and feedback

Progress visibility is everything.

A lightweight /api/stats endpoint reports system-wide counts:

curl -s http://localhost:3002/api/stats | jq
{
  "artists_total": 14293,
  "albums_total": 6058,
  "artists_by_prefix_rows": 28552,
  "tracks_total": 1914,
  "generated_at": "2025-10-24T12:41:08.647Z"
}

Health checks run automatically inside Docker.
Frontend only starts when backend is healthy.

10. Nuxt 3 frontend

Nuxt makes the whole thing feel tactile.
Each page is clean, reactive, and uses $fetch to hit backend endpoints:

// useApi.ts
export function useApi() {
  const backend = useRuntimeConfig().public.backendBase
  const common = { credentials: 'include' }
  return {
    searchArtistsExact: (q: string) =>
      $fetch(`${backend}/search/artists/exact`, { ...common, query: { q } }),
    buildDerived: (by: string) =>
      $fetch(`${backend}/iterate/build`, { method: 'POST', body: { by } }),
    jobStatus: (id: string) =>
      $fetch(`${backend}/iterate/status/${id}`, common),
  }
}

The Alterations page polls /iterate/status/:id and animates progress.
Once done, it fetches /iterate/top and shows the top results inline.

Minimal. Transparent. Fast.

11. Docker and deployment

Each service has its own container:

• Backend (Node 20 slim, npm ci --omit=dev)
• Frontend (Nuxt 3, port 8075)
• Vault (already running)
• Astra DB (managed by DataStax Cloud)

The Compose setup links everything on a single bridge network.
Both frontend and backend share the same .env for consistent ports.

services:
  backend:
    build: ./backend
    ports: ["3002:3002"]
  frontend:
    build: ./frontend
    ports: ["8075:8075"]
    depends_on:
      backend:
        condition: service_healthy

12. Lessons learned

• Cassandra is query-first. Build tables for what you need, not what you wish existed.
• Buckets save you from hotspots.
• Real progress feedback builds trust.
• Vault makes secret hygiene effortless.
• Terraform + Vault + Astra DB = zero-click, zero-risk provisioning.

13. What’s next

• Persist job metadata to Cassandra (so jobs survive restarts)
• Integrate Astra’s vector search for fast similarity lookup
• Extend Alterations for followers, name, and language
• Connect Terraform, Ansible, Apptio, Turbonomic, and Instana
  → so systems start feeding systems, not just dashboards

Final thoughts

This started as a playlist import experiment.

It became a scalable, secure, observable data engine.

Not a dashboard. A living backend.

Vault keeps it clean.
Cassandra keeps it fast.
Terraform keeps it reproducible.
Nuxt keeps it human.

Curiosity keeps it moving.

🧠 Born from How I Use AI as My DevOps Copilot
🤖 Powered by Sally — my AI DevOps copilot
🚀 Because automation should automate itself

📘 Journey recap : Turning From 50 Shades of Red into Blue