i did this with aws kms. i had a root key that encrypted & decrypted data keys, which were unique for each customer. those data keys then encrypted & decrypted sensitive customer info. so called envelope encryption

I've used LiteLLM for my 1k-user service https://LibreThinker.com Works okay-ish, but some vendors now default to MFA on API keys (eg OpenAI) which makes things tricky; I decided to simply let those fail for now.

I am a bit familiar with litellm. Do you proxy server?

BYOK is becoming more common for AI SaaS. Interested to know how people handle rate limiting and key security at scale.

Honestly the API key part ended up being the easy part for me.

The messy stuff was abuse prevention, retries/fallbacks, and sudden cost spikes once real users started using different workflows.