ElevenLabs Agents Platform: Managing User Credentials Programmatically

Building a platform that programmatically creates AI voice agents for users? You'll quickly hit a critical question: How do I securely manage each user's credentials for services like Google, Slack, or Telegram?

After a deep dive into ElevenLabs' Agents Platform, we discovered that while it's incredibly powerful for voice AI, its credential management operates at the workspace level, not per-user. This has significant architectural implications for multi-tenant platforms.

Here's everything we learned.

TL;DR

ElevenLabs stores secrets at the workspace level, not per individual user
To manage per-user OAuth tokens (Google, Slack, Telegram), you need to build your own credential vault
Agents CAN access credentials via dynamic variables with the secret__ prefix
Use the Secrets API (POST /v1/convai/secrets) for workspace-level secrets
Tool calls dynamically inject auth tokens from variables into HTTP headers

🏗️ Understanding the Architecture

ElevenLabs Agents Platform coordinates four core components: Speech-to-Text (ASR), an LLM orchestrator, Text-to-Speech, and a tools system that enables agents to interact with external APIs.

The platform supports multiple deployment channels including web widgets, mobile apps, telephony systems (Twilio, SIP), and WhatsApp—making it ideal for building voice-first AI experiences.

The Credential Hierarchy

┌─────────────────────────────────────────────────────────┐
│                    ElevenLabs Workspace                  │
│  ┌─────────────────────────────────────────────────┐    │
│  │              Workspace Secrets                   │    │
│  │   (Shared across all agents & conversations)    │    │
│  └─────────────────────────────────────────────────┘    │
│                          │                               │
│           ┌──────────────┼──────────────┐               │
│           ▼              ▼              ▼               │
│      ┌─────────┐   ┌─────────┐   ┌─────────┐           │
│      │ Agent A │   │ Agent B │   │ Agent C │           │
│      └─────────┘   └─────────┘   └─────────┘           │
│                                                         │
│  ┌─────────────────────────────────────────────────┐   │
│  │          Workspace Auth Connections              │   │
│  │     (OAuth2, JWT Bearer, Basic Auth)            │   │
│  └─────────────────────────────────────────────────┘   │
└─────────────────────────────────────────────────────────┘

Key insight: Secrets are workspace-scoped. There's no native "User A's Slack token" vs "User B's Slack token" distinction within ElevenLabs itself.

🔐 Authentication Methods Supported

ElevenLabs supports five authentication methods for connecting agents to external APIs:

Method	Use Case	Configuration
OAuth2 Client Credentials	Server-to-server integrations	Client ID, Secret, Token URL, Scopes
OAuth2 JWT Bearer	Service accounts, enterprise SSO	JWT Secret, Token URL, Algorithm, Claims
Basic Auth	Legacy APIs	Username, Password
Bearer Token	API keys, static tokens	Token value stored as secret
Custom Headers	Proprietary auth schemes	Header name + value pairs

These are configured via Workspace Auth Connections in the dashboard under Agents → Workspace Auth Connections → Add Auth.

📡 The Secrets API

ElevenLabs provides a REST API for managing workspace secrets programmatically:

Create a Secret

curl -X POST "https://api.elevenlabs.io/v1/convai/secrets" \
  -H "xi-api-key: YOUR_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "type": "new",
    "name": "slack_bot_token",
    "value": "xoxb-your-token-here"
  }'

Response:

{
  "type": "stored",
  "secret_id": "sec_abc123",
  "name": "slack_bot_token"
}

Other Endpoints

Endpoint	Method	Description
`/v1/convai/secrets`	GET	List all workspace secrets
`/v1/convai/secrets/:id`	PATCH	Update a secret
`/v1/convai/secrets/:id`	DELETE	Delete a secret (if not in use)

🛠️ How Agents Access Credentials

Agents access stored credentials through tool configurations. When you create a server tool (webhook), you can reference secrets in the headers:

{
  "tool_config": {
    "name": "send_slack_message",
    "type": "webhook",
    "url": "https://slack.com/api/chat.postMessage",
    "method": "POST",
    "headers": {
      "Authorization": "Bearer {{ secret__slack_token }}",
      "Content-Type": "application/json"
    }
  }
}

The {{ secret__variable_name }} syntax tells ElevenLabs to:

Inject the secret value into the header
Never send the value to the LLM provider (protecting against prompt injection leaks)

🧩 The Per-User Credential Challenge

Here's where it gets interesting for platform builders. If you're building a multi-tenant application where each user has their own Google/Slack/Telegram credentials, ElevenLabs' workspace-level secrets won't cut it directly.

The Pattern You Need

┌─────────────────┐     ┌──────────────────┐     ┌─────────────────┐
│   Your Users    │────▶│  Your Backend    │────▶│   ElevenLabs    │
│ (OAuth tokens)  │     │ (Token Vault)    │     │   Agent API     │
└─────────────────┘     └──────────────────┘     └─────────────────┘
                              │
                              ▼
                        ┌──────────────────┐
                        │  Your Database   │
                        │  (Encrypted      │
                        │   per-user creds)│
                        └──────────────────┘

Implementation Steps

1. Handle OAuth Flows Server-Side

Your backend manages the OAuth dance with Google/Slack/Telegram:

// Your backend - handle OAuth callback
app.get('/oauth/google/callback', async (req, res) => {
  const { code } = req.query;
  const tokens = await googleOAuth.getTokens(code);

  // Store encrypted tokens in YOUR database
  await db.userCredentials.upsert({
    userId: req.user.id,
    provider: 'google',
    accessToken: encrypt(tokens.access_token),
    refreshToken: encrypt(tokens.refresh_token),
    expiresAt: tokens.expiry_date
  });
});

2. Inject Tokens via Dynamic Variables

When starting an ElevenLabs conversation, pass the user's tokens as secret dynamic variables:

// When initiating a conversation for a specific user
const userTokens = await getUserTokens(userId);

const conversationConfig = {
  agent_id: "agent_xyz",
  dynamic_variables: {
    // Prefix with secret__ to protect from LLM exposure
    "secret__user_google_token": userTokens.google.accessToken,
    "secret__user_slack_token": userTokens.slack.accessToken,
    "user_id": userId,
    "user_name": user.name
  }
};

// Start conversation with ElevenLabs
const response = await elevenlabs.conversations.create(conversationConfig);

3. Reference in Tool Configurations

Your agent's tools use these dynamic variables:

{
  "tool_config": {
    "name": "get_google_calendar_events",
    "url": "https://www.googleapis.com/calendar/v3/calendars/primary/events",
    "method": "GET",
    "headers": {
      "Authorization": "Bearer {{ secret__user_google_token }}"
    }
  }
}

⚠️ Important Limitations

What ElevenLabs DOES Support

Workspace-level secrets via API
OAuth2 Client Credentials flow (for app-level auth)
JWT Bearer flow for service accounts
Dynamic variable injection per conversation
Tool response → variable updates (for token refresh scenarios)

What You Must Build Yourself

Per-user OAuth token storage
Token refresh logic before conversation starts
Secure credential encryption at rest
OAuth flow UI and callbacks
Token expiration handling

Current API Gaps

As of our research, there's no dedicated API endpoint for creating Workspace Auth Connections programmatically. These must be configured via the dashboard. The Secrets API handles simple key-value secrets, but the full OAuth flow configurations (with token URLs, scopes, etc.) are dashboard-only.

🔄 Token Refresh Pattern

For long-running agent interactions, you'll need to handle token expiration:

// Before starting any conversation
async function getValidTokens(userId: string) {
  const creds = await db.userCredentials.findByUserId(userId);

  // Check if token is expired or expiring soon
  if (isExpiringSoon(creds.google.expiresAt)) {
    const newTokens = await googleOAuth.refreshTokens(
      decrypt(creds.google.refreshToken)
    );

    await db.userCredentials.update(userId, {
      google: {
        accessToken: encrypt(newTokens.access_token),
        expiresAt: newTokens.expiry_date
      }
    });

    return newTokens.access_token;
  }

  return decrypt(creds.google.accessToken);
}

Alternatively, create a "token refresh" tool that calls your backend mid-conversation:

{
  "tool_config": {
    "name": "refresh_google_token",
    "url": "https://your-api.com/tokens/refresh",
    "method": "POST",
    "body": {
      "user_id": "{{ user_id }}",
      "provider": "google"
    }
  }
}

Tool responses can update dynamic variables using dot notation paths, allowing the refreshed token to be used in subsequent tool calls.

🏢 Multi-Tenant Architecture Options

Option A: Shared Workspace + Dynamic Variables (Recommended)

Single ElevenLabs workspace
All users share the same agents
Per-user tokens injected at conversation start
Pros: Cost-effective, simple agent management
Cons: Must build credential management layer

Option B: Per-User Workspaces

Each user gets their own ElevenLabs workspace
Secrets are truly isolated
Pros: Native isolation, simpler auth flow
Cons: Expensive, complex workspace provisioning, limited to Scale/Business/Enterprise plans

Option C: Hybrid Approach

Group users into workspaces by tenant/organization
Organization-level secrets in workspace
User-specific tokens via dynamic variables
Pros: Balance of isolation and efficiency
Cons: More complex architecture

Key Takeaways

Build your own credential vault - ElevenLabs secrets are workspace-scoped, not user-scoped. For per-user OAuth tokens, you need your own secure storage.
Use secret__ prefix - Always prefix sensitive dynamic variables with secret__ to prevent LLM exposure.
Handle token lifecycle yourself - Refresh tokens before conversations start, or implement a refresh tool for long-running interactions.
Secrets API is for workspace-level only - Use it for app-level API keys, not individual user credentials.
Dashboard for OAuth configs - Workspace Auth Connections with full OAuth flows must be configured via dashboard (no API yet).