Skip to main content
Version: Next

Prompt Playground

The Prompt Playground is a vanilla-DOM panel for interactive prompt engineering. It supports both multi-message CHAT templates and single-string STR templates with MUSTACHE variable rendering ({{variable}}).

Features​

  • CHAT mode: Multi-message editor with role selectors (system/user/assistant). Add additional turns, edit content inline.
  • STR mode: Single textarea with MUSTACHE variable highlighting.
  • Variable panel: Inputs for each {{variable}} found in the template, auto-extracted.
  • Model selector: Choose from GPT-4o mini, GPT-4o, Claude 3.5 Sonnet, Claude 3 Opus.
  • Streaming response: SSE token-by-token delivery with cursor animation.
  • Version history: Sidebar showing all versions for the current prompt. Click to load, "Save as new version" to create a snapshot.

Model presentation boundary​

The selector presents product-level model names and an accessible AI model label. Developer names are not appended to the visible options. The underlying values still use the existing <model_developer>:<model_name> contract, and a change event splits only on the first colon so stored model names may contain additional colons without corruption.

If a saved prompt references a model that is not in the built-in option list, the editor adds a temporary option using a bounded, sanitized model name. Prompt list metadata applies the same filter. Raw HTTP, SSE, or network errors are converted to application-owned AI-service copy when they contain private infrastructure terms. These changes affect presentation only; database fields, the run-playground payload, and provider selection remain unchanged.

Prompt browser UI notes​

The Prompts list uses the native browser scroll container on .prompt-list-items, with a scoped neutral scrollbar treatment in css/app.css. The scrollbar is thin, keeps a transparent track, hides WebKit scrollbar buttons, and reserves a stable gutter so prompt rows do not shift when the list becomes scrollable.

The scoped prompt-list rules are the source of truth for row sizing, hover, focus, active state, metadata truncation, and scrollbar behavior. The older duplicate shared prompt-list CSS block was removed so the left prompt browser does not inherit conflicting row heights or default scrollbar styling.

This change is visual only. Prompt search, prompt selection, + New, loading states, empty states, and spotlight guide targets keep the same DOM and client behavior.

Architecture​

  • Playground Edge Function (run-playground): SSE streaming with EventSource pattern. Supports DivProxy (OpenAI-compatible) and DivProxy (Anthropic) providers. Keys are Edge Function secrets only.
  • DAL functions: listPrompts, createPrompt, createPromptVersion, setCurrentVersion, getPrompt, getPromptVersion, listPromptVersions.
  • Transactional RPCs: create_prompt_version_and_set_current (atomic insert
    • set-current) and set_prompt_current_version (validated update). These prevent orphan state and cross-prompt version hijacking.
  • Evaluator linkage: evaluators.prompt_version_id FK links an evaluator to a specific prompt version. run-eval loads the template from prompt_versions when the FK is set; falls back to legacy prompt_template column.

Schema​

prompts (registry)
├── id, project_id, name, description, category
├── current_version_id FK → prompt_versions
└── metadata, created_at, updated_at

prompt_versions (configs)
├── id, prompt_id FK → prompts, version_number
├── template_type (CHAT | STR)
├── template_format (MUSTACHE | F_STRING | NONE)
├── template (JSON: messages[] for CHAT, template string for STR)
├── model_developer, model_name
├── invocation_parameters, tools, response_format
├── label, is_archived, created_at, created_by

Mount behavior​

The playground mounts lazily via requestIdleCallback from src/main.ts. Before the panel appears, mountPromptPlayground() performs a preflight query by calling listPrompts(getActiveProject().id). The playground is hosted inside the right-side workbench card, not in a bottom overlay. It is only appended to the DOM when:

  • listPrompts() succeeds (Supabase is configured and the prompts table exists)
  • At least one prompt exists in the registry

If either condition fails during boot, the playground disables itself silently (with a console.warn) and the slot stays display: none. This prevents a blank panel from obstructing the main interface when the backing tables have not been migrated or the prompt registry is empty.

Trigger button and manual open​

The VS Code-style activity bar includes a Prompts button (#activityPromptsTrigger) that opens the playground on demand. When clicked:

  • If the panel is closed, it opens immediately: even if no prompts exist yet.
  • The empty state shows a + New button so the first prompt can be created.
  • If the panel is already open, the button closes it.
  • A close button (×) in the top-right corner of the panel also closes it.

The trigger button highlights with an .active class while the panel is visible.

Project scoping​

The playground reads and writes prompts for the currently active project (getActiveProject().id), not a hard-coded demo ID. This means:

  • Switching projects via the project switcher updates which prompts are visible.
  • Creating a new prompt requires the user to own the active project (enforced by RLS policy prompts_project_owner_insert).
  • If the user is not signed in or is viewing a project they don't own, the playground shows existing prompts (if any are public) but cannot create new ones.

Supabase migration​

The prompts and prompt_versions tables are created by migration 20260605120000_prompt_versioning.sql. The transactional guard RPCs (create_prompt_version_and_set_current and set_prompt_current_version) are added by migration 20260604124328_prompt_version_rpc_guards.sql. Both migrations must be applied to the remote database for the playground to function correctly:

npx supabase db push --yes --include-all

If the migration is not applied, the playground will silently disable itself with no visible panel.

What this means for users (plain English)​

In simple terms:

The Prompt Playground is now a dedicated IDE-style panel in the right workbench. Click the Prompts icon in the far-left activity bar to show it. It lets you:

  • Browse existing prompts for your current project
  • Create new prompts using the + New button
  • Edit prompt templates (either single-string or multi-message chat)
  • Fill in variables like {{name}} and test the prompt with real LLM responses
  • Save different versions of your prompts and roll back to older ones

When it works:

  • You must be signed in
  • You must own the current project (you can't create prompts in someone else's project)
  • The database must have the prompt tables (which are already set up)

When it doesn't work:

  • If you're not signed in, the panel won't open
  • If you're viewing a project you don't own, you can see existing prompts but can't create new ones
  • If the database tables are missing, the panel stays hidden

Edge Function reliability​

The run-playground and run-eval Edge Functions include several safety mechanisms to handle adverse runtime conditions:

Rate limiter memory cleanup (run-playground)​

The per-user rate limiter maintains a sliding window of timestamps in a Map. When a user's window empties (all timestamps expire), the key is deleted from the Map rather than storing an empty array. This prevents unbounded Map growth from users who make occasional requests.

Tools and response_format forwarding (run-playground)​

When the request body includes tools (function definitions) or response_format (JSON schema / structured output), these are forwarded to the underlying provider API. For OpenAI the full tools array and response_format object are passed through. For Anthropic only tools is forwarded (Anthropic does not support response_format).

Provider routing by model_developer (run-playground)​

The function routes to OpenAI or Anthropic based solely on the model_developer field in the request body, not by inspecting the model name for substring matches (e.g. "claude"). This makes routing explicit and unambiguous: a model named custom-claude-v2 under the divproxy developer will not be accidentally routed to the Anthropic API.

Upstream fetch timeouts (both functions)​

All upstream LLM API calls use AbortSignal.timeout:

  • run-playground: 60-second timeout per fetch
  • run-eval: 30-second timeout per fetch

If the provider does not respond within the window, the fetch is aborted and an error is reported to the caller.

SSE disconnect handling (run-playground)​

If the client disconnects mid-stream, the SSE write (controller.enqueue) and close (controller.close) inside the error handler may throw. These calls are wrapped in a nested try-catch so that a client disconnect does not cause an unhandled rejection in the Edge Function.

Non-LLM evaluator guard (run-eval)​

Only evaluators with kind === "llm" will attempt to load a prompt version. Heuristic and code evaluators skip the prompt_versions lookup entirely, avoiding a redundant database query for evaluators that do not call an LLM.

Evaluator seed data updates (seed.sql)​

The seed SQL was corrected to include a proper hallucination evaluator linked to its own prompt version (d5). Previously the comment-to-ID mapping was scrambled: the comments now correctly reflect which ID maps to which evaluator name. The hallucination evaluator uses the same LLM/openai provider pattern as the other quality evaluators.

Usage​

import { listPrompts, createPrompt, createPromptVersion } from '../data/dal'

// List all prompts for the demo project
const prompts = await listPrompts('00000000-0000-0000-0000-000000000001')
if (prompts.ok) {
  console.log(prompts.value)
}

// Create a new prompt
const newPrompt = await createPrompt({
  project_id: '00000000-0000-0000-0000-000000000001',
  name: 'my-prompt',
  description: 'A playground prompt',
  category: 'playground',
})

// Create a version and set it as current
if (newPrompt.ok) {
  await createPromptVersion({
    prompt_id: newPrompt.value.id,
    version_number: 1,
    template_type: 'STR',
    template_format: 'MUSTACHE',
    template: { template: 'Hello {{name}}!' },
    model_developer: 'divproxy',
    model_name: 'gpt-4o-mini',
    invocation_parameters: {},
    label: 'v1',
  }, true)
}