The openai-realtime-api-beta package can be used directly from within a browser but all of the code is designed to work from within a React/Vue app. I thought I’d show you how to use the packages directly from a page using JavaScript Modules.
First, you’ll need a node.js webserver like Express.js and you’ll need to setup a static route the package distributions. Here’s how I have my packages setup on disk:
And here’s the static express route I setup (you’ll need to adjust the path:)
// Serve browser modules
app.use('/browser_modules', express.static(path.join(__dirname, '../../browser_modules')));
Now you just need to import the packages into your browser page as part of a script block that uses JavaScript Modules. Here’s an example:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Realtime Client Example</title>
</head>
<body>
<h1>Realtime Client Example</h1>
<button id="connectClient">Connect Client</button>
<script type="module">
// Import client and create a new instance that connects to a relay server
import { RealtimeClient } from '/browser_modules/realtime-api-beta/index.js';
const client = new RealtimeClient({ url: 'http://localhost:8081' });
// Can set parameters ahead of connecting, either separately or all at once
client.updateSession({ instructions: 'You are a great, upbeat friend.' });
client.updateSession({ voice: 'alloy' });
client.updateSession({
turn_detection: { type: 'none' }, // or 'server_vad'
input_audio_transcription: { model: 'whisper-1' },
});
// Set up event handling
client.on('conversation.updated', (event) => {
const { item, delta } = event;
const items = client.conversation.getItems();
/**
* item is the current item being updated
* delta can be null or populated
* you can fetch a full list of items at any time
*/
});
async function connectClient() {
// Connect to Realtime API
await client.connect();
// Send a item and triggers a generation
client.sendUserMessageContent([{ type: 'input_text', text: `How are you?` }]);
}
// Event listeners for buttons
document.getElementById('connectClient').addEventListener('click', connectClient);
</script>
</body>
</html>
The code is straight from OpenAI’s example and I’ve just updated it to call back to the host server over a socket using what they call “relay server” mode. You can get the code for the relay server from their console example.
Hope that’s useful…