Skip to content

Commit c34b977

Browse files
dlarocquedlarocque
committed
feat(ai): Add LiveSession realtime APIs
1 parent 0bb2fe6 commit c34b977

File tree

9 files changed

+367
-49
lines changed

9 files changed

+367
-49
lines changed

‎.changeset/dull-ligers-bow.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
---
2+
'@firebase/ai': minor
3+
---
4+
5+
Add `sendTextRealtime()`, `sendAudioReatime()`, and `sendVideoRealtime()` to the `LiveSession` class, and deprecate `sendMediaChunks()` and `sendMediaStream()`.

‎common/api-review/ai.api.md‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -994,9 +994,14 @@ export class LiveSession {
994994
isClosed: boolean;
995995
receive(): AsyncGenerator<LiveServerContent | LiveServerToolCall | LiveServerToolCallCancellation>;
996996
send(request: string | Array<string | Part>, turnComplete?: boolean): Promise<void>;
997+
sendAudioRealtime(blob: GenerativeContentBlob): Promise<void>;
997998
sendFunctionResponses(functionResponses: FunctionResponse[]): Promise<void>;
999+
// @deprecated (undocumented)
9981000
sendMediaChunks(mediaChunks: GenerativeContentBlob[]): Promise<void>;
1001+
// @deprecated (undocumented)
9991002
sendMediaStream(mediaChunkStream: ReadableStream<GenerativeContentBlob>): Promise<void>;
1003+
sendTextRealtime(text: string): Promise<void>;
1004+
sendVideoRealtime(blob: GenerativeContentBlob): Promise<void>;
10001005
}
10011006

10021007
// @public

‎docs-devsite/ai.livesession.md‎

Lines changed: 130 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -39,9 +39,12 @@ export declare class LiveSession
3939
| [close()](./ai.livesession.md#livesessionclose) | | <b><i>(Public Preview)</i></b> Closes this session. All methods on this session will throw an error once this resolves. |
4040
| [receive()](./ai.livesession.md#livesessionreceive) | | <b><i>(Public Preview)</i></b> Yields messages received from the server. This can only be used by one consumer at a time. |
4141
| [send(request, turnComplete)](./ai.livesession.md#livesessionsend) | | <b><i>(Public Preview)</i></b> Sends content to the server. |
42+
| [sendAudioRealtime(blob)](./ai.livesession.md#livesessionsendaudiorealtime) | | <b><i>(Public Preview)</i></b> Sends audio data to the server in realtime. |
4243
| [sendFunctionResponses(functionResponses)](./ai.livesession.md#livesessionsendfunctionresponses) | | <b><i>(Public Preview)</i></b> Sends function responses to the server. |
43-
| [sendMediaChunks(mediaChunks)](./ai.livesession.md#livesessionsendmediachunks) | | <b><i>(Public Preview)</i></b> Sends realtime input to the server. |
44-
| [sendMediaStream(mediaChunkStream)](./ai.livesession.md#livesessionsendmediastream) | | <b><i>(Public Preview)</i></b> Sends a stream of [GenerativeContentBlob](./ai.generativecontentblob.md#generativecontentblob_interface)<!-- -->. |
44+
| [sendMediaChunks(mediaChunks)](./ai.livesession.md#livesessionsendmediachunks) | | <b><i>(Public Preview)</i></b> |
45+
| [sendMediaStream(mediaChunkStream)](./ai.livesession.md#livesessionsendmediastream) | | <b><i>(Public Preview)</i></b> |
46+
| [sendTextRealtime(text)](./ai.livesession.md#livesessionsendtextrealtime) | | <b><i>(Public Preview)</i></b> Sends text to the server in realtime. |
47+
| [sendVideoRealtime(blob)](./ai.livesession.md#livesessionsendvideorealtime) | | <b><i>(Public Preview)</i></b> Sends video data to the server in realtime. |
4548

4649
## LiveSession.inConversation
4750

@@ -135,6 +138,45 @@ Promise&lt;void&gt;
135138

136139
If this session has been closed.
137140

141+
## LiveSession.sendAudioRealtime()
142+
143+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
144+
>
145+
146+
Sends audio data to the server in realtime.
147+
148+
The server requires that the audio data is base64-encoded 16-bit PCM at 16kHz little-endian.
149+
150+
<b>Signature:</b>
151+
152+
```typescript
153+
sendAudioRealtime(blob: GenerativeContentBlob): Promise<void>;
154+
```
155+
156+
#### Parameters
157+
158+
| Parameter | Type | Description |
159+
| --- | --- | --- |
160+
| blob | [GenerativeContentBlob](./ai.generativecontentblob.md#generativecontentblob_interface) | The base64-encoded PCM data to send to the server in realtime. |
161+
162+
<b>Returns:</b>
163+
164+
Promise&lt;void&gt;
165+
166+
#### Exceptions
167+
168+
If this session has been closed.
169+
170+
### Example
171+
172+
173+
```javascript
174+
// const pcmData = ... base64-encoded 16-bit PCM at 16kHz little-endian.
175+
const blob = { mimeType: "audio/pcm", data: pcmData };
176+
liveSession.sendAudioRealtime(blob);
177+
178+
```
179+
138180
## LiveSession.sendFunctionResponses()
139181

140182
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
@@ -167,7 +209,12 @@ If this session has been closed.
167209
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
168210
>
169211
170-
Sends realtime input to the server.
212+
> Warning: This API is now obsolete.
213+
>
214+
> Use `sendTextRealtime()`<!-- -->, `sendAudioRealtime()`<!-- -->, and `sendVideoRealtime()` instead.
215+
>
216+
> Sends realtime input to the server.
217+
>
171218
172219
<b>Signature:</b>
173220

@@ -194,7 +241,12 @@ If this session has been closed.
194241
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
195242
>
196243
197-
Sends a stream of [GenerativeContentBlob](./ai.generativecontentblob.md#generativecontentblob_interface)<!-- -->.
244+
> Warning: This API is now obsolete.
245+
>
246+
> Use `sendTextRealtime()`<!-- -->, `sendAudioRealtime()`<!-- -->, and `sendVideoRealtime()` instead.
247+
>
248+
> Sends a stream of [GenerativeContentBlob](./ai.generativecontentblob.md#generativecontentblob_interface)<!-- -->.
249+
>
198250
199251
<b>Signature:</b>
200252

@@ -216,3 +268,77 @@ Promise&lt;void&gt;
216268

217269
If this session has been closed.
218270

271+
## LiveSession.sendTextRealtime()
272+
273+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
274+
>
275+
276+
Sends text to the server in realtime.
277+
278+
<b>Signature:</b>
279+
280+
```typescript
281+
sendTextRealtime(text: string): Promise<void>;
282+
```
283+
284+
#### Parameters
285+
286+
| Parameter | Type | Description |
287+
| --- | --- | --- |
288+
| text | string | The text data to send. |
289+
290+
<b>Returns:</b>
291+
292+
Promise&lt;void&gt;
293+
294+
#### Exceptions
295+
296+
If this session has been closed.
297+
298+
### Example
299+
300+
301+
```javascript
302+
liveSession.sendTextRealtime("Hello, how are you?");
303+
304+
```
305+
306+
## LiveSession.sendVideoRealtime()
307+
308+
> This API is provided as a preview for developers and may change based on feedback that we receive. Do not use this API in a production environment.
309+
>
310+
311+
Sends video data to the server in realtime.
312+
313+
The server requires that the video is sent as individual video frames at 1 FPS. It is recommended to set `mimeType` to `image/jpeg`<!-- -->.
314+
315+
<b>Signature:</b>
316+
317+
```typescript
318+
sendVideoRealtime(blob: GenerativeContentBlob): Promise<void>;
319+
```
320+
321+
#### Parameters
322+
323+
| Parameter | Type | Description |
324+
| --- | --- | --- |
325+
| blob | [GenerativeContentBlob](./ai.generativecontentblob.md#generativecontentblob_interface) | The base64-encoded video data to send to the server in realtime. |
326+
327+
<b>Returns:</b>
328+
329+
Promise&lt;void&gt;
330+
331+
#### Exceptions
332+
333+
If this session has been closed.
334+
335+
### Example
336+
337+
338+
```javascript
339+
// const videoFrame = ... JPEG data
340+
const blob = { mimeType: "image/jpeg", data: videoFrame };
341+
liveSession.sendAudioRealtime(blob);
342+
343+
```
344+

‎packages/ai/integration/live.test.ts‎

Lines changed: 39 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -154,6 +154,45 @@ describe('Live', function () {
154154
});
155155
});
156156

157+
describe('sendTextRealtime()', () => {
158+
it('should send a single text chunk and receive a response', async () => {
159+
const model = getLiveGenerativeModel(testConfig.ai, {
160+
model: testConfig.model,
161+
generationConfig: textLiveGenerationConfig
162+
});
163+
const session = await model.connect();
164+
const responsePromise = nextTurnText(session.receive());
165+
166+
await session.sendTextRealtime('Are you an AI? Yes or No.');
167+
168+
const responseText = await responsePromise;
169+
expect(responseText).to.include('Yes');
170+
171+
await session.close();
172+
});
173+
});
174+
175+
describe('sendAudioRealtime()', () => {
176+
it('should send a single audio chunk and receive a response', async () => {
177+
const model = getLiveGenerativeModel(testConfig.ai, {
178+
model: testConfig.model,
179+
generationConfig: textLiveGenerationConfig
180+
});
181+
const session = await model.connect();
182+
const responsePromise = nextTurnText(session.receive());
183+
184+
await session.sendAudioRealtime({
185+
data: HELLO_AUDIO_PCM_BASE64, // "Hey, can you hear me?"
186+
mimeType: 'audio/pcm'
187+
});
188+
189+
const responseText = await responsePromise;
190+
expect(responseText).to.include('Yes');
191+
192+
await session.close();
193+
});
194+
});
195+
157196
describe('sendMediaChunks()', () => {
158197
it('should send a single audio chunk and receive a response', async () => {
159198
const model = getLiveGenerativeModel(testConfig.ai, {

‎packages/ai/src/methods/live-session-helpers.test.ts‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -65,7 +65,7 @@ class MockLiveSession {
6565
isClosed = false;
6666
inConversation = false;
6767
send = sinon.stub();
68-
sendMediaChunks = sinon.stub();
68+
sendAudioRealtime = sinon.stub();
6969
sendFunctionResponses = sinon.stub();
7070
messageGenerator = new MockMessageGenerator();
7171
receive = (): MockMessageGenerator => this.messageGenerator;
@@ -226,8 +226,8 @@ describe('Audio Conversation Helpers', () => {
226226

227227
await clock.tickAsync(1);
228228

229-
expect(liveSession.sendMediaChunks).to.have.been.calledOnce;
230-
const [sentChunk] = liveSession.sendMediaChunks.getCall(0).args[0];
229+
expect(liveSession.sendAudioRealtime).to.have.been.calledOnce;
230+
const sentChunk = liveSession.sendAudioRealtime.getCall(0).args[0];
231231
expect(sentChunk.mimeType).to.equal('audio/pcm');
232232
expect(sentChunk.data).to.be.a('string');
233233
await controller.stop();

‎packages/ai/src/methods/live-session-helpers.ts‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -184,7 +184,7 @@ export class AudioConversationRunner {
184184
mimeType: 'audio/pcm',
185185
data: base64
186186
};
187-
void this.liveSession.sendMediaChunks([chunk]);
187+
void this.liveSession.sendAudioRealtime(chunk);
188188
};
189189
}
190190

‎packages/ai/src/methods/live-session.test.ts‎

Lines changed: 36 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -110,6 +110,42 @@ describe('LiveSession', () => {
110110
});
111111
});
112112

113+
describe('sendTextRealtime()', () => {
114+
it('should send a correctly formatted realtimeInput message', async () => {
115+
const text = 'foo';
116+
await session.sendTextRealtime(text);
117+
expect(mockHandler.send).to.have.been.calledOnce;
118+
const sentData = JSON.parse(mockHandler.send.getCall(0).args[0]);
119+
expect(sentData).to.deep.equal({
120+
realtimeInput: { text }
121+
});
122+
});
123+
});
124+
125+
describe('sendAudioRealtime()', () => {
126+
it('should send a correctly formatted realtimeInput message', async () => {
127+
const blob = { data: 'abcdef', mimeType: 'audio/pcm' };
128+
await session.sendAudioRealtime(blob);
129+
expect(mockHandler.send).to.have.been.calledOnce;
130+
const sentData = JSON.parse(mockHandler.send.getCall(0).args[0]);
131+
expect(sentData).to.deep.equal({
132+
realtimeInput: { audio: blob }
133+
});
134+
});
135+
});
136+
137+
describe('sendVideoRealtime()', () => {
138+
it('should send a correctly formatted realtimeInput message', async () => {
139+
const blob = { data: 'abcdef', mimeType: 'image/jpeg' };
140+
await session.sendVideoRealtime(blob);
141+
expect(mockHandler.send).to.have.been.calledOnce;
142+
const sentData = JSON.parse(mockHandler.send.getCall(0).args[0]);
143+
expect(sentData).to.deep.equal({
144+
realtimeInput: { video: blob }
145+
});
146+
});
147+
});
148+
113149
describe('sendMediaChunks()', () => {
114150
it('should send a correctly formatted realtimeInput message', async () => {
115151
const chunks = [{ data: 'base64', mimeType: 'audio/webm' }];

0 commit comments

Comments
 (0)