Feature: Add ability to stop/interupt stream

[taylorn-ai]

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [X] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

I would like to have a button that will stop/interrupt the stream being rendered (ChatGPT has this). So if the response is going downhill, the user would be able to stop it before it goes any further. I've tried implementing something myself, but it doesn't quite work with the ASyncGenerator.

[pamelafox]

Interesting. I wonder if you could try ReadableStream.cancel() method, as ReadableStream() is what we use in the frontend:
https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream/cancel

The spec says that sending a cancel event should attempt to cancel the underlying mechanism, so I think it would close the TCP connection? I'm not sure if you'd still pay for the full tokens of the call though. That's worth experimenting with, if you haven't yet.

Otherwise I fear you'd need to move to something with better bidirectional support like web sockets.

[taylorn-ai]

Thanks for the prompt response! I was trying to cancel the asyncio thread in Quart, but cancelling the stream make much more sense :)

I've been able to implement this just now, and it seems to work quite well. As far as I can tell, it also stops the response coming back from the model, but I am not sure if there's a way I can check if I was still billed for the full token count, but it doesn't bother me that much, the user experience is more important :)

const [responseStream, setResponseStream] = useState<ReadableStream|null>(null);
const handleAsyncRequest = async (question: string, answers: [string, ChatAppResponse][], setAnswers: Function, responseBody: ReadableStream<any>) => {
        setResponseStream(responseBody);
// ... other code ... //

Then I've just added a button that calls onStopClick()

const onStopClick = async () => {
    try {
        if (responseStream) {
            responseStream.cancel("User cancelled the stream");
        }
    } catch (e) {
        console.log(e);
    }
};

[pamelafox]

Very cool! You might be able to monitor token usage in your Azure OpenAI metrics dashboard.

[oracle-code]

Looks nice, very cool indeed. We will try it on our end and give you feedback!

[TrentinMarc]

Hi all,
@TaylorN15 don't you face an error such as :

ReadableStream.cancel: Cannot cancel a stream locked by a reader ?

I could not interrupt my chat the way you shared, but I just found a workaround like this :

export class ClassForReadNDJSONStream {
    reader: ReadableStreamDefaultReader;
    responseBody: ReadableStream<any>;

    constructor(responseBody: ReadableStream<any>) {
        this.responseBody = responseBody;
        this.reader = responseBody.getReader();
    }
    async *readStream() {
        let runningText = "";
        let decoder = new TextDecoder("utf-8");
        while (true) {
            const { done, value } = await this.reader.read();
            if (done) break;
            var text = decoder.decode(value, { stream: true });
            const objects = text.split("\n");
            for (const obj of objects) {
                try {
                    runningText += obj;
                    let result = JSON.parse(runningText);
                    yield result;
                    runningText = "";
                } catch (e) {
                    // Not a valid JSON object
                }
            }
        }
    }
}

and

const c4readNDSJonstream = new ClassForReadNDJSONStream(responseBody);
setResponseStream(c4readNDSJonstream); // Using your method with ClassForReadNDJSONStream signature

for await (const event of c4readNDSJonstream.readStream()) {
// other code ...

Based on @pamelafox readNDJSONStream lib but expose reader to allow reader.releaseLock() (and then responseBody.cancel()) in onStopClick() to interrupt chat answer.

What do you think about it ?

[taylorn-ai]

Yes, actually, I did get that error :)

I ended up implementing an abort controller instead.

[oracle-code]

Would be able to share the code ? @TaylorN15

[taylorn-ai]

Add the AbortController
const [abortController, setAbortController] = useState<AbortController | null>(null);

Add the below into makeApiRequest

const makeApiRequest = async (question: string) => {
	const controller = new AbortController();
	setAbortController(controller);

Add the AbortSignal to the props of handleAsyncRequest

const handleAsyncRequest = async (
	question: string,
	answers: [string, ChatAppResponse][],
	responseBody: ReadableStream<any>,
	signal: AbortSignal
) => {

And the logic for handling the abort, also in handleAsyncRequest

setIsStreaming(true);
for await (const event of readNDJSONStream(responseBody)) {
	if (signal.aborted) {
		// Abort the stream if requested
		break;
	}

Create a function for when the button is clicked, and call from where you want, I have mine in the question container, so that the stop button appears instead of the send button, if the answer is streaming

const onStopClick = async () => {
	try {
		if (abortController) {
			abortController.abort();
		}
	} catch (e) {
		console.log("An error occurred trying to stop the stream: ", error);
	}
};

[TrentinMarc]

Thanks a lot @TaylorN15

AbortController works properly for this usage. Just in case, I passed signal directly to chatApi function, it allows you to cancel chat at any time (before you enter the "for await" loop).

export async function chatApi(request: ChatAppRequest, idToken: string | undefined, signal: AbortSignal): Promise<Response> {
    return await fetch(`${BACKEND_URI}/chat`, {
        method: "POST",
        headers: getHeaders(idToken),
        body: JSON.stringify(request),
        signal: signal
    });
}

[taylorn-ai]

Nice, does yours actually stop the stream on the backend? It seems that most of the time, mine doesn’t. It doesn’t bother me much, but it would be nice to reduce the token usage :)

[TrentinMarc]

Hello, sorry for the delay.

Yes, it stops the stream on the backend. Depending on when the chat is stopped, I may save a few tokens.

[taylorn-ai]

Did you add any code to the backend to handle the abort?

[TrentinMarc]

No, I didn't add anything to the backend, as fetch api natively handles AbortController signals.