Skip to content
This repository was archived by the owner on May 29, 2025. It is now read-only.

Stateful chat #2

Draft
wants to merge 2 commits into
base: master
Choose a base branch
from
Draft

Stateful chat #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
ef2bd72
make chat function stateful
bennyschmidt Mar 20, 2024
5968224
Merge branch 'master' into stateful-chat
bennyschmidt Mar 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 25 additions & 10 deletions arthas-api/api/post/prompt/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -28,11 +28,15 @@ const {

const { prefixInput } = require('arthasgpt/src/utils/prefix');

const __statefulChatFunction = {};
Copy link
Owner Author

@bennyschmidt bennyschmidt Mar 20, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can cache the stateful chat method per worker, but it is a bit of an antipattern in a stateless API, and when scaling instances there will be multiple (inconsistent) histories depending on which worker is picking up the request.

With CACHE=true in the .env we already cache the query, prompt, and model responses. For scale, it might be worth the trade-off to keep the prompt endpoint stateless, relying on library-level caching for speed, and an application-level solution for chat history/context.

Note: __statefulChatFunction is instantiated as an object (hash) because it maintains a chat method for each active persona, not just 1, like this:

__statefulChatFunction = {
  [persona1Key]: persona1ChatFunction,
  [persona2Key]: persona2ChatFunction,
    
  ...
}

to be called like this:

__statefulChatFunction[persona1Key]("what did you just say?")


/**
* Prompt
*/

module.exports = asyncCache => async (req, res) => {
const config = await asyncCache.getItem('config');

const timeout = await asyncCache.getItem('timeout');

let answer = await asyncCache.getItem('answer');
Expand All @@ -44,8 +48,6 @@ module.exports = asyncCache => async (req, res) => {
body.push(chunk);
})
.on('end', async () => {
const config = await asyncCache.getItem('config');

body = Buffer.concat(body).toString();

const { key, input } = JSON.parse(body || '{}');
Expand Down Expand Up @@ -112,16 +114,29 @@ module.exports = asyncCache => async (req, res) => {
await delay(timeout);
}

if (isVerbose) {
log(CREATING_AGENT);
}
const currentChat = __statefulChatFunction?.[key];

answer = await ArthasGPT({
...currentConfig,
if (currentChat) {
await currentChat(messageResponse);
} else {
if (isVerbose) {
log(CREATING_AGENT);
}

query: messageResponse,
cache: false
});
answer = await ArthasGPT({
...currentConfig,

query: messageResponse,
cache: true
});

await asyncCache.setItem('answer', answer);

// Cache the stateful chat method in the API
// (antipattern)

__statefulChatFunction[key] = answer?.chat;
}

res.end(JSON.stringify({
success: true,
Expand Down
3 changes: 2 additions & 1 deletion arthas-api/index.js
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -31,7 +31,8 @@ module.exports = (routes = { GET: {}, POST: {}}, init) => {
* Config
*/

const numCPUs = availableParallelism();
// const numCPUs = availableParallelism();
const numCPUs = 1;

const onCluster = require('./events/cluster');
const onWorker = require('./events/worker');
Expand Down