Pass context/metadata to VectorStore search

[rcwsr]

Hi, really enjoying this library!

Is there a way to pass more context to a vector store when doing a similarity search? I am using Pinecone and want to filter the data being searched using their metadata filters feature (e.g. user permissions). Right now I have my own implementation of UserMessage which i set some metadata on, and I have overridden \NeuronAI\RAG\RAG::retrieveDocuments (copy/paste) to pass a $metadata array to \NeuronAI\RAG\VectorStore\VectorStoreInterface::similaritySearch- which I have implemented in my own PineconeVectorStore.

Am I missing something or is there just not a nicer way to do that yet?

Cheers, Rob

[ilvalerione]

Can you share the snippet of the code performing the QUERY request with this hybrid approach?

[rcwsr]

This is a non working example, in terms of the actual filtering, but this is the kind of approach i'm looking at.

Potentially also looking at using another agent to produce some other filters based on the user query - but that's out of scope at the moment while I get the basics working!

// controller:

$questionAgent = $this->agentFactory->createAgent();

$answerResponse = $questionAgent->answer(
    new UserMessage($question)->addMetadata('filters', ['some_filters'])
);

// agent factory:

public function createAgent(): RAG
    return RagAgent::make()
        ->withInstructions(<<<INSTRUCTIONS
...
INSTRUCTIONS
)
    ->withProvider($this->aiProviderFactory->create())
    ->setEmbeddingsProvider($this->embeddingsProvider)
    ->setVectorStore(new PineconeVectorStore($this->pineconeClient, namespace: 'acme'));
}

// pinecone vector store:

public function similaritySearch(array $embedding, array $metadata): iterable
{
    $result = $this->pineconeClient->data()->vectors()->query(
        vector: $embedding,
        namespace: $this->namespace,
        filter: $metadata['filters'],
    );
    ...
    
// override retrieveDocuments in my RagAgent class in order to pass extra arg to similaritySearch() :

class RagAgent extends RAG
{
    public function retrieveDocuments(Message $question): array
    {
        $this->notify('rag-vectorstore-searching', new VectorStoreSearching($question));

        $docs = $this->resolveVectorStore()->similaritySearch(
            $this->resolveEmbeddingsProvider()->embedText($question->getContent()),
            $question instanceof ScopedUserMessage ? $question->getMetadata()['filters'] : []
        );
        ...

I guess one alternative would be to instantiate the agent and vector store where i need it, and pass the metadata/filters to the VectorStore contructor, so either do that via my AgentFactory or get rid of the factory altogether.

[ilvalerione]

A built-in solution could be to accept an additional configuration "filters" when vector store instance is created, like this:

class MyChatBot extends RAG
{
    public function __construct(protected array $pineconeFilters = [])
    {
    }

    ...
    
    protected function vectorStore(): VectorStoreInterface
    {
        return (new PineconeVectoreStore(
            key: 'PINECONE_API_KEY',
            indexUrl: 'PINECONE_INDEX_URL'
        ))->withFitlers($this->pineconeFilters);
    }

    public function withPineconeFilters(array $filters): self
    {
        $this->pineconeFilters = $filters;
        return $this;
    }
}

With this implementation you have two ways to pass the filters:

// When the agent instance is created
$agent = RagAgent::make([...]);

// Or at runtime
$agent->withPineconeFilters([...])->answer( new UserMessage(...));

If you are getting your agent instance from a factory you could go for the second and avoid the constructor parameter.

What do you think?

We need to implement the support for filters into the Pinecone component.

[rcwsr]

Yeah I think that will be what i do, and it's what i was getting at in my edits above. I'm coming from an app where these types of classes are usually services.

If i get time i'll put a PR together for filters.

[ilvalerione]

It's already available in the latest 1.12.12

https://docs.neuron-ai.dev/components/vector-store#pinecone

Let me know if you face any problem.

What's the library you are coming from?

[rcwsr]

Nice i'll have a play with it, now just need a way to set the metadata via addDocuments().

I meant that I'm working on a new app at work right now, but traditionally I'm used to a big Symfony app that is very heavy on DI where basically everything is a service. Neuron-ai doesn't seem to encourage use of services, instead encapsulating your agent as a class when needed, and calling setters for deps. E.g. i couldnt make the PineconeVectorStore a service since I'd want to set those filters at runtime in a controller action. But that's fine just not what I'm used to 😄

[ilvalerione]

Ok, I understand. Symfony developers usually use the agent class as a service to automatically inject other services into the agent instance. These services can also be vector store instances or other agent components. But these are quite advanced use cases where it's needed to extend default implementations with custom ones.

I think Neuron has a lot of margin to improve its components and support more and more use cases out of the box.

For your implementations, yes, probably the Document class should support the possibility to carry additional fields in and out of the vector store.