Add a high level Python query library ontop of gafferpy

[t92549]

gafferpy is great for directly sending json to the Gaffer rest-api, and to do this creates Python objects that map 1 to 1 to that json. However, these queries can become very long and require knowledge of Gaffer's verbose query language/json.
For a better user experience, an additional library should be made to sit on top of gafferpy, allowing users to specify more Pythonic, user friendly queries that would get translated to gafferpy queries and sent.

For example, here is a json query for a GetElements operation on the road traffic api. It gets Edges connected to the Entity with vertex M32:1. These are then filtered on the count property being more than 1, and the group by removed. It is also using the gaffer.federatedstore.operation.graphIds option to assert that this only gets executes on sub-graph graph1:

{
    "class": "uk.gov.gchq.gaffer.operation.impl.get.GetElements",
    "input": [
        {
            "class": "uk.gov.gchq.gaffer.operation.data.EntitySeed",
            "vertex": "M32:1"
        }
    ],
    "view": {
        "edges": {
            "RoadUse": {
                "preAggregationFilterFunctions": [
                    {
                        "selection": [
                            "count"
                        ],
                        "predicate": {
                            "class": "uk.gov.gchq.koryphe.impl.predicate.IsMoreThan",
                            "value": {
                                "java.lang.Long": 1
                            }
                        }
                    }
                ],
                "groupBy": []
            }
        }
    },
    "directedType": "EITHER",
    "options": {
        "gaffer.federatedstore.operation.graphIds": "graph1"
    }
}

This is a very long and verbose mapping to the Java api. The gafferpy code to perform this query is a verbose map to this json:

from gafferpy import gaffer as g
from gafferpy import gaffer_connector

gc = gaffer_connector.GafferConnector("http://localhost:8080/rest/latest")
op = g.GetElements(
    input=['M32:1'],
    view=g.View(
        edges=[
            g.ElementDefinition(
                group='RoadUse',
                group_by=[],
                pre_aggregation_filter_functions=[
                    g.PredicateContext(
                        selection=['count'],
                        predicate=g.IsMoreThan(
                            value=g.long(1)
                        )
                    )
                ]
            )
        ]
    ),
    directed_type=g.DirectedType.EITHER,
    options=["graph1"]
)
results = gc.execute_operation(op)

A more usable query library based in Python could look something like this:

from gafferpy import gaffer_query as gq
from gafferpy import gaffer_connector

gc = gaffer_connector.GafferConnector("http://localhost:8080/rest/latest")
results = gq.GetElements(using=gc, graphs="graph1") \
            .input("M32:1") \
            .view(edge="RoadUse", group_by=[], pre_agg_filter="count > 1") \
            .directed("either")

Most of this simplification could be achieved by restructuring operations so that objects like ElementDefinitions don't have to be created in such a verbose way.
For the simplification of the predicate however, a parser would have to be written to map the string to the relevant Predicate.

[t92549]

This would make querying with gafferpy significantly smoother for users. Together with issue #4, which makes using gaffer's results smoother also, the user experience could be greatly enhanced.

[n3101]

@t92549 How does this align with the alternative use of Jupyter notebooks as Gaffer clients.

[t92549]

It would make them a lot easier to use and accessible

[n3101]

@t92549 sorry I meant - would this library also be available in Jupyter, or unnecessary, or equivalent to another thing or what?

[t92549]

Currently you can use gafferpy in jupyter notebooks, and you would be able to use this high level pythonic library in a notebook also

[t92549]

For reference, the Java code for this query looks like this:

final GetElements query = new GetElements.Builder()
        .input(new EntitySeed("M32:1"))
        .view(new View.Builder()
                .edge("RoadUse", new ViewElementDefinition.Builder()
                        .preAggregationFilter(new ElementFilter.Builder()
                                .select("count")
                                .execute(new IsMoreThan(1))
                                .build())
                        .groupBy((String[]) null)
                        .build())
                .build())
        .directedType(DirectedType.EITHER)
        .option("gaffer.federatedstore.operation.graphIds", "graph1")
        .build();

[t92549]

Is this now redundant with gremlin being added to Gaffer?

[GCHQDeveloper314]

I wouldn't say so. Gremlin doesn't fulfil your request in the original post which asks for "an additional library ... to sit on top of gafferpy, allowing users to specify more Pythonic, user friendly queries".

I don't think the Python API offered by Gremlin is sufficient to meet this requirement as it's not possible to interact with that in the same way (i.e. using Gaffer specific terminologies and functions).