Setup integration tests (with databases in container, Java app for SpEL)

[ukrbublik]

We need to have a setup to write integration tests to check that export utils to SQL, MongoDb, ElasticSearch, SpEL works correctly - generates queries that will produce expected results on real databases on sample data.

Recommendation is to use GitHub actions (eg. with mongodb-in-github-actions, elasticsearch-action, setup-postgresql, setup-mysql ..)
Also need a setup to run new integration tests locally (with Docker containers, docker-compose ?).

Current tests internal package is set up to run unit tests with Enzyme on Karma. So either a separate internal package (like tests-integration) could be created or the current one can be extended to run integration tests as well.

On GitHub tests are running in smoke.yml workflow action.
For integration tests it worth having a separate action (using mongodb-in-github-actions, ... mentioned above).

First minimum test

See sample code that declares a config with 2 fields (name and age), crates query age >= 18 && name == 'denis' and exports it to multiple formats: SQL, Mongo, Elastic, SpEL, JsonLogic.
Given 3 persons {name: 'denis', age: 37}, {name: 'denis', age: 3}, {name: 'oleg', age: 20} we expect that this query will result in 1 record when running this query on MySQL server, PostgreSQL, MongoDB, Elastic and in Java app using SpEL.

Databases (SQL, MongoDb, Elastic)

For SQL we need to have a server running in container and JS API to run queries agains this server.
We need to:

• create a table with name and age
• add 3 records mentioned above
• run query generated by RAQB with Utils.sqlFormat(state.tree, config)

And have an ability to run these steeps against MySQL server and PostgreSQL server

Similar steps should be done for MongoDb and Elastic.

SpEL

For SpEL I think it would be nice to have a Java app on Spring Framework that can parse and evaluate given SpEL expression with given context using some simple REST API available to JS app.

I am not familiar with Java development so used sample app like this to test SpEL expressions generated by RAQB:

import org.springframework.expression.Expression;  
import org.springframework.expression.ExpressionParser;  
import org.springframework.expression.spel.standard.SpelExpressionParser; 
import org.springframework.expression.spel.support.StandardEvaluationContext;

class Person {
    public String name;
    public int age;
}

public class App {
  public static void main( String[] args )  {
    try {
        ExpressionParser parser = new SpelExpressionParser();
        String expr = "age >= 18 && name == 'denis'"; // copy-paste output of `Utils.Export.spelFormat (tree, config)` 
        Expression exp = parser.parseExpression(expr);
        Person person = new Person();
        person.age = 18; // fill manually
        person.str = "denis";  // fill manually
        StandardEvaluationContext ctx = new StandardEvaluationContext(person);
        boolean res = exp.getValue(demo, Boolean.class);
        System.out.println(res); // expect to be "true"
    } catch (Exception e) {
      System.err.println(e);
    }
  }
}

It's an open question - how can we generate a context for SpEL (instance of Person class in code) dynamically by REST API?
Maybe preliminarily on compile-time create all classes (like Person in example) that can be used in contexts for SpEL expressions in Java app.
And only have ability to pass serialized instance of one of such classes as context + SpEL expression string to evaluate the result.

[Blue-smoke007]

I can build a robust integration testing setup to verify export utilities for SQL, MongoDB, ElasticSearch, and SpEL. I’ll use Docker and Docker Compose to run containerized databases locally and in GitHub Actions CI workflows, ensuring queries produce expected results on real sample data.

For SpEL, I propose developing a lightweight Java Spring REST API that evaluates expressions dynamically against JSON-serialized context objects, enabling seamless integration with JS tests.

I’ll create a modular, maintainable test framework that integrates smoothly with your current setup, delivering reliable validation across all supported databases and formats.

[ukrbublik]

@Blue-smoke007 Sounds perfect!

[Blue-smoke007]

SQL tests are fully implemented, MongoDB tests are fully implemented, Elasticsearch tests are fully implemented, SpEL tests are fully implemented, GitHub Actions workflow is properly configured all thats left is the gradle wrapper its kind of confusing me lol😂😂
Gradle wrapper setup is where i stopped what do you think

[ukrbublik]

@Blue-smoke007 Sounds good.
Do you have a PR or branch?
I'm not familiar with Java development but can try to assist with your PR

[Blue-smoke007]

I’ve created a separate branch for the integration test setup work:
👉 feature/integration-tests