Skip to content

1. Structure of the REopt API

Jump to bottom
adfarth edited this page Aug 8, 2023 · 30 revisions

Structure Overview

The REopt® API is built primarily in Python 3.6, with the exception of the code used to call the optimization kernel, which is written in Julia. In the REopt_API version 3 and higher, the Julia JuMP model (and all of the supporting functions) are now housed in the publicly-registered Julia package, REopt.jl. The API is served using the Django 2.2 framework, with a PostgreSQL database backend. Task management is achieved with Celery 4.3, which uses Redis as a message broker. The figure below shows a how each of these pieces interact.

In brief,

  • the Django server provides the API itself,
  • the Celery server manages the tasks (functions within the API),
  • the Redis server is a database for storing and retrieving Celery tasks (aka a message broker),
  • REopt.jl is the Julia package that contains the REopt optimization model,
  • and the PostgreSQL server is a database for all of the data models in the API (stores inputs, outputs, and errors).

API Versioning

We typically update the API version when we make major changes to default values or other breaking changes. Please see the documentation at [https://developer.nrel.gov/docs/energy-optimization/reopt/] for REopt API versions and endpoints. In general, this documentation reflects the API v3, which uses the REopt Julia Package as the optimization back-end and is concentrated mainly in the reoptjl endpoint of the REopt API.

Workflow for REopt Jobs

v1-v2 Workflow for REopt jobs

The REopt API has many endpoints. The endpoint for optimizing an energy system is described here.

The steps behind an optimization are:

  1. A POST is made at <host-url>/v1/job (for a description of all input parameters GET <host-url>/v1/help)
  2. reo/api.py validates the POST, sets up the four celery tasks, and returns a run_uuid
  3. reo/scenario.py sets up all the inputs for the two optimization tasks
  4. Two reo/src/run_jump_model.py tasks are run in parallel, one for the business-as-usual costs, and one for the minimum life cycle cost system
  5. reo/process_results.py processes the outputs from the two REopt tasks and saves them to database.

v3 Structural changes for handling optimization jobs

From the user's perspective the transition from to the new job endpoint is manifested in the nested inputs and outputs getting "flattened", as well as some name clarifications and reorganizing of a few inputs (see the analysis wiki for more).

For developers of the API there are more significant changes that will hopefully make development easier.

Django Models and Input Validation

In v1 we stored all of the inputs and outputs for any high level model, such as the ElectricTariff, in one table (which translates to one Django Model). In the new job endpoint the inputs and outputs are split into two separate models, eg. ElectricTariffInputs and ElectricTariffOutputs. This division greatly simplifies creating the API response as well as saving and updating the data models.

In v1 we filled in rows of every Django Model, for every possible input and output, for every job. In the new job endpoint only the tables for objects that are under consideration have rows inserted. For example, if a scenario is not modeling Wind then no WindInputs nor WindOutputs are instantiated. This also implies that the results endpoint does not return any information related to Wind in this example (whereas in v1 we return a "Wind" key with all defaults in the "inputs" and also a "Wind" key in the "outputs" with many null's).

InputValidator.clean_fields()

In v1 of the API the nested_inputs_definitions is used as a source of data types, limits, defaults, and descriptions, which is used in ValidateNestedInput to validate user's POSTs. In the new job endpoint we take advantage of Django's built-in data validation to accomplish most of what ValidateNestedInput does. Using Django's built in methods we can:

  1. validate field types
  2. validate field min/max values
  3. limit fields to certain values
  4. provide help_text for each field
  5. set defaults

For example, the time_steps_per_hour field in the new job endpoint looks like:

time_steps_per_hour = models.IntegerField(
    default=TIME_STEP_CHOICES.ONE,
    choices=TIME_STEP_CHOICES.choices,
    help_text="The number of time steps per hour in the REopt model."
)

where TIME_STEP_CHOICES is defined as:

class TIME_STEP_CHOICES(models.IntegerChoices):
    ONE = 1
    TWO = 2
    FOUR = 4

And for an example of a min/max validation:

timeout_seconds = models.IntegerField(
    default=420,
    validators=[
        MinValueValidator(1),
        MaxValueValidator(420)
    ],
    help_text="The number of seconds allowed before the optimization times out."
)

After instantiating a Django model with a user's input values in the new job endpoint we call the Model.clean_fields() method on all of the input data models. The Model.clean_fields() method checks data types, min/max values, choices, and sets defaults. Calling the Model.clean_fields method is done in the new InputValidator (in job/validators.py) for every input model.

InputValidator.clean()

The Django Model also has a clean method that by default is empty. In the new job endpoint we "over-ride" this method to perform custom validation as needed. For example, the ElecricLoadInputs class has a clean method defined as follows:

    def clean(self):
        error_messages = []

        # possible sets for defining load profile
        if not at_least_one_set(self.dict, self.possible_sets):
            error_messages.append((
                "Must provide at valid at least one set of valid inputs from {}.".format(self.possible_sets)
            ))

        if error_messages:
            raise ValidationError(' & '.join(error_messages))

where:

possible_sets = [
    ["loads_kw"],
    ["doe_reference_name", "monthly_totals_kwh"],
    ["annual_kwh", "doe_reference_name"],
    ["doe_reference_name"]
]

The InputValidator calls the Model.clean() method on all of the input models (after calling the Model.clean_fields() method).

InputValidator.cross_clean()

Finally, there are some cases for input validation that require comparing fields from two different Django Models. For example, the length of any time-series input value, such as ElectricLoadInputs.loads_kw, needs to align with the Settings.time_steps_per_hour. This type of cross-model validation is done after the clean_fields and clean methods, and is called cross_clean. The cross_clean method is part of the InputValidator and is called in job/api.py as the final step for validating the job inputs.

Celery Tasks

In v1 each optimization job consists of four celery tasks (setup_scenario, one optimal and one baurun_jump_model, and process_results). In the new job endpoint there is only one celery task per optimization job. This has been accomplished by running the BAU and optimal scenarios in parallel in the Julia code and combining the scenario set up and post process steps into a single celery task.

Julia code

In v1 of the API the julia_src/ directory includes the code to build and optimize the JuMP model. In the new job endpoint all of the Julia code is housed in a Julia Module that is publicly registered. This means that the JuMP model (and all of the supporting functions) are now in a separate repository and that the REopt model can be used in Julia with just a few lines of code. For example:

using REoptLite, Cbc, JuMP
model = JuMP.Model(Cbc.Optimizer)
results = REoptLite.run_reopt(m, "path/to/scenario.json")

In the new job endpoint of the API we use the Julia package for REoptLite in a similar fashion to the last example. (See julia_src/http.jl for more details). Note that in the new job endpoint the BAU and Optimal scenarios are run in parallel in Julia, and running the BAU scenario is optional (via Settings.run_bau).

Migration of Python code to Julia

In creating the Julia Module (or "package") for REoptLite much of the inputs set up code and post-processing has shifted from the API to the Julia package. By porting the setup and post-processing code to the Julia package anyone can now use REoptLite in Julia alone, which makes it much easier to add inputs/outputs, create new constraints, and in general modify REoptLite as one sees fit.

For past and recent developers of the REopt Lite API, here is a pseudo-map of where some of API code responsibilities has been moved to in the Julia package:

  • reo/src/techs.py -> broken out into individual files for each tech in the src/core/ directory, e.g. pv.jl, wind.jl, generator.jl
  • reo/scenario.py -> src/core/scenario.jl
  • reo/src/pvwatts.py + wind_resource.py + wind.py + sscapi.py -> src/core/prodfactor.jl
  • reo/src/data_manager.py + julia_src/utils.jl -> src/core/reopt_inputs.jl
  • reo/process_results.py -> src/results/ contains a main results.jl as well as results generating functions for each data structure (e.g. src/results/pv.jl)
  • julia_src/reopt_model.jl -> src/core/reopt.jl + src/constraints/*

Also, here is a pseudo-map of where some of the reo/ code is now handled in the job/ app:

  • reo/nested_inputs.py + nested_outputs.py + models.py -> job/models.py
  • reo/process_results.py -> job/src/process_results.py (only saving results to database now since results are created in Julia)
  • reo/api.py -> job/api.py
  • reo/views.py -> job/views.py
  • reo/src/run_jump_model.py -> job/src/run_jump_model.py

For more information on the Julia package please see the developer section in the Julia package documentation.

Modifications of mathematical model

The primary change to the mathematical model in the new job endpoint is that all binary variables (and constraints) are added conditionally. For example, when modeling PV, Wind, and/or Storage without a tiered ElectricTariff no binary variables are necessary. However, adding a Generator or a tiered ElectricTariff does require binaries.

Also related to binary variables, the approach to modeling the net metering vs. wholesale export decision has been simplified:

In v1 the approach to the NEM/WHL binaries is as follows. First, binNMIL is used to choose between a combined tech capacity that is either above or below net_metering_limit_kw. There are two copies of each tech model made for both the above or below net_metering_limit_kw. If the combined tech capacity is below net_metering_limit_kw then the techs can export into the NEM bin (i.e. take advantage of net metering). If the combined tech capacity is greater than net_metering_limit_kw then the techs can export into the WHL bin. In either case the techs can export into the EXC bin. Second, this approach also requires a tech-to-techclass map and binaries for constraining one tech from each tech class - where the tech class contains the tech that can net meter and the one that can wholesale.

In the new approach of the new job endpoint there there is no need for the tech-to-techclass map and associated binaries, as well as the duplicate tech models for above and below net_metering_limit_kw. Instead, indicator constraints are used (as needed) for binNEM and binWHL variables, whose sum is constrained to 1 (i.e. the model can choose only NEM or WHL, not both). binNEM is used in two pairs of indicator constraints: one for the net_metering_limit_kw vs. interconnection_limit_kw choice and another set for the NEM benefit vs. zero NEM benefit. The binWHL is also used in one set of indicator constraints for the WHL benefit vs. zero WHL benefit. The EXC bin is only available if NEM is chosen.

See https://github.com/NREL/REoptLite/blob/d083354de1ff5d572a8165343bfea11054e556fc/src/constraints/electric_utility_constraints.jl for details.

Clone this wiki locally