Support for nested Attributes

[thomaschn]

As already mentioned here, I noticed that nested attributes cannot be accessed in the lambda function. For example, I want to make a plot from weather forecast data that looks like this:

temperature: 21.7
humidity: 32
pressure: 1019.2
wind_bearing: 131
wind_speed: 9
visibility: 40.2
attribution: Powered by WeatherFlow
friendly_name: Weatherflow Hourly based Forecast
forecast:
  - datetime: '2022-06-02T14:00:00+00:00'
    temperature: 22
    precipitation: 0
    precipitation_probability: 0
    condition: partlycloudy
    wind_speed: 7.2
    wind_gust: 7.2
    wind_bearing: 29
    feels_like: 22
    uv_index: 4
  - datetime: '2022-06-02T15:00:00+00:00'
    temperature: 22
    precipitation: 0
    precipitation_probability: 0
    condition: partlycloudy
    wind_speed: 7.2
    wind_gust: 7.2
    wind_bearing: 39
    feels_like: 22
    uv_index: 3
  - datetime: '2022-06-02T16:00:00+00:00'
    temperature: 22
    precipitation: 0
   ...

Similar to this ApexCharts example, the lambda function should look something like this:

data_generator: |
        return entity.attributes.forecast.map((entry) => {
          return [new Date(entry.datetime).getTime(), entry.temperature];
        });

However, I found no way to make it work because forecast doesn't even show up in the attributes. How can I get this to work?

[dbuezas]

What weather integration is this one?
I'll need to reproduce this first.
Btw, thanks for the kind words in the form! Really happy to hear you like the card! :)

[thomaschn]

I'm using the WeatherFlow integration. You need a developer API token to pull data from any station.
I would guess that the data format is the same for other weather integrations, too. Not sure though.

Sure thing. I really do love myself a good, customizable tool to visualize my data! It's the best I've come across for Home Assistant ;)

[dbuezas]

Have you tried using lambdas? I unfortunately have no sensors with nested attributes and can therefore not test it myself

[thomaschn]

Yes, I have tried it using lambdas. It does not work unfortunately. It seems like only attributes with a single value are available. Maybe this has to do with how entities are stored in your history?

You can test it with the Met.no Integration. It provides a nested forecast attribute. Here is an example config, illustrating that only 11 out of 13 attributes are available:

type: custom:plotly-graph
entities:
  - entity: weather.home
    lambda: |-
      (ys, xs, entity) => {
        var attr = entity[0].attributes;
        var x_val = []
        var y_val = []
        for (const id in attr){
          x_val.push(id)
          y_val.push(Object.keys(attr).length)
        }
        return ({x: x_val, y: y_val})
      }
refresh_interval: 10

[midiwidi]

I'm glad I found this discussion. I'm trying to achieve the same thing. I'm using data from the Australian Bureau of Meteorology (HACS integration). This one doesn't require a API key. The entity has similar attributes

temperature: 13.7
temperature_unit: °C
humidity: 53
pressure_unit: hPa
wind_bearing: WNW
wind_speed: 19
wind_speed_unit: km/h
visibility_unit: km
precipitation_unit: mm
forecast:
  - datetime: '2022-11-01T23:00:00+11:00'
    condition: sunny
    precipitation_probability: 0
    wind_bearing: WNW
    wind_gust_speed: 31
    humidity: 55
    uv: 0
    temperature: 14
    wind_speed: 19
    precipitation: 0
  - datetime: '2022-11-02T00:00:00+11:00'
    condition: sunny
    precipitation_probability: 0
    wind_bearing: WNW
    wind_gust_speed: 30
    humidity: 54
    uv: 0
    temperature: 13
    wind_speed: 19
    precipitation: 0
  - datetime: '2022-11-02T01:00:00+11:00'
    condition: sunny
    precipitation_probability: 0
    . . .

I extracted the temperature and timestamp information from the forecast and created a list variable using this template.

{% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
{% set ns = namespace(temperature = []) %}
{% for hour_value in weather_forecast %}
  {% set ns.temperature = ns.temperature + [[hour_value.datetime, hour_value.temperature | float]] %}
{% endfor %}
{{ ns.temperature }}

The created list looks like this

[
  [
    "2022-11-01T23:00:00+11:00",
    14
  ],
  [
    "2022-11-02T00:00:00+11:00",
    13
  ],
  [
    "2022-11-02T01:00:00+11:00",
    13
  ],
  . . .
]

Is there a way to hand that list over to the plotly graph card or can the card only handle history data of an entity.
If so, do you know a way to manually set history data?

[dbuezas]

You could use a lambda :)
And while you are at it, can use this one, open the dev console and post how the entity looks like?

lambda: |-
    (ys, xs, entity) => 
       console.log(entity);
       return ys

[midiwidi]

It was not completely clear to me what you asked me to do and post. So I hope this is what you meant.

I placed a plotly graph card on one of my dashboards and configured it like so

type: custom:plotly-graph
entities:
  - entity: weather.marsfield_hourly
    attribute: forecast
    unit_of_measurement: °C
    lambda: |-
      (ys, xs, entity) => 
        console.log(entity);
        return ys
hours_to_show: 24
refresh_interval: 10

This generated the following error in the log

Logger: frontend.js.latest.202210100
Source: components/system_log/__init__.py:227
First occurred: 09:05:20 (4 occurrences)
Last logged: 09:11:02

http://192.168.2.6:8123/hacsfiles/lovelace-plotly-graph-card/plotly-graph-card.js?hacstag=413812496153:54:1343 Uncaught TypeError: Cannot read properties of undefined (reading 'hours_to_show')

I get the same error with any working plot as soon as I add the lambda you provided.

[dbuezas]

I mean the browser console:
https://balsamiq.com/support/faqs/browserconsole/

[midiwidi]

Ups, sorry. I got it now.

Here is the output on the log console

Interestingly, the attributes it shows are slightly different to the attributes of the weather entity I fed into the card. There are some additional attributes like icon and friendly_name, but the one we are after (forecast) is missing. I also wonder why I get an array with 3 elements. Maybe historical data?

I tried something else. In the plotly-graph-card configuration I added attribute: forecast and I got this

A much bigger array of 138 elements. But the elements don't have the forecast values (e.g. precipitation_probability) in them. Instead, the same attributes values as in the first try above. I have no idea what the 138 elements are.
I might analyse the timestamps and try to match them.

Maybe we have a chance to create a new sensor with the forecast data arranged differently like 1 attribute for each hour.
How do you think shout it look like for the card to be able to display it?

[dbuezas]

I also wonder why I get an array with 3 elements. Maybe historical data?

Yes, exactly. These are the state an attribute values for each point in time. The more you zoom out, the more get fetched

[dbuezas]

As to why the forecast attribute is no not showing up, I don't know. Maybe there is a different API I could use to get a more complete picture of the entities

[midiwidi]

Ah, ok. Maybe specifying attribute: forecast didn't do anything because the entity doesn't include it (from the JavaScript perspective). Maybe HA strips all attributes which are arrays and only keeps single value ones before it hands it over to the card.
That the number of array elements changed from 3 to 138 might have just been me accidentally zooming out when I changed the card configuration to add attribute: forecast.
I'll play a bit more around with it. I might be able to create a sensor which has all forecast data and timestamps as seperate attributes (or at least the ones for the next 24 hours) and then use a lambda to combine these to xs and ys.
Can xs and ys be arrays representing a whole series or only a single data point?

[midiwidi]

I think I'm almost there.

I created a template sensor which has different forecast values and the timestamps stored as attributes

- platform: template
  sensors:
    weather_forecast:
      friendly_name: "Weather Forecast Marsfield"
      value_template: "no state"
      attribute_templates:
        datetime: >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(t = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.t = ns.t + [hour_value.datetime] %}
            {% endif %}
          {% endfor %}
          {{ ns.t }}
        temperature:  >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(temperature = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.temperature = ns.temperature + [hour_value.temperature | float] %}
            {% endif %}
          {% endfor %}
          {{ ns.temperature }}

I have to figure out if I can optimise that by not having to repeat the code for each attribute (but that's a different story). The sensor entity looks like this

This is the plotly graph card configuration

type: custom:plotly-graph
entities:
  - entity: sensor.weather_forecast
    type: bar
    unit_of_measurement: °C
    lambda: |-
      (ys, xs, entity) => {
        last = entity[entity.length - 1];
        xs = last['attributes']['datetime'].map(x => new Date(x));
        ys = last['attributes']['temperature'];
        console.log(xs)
        console.log(ys)
        return (xs, ys)
        }
hours_to_show: 24
refresh_interval: 10

The browser developer console shows

For comparison I also printed the xs and ys data from a standard temperature sensor to the console. To me, xs and ys of the weather forecast sensor (the one with an array length of 24 ) and of the standard sensor look similar.

The Y values of the plot are correct, but the X values are not. The timestamps in the console log show 1 timestamp each hour. The plot shows different values.
Do I return it correctly in the JavaScript code?

[dbuezas]

Almost there!
It may be that the date from home assistant is in a format incompatible with new Date.
Why not use the attribute yaml key instead?

Anyway there is something fishy in the JavaScript, it should say: return {x: xs, y: ys}

[dbuezas]

Also, why are you computing the xs yourself?

[midiwidi]

Bingo !

Thanks for your hint. It was the return statement. Now it is working as expected.

Before I show my final configuration, let me answer your question why I calculate the x values myself. Hopefully I can explain it well enough :)
If you look at the very first post from @thomaschn or in my first post which shows the original weather data, you notice that all data points of the weather forecast is contained in the forecast attribute of a single weather entity data point.
It is not like usual sensor data which have 1 data point (usually the state) per entity which evolves over time so that the data series is contained in the history of the entity.
When I create my custom template sensor to re-organise the forecast data so that the graph card can access it, I can't generate history data or at least I don't know how. Also, the timestamps are in the future and not in the past. Instead I'm doing the same thing and write all data points into the attributes of my sensor (datetime, temperature, humidity ...). At the moment I haven't turned off recording the history of my sensor. So I get an array (each element containing the whole forecast data series) and I only use the last element. This means that xs only contains 1 timestamp, the time of when the custom sensor state was captured. Therefore I replace it with the timestamps which belong to the forecast series which are stored in the attribute datetime.

OK, here my final configuration:

Template sensor

- platform: template
  sensors:
    weather_forecast:
      friendly_name: "Weather Forecast Marsfield"
      value_template: "no state"
      attribute_templates:
        datetime: >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(t = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.t = ns.t + [hour_value.datetime] %}
            {% endif %}
          {% endfor %}
          {{ ns.t }}
        temperature:  >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(temperature = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.temperature = ns.temperature + [hour_value.temperature | float] %}
            {% endif %}
          {% endfor %}
          {{ ns.temperature }}
        condition: >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(condition = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.condition = ns.condition + [hour_value.condition | string] %}
            {% endif %}
          {% endfor %}
          {{ ns.condition }}          
        rain_probability: >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(rain_probability = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.rain_probability = ns.rain_probability + [hour_value.precipitation_probability | float] %}
            {% endif %}
          {% endfor %}
          {{ ns.rain_probability }}
        rain: >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(rain = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.rain = ns.rain + [hour_value.precipitation | float] %}
            {% endif %}
          {% endfor %}
          {{ ns.rain }}
        humidity:  >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(humidity = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.humidity = ns.humidity + [hour_value.humidity | float] %}
            {% endif %}
          {% endfor %}
          {{ ns.humidity }}
        wind_speed:  >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(wind_speed = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.wind_speed = ns.wind_speed + [hour_value.wind_speed | float] %}
            {% endif %}
          {% endfor %}
          {{ ns.wind_speed }}
        wind_gust_speed:  >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(wind_gust_speed = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.wind_gust_speed = ns.wind_gust_speed + [hour_value.wind_gust_speed | float] %}
            {% endif %}
          {% endfor %}
          {{ ns.wind_gust_speed }}
        wind_bearing:  >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(wind_bearing = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.wind_bearing = ns.wind_bearing + [hour_value.wind_bearing | string] %}
            {% endif %}
          {% endfor %}
          {{ ns.wind_bearing }}
        uv:  >-
          {% set weather_forecast = state_attr('weather.marsfield_hourly','forecast') %}
          {% set ns = namespace(uv = [], count = 0) %}
          {% for hour_value in weather_forecast %}
            {% set ns.count = ns.count + 1 %}
            {% if ns.count <= 24 %}
              {% set ns.uv = ns.uv + [hour_value.uv | float] %}
            {% endif %}
          {% endfor %}
          {{ ns.uv }}

ToDo: Find a way to have only 1 for loop in which all variables needed are created. Then use the namespace variables in the attribute_templates.

Plotly Graph Card (1 example)

Plotly Graph Card Configuration

type: custom:plotly-graph
entities:
  - entity: sensor.weather_forecast
    type: bar
    name: Temperature
    unit_of_measurement: °C
    yaxis: y1
    offsetgroup: 1
    lambda: |-
      (ys, xs, entity) => {
        last = entity[entity.length - 1];
        xs = last['attributes']['datetime'].map(x => new Date(x));
        ys = last['attributes']['temperature'];
        //console.log(xs)
        //console.log(ys)
        return {x: xs, y: ys}
        }
  - entity: sensor.weather_forecast
    type: bar
    name: Humidity
    unit_of_measurement: '%'
    yaxis: y2
    offsetgroup: 2
    lambda: |-
      (ys, xs, entity) => {
        last = entity[entity.length - 1];
        xs = last['attributes']['datetime'].map(x => new Date(x));
        ys = last['attributes']['humidity'];
        //console.log(xs)
        //console.log(ys)
        return {x: xs, y: ys}
        }
layout:
  barmode: group
  yaxis:
    range:
      - 10
      - 24
hours_to_show: 6
refresh_interval: 10

ToDo: Come up with graph card layouts and settings for the different values. For example how to dynamically set the lower limit of the y axis for the temperature, because I don't want that to be 0 which is the default.

[dbuezas]

Really good!
btw, you can specify the yaxis range like this:

layout:
  yaxis:
    range: [10, 30] # https://plotly.com/javascript/reference/layout/yaxis/#layout-yaxis-range
    # fixedrange: true # https://plotly.com/javascript/reference/layout/yaxis/#layout-yaxis-fixedrange

fixedrange will make zooming and scrolling only apply to the xaxis, which is very convenient to navigate.

I think you will also like this feature when it's done: #33
It will allow you to tell plotly to show you some hours into the future by default.

And fiinally a question: since you have older forecasts, what about adding traces for those too? then you can see how the forecast gets corrected over time (and I'm very curious to see some screenshots of that myself)

[dbuezas]

ToDo: Find a way to have only 1 for loop in which all variables needed are created. Then use the namespace variables in the attribute_templates.

- platform: template
  sensors:
    weather_forecast:
      friendly_name: "Weather Forecast Marsfield"
      value_template: "no state"
      attribute_templates:
        {%- set weather_forecast = state_attr('weather.openweathermap','forecast') %}
        {%- for attribute in weather_forecast[0] %}
        {{ attribute }}: >-
          {%- set ns = namespace(t = [], count = 0) %}
          {%- for hour_value in weather_forecast %}
            {%- set ns.count = ns.count + 1 %}
            {%- if ns.count <= 24 %}
              {%- set ns.t = ns.t + [hour_value[attribute]] %}
            {%- endif %}
          {%- endfor %}
          {{ ns.t }}
        {%- endfor %}

I'm, not sure if this would work, but I couldn't ignore the challenge :)

[midiwidi]

Regarding the template sensor:

The output looks good in the template sandbox in the developer tools, but HA doesn't like it in the configuration.

    testsensor:
      friendly_name: "Weather Forecast Marsfield"
      value_template: "no state"
      attribute_templates:
        {%- set weather_forecast = state_attr('weather.marsfield','forecast') %}
        {%- for attribute in weather_forecast[0] %}
        {{ attribute }}: >-
          {%- set ns = namespace(t = [], count = 0) %}
          {%- for hour_value in weather_forecast %}
            {%- set ns.count = ns.count + 1 %}
            {%- if ns.count <= 24 %}
              {%- set ns.t = ns.t + [hour_value[attribute]] %}
            {%- endif %}
          {%- endfor %}
          {{ ns.t }}
        {%- endfor %}

results in

Error loading /config/configuration.yaml: while scanning for the next token
found character '%' that cannot start any token
  in "/config/sensor.yaml", line 188, column 10

Line 188 is this one {%- set weather_forecast = state_attr('weather.marsfield','forecast') %}

[dbuezas]

HA doesn't like it in the configuration.

Oh, what a pitty.

[dbuezas]

I found out yesterday that that the full set of attributes of the last entry is available internally to the frontend at runtime. I may add an extra modifier in the future for this

[midiwidi]

For the weather forecast I would like to use bar plots, because the data points have a fix interval (every hour) and the bars make it easy to distinguish between them.
Unfortunately the bars always start at y=0 and small temperature changes can not easily be seen. Is there a way to configure the y-axis autoscale to ignore the 0 and scale to the actual minimum of the data?
Alternatively I could use the range setting for the y axis, but it looks like I can only use fix values.
I tried to use a template

layout:
  yaxis:
    range: {{ [10,30] }}

but it doesn't work.
Is there a way of specifying the frame dynamically? Maybe with a lambda function?

[dbuezas]

Please send screenshots and open a new issue or discussion :)

[midiwidi]

Upps, sorry. I created a new discussion here.

[dbuezas]

Good news!
v1.7.0 now updates the sensors directly from the frontend and uses the latest active entity state. This means the entity should now have ALL the attributes, and you should be able to achieve what you did without the need of templates.

Almost... it may only start working after 1 automatic update.
If you give it a try, I could force to forcibly replace the latest datapoint with the front end data.

[dbuezas]

Oh, I think one of the optimisations broke your plot. In my defence, it shouldn't have worked before either 😁
You have 3 options:

• Set refresh_interval: auto: this will get the latest state from directly from the front end, which contains all attributes.
• Add attribute: xxx to the trace (inside entity:): this will force the card to fetch attributes. (this is avoided by default to reduce the data transferred)
• Set minimal_response to false

Sorry for breaking your plot!
Let me know if this doesn't fix it :)

[dbuezas]

Just found out I also made a mistake and added a bug. I'll fix and release a patch in 2 hours.
I'm adding a test dashboard so this shouldn't happen again. Sorry for that!

[midiwidi]

No worries at all, those things happen. It's good that you make changes to improve functionality rather than follow the saying "never touch a running system".
I'm a bit restricted with my time right now, but will follow your advice and get my plots back working soon.

[midiwidi]

Option 1 (refresh_interval: auto) was working perfectly fine, but after updating from v1.7.5 to v1.8.2 it doesn't work any more. There is no attributes returned.

Option 3 (minimal_response: false) doesn't work either in v1.8.5.. There are not attributes returned within the entity. I didn't test it in v1.7.5.

Option 2 is working in v1.8.2. As soon as 1 attribute is requested (inside entity:), all attributes are delivered.
But depending on the zoom level, a (big) array of entities is returned (historical data) with all of them containing all attributes. I ignore all and only use the last one, but that is unnecessary data traffic. I have tried to use it in combination with refresh_interval: auto to get it to only return the last entity (including the attributes which contain my traces), but that didn't work. I still got an array of entities. Is there any YAML setting to achieve that? If not, then I can probably just disable recording the history for my sensor.

[dbuezas]

Ok, thanks for the report.
I'll duplicate your plot during the weekend and iron this out.

My current guess is that:

• Option 1 (refresh_interval: auto): stopped storing the attributes because the last element is already fetched via the rest api and that now may take precedence over the current state. To confirm you could leave the plot open and when a new datapoint appears, it should have all attributes. If true, I'll make sure the current state gets a "higher priority"
• Option 3 (minimal_response: false) doesn't work: This should actually work, I'll need to debug
• Is there any YAML setting to achieve that (fetch only 1 entry, irrespective of zoom): not at the moment, but there will be once I implement Make lambdas able to change other trace attributes #130. It will take a while though.

[dbuezas]

@midiwidi
I think you will appreciate this example: #168 (comment)

:)