Local runtime (Java) memory leak with Inspector opened

[No3371]

Outdated description. See comments below first.

Description

I am working on managed AI worker which spawns 715 entities, modify the position components and let SpatialOS update the data to runtime, every frame.

If I disable the worker (by disabling the worker connector before building it), and run the worker in Unity Editor, the memory usage stay stable at about 4GB memory working fine.

If I enable the worker connector and let it managed by the local runtime, the Java application consumes all my memory.

I've isolated the issue by testing the manged worker with/without the system that assign new value to Position.

Expected behaviour

The worker works normally whether it's managed/external.
(I expected workers running inside Unity should have more overhead)

Environment

i7-9700 / 32GB
Windows 10 1909 / Unity 2019.3.7f1
OpenJDK Runtime Environment (build 14.0.1+7)
OpenJDK 64-Bit Server VM (build 14.0.1+7, mixed mode, sharing)

[No3371]

I am working on managed AI worker which spawns 715 entities, modify the position components and let SpatialOS update the data to runtime, every frame.

So it turns out it's because a managed worker runs way faster then one running inside UnityEditor, so it generate enormous traffic toward the runtime.

After limit targetFrameRate to 60, the memory usage stays at about 4GB.

[No3371]

Unfortunately I have to reopen this since the memory usage will randomly grows, like, after staying at 4GB for 5 minutes, it suddenly goes up to 13GB, then after 2 minutes, it goes up to 15GB.

Continuing to do these tests, will let you know the results.

• [DONE] External worker in UnityEditor without opening inspector (Issue not reproduced in 25 mins)
• [DONE] Managed worker without opening inspector (Issue not reproduced in 30 mins)
• [PEDNING] External worker in UnityEditor opening inspector
• [DONE] Managed worker opening inspector (Issue reproduced in 8 mins, memory usage grown to 9GB)

The code of the system setting Position for reproducing purpose:

using Improbable;
using Improbable.Gdk.Core;
using Unity.Burst;
using Unity.Collections;
using Unity.Entities;
using Unity.Transforms;

[UpdateInGroup(typeof(SpatialOSUpdateGroup))]
public class SpatialPositionFromTranslationSystem : SystemBase
{
    EntityQuery query;
    protected override void OnCreate()
    {
        query = GetEntityQuery(ComponentType.ReadOnly<Translation>(), ComponentType.ReadWrite<Position.Component>(), ComponentType.Exclude<NewlyAddedSpatialOSEntity>());
        query.SetChangedVersionFilter(typeof(Translation));
    }
    protected override void OnUpdate()
    {
        SyncJob job = new SyncJob
        {
            Translation_Type = GetArchetypeChunkComponentType<Translation>(),
            Position_Type = GetArchetypeChunkComponentType<Position.Component>()
        };
        job.ScheduleParallel(query).Complete();
    }

    [BurstCompile]
    protected struct SyncJob : IJobChunk
    {
        [ReadOnly] public ArchetypeChunkComponentType<Translation> Translation_Type;
        public ArchetypeChunkComponentType<Position.Component> Position_Type;
        public void Execute(ArchetypeChunk chunk, int chunkIndex, int firstEntityIndex)
        {
            var translations = chunk.GetNativeArray<Translation>(Translation_Type);
            var positions = chunk.GetNativeArray<Position.Component>(Position_Type);
            var cache = new Position.Component();
            var coordCache = new Coordinates();
            for (int i = 0; i < translations.Length; i++)
            {
                coordCache.CopyFromUnityFloat3(translations[i].Value);
                cache.Coords = coordCache;
                positions[i] = cache;
            }
        }
    }
}

public static class SpatialCoordsExtentsion
{
    public static float3 ToUnityFloat3 (this global::Improbable.Coordinates source)
    {
        return new float3((float) source.X, (float) source.Y, (float) source.Z);
    }
    public static global::Improbable.Coordinates FromUnityFloat3 (float3 source)
    {
        return new global::Improbable.Coordinates(source.x, source.y, source.z);
    }
    public static void CopyFromUnityFloat3 (this ref global::Improbable.Coordinates dest, float3 source)
    {
        dest.X = source.x;
        dest.Y = source.y;
        dest.Z = source.z;
    }
}

[No3371]

[DONE] Managed worker opening inspector (Issue reproduce in 8 mins, memory usage grows to 9GB)

I've reproduced the random memory leak, it happens when Inspector opened.

Another minor issue along with this:
Under heavy load Entity count in Inspector is being decreased overtime. The entity count decreased to 6XX, but shows 715 if I opens with Chrome at the time, then start decreasing as well after minutes.

Not sure if these only happens when CPU is 100% loaded. (When inspector opens running these tests, CPU are at 100% all the time)

[zeroZshadow]

Does this happen with just the Inspector, or also Inspector V2?
When you mention CPU and memory usage, is this for the worker or the runtime process?

Note that sening an update on Position is special and rather heavy, as this will cause the runtime to re-evaluate all subscriptions made through QBI queries.

[No3371]

Does this happen with just the Inspector, or also Inspector V2?

I did not test with InspectorV2, I've tried it multiple times though, but it's not as intuitive as V1 😅

When you mention CPU and memory usage, is this for the worker or the runtime process?

It's the java process. It always occupy a lot of CPU usage, I am thinking it may be related to what you said below.

Note that sening an update on Position is special and rather heavy, as this will cause the runtime to re-evaluate all subscriptions made through QBI queries.

Thank you, that's very valuable info!
Does this means I can give up Immediate updating of Inspector positions and use custom position component for performance?

[zeroZshadow]

Yes it is recommended to only update Position at a low rate, and use a custom component for the actual position of the entity.
Position is used for things like queries, the inspector, and for the initial prefab spawn if you use the default gameobject creator.

[No3371]

OK, thanks for the heads up.

I am in the progress to finish up a demo project and moving to cloud, so may not be able to reproduce the issue in near future, as we yet have a conclusion, I'll let this remain opened, will definitely report back if it happens again.