Implementing Nerdbank.MessagePack for serializing and deserializing messages

[AlleSchonWeg]

Is your feature request related to a problem? Please describe.
Nerdbank.MessagePack library is the successor to MessagePack-CSharp. I created a custom marshaller to implement Nerdbank.MessagePack into ServiceModel.Grpc:

public sealed class NerdbankMessagePackMarshallerFactory : IMarshallerFactory
{
    private Nerdbank.MessagePack.MessagePackSerializer _serializer;
    public NerdbankMessagePackMarshallerFactory()
    {
        _serializer = new MessagePackSerializer { PerfOverSchemaStability = true };
    }

    public Marshaller<T> CreateMarshaller<T>()
    {
        return new Marshaller<T>(Serialize, Deserialize<T>);
    }

    private void Serialize<T>(T value, SerializationContext context)
    {

        var shape = ReflectionTypeShapeProvider.Default.GetShape<T>();
        _serializer.Serialize<T>(context.GetBufferWriter(), value, shape);
        context.Complete();
    }

    private T Deserialize<T>(DeserializationContext context)
    {
        var shape = ReflectionTypeShapeProvider.Default.GetShape<T>();
        return _serializer.Deserialize<T>(context.PayloadAsReadOnlySequence(), shape);
    }
}

Nerdbank.MessagePack uses PolyType as a source generator. The models must annotated with the GenerateShape attribute or by adding a witness type for a type that isn't declared in the project.
My problem is that ServiceModel.Grpc also uses a source generator for the messages. The type of the value parameter in the Serialize method are like this:

ServiceModel.Grpc.Channel.Message<System.DateTime>
ServiceModel.Grpc.Channel.Message<Contract.MyModel>
Client.MyGrpcClients.Message<string, int, int, string, string, int, bool, int, bool, int, int>

depending on the method signatures in the service contract interface:

[ServiceContract]
public interface IDemo
{
      [OperationContract]
      string TestDateTime(DateTime dt);
      [OperationContract]
      void TestObjectProperty(MyModel myModel);
      [OperationContract]
      void MultipleParameters(string p1, int p2, int p3, string p4, string p5, int p6, bool p7, int p8, bool p9, int p10, int p11);
}

Client:
//Client:
// configure ServiceModel.Grpc.DesignTime to generate a source code for IDemo client proxy
[ImportGrpcService(typeof(IDemo))]
internal static partial class MyGrpcClients
{}

In the NerdbankMessagePackMarshallerFactory above i use the ReflectionTypeShapeProvider.Default, because the models are wrapped as generic types in source generated ServiceModel.Grpc.Channel.Message<T1, T2, ..> or Client.MyGrpcClients.Message<T1, T2, ..>.

Is there a way make this work without reflection? Using a witness is also not possible.

[GenerateShapeFor<Client.MyGrpcClients.Message<string, int, int, string, string, int, bool, int, bool, int, int>>] //Compiler error
partial class Witness;

CC: @AArnott

[AArnott]

Nerdbank.MessagePack uses PolyType as a source generator. The models must annotated with the GenerateShape attribute or by adding a witness type for a type that isn't declared in the project.

This is only true when using the source generator. If you're using the ReflectionTypeShapeProvider you don't need the GenerateShape attribute. Just FYI.

Just yesterday I documented a few approaches for using Nerdbank.MessagePack (or PolyType in general) with data models that are source generated:
https://aarnott.github.io/Nerdbank.MessagePack/docs/type-shapes.html?tabs=net#source-generated-data-models

The most straightforward one is to produce your data model in one project, and write your Grpc marshaler in another project that references your data model project. In the serializing project you'll use a Witness type or some envelope type with [GenerateShape] on it.

[AlleSchonWeg]

Hi @AArnott ,

The most straightforward one is to produce your data model in one project, and write your Grpc marshaler in another project that references your data model project. In the serializing project you'll use a Witness type or some envelope type with [GenerateShape] on it.

I tried it and it is working with the Witness type in the data model project. But this approach needs a lot of manual work. The data model code is produced for client and server.
Client:

[ImportGrpcService(typeof(IDemo))]
public static partial class MyGrpcClients;

Server:

[ExportGrpcService(typeof(Demo), GenerateSelfHostExtensions = true)]
public static partial class MyGrpcServices;

As a result we need a grpc marshaller for client and server, because the client needs a Witness with the shapes for the client

[GenerateShapeFor<MyGrpcClients.Message<string, int, int, string, string, int, bool, int, bool, int, int>>]
public partial class Witness;

and the server a Witness for the server

[GenerateShapeFor<MyGrpcServices.Message<string, int, int, string, string, int, bool, int, bool, int, int>>]
public partial class Witness;

The types MyGrpcClients.Message and MyGrpcServices.Message are generated via source generators with their generic implementations:

 public sealed partial class Message<T1>
 public sealed partial class Message<T1, T2>
 ...
 public sealed partial class Message<T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11>

Then i need to set GenerateShapeFor for every Message (Client and Server). Example for server:

[GenerateShapeFor<MyGrpcServices.Message<string>>]
[GenerateShapeFor<MyGrpcServices.Message<int>>]
[GenerateShapeFor<MyGrpcServices.Message<int, string>>]
[GenerateShapeFor<MyGrpcServices.Message<string, int, int, string, string, int, bool, int, bool, int, int>>]

This becomes unusable, if you have lot of method signatures.
Perhaps i understand something wrong and there is a easier a approach.

[AArnott]

Wow. That does seem tedious.

If there were a (generated?) type that had these many Message generic types as fields/properties, you could just add that other type to your witness class and that would propagate to the rest. Is that a possibility?

Does this other source generator emit interfaces in which these Message generic types appear as parameters and return types? If so, you're in luck, because the latest PolyType adds better RPC support which includes method shapes. And while you may not need the method shapes, it means you can automatically get type shapes for all types referenced by methods on an interface.

[AArnott]

CC: @eiriktsarpalis

[AlleSchonWeg]

This is an example of the generated class MyGrpcServices.Messages.g.cs:

static partial class MyGrpcServices
{
    [global::System.CodeDom.Compiler.GeneratedCode("ServiceModel.Grpc", "1.16.0")]
    [global::System.Runtime.CompilerServices.CompilerGenerated]
    [global::System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
    [global::System.Reflection.Obfuscation(Exclude = true)]
    [global::System.Serializable]
    [global::System.Runtime.Serialization.DataContract(Name = "m", Namespace = "s")]
    public sealed partial class Message<T1, T2, T3, T4, T5, T6>
    {
        public Message() { }
        
        public Message(T1 value1, T2 value2, T3 value3, T4 value4, T5 value5, T6 value6)
        {
            Value1 = value1;
            Value2 = value2;
            Value3 = value3;
            Value4 = value4;
            Value5 = value5;
            Value6 = value6;
        }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v1", Order = 1)]
        public T1 Value1 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v2", Order = 2)]
        public T2 Value2 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v3", Order = 3)]
        public T3 Value3 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v4", Order = 4)]
        public T4 Value4 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v5", Order = 5)]
        public T5 Value5 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v6", Order = 6)]
        public T6 Value6 { get; set; }
    }
    
    public sealed class MessageAccessor<T1, T2, T3, T4, T5, T6> : global::ServiceModel.Grpc.Internal.IMessageAccessor
    {
        public MessageAccessor(string[] names)
        {
            if (names == null) throw new ArgumentNullException("names");
            if (names.Length != 6) throw new ArgumentOutOfRangeException("names");
            Names = names;
        }
        
        public string[] Names { get; }
        
        public object CreateNew() => new Message<T1, T2, T3, T4, T5, T6>();
        
        public Type GetInstanceType() => typeof(Message<T1, T2, T3, T4, T5, T6>);
        
        public Type GetValueType(int property)
        {
            switch (property)
            {
                case 0:
                    return typeof(T1);
                case 1:
                    return typeof(T2);
                case 2:
                    return typeof(T3);
                case 3:
                    return typeof(T4);
                case 4:
                    return typeof(T5);
                case 5:
                    return typeof(T6);
            }
            
            throw new ArgumentOutOfRangeException("property");
        }
        
        public object GetValue(object message, int property)
        {
            if (message == null) throw new ArgumentNullException("message");
            var instance = (Message<T1, T2, T3, T4, T5, T6>)message;
            switch (property)
            {
                case 0:
                    return instance.Value1;
                case 1:
                    return instance.Value2;
                case 2:
                    return instance.Value3;
                case 3:
                    return instance.Value4;
                case 4:
                    return instance.Value5;
                case 5:
                    return instance.Value6;
            }
            
            throw new ArgumentOutOfRangeException("property");
        }
        
        public void SetValue(object message, int property, object value)
        {
            if (message == null) throw new ArgumentNullException("message");
            var instance = (Message<T1, T2, T3, T4, T5, T6>)message;
            switch (property)
            {
                case 0:
                    instance.Value1 = (T1)value;
                    return;
                case 1:
                    instance.Value2 = (T2)value;
                    return;
                case 2:
                    instance.Value3 = (T3)value;
                    return;
                case 3:
                    instance.Value4 = (T4)value;
                    return;
                case 4:
                    instance.Value5 = (T5)value;
                    return;
                case 5:
                    instance.Value6 = (T6)value;
                    return;
            }
            
            throw new ArgumentOutOfRangeException("property");
        }
    }
    
    [global::System.CodeDom.Compiler.GeneratedCode("ServiceModel.Grpc", "1.16.0")]
    [global::System.Runtime.CompilerServices.CompilerGenerated]
    [global::System.Diagnostics.CodeAnalysis.ExcludeFromCodeCoverage]
    [global::System.Reflection.Obfuscation(Exclude = true)]
    [global::System.Serializable]
    [global::System.Runtime.Serialization.DataContract(Name = "m", Namespace = "s")]
    public sealed partial class Message<T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11>
    {
        public Message() { }
        
        public Message(T1 value1, T2 value2, T3 value3, T4 value4, T5 value5, T6 value6, T7 value7, T8 value8, T9 value9, T10 value10, T11 value11)
        {
            Value1 = value1;
            Value2 = value2;
            Value3 = value3;
            Value4 = value4;
            Value5 = value5;
            Value6 = value6;
            Value7 = value7;
            Value8 = value8;
            Value9 = value9;
            Value10 = value10;
            Value11 = value11;
        }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v1", Order = 1)]
        public T1 Value1 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v2", Order = 2)]
        public T2 Value2 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v3", Order = 3)]
        public T3 Value3 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v4", Order = 4)]
        public T4 Value4 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v5", Order = 5)]
        public T5 Value5 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v6", Order = 6)]
        public T6 Value6 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v7", Order = 7)]
        public T7 Value7 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v8", Order = 8)]
        public T8 Value8 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v9", Order = 9)]
        public T9 Value9 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v10", Order = 10)]
        public T10 Value10 { get; set; }
        
        [global::System.Runtime.Serialization.DataMember(Name = "v11", Order = 11)]
        public T11 Value11 { get; set; }
    }
    
    public sealed class MessageAccessor<T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11> : global::ServiceModel.Grpc.Internal.IMessageAccessor
    {
        public MessageAccessor(string[] names)
        {
            if (names == null) throw new ArgumentNullException("names");
            if (names.Length != 11) throw new ArgumentOutOfRangeException("names");
            Names = names;
        }
        
        public string[] Names { get; }
        
        public object CreateNew() => new Message<T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11>();
        
        public Type GetInstanceType() => typeof(Message<T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11>);
        
        public Type GetValueType(int property)
        {
            switch (property)
            {
                case 0:
                    return typeof(T1);
                case 1:
                    return typeof(T2);
                case 2:
                    return typeof(T3);
                case 3:
                    return typeof(T4);
                case 4:
                    return typeof(T5);
                case 5:
                    return typeof(T6);
                case 6:
                    return typeof(T7);
                case 7:
                    return typeof(T8);
                case 8:
                    return typeof(T9);
                case 9:
                    return typeof(T10);
                case 10:
                    return typeof(T11);
            }
            
            throw new ArgumentOutOfRangeException("property");
        }
        
        public object GetValue(object message, int property)
        {
            if (message == null) throw new ArgumentNullException("message");
            var instance = (Message<T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11>)message;
            switch (property)
            {
                case 0:
                    return instance.Value1;
                case 1:
                    return instance.Value2;
                case 2:
                    return instance.Value3;
                case 3:
                    return instance.Value4;
                case 4:
                    return instance.Value5;
                case 5:
                    return instance.Value6;
                case 6:
                    return instance.Value7;
                case 7:
                    return instance.Value8;
                case 8:
                    return instance.Value9;
                case 9:
                    return instance.Value10;
                case 10:
                    return instance.Value11;
            }
            
            throw new ArgumentOutOfRangeException("property");
        }
        
        public void SetValue(object message, int property, object value)
        {
            if (message == null) throw new ArgumentNullException("message");
            var instance = (Message<T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11>)message;
            switch (property)
            {
                case 0:
                    instance.Value1 = (T1)value;
                    return;
                case 1:
                    instance.Value2 = (T2)value;
                    return;
                case 2:
                    instance.Value3 = (T3)value;
                    return;
                case 3:
                    instance.Value4 = (T4)value;
                    return;
                case 4:
                    instance.Value5 = (T5)value;
                    return;
                case 5:
                    instance.Value6 = (T6)value;
                    return;
                case 6:
                    instance.Value7 = (T7)value;
                    return;
                case 7:
                    instance.Value8 = (T8)value;
                    return;
                case 8:
                    instance.Value9 = (T9)value;
                    return;
                case 9:
                    instance.Value10 = (T10)value;
                    return;
                case 10:
                    instance.Value11 = (T11)value;
                    return;
            }
            
            throw new ArgumentOutOfRangeException("property");
        }
    }
}

From this source generator attribut:

[ExportGrpcService(typeof(Demo), GenerateSelfHostExtensions = true)]
public static partial class MyGrpcServices;

Perhaps i can do something like this:

  [ExportGrpcService(typeof(Demo), GenerateSelfHostExtensions = true)]
  public static partial class MyGrpcServices
  {
      [GenerateShape]
      public sealed partial class Message<T1>
      {
      }
      [GenerateShape]
      public sealed partial class Message<T1, T2>
      {
      }
      [GenerateShape]
      public sealed partial class Message<T1, T2, T3, T4, T5, T6>
      {
      }
     [GenerateShape]
     public sealed partial class Message<T1, T2, T3, T4, T5, T6, T7, T8, T9, T10, T11>
     {
     }
  }

I don't know if this could work. Not tested.

[AArnott]

That wouldn't work because you cannot generate shapes for open generic types. Only closed generics.
I'll study the rest of the generated code sample you shared later today to see if I can make any further recommendations.

[AArnott]

Looking more at the sample generated code, I'm left with lots of questions. Why all this trouble to emit what seems to amount to a ValueTuple? What is the API like to actually use these types?

[AlleSchonWeg]

I don't know. Perhaps @max-ieremenko can help us.

[AArnott]

If you're not totally set on gRPC, you might also consider StreamJsonRpc, which has built-in support for Nerdbank.MessagePack (in its upcoming version).

[AlleSchonWeg]

If you're not totally set on gRPC, you might also consider StreamJsonRpc, which has built-in support for Nerdbank.MessagePack (in its upcoming version).

In our company it is not so easy to switch from gRPC to something other. We need to create a lot of docs, tests etc. I don't want to do this. :)
The workaround is the ReflectionTypeShapeProvider.

[max-ieremenko]

Looking more at the sample generated code, I'm left with lots of questions. Why all this trouble to emit what seems to amount to a ValueTuple? What is the API like to actually use these types?

This is what the library does for you, see gRCP protocol limitation described on the main page.

[OperationContract]
int Sum(long x, int y, int z)

// behind the scene
// gRPC.Post(new Message<long, int, z>(x, y, z))

[max-ieremenko]

release 1.17.0, new package ServiceModel.Grpc.Nerdbank.MessagePackMarshaller.

• Examples/NerdbankMessagePackMarshaller.AOT (.net9)
• Examples/NerdbankMessagePackMarshaller.Grpc.Core (.net462)

[AlleSchonWeg]

Thank you. I will test it. Perhaps in two or three weeks.

[AlleSchonWeg]

Hi @max-ieremenko
i cloned the repository but compiling fails: