Xamarin Tizen Networking: Under the covers of HTTP/2 in .NET

My current side/passion project requires the use of HTTP/2: It’s a .NET implementation of the Alexa Voice Service and I use it to drive Voice in a Can: Alexa for iOS, Apple Watch, Mac, Android, Android Wear, and … Tizen.

This isn’t an advert, but I do want to set the context. The Alexa Voice Service requires the use of HTTP/2 and this is a real-world product, not an academic excercise.

Why HTTP/2?

The reason the Alexa Voice Service requires HTTP/2 is that as well as the normal requests a client makes (send a request and get a response), the Alexa Voice Service specifies that a client keep a long-running downchannel HTTP/2 connection open so that it can use the HTTP/2 server push mechanism to send directives to the client.

For example when the client sends a request to recognize what is being said, it sends Recognise event to the Alexa Voice Service. This consists of a multipart mime message, the first part being JSON indicating that a recognize request is being sent, and the second part is binary data containing the audio samples from microphone (streamed).

Whilst the microphone data is being streamed, the Alexa Voice Service can detect that the person has stopped speaking (silence detection) and it uses the downchannel to asynchronously send a StopCapture directive, at which point the client stops recording and finishes the request.

So the HTTP/2 is a must. You can’t create an AVS client without supporting HTTP/2.

On platforms such as iOS, WatchOS, MacOS and Android I’ve abstacted out the HTTP functionality behind an interface, and used platform-specific code to implement the interface (NSUrlSession, OkHttp etc).

On Tizen I wanted to see if I could just use the .NET platform.

Forcing HTTP/2 to be used by the .NET HttpClient

The first challenge was to make the .NET HttpClient use HTTP/2.

This turned out to be surprisingly easy. I needed to specify the HttpRequestMessage.Version.

This was my original code for sending a message:

var content = stream == null ? null : new StreamContent(stream);
var request = new HttpRequestMessage(httpMethod, url) {
  Content = content,
  Version = new Version(2,0)
};
var response = await _httpClient.SendAsync(request, cancellationToken);

Notice how I’m setting the Version property.

Handling streamed responses as the data arrives

The second challenge is that by default the HttpClient waits for the complete response. This doesn’t work with the Alexa Voice Service because it streams responses. If you ask “Alexa, what is PI to 100 decimal places” you don’t want for the complete response to return before you start hearing the response … you want the response to stream and be played as it is received.

The solution to this was an additional parameter when calling SendAsync. You can specify whether you want the HttpClient to wait until the complete response is received, or just the HTTP headers, using the HttpCompletionOption.

var response = await _httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, cancellationToken);

Tizen uses the .NET Core UNIX HttpClient implementation

There were times when I wanted to look at how the Tizen HttpClient was implemented. One of the many delights of Xamarin and the “new” Microsoft is that pretty much everything is open source.

I went digging, expecting to find a Tizen HttpClientHandler but to my surprise, I found it was using the .NET Core UNIX HttpClient. The source is here (it uses Curl).

Enabling logging

One final tip. Sometimes you want to see what is happening under the hood. When looking through the source I found logging statements, and I wanted to see the logs, such as this code from the CurlHandler:

      static CurlHandler()
        {
            // curl_global_init call handled by Interop.LibCurl's cctor

            Interop.Http.CurlFeatures features = Interop.Http.GetSupportedFeatures();
            s_supportsSSL = (features & Interop.Http.CurlFeatures.CURL_VERSION_SSL) != 0;
            s_supportsAutomaticDecompression = (features & Interop.Http.CurlFeatures.CURL_VERSION_LIBZ) != 0;
            s_supportsHttp2Multiplexing = (features & Interop.Http.CurlFeatures.CURL_VERSION_HTTP2) != 0 && Interop.Http.GetSupportsHttp2Multiplexing() && !UseSingletonMultiAgent;

            if (NetEventSource.IsEnabled)
            {
                EventSourceTrace($"libcurl: {CurlVersionDescription} {CurlSslVersionDescription} {features}");
            }

To see these log messages I first declared a member _myEventListener member which is an EventListener:

private MyEventListener _myEventListener;

Then later in my code I initialized the _myEventListener:

  var netEventSource = EventSource.GetSources().FirstOrDefault(es => es.Name == "Microsoft-System-Net-Http");
  if (netEventSource != null && _myEventListener == null) {
    _myEventListener = new MyEventListener();
    _myEventListener.EnableEvents(netEventSource, EventLevel.LogAlways);
  }

The event listener is declared like this. Note the filtering of a couple of hard-coded strings that were poluting my output:

class MyEventListener : EventListener {
  protected override void OnEventWritten(EventWrittenEventArgs eventData) {
    var memberNameIndex = eventData.PayloadNames.IndexOf("memberName");

    var memberName = memberNameIndex == -1 ? null : eventData.Payload[memberNameIndex].ToString();

    var message = new StringBuilder();
    for (var i = 0; i < eventData.Payload.Count; i++) {
      if(i == memberNameIndex) continue;
      if (i > 0) {
        message.Append(", ");
      }
      message.Append(eventData.PayloadNames[i] + "=" + eventData.Payload[i]);
    }

    var last = eventData.Payload.Last().ToString();

    if(last == "Ask libcurl to perform any available work...") return;
    if (last == "...done performing work: CURLM_OK") return;
    if(string.IsNullOrWhiteSpace(last)) return;

    if (memberName == null) {
      Log.D(message);
    } else {
      // ReSharper disable once ExplicitCallerInfoArgument
      Log.D(message, memberName, "CurlHandler");
    }
  }
}

My logger uses Tizen.Log.Debug("viac", message, "", "",0); to output to the log, using the Tizen system Log class.

I used this command line to view the log:

sdb dlog viac:D" or "sdb dlog viac:D`

An extract of the output it all its glory:

D/viac    ( 7582):  18:30:26 []  TizenNetworkImpl MakeHttpRequest Sending...
D/viac    ( 7582):  18:30:26 []  CurlHandler SendAsync thisOrContextObject=HttpClient#52727599, parameters=(Method: GET, RequestUri: 'https://avs-alexa-na.amazon.com/v20160207/directives', Version: 2.0, Content: <null>, Headers:
D/viac    ( 7582): {
D/viac    ( 7582):   Authorization: Bearer ...
D/viac    ( 7582): })
D/viac    ( 7582):  18:30:26 []  CurlHandler .ctor thisOrContextObject=CurlResponseMessage#51192825, parameters=(OK)
D/viac    ( 7582):  18:30:26 []  CurlHandler RequestMessage thisOrContextObject=CurlResponseMessage#51192825, first=CurlResponseMessage#51192825, second=HttpRequestMessage#38539564
D/viac    ( 7582):  18:30:26 []  CurlHandler Content thisOrContextObject=CurlResponseMessage#51192825, first=CurlResponseMessage#51192825, second=NoWriteNoSeekStreamContent#64971671
D/viac    ( 7582):  18:30:26 []  CurlHandler SendAsync handlerId=26756241, workerId=4, requestId=5, message=Method: GET, RequestUri: 'https://avs-alexa-na.amazon.com/v20160207/directives', Version: 2.0, Content: <null>, Headers:
D/viac    ( 7582): {
D/viac    ( 7582):   Authorization: Bearer ...
D/viac    ( 7582): }
D/viac    ( 7582):  18:30:26 []  CurlHandler SendAsync thisOrContextObject=HttpClient#52727599, result=System.Threading.Tasks.Task`1[System.Net.Http.HttpResponseMessage]

Final thoughts

When I first learned to program I spent evening after evening of focused hours trying to break the copy-protection on 8-bit games, not to steal them (I’d already bought them), but to try to disssassemble them in order to work out how to get infinite lives.

I often think that despite the formal training I later received getting a degree in computer science, those childhood hours of fierce focused concentration, trying to accomplish something I wasn’t even sure was possible, was the best training I ever had.

I had no idea whether I could get the Alexa Voice Service running on Tizen, whether I could get HTTP/2 working, or a myriad other things. Sometimes you just have to keep trying, having faith in your abilities, continually trying different approaches, until eventually, one day: