Streaming HTTP request data live – .NET to Cloudflare

After some careful code crafting to CPU- and memory-efficiently process incoming telemetry, batch it up and store in “real-time”, a hurdle presented itself.

I’m using Cloudflare to act as a front-end web server, handling the SSL aspect, hiding the “real” server and potentially using its DDoS protection / analytics features.

However, Cloudflare doesn’t support streaming requests: requests are read from the client in their entirety, then transmitted to the origin server. A support forum thread from October 2018 confirmed that this wasn’t something that’s configurable. 😥

So if I want to have this feature, I’ll need to use another front-end “protective” web service since Kestrel, the ASP.NET Core web server tech, isn’t recommended to be directly exposed itself. Most likely this will take the form of an NGINX instance also running on GCP.

There were some other hurdles, too, specifically using the .NET Framework (the mod is on Unity – so not .NET Core yet!) to stream HTTP request data. This is only possible under HTTP/1.1 by either specifying a Content-Length header, or using “chunked” transfers.

The former isn’t possible – we don’t know the length in advance! The latter should work, although it does depend, of course, on sending up individual chunks. I was hoping for something a little more “real-time”. It worked, but with the Cloudflare hurdle, plus the need to send the Stream-writing code to a background thread (things got jerky! I think each and every write to the Stream – each “float”, for instance, is blocking), for now I’m just uploading data after the track has finished.

The Pointer

It’s time to introduce another member of the MegaDemo family! Here’s The Pointer, as it stands right now. It’s going to be a physical hits-per-second meter 🙂