2021-01-26

Dotnet framework 4.8 throwing "The server committed a protocol violation. Section=ResponseStatusLine"

Today i had made a static file to be served from haproxy. It contained both http headers and a body in the following format:

HTTP/1.0 503 Service Unavailable
Connection: close
Content-Type: text/html

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html>
    <head>
        <title>... some title ...</title>
    </head>
    <body>
		... some content here ...
    </body>
</html>

The file contains the http headers along with a body. Our deployment flow is, that the static file is checked into an ansible repo on github and then pull from an ansible server and deployed to our haproxy load balancer server.

.Net 4.8 was not happy

When loading the url from a browser, everything looked fine. Both Edge and Firefox was showing the expected response headers.

But....

I ran a script checking tool made in dotnet framework 4.8 where the new endpoint was added and this immediately throw the following WebException:

"The server committed a protocol violation. Section=ResponseStatusLine"

I was digging a bit into my file, trying to save it using ASCII explicitly. Also trying to remove and add headers to make my .net 4.8 project accept the response - nothing worked.

It worked in .Net 5

Then I moved the logic into a dotnet standard project and created a dotnet 5 project and ran the same logic. The funny thing here was: This did not throw the same exception - in fact it ran successfully and downloaded the expected content.

I now had to find out what .net 4.8 was unhappy about in the http response from my HaProxy server.

I gave it a go with PostMan, and this also worked fine.

UseUnsafeHeaderParsing

After googling some more, I found the following setting described here https://docs.microsoft.com/en-us/dotnet/api/system.net.configuration.httpwebrequestelement.useunsafeheaderparsing?view=netframework-4.8

The setting is called UseUnsafeHeaderParsing. I enabled this feature in code using

And now the .net 4.8 project did not throw an error!

So now I knew that there must be some validations that do not pass. According to the aforementioned MS docs, the following reasons are possible:

  • In end-of-line code, use CRLF; using CR or LF alone is not allowed.
  • Headers names should not have spaces in them.
  • If multiple status lines exist, all additional status lines are treated as malformed header name/value pairs.
  • The status line must have a status description, in addition to a status code.

The last option "Header names cannot have non-ASCII chars in them. This validation is performed whether this property is set to true or false." both applies with and without the setting enabled, so that could not be the reason in this case.

I started checking my file again - there was indeed CRLF endings on the lines, there were no spaces in the header names. I also double checked that there was the Service Unavailable after the 503 in the status line.

I tried to use fiddler and compared the response from this broken call with the response headers from other, working endpoints. Fiddler did not show any sign of errors either.

I did actually download wireshark, but I regret, as I have only use this tool once before for debugging a certificate issue case, and I didn't actually know what I was doing :p

I removed my ssl handshake on the haproxy, so I could load the url with just plain http. The problem was still the same.

Using a public http header validation tool

I now created a public domain for the site and configured it to be visible to the outside world. Then I used the online scanning tool https://securityheaders.com/

I typed in my own and another website and compared. The other website reported the following header overview:

image-20210127001517982

Then I did the same scan with my own site. And the result looked like this:

image-20210127001612087

The weird thing is now, that there seems to be only 1 header and all the expected header names and values are listed in a single line. This lead me to the CRLF issue again, so I tripple checked in the http file locallly, and verified that the CRLF was actually there.

Http request using a socket

I now had to know exactly what was returned by my server. I google around and found some C# socket snippets for making a manual http call. I changed some bits and ended up with the following C# code:

var hostname = "[my domain was here]";

Socket socket = null;
var ipAddress = IPAddress.Parse("?.?.?.?"); // My ip was here
var endPoint = new IPEndPoint(ipAddress, 80);

using (socket = new Socket(ipAddress.AddressFamily, SocketType.Stream, ProtocolType.Tcp))
{
    socket.Connect(endPoint);
    using (var networkStream = new NetworkStream(socket))
    {
        Write(networkStream, "GET / HTTP/1.1" + Environment.NewLine);
        Write(networkStream, "Host: " + hostname + Environment.NewLine);
        Write(networkStream, Environment.NewLine);
        networkStream.Flush();

        // Data buffer for incoming data.  
        byte[] bytes = new byte[10 * 1024];
        // Receive the response from the remote device.  
        int bytesRec = socket.Receive(bytes);
        var stringReceived = Encoding.ASCII.GetString(bytes, 0, bytesRec);
        Console.WriteLine("Echoed test = {0}", stringReceived);

        // Release the socket.  
        socket.Shutdown(SocketShutdown.Both);
        socket.Close();
    }
}

The result of the "stringReceived" variable now showed the following content:

"HTTP/1.0 503 Service Unavailable\nConnection: close\nContent-Type: text/html\n\n<!DOCTYPE html PUBLIC...

It now became a sad panda as I realized, that haproxy was not serving the CR character!

Git and line endings

First I was angry with HaProxy for doing such a thing to me, but seconds later it hit me: Git is configured to check in as LF and check out as CRLF!!

This problem took me 8 hours of solving today and as the time of writing it is 00:23 on a tuesday. I will now go to bed and pray that tomorrow will be a coding day for me.

Allowing carriage returns in git

I now had to ensure CRLF in my static http file but I did not want git to change its behaviour for any other of my files in the same repo. The solution is to ad a .gitattributes file in the root of the git repo. I configured the file to define CRLF to be used, but only for .http files:

*.http text eol=crlf

After adding this file, I ran my deploy script in ansible again, checking out the latest files form the repo, let it deploy the file to the haproxy server and re-ran my C# socket code. I could now confirm, that CRLF was actually in the headers section.

Ref: https://docs.github.com/en/github/using-git/configuring-git-to-handle-line-endings#per-repository-settings

After rerunning the online scanning tool, the headers now looked fine:

image-20210127005253028

(I changed "503 Service Unavailable" to "200 OK" in the source file as this was the desired final response status in this case)

.Net 4.8 now also runs successfully when using either of the HttpWebRequest or WebClient.