HTTP/2 and CONTINUATION Flood

Table of Contents

Introduction
#

The reason for writing this article is that I saw a CONTINUATION Flood problem based on the HTTP/2 protocol on Hackernews, and I wanted to understand the cause and review the HTTP/2 specification.

To make the protocol specification and the security issues it raises seem more intuitive, this article will be supplemented with the golang golang.org/x/net source code to explain.

HTTP/2
#

Overview
#

Let’s recap the HTTP/2 protocol. The biggest difference between HTTP/2 and HTTP/1.1 is that

HTTP/1.1 is a text-based protocol, and the basic unit of the protocol is the message, which is separated from the next message with CRLF (\r\n). For example: POST /foo? name=menu&value= HTTP/1.1\ r\nHost: google.com\r\nTransfer-Encoding: chunked\r\nContent-Type: aa/bb\r\n\r\n3 \r\nabc\r\n0\r\n\r\n
HTTP/2 is a binary protocol, and the basic unit of the protocol is the frame. The start content of an HTTP/2 connection is PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n , followed by frame. HTTP/2 also uses HPACK to compress the transmitted content.

💡 For HTTPS, in fact, an SSL/TLS layer is added to the HTTP protocol. You need to decrypt it with a certificate to get binary and text information

Frame format
#

First, let’s take a look at the structure of a frame. The first 9 bytes are fixed and represent Length (3 bytes/24 bits), Type (1 byte/8 bits), Flags (1 byte/8 bits), Reserved (1 bit) and Stream Identifier (31 bits), followed by the specific payload content. It should be noted that the 1 bit of Reserved is often negligible. The following shows a specific format and example:

HTTP Frame {
	Length (24), // 00 00 0C; Frame length: 12 
	Type (8), // 01; Frame type: HEADERS
	
	Flags (8), // 04; Flags: END_HEADERS
	
	Reserved (1),
	Stream Identifier (31), // 00 00 00 01; Stream Identifier: 1
	
	Frame Payload (..), // 87 01 84 8D 4E 3D 6F C8; Binary data for request header information
}

After understanding the definition of the protocol’s own Frame, let’s take the golang.org/x/net source code as an example and see how it is defined in the code during parsing


type FrameType uint8
type Flags uint8

type FrameHeader struct {
  valid bool // caller can access []byte fields in the Frame

  // Type is the 1 byte frame type. There are ten standard frame types
  Type FrameType

  // Flags are the 1 byte of 8 potential bit flags per frame.
  Flags Flags

  // Length is the length of the frame, not including the 9 byte header.
  // The maximum size is one byte less than 16MB (uint24), but only
  // frames up to 16KB are allowed without peer agreement.
  Length uint32

  // StreamID is which stream this frame is for. Certain frames
  // are not stream-specific, in which case this field is 0.
  StreamID uint32
}

Frame types
#

There are 10 types of Frame type, and the different types are distinguished by the Type in the Frame. It is defined as follows in golang.org/x/net:

const (
	FrameData FrameType = 0x0
	FrameHeaders FrameType = 0x1
	FramePriority FrameType = 0x2
	FrameRSTStream    FrameType = 0x3
	FrameSettings     FrameType = 0x4
	FramePushPromise  FrameType = 0x5
	FramePing         FrameType = 0x6
	FrameGoAway       FrameType = 0x7
	FrameWindowUpdate FrameType = 0x8
	FrameContinuation FrameType = 0x9
)

var frameName = map[FrameType]string{
	FrameData:         "DATA", 
	FrameHeaders:      "HEADERS", 
	FramePriority:     "PRIORITY", 
	FrameRSTStream:    "RST_STREAM", 
	FrameSettings:     "SETTINGS",
	FramePushPromise:  "PUSH_PROMISE", 
	FramePing:         "PING", 
	FrameGoAway:       "GOAWAY", 
	FrameWindowUpdate: "WINDOW_UPDATE", 
	FrameContinuation: "CONTINUATION",
}

Let’s discuss these 10 types in more detail:

DATA: a Frame that contains the request body or response body. This Frame must have a Stream Identifier because the overall payload will be streamed in chunks during transmission
HEADERS: contains the request or response header. This frame must also have a Stream Identifier
PRIORITY: this frame is currently deprecated and was previously used to specify the dependency and priority of a stream
RST_STREAM: used to immediately terminate a stream. This frame is sent when a request is cancelled or an error occurs. RST_STREAM is the last frame in the stream.

SETTINGS: used to send connection parameter configurations such as the flow control window size, maximum frame size, etc. when establishing a connection. Among the Flags defined in SETTINGS, one bit is used as the ACK identifier and the remaining 7 bits are not used.

If one party to the connection has received the other party’s parameter configuration, it needs to set ACK to 1 and not transmit the rest of the content in SETTINGS; if neither party transmits ACK, it is assumed that the negotiation of this parameter configuration has failed, and an error will be reported SETTINGS_TIMEOUT.

In the server and client scenario, the client must often confirm and send an ACK, otherwise the server can simply end the connection. For example, server.go in golang.org/x/net is implemented as follows:

func (sc *serverConn) processSettings(f *SettingsFrame) error {
  sc.serveG.check()
  if f.IsAck() {
   sc.unackedSettings--
   if sc.unackedSettings < 0 {
    // Why is the peer ACKing settings we never sent?
    // The spec doesn't mention this case, but
    // hang up on them anyway.
    return ConnectionError(ErrCodeProtocol)
   }
   return nil
  }
  if f.NumSettings() > 100 || f.HasDuplicates() {
   // This isn't actually in the spec, but hang up on
   // suspiciously large settings frames or those with
   // duplicate entries.
   return ConnectionError(ErrCodeProtocol)
  }
  if err := f.ForeachSetting(sc.processSetting); err != nil {
   return err
  }
  sc.needToSendSettingsAck = true
  sc.scheduleFrameWrite()
  return nil
}

PUSH_PROMISE: used when the connection is in the open or half-closed (remote) state, and the server actively pushes the frame
PING: used to measure the minimum round-trip time between the two communicating parties. PING is divided into a sender and a responder, and the responder needs to return the identifier ACK
GOAWAY: used to signal the closure of a connection or a serious error. Compared to RST_STREAM, GOAWAY allows for a more graceful exit and is generally initiated by the server
WINDOW_UPDATE: used to throttle the content in DATA only. WINDOW_UPDATE is generally initiated by the server to tell the client how much data can be transferred. Flow control is implemented in two dimensions: the entire connection serverConn and each individual stream stream. This means that when processing DATA, we can control the server reads and client sends either on the entire connection or on each stream. For example, in the func (sc *serverConn) processData(f *DataFrame) error method in golang.org/x/net, it is handled like this:


func (sc *serverConn) processData(f *DataFrame) error {
  ...
  
  if st == nil || state != stateOpen || st.gotTrailerHeader || st.resetQueued {
   ...
   // Temporarily subtract the length of the DATA frame from the server-side flow control window
   // Then tell the client to continue processing the content of length
   // and then restore the length of the server's flow control window 
   // The reason for taking and then adding is to prevent additional content from being read while sending WINDOW_UPDATE to the client
   sc.inflow.take(int32(f.Length))
   sc.sendWindowUpdate(nil, int(f.Length))
  }
  
  if f.Length > 0 {
	  ...
	  
	  if pad := int32(f.Length) - int32(len(data)); pad > 0 {
		  sc.sendWindowUpdate32(nil, pad)
		  sc.sendWindowUpdate32(st, pad)
	  }
  }
  ...
}

func (sc *serverConn) sendWindowUpdate(st *stream, n int) {
  sc.serveG.check()
  const maxUint31 = 1<<31 - 1
  for n >= maxUint31 {
   sc.sendWindowUpdate32(st, maxUint31)
   n -= maxUint31
  }
  sc.sendWindowUpdate32(st, int32(n))
}

func (sc *serverConn) sendWindowUpdate32(st *stream, n int32) {
  sc.serveG.check()
  if n == 0 {
   return
  }
  if n < 0 {
   panic("negative update")
  }
  var streamID uint32
  if st != nil {
   streamID = st.id
  }
  sc.writeFrame(FrameWriteRequest{
   write:  writeWindowUpdate{streamID: streamID, n: uint32(n)},
   stream: st,
  })
  var ok bool
  if st == nil {
   // Restore the flow control window for the server-side conn
   ok = sc.inflow.add(n)
  } else {
   // Restore the flow control window for the server stream
   ok = st.inflow.add(n)
  }
  if !ok {
   panic("internal error; sent too many window updates without decrements?")
  }
}

CONTINUATION: As long as the HEADER, PUSH_PROMISE or CONTINUATION has not been set to the END_HEADERS flag, CONTINUATION can be used to continue sending any number of data blocks, which will be used as Header data. We will talk about this in more detail below.

CONTINUATION Flood attack
#

In fact, clues can be seen in the above source code section.

Let’s first summarize the description of the HTTP/2 protocol for reconstructing HTTP headers, that is, the Header section can be represented in two ways (quoted from name-field-section-compression-a):

a HEADERS or PUSH_PROMISE frame with the END_HEADERS flag set
a HEADERS or PUSH_PROMISE frame without the END_HEADERS flag set, and one or more CONTINUATION frames, the last of which needs to have the END_HEADERS flag set

CONTINUATION Flood attacks target this second point. Before sending the last CONTINUATION, the HTTP/2 server stores the parts that need to be parsed and combined in memory

This attack can lead to three security risks:

CPU exhaustion. Reading additional headers can lead to high CPU usage, which slows down other responses. This risk often occurs because the server cannot respond to other requests in time due to too many active connections. The solution is to optimize the number of active connections, improve the efficiency of connection processing, and release inactive connections.
OOM memory overflow: The implementation of some HTTP/2 servers is relatively simple, and they simply read CONTINUATION into memory, which can cause an OOM with a single connection. If the server only limits the size of the headers but does not limit the timeout, an attacker can also request multiple connections to trigger an OOM.
Crash the server after only a few frames have been sent. This is a more specific and extremely serious problem. If the server does not handle the CONTINUATION mid-stream disconnection properly, then only a few frames are needed to crash the server.

Golang CASE
#

Let’s now take golang.org/x/net (versions v0.22.0 and earlier) as an example to see how the CONTINUATION Flood problem is triggered. Locate the func (fr *Framer) readMetaFrame(hf *HeadersFrame) (*MetaHeadersFrame, error) method in frame.go:

type ContinuationFrame struct {
  http2.FrameHeader
  headerFragBuf []byte
}

func (f *ContinuationFrame) HeaderBlockFragment() []byte {
  return f.headerFragBuf
}

func (fr *Framer) readMetaFrame(hf *HeadersFrame) (*MetaHeadersFrame, error) {
  ... 
  // MAX_HEADER_LIST_SIZE
  var remainSize = fr.maxHeaderListSize()
  hdec := fr.ReadMetaHeaders
  hdec.SetEmitFunc(func(hf hpack.HeaderField) {
	  ...
	  if !httpguts.ValidHeaderFieldValue(hf.Value) {
		  invalid = headerFieldValueError(hf.Value)
	  }
	  isPseudo := strings.HasPrefix(hf.Name, ":")
	  if isPseudo {
		  if sawRegular {
			  invalid = errPseudoAfterRegular
		  }
	  } else {
		  sawRegular = true
		  if !validWireHeaderFieldName(hf.Name) {
			  invalid = headerFieldNameError(hf.Name)
		  }
	  }
	  
	  if invalid != nil {
		  hdec.SetEmitEnabled(false)
		  return
	  }
	  
	  // Limit header size 
	  size := hf.Size()
	  if size > remainSize {
		  hdec.SetEmitEnabled(false)
		  mh.Truncated = true
    return
	  }
	  remainSize -= size
	
	  mh.Fields = append(mh.Fields, hf)
  })
  
  ...
  var hc headersOrContinuation = hf
  for {
	  frag := hc.HeaderBlockFragment()
    // Decoder writes
    if _, err := hdec.Write(frag); err != nil {
		  return nil, ConnectionError(ErrCodeCompression)
    }
	
	  // END_HEADERS flag
    if hc.HeadersEnded() {
		  break
    }
	
    if f, err := fr.ReadFrame(); err != nil {
      return nil, err
	  } else {
      hc = f.(*ContinuationFrame) // guaranteed by checkFrameOrder
    }
    ...
}

You can see that a loop is started here to construct the headers, and there are three exit conditions:

The hdec.Write method returns an exception: hdec is the HPACK decoder, and an error will be generated when a decoding exception occurs.
The END_HEADERS flag: This is easy to understand, i.e., HEADER or CONTINUATION will exit the loop after adding the END_HEADERS flag.
fr.ReadFrame method returns an exception: fr is the current Framer object, so here the main task is to read the content in the frame. The main reasons for an error are failure to verify the length of the content being read, frame ordering problems, connection problems, etc.

Here, I would also like to explain in detail EmitFunc in hdec.

When the hdec.Write method is executed, it will call the emit callback method, which determines that if the length of the headers exceeds MAX_HEADER_LIST_SIZE, then the emit callback will be closed—hdec.SetEmitEnabled(false)—and no data will be added to MetaHeadersFrame. But this does not stop the for loop above!

Not only that, but other exceptions thrown in the emit callback also do not return or interrupt the loop, such as headerFieldNameError, errPseudoAfterRegular, and headerFieldValueError, which also just set emitEnabled to false.

This allows an attacker to send CONTINUATION frames after MAX_HEADER_LIST_SIZE, without the server stopping receiving CONTINUATION frames. This means that an attacker can send an arbitrary number of CONTINUATION frames without ever delivering the END_HEADERS marker, which can be used to continuously drain server resources.

GO-2024-2687
#

This issue has been addressed in the latest golang.org/x/net, as detailed in GO-2024-2687.

We will mainly look at the part of the source code that has been modified, which can be seen in https://go-review.googlesource.com/c/net/+/576155. We will post it below:

func (fr *Framer) readMetaFrame(hf *HeadersFrame) (*MetaHeadersFrame, error) {
  ...
  var remainSize = fr.maxHeaderListSize()
  var invalid error // pseudo header field errors
  hdec := fr.ReadMetaHeaders
  hdec.SetEmitEnabled(true)
  hdec.SetMaxStringLength(fr.maxHeaderStringLen())
  hdec.SetEmitFunc(func(hf hpack.HeaderField) {
   if VerboseLogs && fr.logReads {
    fr.debugReadLoggerf("http2: decoded hpack field %+v", hf)
   }
   if !httpguts.ValidHeaderFieldValue(hf.Value) {
    // Don't include the value in the error, because it may be sensitive.
    invalid = headerFieldValueError(hf.Name)
   }
   isPseudo := strings.HasPrefix(hf.Name, ":")
   if isPseudo {
    if sawRegular {
     invalid = errPseudoAfterRegular
    }
   } else {
    sawRegular = true
    if !validWireHeaderFieldName(hf.Name) {
     invalid = headerFieldNameError(hf.Name)
    }
   }

   if invalid != nil {
    hdec.SetEmitEnabled(false)
    return
   }

   size := hf.Size()
   if size > remainSize {
    hdec.SetEmitEnabled(false)
    mh.Truncated = true
    remainSize = 0
    return
   }
   remainSize -= size

   mh.Fields = append(mh.Fields, hf)
  })
  // Lose reference to MetaHeadersFrame:
  defer hdec.SetEmitFunc(func(hf hpack.HeaderField) {})

  var hc headersOrContinuation = hf
  for {
   frag := hc.HeaderBlockFragment()

   // The break condition adds a check for remainSize
   // After the header exceeds the **MAX_HEADER_LIST_SIZE** limit, remainSize will become 0
   if int64(len(frag)) > int64(2*remainSize) {
    if VerboseLogs {
     log.Printf("http2: header list too large")
    }
    return nil, ConnectionError(ErrCodeProtocol)
   }

// Added break for loop for other exceptions in the emit callback method
if invalid != nil {
if VerboseLogs {
log.Printf("http2: invalid header: %v", invalid)
}
return nil, ConnectionError(ErrCodeProtocol)
}
if _, err := hdec.Write(frag); err != nil {
    return nil, ConnectionError(ErrCodeCompression)
   }

   if hc.HeadersEnded() {
    break
   }
   if f, err := fr.ReadFrame(); err != nil {
    return nil, err
   } else {
    hc = f.(*ContinuationFrame) // guaranteed by checkFrameOrder
   }
}

}

References
#

HTTP/2 CONTINUATION Flood: Technical Details

RFC 9113: HTTP/2

RFC 9112: HTTP/1.1

Introduction #

HTTP/2 #

Overview #

Frame format #

Frame types #

CONTINUATION Flood attack #

Golang CASE #

GO-2024-2687 #

References #