Introduction #
The reason for writing this article is that I saw a CONTINUATION Flood problem based on the HTTP/2 protocol on Hackernews, and I wanted to understand the cause and review the HTTP/2 specification.
To make the protocol specification and the security issues it raises seem more intuitive, this article will be supplemented with the golang golang.org/x/net source code to explain.
HTTP/2 #
Overview #
Let’s recap the HTTP/2 protocol. The biggest difference between HTTP/2 and HTTP/1.1 is that
- HTTP/1.1 is a text-based protocol, and the basic unit of the protocol is the message, which is separated from the next message with CRLF (
\r\n
). For example:POST /foo? name=menu&value= HTTP/1.1\ r\nHost: google.com\r\nTransfer-Encoding: chunked\r\nContent-Type: aa/bb\r\n\r\n3 \r\nabc\r\n0\r\n\r\n
- HTTP/2 is a binary protocol, and the basic unit of the protocol is the frame. The start content of an HTTP/2 connection is
PRI * HTTP/2.0\r\n\r\nSM\r\n\r\n
, followed by frame. HTTP/2 also uses HPACK to compress the transmitted content.
Frame format #
First, let’s take a look at the structure of a frame. The first 9 bytes are fixed and represent Length (3 bytes/24 bits), Type (1 byte/8 bits), Flags (1 byte/8 bits), Reserved (1 bit) and Stream Identifier (31 bits), followed by the specific payload content. It should be noted that the 1 bit of Reserved is often negligible. The following shows a specific format and example:
HTTP Frame {
Length (24), // 00 00 0C; Frame length: 12
Type (8), // 01; Frame type: HEADERS
Flags (8), // 04; Flags: END_HEADERS
Reserved (1),
Stream Identifier (31), // 00 00 00 01; Stream Identifier: 1
Frame Payload (..), // 87 01 84 8D 4E 3D 6F C8; Binary data for request header information
}
After understanding the definition of the protocol’s own Frame, let’s take the golang.org/x/net source code as an example and see how it is defined in the code during parsing
type FrameType uint8
type Flags uint8
type FrameHeader struct {
valid bool // caller can access []byte fields in the Frame
// Type is the 1 byte frame type. There are ten standard frame types
Type FrameType
// Flags are the 1 byte of 8 potential bit flags per frame.
Flags Flags
// Length is the length of the frame, not including the 9 byte header.
// The maximum size is one byte less than 16MB (uint24), but only
// frames up to 16KB are allowed without peer agreement.
Length uint32
// StreamID is which stream this frame is for. Certain frames
// are not stream-specific, in which case this field is 0.
StreamID uint32
}
Frame types #
There are 10 types of Frame type, and the different types are distinguished by the Type in the Frame. It is defined as follows in golang.org/x/net:
const (
FrameData FrameType = 0x0
FrameHeaders FrameType = 0x1
FramePriority FrameType = 0x2
FrameRSTStream FrameType = 0x3
FrameSettings FrameType = 0x4
FramePushPromise FrameType = 0x5
FramePing FrameType = 0x6
FrameGoAway FrameType = 0x7
FrameWindowUpdate FrameType = 0x8
FrameContinuation FrameType = 0x9
)
var frameName = map[FrameType]string{
FrameData: "DATA",
FrameHeaders: "HEADERS",
FramePriority: "PRIORITY",
FrameRSTStream: "RST_STREAM",
FrameSettings: "SETTINGS",
FramePushPromise: "PUSH_PROMISE",
FramePing: "PING",
FrameGoAway: "GOAWAY",
FrameWindowUpdate: "WINDOW_UPDATE",
FrameContinuation: "CONTINUATION",
}
Let’s discuss these 10 types in more detail:
-
DATA: a Frame that contains the request body or response body. This Frame must have a Stream Identifier because the overall payload will be streamed in chunks during transmission
-
HEADERS: contains the request or response header. This frame must also have a Stream Identifier
-
PRIORITY: this frame is currently deprecated and was previously used to specify the dependency and priority of a stream -
RST_STREAM: used to immediately terminate a stream. This frame is sent when a request is cancelled or an error occurs. RST_STREAM is the last frame in the stream.
-
SETTINGS: used to send connection parameter configurations such as the flow control window size, maximum frame size, etc. when establishing a connection. Among the
Flags
defined in SETTINGS, one bit is used as theACK
identifier and the remaining 7 bits are not used.If one party to the connection has received the other party’s parameter configuration, it needs to set
ACK
to 1 and not transmit the rest of the content in SETTINGS; if neither party transmits ACK, it is assumed that the negotiation of this parameter configuration has failed, and an error will be reportedSETTINGS_TIMEOUT
.In the server and client scenario, the client must often confirm and send an
ACK
, otherwise the server can simply end the connection. For example,server.go
in golang.org/x/net is implemented as follows:func (sc *serverConn) processSettings(f *SettingsFrame) error { sc.serveG.check() if f.IsAck() { sc.unackedSettings-- if sc.unackedSettings < 0 { // Why is the peer ACKing settings we never sent? // The spec doesn't mention this case, but // hang up on them anyway. return ConnectionError(ErrCodeProtocol) } return nil } if f.NumSettings() > 100 || f.HasDuplicates() { // This isn't actually in the spec, but hang up on // suspiciously large settings frames or those with // duplicate entries. return ConnectionError(ErrCodeProtocol) } if err := f.ForeachSetting(sc.processSetting); err != nil { return err } sc.needToSendSettingsAck = true sc.scheduleFrameWrite() return nil }
-
PUSH_PROMISE: used when the connection is in the
open
orhalf-closed (remote)
state, and the server actively pushes the frame -
PING: used to measure the minimum round-trip time between the two communicating parties. PING is divided into a sender and a responder, and the responder needs to return the identifier
ACK
-
GOAWAY: used to signal the closure of a connection or a serious error. Compared to RST_STREAM, GOAWAY allows for a more graceful exit and is generally initiated by the server
-
WINDOW_UPDATE: used to throttle the content in DATA only. WINDOW_UPDATE is generally initiated by the server to tell the client how much data can be transferred. Flow control is implemented in two dimensions: the entire connection
serverConn
and each individual streamstream
. This means that when processing DATA, we can control the server reads and client sends either on the entire connection or on each stream. For example, in thefunc (sc *serverConn) processData(f *DataFrame) error
method in golang.org/x/net, it is handled like this:
func (sc *serverConn) processData(f *DataFrame) error {
...
if st == nil || state != stateOpen || st.gotTrailerHeader || st.resetQueued {
...
// Temporarily subtract the length of the DATA frame from the server-side flow control window
// Then tell the client to continue processing the content of length
// and then restore the length of the server's flow control window
// The reason for taking and then adding is to prevent additional content from being read while sending WINDOW_UPDATE to the client
sc.inflow.take(int32(f.Length))
sc.sendWindowUpdate(nil, int(f.Length))
}
if f.Length > 0 {
...
if pad := int32(f.Length) - int32(len(data)); pad > 0 {
sc.sendWindowUpdate32(nil, pad)
sc.sendWindowUpdate32(st, pad)
}
}
...
}
func (sc *serverConn) sendWindowUpdate(st *stream, n int) {
sc.serveG.check()
const maxUint31 = 1<<31 - 1
for n >= maxUint31 {
sc.sendWindowUpdate32(st, maxUint31)
n -= maxUint31
}
sc.sendWindowUpdate32(st, int32(n))
}
func (sc *serverConn) sendWindowUpdate32(st *stream, n int32) {
sc.serveG.check()
if n == 0 {
return
}
if n < 0 {
panic("negative update")
}
var streamID uint32
if st != nil {
streamID = st.id
}
sc.writeFrame(FrameWriteRequest{
write: writeWindowUpdate{streamID: streamID, n: uint32(n)},
stream: st,
})
var ok bool
if st == nil {
// Restore the flow control window for the server-side conn
ok = sc.inflow.add(n)
} else {
// Restore the flow control window for the server stream
ok = st.inflow.add(n)
}
if !ok {
panic("internal error; sent too many window updates without decrements?")
}
}
- CONTINUATION: As long as the HEADER, PUSH_PROMISE or CONTINUATION has not been set to the
END_HEADERS
flag, CONTINUATION can be used to continue sending any number of data blocks, which will be used as Header data. We will talk about this in more detail below.
CONTINUATION Flood attack #
In fact, clues can be seen in the above source code section.
Let’s first summarize the description of the HTTP/2 protocol for reconstructing HTTP headers, that is, the Header section can be represented in two ways (quoted from name-field-section-compression-a):
- a HEADERS or PUSH_PROMISE frame with the
END_HEADERS
flag set - a HEADERS or PUSH_PROMISE frame without the
END_HEADERS
flag set, and one or more CONTINUATION frames, the last of which needs to have theEND_HEADERS
flag set
CONTINUATION Flood attacks target this second point. Before sending the last CONTINUATION, the HTTP/2 server stores the parts that need to be parsed and combined in memory
This attack can lead to three security risks:
- CPU exhaustion. Reading additional headers can lead to high CPU usage, which slows down other responses. This risk often occurs because the server cannot respond to other requests in time due to too many active connections. The solution is to optimize the number of active connections, improve the efficiency of connection processing, and release inactive connections.
- OOM memory overflow: The implementation of some HTTP/2 servers is relatively simple, and they simply read CONTINUATION into memory, which can cause an OOM with a single connection. If the server only limits the size of the headers but does not limit the timeout, an attacker can also request multiple connections to trigger an OOM.
- Crash the server after only a few frames have been sent. This is a more specific and extremely serious problem. If the server does not handle the CONTINUATION mid-stream disconnection properly, then only a few frames are needed to crash the server.
Golang CASE #
Let’s now take golang.org/x/net (versions v0.22.0 and earlier) as an example to see how the CONTINUATION Flood problem is triggered. Locate the func (fr *Framer) readMetaFrame(hf *HeadersFrame) (*MetaHeadersFrame, error)
method in frame.go
:
type ContinuationFrame struct {
http2.FrameHeader
headerFragBuf []byte
}
func (f *ContinuationFrame) HeaderBlockFragment() []byte {
return f.headerFragBuf
}
func (fr *Framer) readMetaFrame(hf *HeadersFrame) (*MetaHeadersFrame, error) {
...
// MAX_HEADER_LIST_SIZE
var remainSize = fr.maxHeaderListSize()
hdec := fr.ReadMetaHeaders
hdec.SetEmitFunc(func(hf hpack.HeaderField) {
...
if !httpguts.ValidHeaderFieldValue(hf.Value) {
invalid = headerFieldValueError(hf.Value)
}
isPseudo := strings.HasPrefix(hf.Name, ":")
if isPseudo {
if sawRegular {
invalid = errPseudoAfterRegular
}
} else {
sawRegular = true
if !validWireHeaderFieldName(hf.Name) {
invalid = headerFieldNameError(hf.Name)
}
}
if invalid != nil {
hdec.SetEmitEnabled(false)
return
}
// Limit header size
size := hf.Size()
if size > remainSize {
hdec.SetEmitEnabled(false)
mh.Truncated = true
return
}
remainSize -= size
mh.Fields = append(mh.Fields, hf)
})
...
var hc headersOrContinuation = hf
for {
frag := hc.HeaderBlockFragment()
// Decoder writes
if _, err := hdec.Write(frag); err != nil {
return nil, ConnectionError(ErrCodeCompression)
}
// END_HEADERS flag
if hc.HeadersEnded() {
break
}
if f, err := fr.ReadFrame(); err != nil {
return nil, err
} else {
hc = f.(*ContinuationFrame) // guaranteed by checkFrameOrder
}
...
}
You can see that a loop is started here to construct the headers, and there are three exit conditions:
- The
hdec.Write
method returns an exception:hdec
is the HPACK decoder, and an error will be generated when a decoding exception occurs. - The
END_HEADERS
flag: This is easy to understand, i.e., HEADER or CONTINUATION will exit the loop after adding theEND_HEADERS
flag. fr.ReadFrame
method returns an exception:fr
is the currentFramer
object, so here the main task is to read the content in the frame. The main reasons for an error are failure to verify the length of the content being read, frame ordering problems, connection problems, etc.
Here, I would also like to explain in detail EmitFunc
in hdec
.
When the hdec.Write
method is executed, it will call the emit
callback method, which determines that if the length of the headers exceeds MAX_HEADER_LIST_SIZE, then the emit
callback will be closed—hdec.SetEmitEnabled(false)
—and no data will be added to MetaHeadersFrame
. But this does not stop the for loop above!
Not only that, but other exceptions thrown in the emit
callback also do not return or interrupt the loop, such as headerFieldNameError, errPseudoAfterRegular, and headerFieldValueError, which also just set emitEnabled
to false
.
This allows an attacker to send CONTINUATION frames after MAX_HEADER_LIST_SIZE, without the server stopping receiving CONTINUATION frames. This means that an attacker can send an arbitrary number of CONTINUATION frames without ever delivering the END_HEADERS
marker, which can be used to continuously drain server resources.
GO-2024-2687 #
This issue has been addressed in the latest golang.org/x/net, as detailed in GO-2024-2687.
We will mainly look at the part of the source code that has been modified, which can be seen in https://go-review.googlesource.com/c/net/+/576155. We will post it below:
func (fr *Framer) readMetaFrame(hf *HeadersFrame) (*MetaHeadersFrame, error) {
...
var remainSize = fr.maxHeaderListSize()
var invalid error // pseudo header field errors
hdec := fr.ReadMetaHeaders
hdec.SetEmitEnabled(true)
hdec.SetMaxStringLength(fr.maxHeaderStringLen())
hdec.SetEmitFunc(func(hf hpack.HeaderField) {
if VerboseLogs && fr.logReads {
fr.debugReadLoggerf("http2: decoded hpack field %+v", hf)
}
if !httpguts.ValidHeaderFieldValue(hf.Value) {
// Don't include the value in the error, because it may be sensitive.
invalid = headerFieldValueError(hf.Name)
}
isPseudo := strings.HasPrefix(hf.Name, ":")
if isPseudo {
if sawRegular {
invalid = errPseudoAfterRegular
}
} else {
sawRegular = true
if !validWireHeaderFieldName(hf.Name) {
invalid = headerFieldNameError(hf.Name)
}
}
if invalid != nil {
hdec.SetEmitEnabled(false)
return
}
size := hf.Size()
if size > remainSize {
hdec.SetEmitEnabled(false)
mh.Truncated = true
remainSize = 0
return
}
remainSize -= size
mh.Fields = append(mh.Fields, hf)
})
// Lose reference to MetaHeadersFrame:
defer hdec.SetEmitFunc(func(hf hpack.HeaderField) {})
var hc headersOrContinuation = hf
for {
frag := hc.HeaderBlockFragment()
// The break condition adds a check for remainSize
// After the header exceeds the **MAX_HEADER_LIST_SIZE** limit, remainSize will become 0
if int64(len(frag)) > int64(2*remainSize) {
if VerboseLogs {
log.Printf("http2: header list too large")
}
return nil, ConnectionError(ErrCodeProtocol)
}
// Added break for loop for other exceptions in the emit callback method
if invalid != nil {
if VerboseLogs {
log.Printf("http2: invalid header: %v", invalid)
}
return nil, ConnectionError(ErrCodeProtocol)
}
if _, err := hdec.Write(frag); err != nil {
return nil, ConnectionError(ErrCodeCompression)
}
if hc.HeadersEnded() {
break
}
if f, err := fr.ReadFrame(); err != nil {
return nil, err
} else {
hc = f.(*ContinuationFrame) // guaranteed by checkFrameOrder
}
}
}