Tracing H2O with DTrace/BPF

Tracing expert? you can stop reading and point your browser to: h2o-probes.d

The User Statically Defined Trace (USDT) mechanism enables system operators and support engineers to gain insight into a running system with low performance overhead. This document walks through the USDT probes that the H2O server provides out of the box.

Enabling USDT Probes

USDT probes in H2O are disabled by default. To enable, provide the -DWITH_DTRACE=ON option to CMake at build time. If you are using Linux, you will likely need to install SystemTap in advance.

Available USDT Probes

Signatures are abbreviated. See h2o-probes.d for the full interface. Expect more probes to appear in the future.

usdt::h2o:h1_accept

Triggered when an HTTP/1 connection is accepted. Emits the underlying socket and connection.

usdt::h2o:h1_close

Triggered when an HTTP/1 connection is closed. Emits the closed connection ID.

usdt::h2o:receive_request

Triggered when an HTTP request is received. Emits the received HTTP version. Protocol agnostic.

usdt::h2o:receive_request_header

Triggered when an HTTP header is read. Emits the header name/value pair. Protocol agnostic.

usdt::h2o:send_response

Triggered before sending the HTTP response. Emits the response status. Protocol agnostic.

usdt::h2o:send_response_header

Triggered before sending the HTTP response. Emits the header name/value pair. Protocol agnostic.

Client Tools

bpftrace

In Linux, bpftrace is undoubtedly the easiest way to interact with the underlying BPF virtual machine. Here's how you might attach to a single probe.

$ sudo bpftrace -p $(pgrep -o h2o) -e 'usdt::h2o:h1_accept {printf("Connection accepted! ID: %llu\n", arg0)}'

Attaching 1 probe...
Connection accepted! ID: 1
Connection accepted! ID: 2
Connection accepted! ID: 3

h2olog

h2olog is an operator friendly varnishlog-like request logging client for the H2O server.

$ sudo h2olog -p $(pgrep -o h2o)

3 1 RxProtocol HTTP/2.0
3 1 RxHeader   :authority torumk.com
3 1 RxHeader   :method GET
3 1 RxHeader   :path /
3 1 RxHeader   :scheme https
... more ...