Packet Mirroring with XDP

There are several ways to achieve packet mirroring in Linux, but I had thought that XDP alone could not implement packet mirroring. In other words, I believed it required combining with other features like TC.

After some experimentation, I found that packet mirroring can be achieved using the devmap broadcast feature (simultaneous transmission to all devmap entries). The code is available at xdp_mirror.py.

The overall architecture looks like this. There are 3 hosts (h1-3) and a gateway (gw) connecting them. The hosts and gateway are virtually created using netns and connected via veth pairs. In a scenario where Packet1 is being forwarded from h1 to h2, if the packet matches a pre-configured 5-tuple at gw, it is forwarded to h2 and also mirrored to h3 with some packet modification. To reiterate, this is implemented using XDP alone without external features like TC. The gray blocks represent XDP programs. There are 4 types in total. host_pass.c was created for veth but is not essential and can be ignored.

filter.c performs 5-tuple matching, and if matched, sends the packet to all entries in the devmap.

// filter.c
SEC("xdp")
int xdp_filter(struct xdp_md *ctx)
{
    ip = ... ;
    udp = ... ;

    struct five_tuple key = {
        .saddr = ip->saddr,
        .daddr = ip->daddr,
        .sport = udp->source,
        .dport = udp->dest,
        .proto = ip->protocol,
    };

    // Mirror Enabled: the packet is sent to
    // all entries (for h2 & h3) in the devmap.
    if (bpf_map_lookup_elem(&mirror_targets, &key))
        return bpf_redirect_map(&tx_port, 0, BPF_F_BROADCAST);

    // Mirror Disabled: the packet is sent to
    // the first entry (for h2) in the devmap.
    return bpf_redirect_map(&tx_port, 0, 0);
}

Note that mirror.c, which is called from filter.c via bpf_redirect_map, must be compiled with the BPF_XDP_DEVMAP attach type. You can modify packets using functions like bpf_xdp_adjust_head, and in this example, a VXLAN header is added.

That’s all.