Routing Talos cluster traffic over specific NIC

When your Talos nodes have multiple NICs attached to them and you'd like to route in-cluster traffic over a specific NIC. How would you go about doing that?

There can be various reasons why you'd want to do that. For example, the NICs available could offer differ speeds, and the with the given workloads it could make sense to route in-cluster traffic through the faster one and the egress over the slower one.

The following assumes that Cilium CNI is being used and it has been configured to use native routing. Let's also assume that the nodes are configured in the following fashion:

First, we need to assign both of the network interfaces IPs in different subnets. For example, all the NIC-1's would get 10.1.0.0/24 and all the NIC-2's would get 10.2.0.0/24. The Talos patch for the node-1 would be the following:

machine:
  network:
    interfaces:
      - interface: enp0s31f6
        addresses:
          - 10.1.0.1/24
        routes:
          - network: 0.0.0.0/0
            gateway: 10.0.0.254 # for example
      - interface: enp2s0
        addresses:
          - 10.2.0.1/24

At this point, the nodes are reachable to each other via either of the links. The traffic may or may not flow through your desired NIC. To make it explicit, we need to tell kubelet which of the subnets it is meant to use. We can do that with the following patch file:

machine:
  kubelet:
    nodeIP:
      validSubnets:
        - 10.2.0.0/24

And that's it. After applying the configuration to nodes, the in-cluster traffic will now use NIC-2 and all the egress traffic will get routed via NIC-1. It can be verified with talosctl pcap command.