I've been having trouble with a little RPI Kubernetes cluster on my school's network, and I think the issue has to do with DHCP. Problem is, I don't really understand a whole lot of networking stuff, let alone DHCP. Could anyone ELI5?
EDIT: Thanks to the help of the campus "networking guy," I believe we've fixed the issue. Apparently wired interfaces get shut off after a while if they aren't "chatty." While my cluster is chatty by virtue of being a cluster, since it's all on the same ethernet switch, that doesn't really matter. While an actual fix would be to investigate interface shutdowns on non-chattiness, the temporary fix? Curl google.com every minute! https://twitter.com/dvdmuckle/status/958019950001745922 Full Twitter thread of the issue solution! Turns out it wasn't exactly DHCP!
Top comments (8)
It is like you show up to a class for the first day. But you are late and apparently everyone is already split up in groups working, including the professor, who you do not immediately recognize. You shout in the classroom "Hey, what group am I in?" Everybody turns around and looks at you funny for a second. But the professor pipes up and calls for you to come over. He hands out your group assignment so you can begin to participate.
Edit: This is an IPv4 description. IPv6 is more complicated.
A DHCP server listens for broadcast requests going to a specific address (usually 255.255.255.255) and port (67). It then responds to those requests, possibly with network configuration information like ("I have assigned you IP address 10.0.1.37, netmask 255.255.255.0, and the router for this network is 10.0.1.1, and I recommend using DNS servers 8.8.8.8 and 8.8.4.4 so Google can record all the websites you visit."). Or maybe by policy it only hands out network info to certain MAC addresses. MAC addresses are (supposed to be) unique to every network device, so the DHCP server can essentially allow or block individual computers.
DHCP servers are setup with a specific range of IP addresses. Like 10.0.3.10 - 10.0.3.49. The server keeps track of who (which MAC address) has been assigned which IP. And each IP address is only good for a limited amount of time (called a "lease"). Once the lease expires, the client either has to renew the lease or the address is free for another client to use.
Common Issues
If a DHCP server runs out of IP addresses, new DHCP clients will not be able to connect to the network.
Usually having two DHCP servers on the same network will cause severe problems and lack of connectivity. Unless you have specifically setup your DHCP servers to cooperate (e.g. clustered).
DHCP server must be on the same broadcast domain as the DHCP client in order to hear its request. (I.e. there are switches but no routers in between the DHCP client and server.) You can work around this by having a DHCP relay or DHCP helper agent setup on the client's broadcast domain. This will listen for DHCP client requests and forward them to the network where the DHCP server lives.
A device that fails to receive a DHCP address from the server will often assign itself a 169.x.x.x IP address. If you see a DHCP-assigned IP address starting with 169, that means DHCP failed.
This is really helpful! Is there any reason a client wouldn't renew a lease? I think that might be my issue.
There could be some low-possibility scenarios where the client did not renew. Maybe the DHCP client service was stopped (or crashed) so it did not renew. Maybe the DHCP server's usable IP addresses have all be used, and another client grabbed the IP before the original client got a chance to renew. If a lot of leases were issued at the same time (i.e. cluster startup), maybe the server was flooded with renewals and dropped some.
There are a lot of other even-less-likely things: the client being in the process of shutting down, network interface turns off, etc.
In this case, we believe the network interface was being turned off. Still not completely sure, but we should see in a couple of days. Bless the patience of the campus networking guy.
That's a pretty good explanation, and it gets into a 'bit' more depth than I would have expected, given how much you simplified it. Nice work!
Nice explanation, it's clear enough, thanks!
For an easy to implement example, try RFC 2322 - Peg DHCP
Ah, yes! And of course we can use RFC 1149 to send our packets!