DEV Community

loading...
Cover image for Why all developers should learn how to perform basic network troubleshooting

Why all developers should learn how to perform basic network troubleshooting

memattchung
Former Amazonian who helped built the AWS Cloud. Now solo-entrepeneur. @memattchung
Originally published at blog.mattchung.me ・2 min read

Regardless of whether you work on the front-end or back-end, I think all developers should gain some proficiency in network troubleshooting. This is especially true if you find yourself gravitating towards lower level systems programming.

The ability to troubleshoot the network and systems separates good developers from great developers. Great developers understand not just code abstraction, but understand the TCP/IP model:

OSI and TCP/IP model

Source: https://www.guru99.com/tcp-ip-model.html

Some basic network troubleshooting skills

If you are just getting into networking, here are some basic tools you should add to your toolbelt:

  • Perform a DNS query (e.g. dig or nslookup command)
  • Send an ICMP echo request to test end to end IP connectivity (i.e. ping command)
  • Analyze the various network hops (i.e. traceroute X.X.X.X)
  • Check whether you can establish a TCP socket connection (e.g. telnet X.X.X.X [port])
  • Test application layer (i.e. curl https://somedomain)
  • Perform a packet capture (e.g. tcpdump -i any) and what bits are sent on the wire

What IP address is my browser connecting to?

% dig dev.to

; <<>> DiG 9.10.6 <<>> dev.to
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 39029
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;dev.to.                IN  A

;; ANSWER SECTION:
dev.to.         268 IN  A   151.101.2.217
dev.to.         268 IN  A   151.101.66.217
dev.to.         268 IN  A   151.101.130.217
dev.to.         268 IN  A   151.101.194.217
Enter fullscreen mode Exit fullscreen mode

Is the web server listening on the HTTP port?

% telnet 151.101.2.217 443
Trying 151.101.2.217...
Connected to 151.101.2.217.
Escape character is '^]'.
Enter fullscreen mode Exit fullscreen mode

Each of the above tools helps you isolate connectivity issues. For example, if your client receives an HTTP 5XX error, you can immediately rule out any TCP level issue. That is, you don't need to use telnet to check whether there's a firewall issue or whether the server is listening in on the right socket: the server already sent an application level response.

Summary

Learning more about the network stack helps you quickly pinpoint and isolate problems:

  • Is it my client-side application?
  • Is it a firewall blocking certain ports?
  • Is there a transient issue on the network?
  • Is the server up and running?

Let's chat more about network engineering and software development

If you are curious about learning how to move from front-end to back-end development, or from back-end development to low level systems programming, hit me up on Twitter: @memattchung

Discussion (14)

Collapse
aghost7 profile image
Jonathan Boudreau

Personally I prefer tcpflow over tcpdump since it provides colouring which can make things easier to view. This is especially useful on systems where you can't easily transfer files (ie, the packet capture file) from.

In my experience ICMP is usually blocked. nmap can instead be used to test connectivity on the relevant port with a syn scan.

Collapse
memattchung profile image
memattchung Author

Cool, I never heard of tcpflow. However, it appears this tool only handles tcp, not other transport protocols like udp, right?

Also, regarding icmp, although it is often blocked, you can modify the encapsulation so instead of ip->icmp, you can modify it such that it rides on udp.

Collapse
aghost7 profile image
Jonathan Boudreau

Cool, I never heard of tcpflow. However, it appears this tool only handles tcp, not other transport protocols like udp, right?

Yea, in that case you should use tcpdump or something else. I just like the colouring.

Also, regarding icmp, although it is often blocked, you can modify the encapsulation so instead of ip->icmp, you can modify it such that it rides on udp.

Only what is absolutely necessary is let through by the firewall in my past experience. This includes disabling inbound UDP if the application does not need it. SYN scans work because you can poke the ports being used by the application.

Collapse
nitinkatageri profile image
Niitn Katageri

Thanks Jonathan, will use tcpflow next time I deal with the packets and the directions.

Collapse
nitinkatageri profile image
Niitn Katageri

Completely agree, developers not having a clue about how different components of the system talk to eachother is really sad.

During college days I had inclination towards system admin but ended up on the programming side of things, but my inclination has helped me triage issues in development much faster.

Recently got into tcpdumps and port monitoring to analyse why a webservice request to one of engines was randomly erroring with connection reset.

Collapse
memattchung profile image
memattchung Author

Agree. The best developers I've worked with weren't just great at programming, but also intimately aware of the underlying system too.

webservice request to one of engines was randomly erroring with connection reset

Another great tool you might want to use is strace, assuming your application is running on linux.

Collapse
nitinkatageri profile image
Niitn Katageri

Sure will try it out. Thanks. Yes it runs on Linux.

Collapse
jonasjohnson profile image
Jonas Johnson

This is so true, I just ran into this on a project I was working on and didn’t realize my isp was auto resolving sites for me when the weren’t giving me all the wrong results. Thanks for sharing this!

Collapse
memattchung profile image
memattchung Author

Interesting ! Can you elaborate on this — was your DNS pointing at your ISP's resolver?

Collapse
jonasjohnson profile image
Jonas Johnson

I searched for something and the dns responded, even though the site didn’t exist. I then called the non existent thing and it came back with a 200 response. You can probably tell I’m very new to this and so I can’t elaborate much. My mentor caught it because it just didn’t make any sense.

Thread Thread
memattchung profile image
memattchung Author

I think I have a better understanding. Likely the DNS record resolved returned an IP address pointed at a different address. Despite hitting that IP and receiving an HTTP 200 OK, the web site does not exist. Well, that's my guess at least.

Thread Thread
jonasjohnson profile image
Jonas Johnson

Yeah that sounds about right.

Collapse
funkysi1701 profile image
Simon Foster

As a developer that started his career as a sysadmin I agree with this article. Having a basic understanding of DNS and other network stuff has certainly been very useful to me.

Collapse
memattchung profile image
memattchung Author

Hey Simon,

I had a similar career trajectory, also starting off with systems administration. It definitely goes both ways: great sysadmins know some elements of programming (e.g. using a debugger, analyzing a core dump) and great programmers know some systems/network administration (e.g. checking for host-to-host connectivity, inspecting packet payloads). In this context, the 80/20 rule — 20% of activities contribute to 80% of results — rings true.

These days, what are you working on?