Real World Example: Why can’t I talk to my blackbox?

You are probably getting tired of hearing about TCPdump, but I just can’t get over how handy it is.

After a recent upgrade to one of our simulator systems, we needed to re-align the projectors. This re-alignment is quite complicated, where you have multiple projectors overlayed on a curved screen which reflects off of a curved mirror to point all of the image to the focal point of the pilot who is driving in the simulated world. Fortunately for us, we have a computer program that does the heavy math for “warping” the image on the projectors to make them line up on the curved surface. To do that, however, we first have to turn on a series of LED lights to guide the projector and camera system to the edges of the screen. Here is where we ran into a problem.

The LED lights are controlled by a simple blackbox controller with the IP address of 192.168.2.65, which you can open in a browser on several hosts to reach the web gui and turn on and off the various LEDs. When we tried to connect to the web gui, our connection would time out, and we were unable to communicate with the box.

Unfortunately, I did not tee it to a log, so I don’t have the log example to paste here, but here is what we did.

First we tried to ping it:

$su
#ping 192.168.2.65

All of the pings whent through. However, the pings had unusually high return times. Something was creating a problem. We then ran two different TCPdump instances.

#tcpdump -i eth0 |grep 192.168.2.65

There are better ways to do this, but it was the quick and dirty way to see who the black box was talking to, and who was talking to the black box. As it turns out, the Image Generation system (192.168.2.14) was incredibly busy talking to the blackbox. What we found was that the IG system was set to talk to all of the projectors and cameras on the 192.168.2.50+ addresses and was constantly polling the projectors for status updates and consistently sending the projectors a time stamped request to display the image, a sort of lock to keep them in sync. Because the blackbox was above 50, it was also receiving this data, but it was useless to the blackbox. Unfortunately, the blackbox was not able to keep up with the constant amount of useless traffic, and would falter on loading the gui because it was too busy processing the other traffic.

Since we are not allowed to change the software settings, we simply changed our procedure to first stop the IG system, which was not needed for projector alignment. Problem was solved, and we were once again able to communicate with the blackbox. Just another easy real world example of using TCPdump with Linux.

Linux – keep it simple.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s