What’s inside X-Forwarded-For and how to handle

Cache, HTTP, Web June 30th, 2007

X-Forwarded-For

In one of my project, I need to handle the http header part, in X-Forwarded-For header there contain IP addresses and some names such as “unknown”, what does this mean? Is it possible to contain a domain name in this list? Do we need to run a DNS lookup (which may low down the performance) or we just need to validate the IP address? I searched across the Internet and here some information I collected.

X-Forward-For was originally invented by Squid, and become a de fact standard for most of other proxies implementation.

From Wikipedia: http://en.wikipedia.org/wiki/X-Forwarded-For

The X-Forwarded-For (XFF) HTTP header is a de facto standard for identifying the originating IP address of a client connecting to a web server through an HTTP proxy. XFF headers are supported by most proxy servers, notably Squid, Apache mod_proxy, Blue Coat ProxySG, Cisco Cache Engine, and NetApp NetCache.

In this context, the caching servers are most often those of large ISPs who either encourage or force their users to use proxy server for access to the World Wide Web, something which is often done to reduce external bandwidth through caching. In some cases, these proxy servers are transparent proxies, and the user may be unaware that they are using them.

Without the use of XFF or another similar technique, any connection through the proxy would reveal only the originating IP address of the proxy server, effectively turning the proxy server into an anonymizing service, thus making the detection and prevention of abusive accesses significantly harder than if the originating IP address was available. The usefulness of XFF depends on the proxy server truthfully reporting the original host’s IP address; for this reason, effective use of XFF requires knowledge of which proxies are trustworthy, for instance by looking them up in a whitelist of servers whose maintainers can be trusted.

The general format of the header is:

X-Forwarded-For: client1, proxy1, proxy2

From SQUID’s FAQ: (http://www.comfsm.fm/computing/squid/FAQ.html#toc4.17)

When a proxy-cache is used, a server does not see the connection coming from the originating client. Many people like to implement access controls based on the client address. To accommodate these people, Squid adds its own request header called “X-Forwarded-For” which looks like this:

X-Forwarded-For: 128.138.243.150, unknown, 192.52.106.30

Entries are always IP addresses, or the word unknown if the address could not be determined or if it has been disabled with the forwarded_for configuration option.

We must note that access controls based on this header are extremely weak and simple to fake. Anyone may hand-enter a request with any IP address whatsoever. This is perhaps the reason why client IP addresses have been omitted from the HTTP/1.1 specification.

Conclusion

By read the source code of SQUID and some apache mod, I found the implementation of this header had never try to put any name in this header tag. It could put “unknown”, but this is surely not intend to be a name.

From Squid source code: http://squid.cvs.sourceforge.net/squid/squid/src/http.c?view=markup

1212 /* append X-Forwarded-For */

1213 if (opt_forwarded_for) {

1214 strFwd = httpHeaderGetList(hdr_in, HDR_X_FORWARDED_FOR);

1215 strListAdd(&strFwd,

1216 (((orig_request->client_addr.s_addr != no_addr.s_addr) && opt_forwarded_for) ?

1217 inet_ntoa(orig_request->client_addr) : ”unknown”), ’,’);

1218 httpHeaderPutStr(hdr_out, HDR_X_FORWARDED_FOR, strBuf(strFwd));

1219 stringClean(&strFwd);

1220 }

By searched and read a bunch of source codes which process this header (http://www.google.com/codesearch?hl=zh-CN&q=+x-forwarded-for&start=20&sa=N) , none of the code I read check the IP by name lookup.

So, we can simply change the code, remove the DNS lookup and just valid the IP address by itself. This will dramatically improve the speed of this function.