WebEQ - Web Farm Equaliser
Born out of necessity due to our involvement with high profile websites,
WebEQ is a cost effective solution for managing web server farms (or even
just a single server). By
filtering unwanted web requests before they reach the real web server
WebEQ eliminates the load that web servers need to expend in handling invalid
(malicious or accidental) requests. Adding this with automated
failover/recovery, load balancing and keepalive intelligence amongst multiple web servers, WebEQ delivers
superior performance and availability of mission critical web services.
WebEQ provides similar core functionality found in solutions from the big
networking vendors costing much more.
How does it work?
WebEQ can be deployed as a separate hardware module or, as a 'bolt-on'
to your current Unix based web server. It listens to incoming requests for
web services (usually on port 80), and assesses each request for validity
(see below for details). Valid requests are then passed onto a real web
server which is listening on a restricted internal port, preferably not
viewable to the outside world, or ideally on an internal network segment. WebEQ operates at the http request level
allowing much more flexibility in defining what are valid transactions to be
honoured than packet filtering products.
Here is the process (simplified) in which every incoming request is handled (this
happens in parallel - WebEQ can handle many requests simultaneously):
- An incoming request is received. WebEQ opens the connection and waits
for the request header. If this does not happen within a few (configurable)
seconds, the request is discarded and the port closed and re-used
mechanism deals with denial of service attacks.
- The request's URL is matched against an internal access list. For best
results, this list should be configured for 'closed access' (ie. disallow
everything except known valid URL's) for this web server. Requests that are
rejected are returned (or alternatively simply discarded) as 'unknown' to
the request originator. This prevents malcious or accidental requests (eg.
'Script Kiddies', nimda, code red) from reaching the web server.
- The request is passed to the next available web server in the farm in a
transparent failover 'chain'. If a server in the chain cannot honour the
request, then the request is passed to the next web server. Failed web
servers are removed from the chain and support staff are notified.
- WebEQ captures the response from the web server, and returns this to the
WebEQ is written in multi-threaded POSIX compliant C for speed and platform portability. Request handling for
WebEQ is very economical in CPU terms compared to that of a web server. WebEQ can
still pass through valid requests successfully with no noticeable overall
performance hit during an attempted
denial-of-service attack if configured appropriately.
- Request flow management
By denying and allowing requests based on the
amount of time taken for the request to be received, WebEQ
protects against denial
of service type attacks. The characteristic of these types of attacks is
for a would-be attacker to
send forged SYN packets to web server, which responds by opening a new
connection and awaiting the actual request. These are sent in rapid
succession, causing the web server to spend resources in handling these
bogus requests. WebEQ will protect against these attacks by timing the
interval between the connection open and the request data. If the timeout is
exceeded, the connection is dropped. The intervals are configurable to
ensure site-specific requirements.
- Buffer size management
Requests are processed in small chunks to a maximum limit. This reduces the
possibility of exploitation of web server buffer overflow bugs.
- URL access list
Similar in style to a router or firewall access list, WebEQ allows the
administrator to set up a filter based on IP address and a URL pattern.
These patterns are written as unix style regular expressions. WebEQ is
shipped with some sample access list templates that can be used 'as-is' or
customised by the administrator if required. We also keep up-to-date copies
of the templates online, to cater for new methods of attack.
Performance and Fortification:
- Load balancing
In the case of multiple web servers, WebEQ can be configured to direct valid
requests to each of these on a round robin basis. This allows the load of the site
to be distributed, enabling increased overall performance, as each web
server is only accepting a share of the total requests. Servers can also be
weighted, so that some servers can also do more work than others where the
server facilities are not matched.
- Transparent failover and recovery
- Web server 'keepalive'
Modern web servers have the ability to keep client connections alive if
requested to do so. However, many browsers in use today do not support this,
which results in the web server needing to close and open new connections to
the same client. WebEQ incorporates a mechanism where it will always ask the
web server to remain open regardless, and will even re-use a currently open
connection for a completely different client. The result of this technique
is that the web server opens and closes connections far less often, and when
the site starts to become loaded will result in a significant overall
throughput increase. As WebEQ runs in a very small resource footprint,
handling the task of connection turnover is much more efficient. Of course,
WebEQ handles a browser's request to keepalive also.
Multiple WebEQ engines may be deployed for the same (or multiple) web server farm, by
configuring the DNS in a 'round robin' fashion. This causes traffic to first
of all be shared (approximately) between the available WebEQ engines which
in turn balances load amongst web servers. This architecture scales well and
adds increased availability as there is not a single point of failure (as
there would be with just one WebEQ). Here, network architectures (eg.
separate fabrics) can also be employed to increase availability even
- Web based dynamic configuration
WebEQ provides a built-in browser interface that can be used by an
administrator to configure it (or via a text configuration file).
Changes may be made while WebEQ is operating, as it includes a feature to
apply changes in a controlled fashion without affecting current sessions.
This also implies that it does not need to be restarted for changes to take
effect. WebEQ also responds to a SIGHUP signal to reconfigure itself, which
can be useful for configuring from a script or the command line.
- Centralised logging and notification
In the case of multiple web servers, managing the logs that these produce
requires some effort. WebEQ logs centrally in standard apache format, which can be fed
into third party statistics packages (eg. pwebstats/MRTG). When multiple
WebEQ engines are deployed, an internal logging synchroniser ensures central
integrity of the logs.
- Statistics gathering
WebEQ maintains a set of internal current and historical statistics on its
own performance. These can be viewed and browsed via the administration
browser interface, or imported into other packages (eg. spreadsheet or
database). The statistics are stored in plain text files.
- Virtual server support
Virtual servers are supported, and the logs may be merged or separated via a
- Secure server support
WebEQ may also be deployed to validate https (secure sockets layer)
Pilot and partner program
We are always looking to establish reference sites and construct
relationships with suitable sales partners. If your organisation is
interested in participating in running a WebEQ pilot site, we would be happy
to talk to you and provide a custom installation to you at a significantly
reduced cost. In return we would benefit from beng able to refer potential
customers to live reference sites.
As WebEQ is an Internet based product, we are able to remotely deploy and
support our customers, no matter where they are geographically. We offer
WebEQ as a standalone turnkey hardware and software package, or if you are
currently running a unix based platform, we can ship WebEQ as a software
only product. We would welcome the opportunity to perform an obligation free
analysis of your current and future installation and recommend (and quote)
suitable configuration to you, based on your specific needs. Please use our
page to get in touch.
WebEQ Hardware Engine
Supplied in an industrial strength 1RU rack mountable computer, pre-installed
with WebEQ, this option would suit organisations running either non-unix web
services or who wish to seggregate their architecture into 'black boxes'.
WebEQ Software 'bolt-on'
Suited to customers already running their own unix platforms who are
comfortable in installing and maintaining their own platforms. The software
only version is installed on the customer's own hardware, either on the same
physical machine as the web server or on a separate host.
Minimum System Requirements
(for standalone unix bolt-on)
- POSIX compliant unix operating system (eg. Solaris, Linux, HP-UX, Mac OS
- WebEQ memory footprint is approximately 2Mb
- The faster the CPU, the better, however WebEQ will run on lower spec.
machines. (eg. Sun Sparc 10, Intel Pentium II) for smaller installations.
- Approx. 500Mb disk space for software (only 100K) and log space (the rest)
development - consultancy - solutions
Solaris - Linux - Win32 - Mac OS X - HP-UX