Load-Balancing using HostStated

Pierre-Yves Ritschard

Introducing HostStated

Talk Overview

  • Available tools and Concepts.
  • Design.
  • Hoststatectl features.
  • Layer 3 configuration.
  • Layer 7 configuration.
  • The future.

What is Load-Balancing ?

  • Span a service across multiple machines.
  • Functions as an application gateway or proxy.
  • Related: High-Availability.

Load-Balancing tools in OpenBSD: PF

  • Provides address tables stored in kernel for fast access.
  • Provides an interface to allow the userland to modify tables.
  • Provides different kind of load-balancing methods.

Example rule: rdr from any to $public_ip port http -> <webservers>

High-Availability tools in OpenBSD: Carp and Pfsync

  • Carp provides virtual adresses.
  • Pfsync provides shared firewall states across firewalls.
  • Carp interfaces can be grouped together.
  • A Carp group can be manipulated from userland to force failover.

Types of load-balancing: Layer 3

  • Packet level.
  • Extension of NAT.

Types of load-balancing: Layer 7

  • Application level.
  • Man-in-the-middle approach.
  • Allows packet processing.

The need

  • Knowing when a service fails to stop directing requests at it.
  • A tool that is able to maintain address tables up to date with regard to host availability.
  • A tool that is able to create and remove pf rules based on the status of the hosts it monitors.
  • A tool that can provide a solution to a general failure.

Similar or related applications

  • Many vendors provide commercial solutions: F5, Nortel and Cisco for instance.
  • A few linux projects provide limited load-balancing solutions such as keepalived.

Design goals

  • Security: use privilege separation and separate the different aspects of the daemon into several processes.
  • Efficiency: create a fast and asynchronous host checking engine with various commonly used checking methods.
  • Simplicity: provide a clean and familiar looking configuration file syntax. Consistent syntax across layer 3 and layer 7, standard and SSL.
  • Administrator Friendly: provide a simple control tool to report host statuses and manually set status.

Design

Parent Process

  • Handle configuration loading and reloading.
  • Handle external script execution.
  • Handle carp demotion requests.

HCE: Host check engine

  • Mono-process, fully asynchronous checks.
  • Schedule checks and notify PFE of state transitions.

Design (cont.)

PFE: PF Engine

  • Create and destroy PF rules.
  • Update PF tables based on HCE notifications.

Relay Engine

  • Create listening sockets for services.
  • Filter protocols before relaying.

Design details

  • Steal as much as possible from recent OpenBSD daemons: ospfd, bgpd.
  • Use safe buffer routines.
  • Use the imsg protocol to communicate between the different processes.
  • Create a set of easy asynchronous SSL routines.
  • Use libevent to facilitate asynchronous socket programming.

Configuration elements

  • hosts: real service providers.
  • tables: groups of hosts providing a common service.
  • services: layer 3 load-balancing declarations.
  • protocols: protocol specific parameters for relays.
  • relays: layer 7 load-balancing declarations.

Simple layer 3 setup

  • One internet reachable host.
  • Two web servers using private addressing.

Initial Configuration

In pf.conf

rdr-anchor "hoststated/*"

In hoststated.conf

public_addr=81.81.81.81

webhost1=10.1.1.100
webhost2=10.1.1.101

table webhosts {
	real port http
	check http "/" code 200
	host $webhost1
	host $webhost2
}

service www {
	virtual host $public_addr port http interface trunk0
	table webhosts
}

Retrieving status information

# hoststatectl show summary
Type      Id    Name                            Avlblty Status
service    0    www                                     active
table      0    webhosts                                active (2 hosts up)
host       1    10.1.100.2                      100.00% up
host       0    10.1.100.1                      100.00% up

Hoststatectl features

  • Host disable/enable.
  • Service disable/enable.
  • Table disable/enable.
  • Reload.
  • Host, tables, services and relay display.
  • Summary display.
  • Monitor mode.

More table options

checking methods

  • HTTP(S) code
  • HTTP(S) digest
  • ICMP
  • send / expect
  • script
  • TCP / SSL

other options

  • real port
  • timeout
  • disable

What if every host goes down

+table sorry_server {
+       check icmp
+       host 127.0.0.1
+}
+
 service www {
         virtual host $public_addr port http interface trunk0
         table webhosts
+       backup table sorry_server
 }

result in hoststatectl's output:

Type      Id    Name                            Avlblty Status
service    0    www                                     active (using backup table)
table      0    webhosts                                empty
host       1    10.1.1.101                      0.00%   down
host       0    10.1.1.100                      0.00%   down
table      2    sorry_server:80                         active (1 hosts up)
host       3    127.0.0.1                       100.00% up

Bringing things down

Forcible disabling of hosts is useful for maintenance updates

# hoststatectl table disable webhosts:80 
command succeeded
# hoststatectl show su                   
Type      Id    Name                            Avlblty Status
service    0    www                                     active (using backup table)
table      2    webhosts:80                             disabled
table      3    sorry_server:80                         active (1 hosts up)
host       5    127.0.0.1                       100.00% up

Layer 7 Features

  • Generic TCP relaying.
  • HTTP/HTTPS relaying.
  • DNS relaying.

Generic UDP relaying can not be achieved due to the stateless datagram based nature of the protocol.

Sample HTTPS Relaying configuration

ext_addr="172.16.1.1"
webhost1="10.1.1.100"
webhost2="10.1.1.101"

table webhosts {
        check http "/" code 200
        host $webhost1
        host $webhost2
}

protocol http {
        protocol http
        header append "$REMOTE_ADDR" to "X-Forwarded-For"
        header append "$SERVER_ADDR:$SERVER_PORT" to "X-Forwarded-By"
        header change "Connection" to "close"
}

relay wwwssl {
        listen on $ext_addr port http
        protocol http

        table webhosts loadbalance
}

Relaying methods

  • Fixed forwarding.
  • Table forwarding, with either roundrobin, loadbalance or hash method.
  • Service: use address of specified service.

Available HTTP actions

  • Select connection direction (request / response).
  • Manipulate cookies and urls (in the request), headers and path.
  • Append key/value pairs.
  • Change key/value pairs.
  • Remove a key and its value.
  • Expect a value to be present in a key/value pair.
  • Filter connections which contain a key/value pair.
  • Feed a value to a table's hash.
  • Log a key/value pair.

Available variables for the HTTP protocol

  • $REMOTE_ADDR: The IP address of the connected client.
  • $REMOTE_PORT: The TCP source port of the connected client.
  • $SERVER_ADDR: The configured IP address of the relay.
  • $SERVER_PORT: The configured TCP server port of the relay.
  • $TIMEOUT: The configured session timeout of the relay.

Other protocol options

TCP

  • Backlog.
  • IP minttl, ip ttl.
  • Nodelay, sack and socket buffer.

SSL

  • Ciphers.
  • Session cache.
  • SSLv2, SSLv3, TLSv1.

Tips and Tricks

  • Load Balancing across networks.
  • Combining hoststated and carp.
  • Following development.

The Future

  • Layer 7 reloading.
  • More layer 7 protocols.
  • Conditionnal tables.
  • More reverse proxy features.
  • Layer 3 weighted hosts and other methods.

The Future (cont.)

  • GSLB: dns.
  • GSLB: bgpd.
  • Direct Server Return.
  • TCP Splicing.

Thank you

  • http://www.OpenBSD.org
  • Questions ?