Openstack devstack setup with VXLAN as overlay

Good News is that OVS now supports VXLAN tunnel protocol. This makes opensource Openflow Controllers equipped with power to get ready enter into new markets where Overlay networks are preferred. Here is the release notes for Openvswitch release 1.10.

I wanted to build a Openstack setup with VXLAN and try it out myself. But there was no proper documentation put together. Hope this post helps folks who want to build VXLAN overlay network based Openstack setup.

Setup:

I have two Ubuntu Servers and a Windows desktop. I used Ubuntu Servers as compute nodes , twos VMs on virtual box in Windows. One VM as a Controller and second VM as a Network node.

Since VXLAN support in OVS is in master branch, I chose to use devstack to setup Openstack.

For simplicity I will not mention second Compute Node in the config section as the settings are same as other. And each system has only one NIC.

Controller: 192.168.1.121
Compute Node: 192.168.1.112
Network Node: 192.168.1.123

localrc for Controller:

#SCHEDULER=nova.scheduler.simple.SimpleScheduler
SCHEDULER=nova.scheduler.filter_scheduler.FilterScheduler
LOGFILE=/opt/stack/data/stack.log
SCREEN_LOGDIR=/opt/stack/data/log
RECLONE=yes
#disable_service n-net, n-cpu
#enable_service q-svc, q-agt, q-l3, q-meta, q-dhcp, neutron
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,horizon,rabbit,mysql,neutron,q-svc
Q_SRV_EXTRA_OPTS=(tenant_network_type=vxlan)
Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_types=vxlan vxlan_udp_port=8472)
ENABLE_TENANT_TUNNELS=True

localrc for compute

ENABLED_SERVICES=n-cpu,rabbit,neutron,q-agt
LOGFILE=/opt/stack/data/stack.log
SCREEN_LOGDIR=/opt/stack/data/log
RECLONE=yes
# Openstack services running on controller node
SERVICE_HOST=192.168.1.121 # replace this with the IP address of the controller node
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
Q_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_types=vxlan vxlan_udp_port=8472)
Q_SRV_EXTRA_OPTS=(tenant_network_type=vxlan)
ENABLE_TENANT_TUNNELS=True 

localrc for network node

SERVICE_HOST=192.168.1.121
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
Q_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
#SCHEDULER=nova.scheduler.simple.SimpleScheduler
SCHEDULER=nova.scheduler.filter_scheduler.FilterScheduler
LOGFILE=/opt/stack/data/stack.log
SCREEN_LOGDIR=/opt/stack/data/log
RECLONE=yes
ENABLED_SERVICES=q-agt,q-l3,q-dhcp,q-meta,rabbit
Q_SRV_EXTRA_OPTS=(tenant_network_type=vxlan)
Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_types=vxlan vxlan_udp_port=8472)
ENABLE_TENANT_TUNNELS=True

Other Changes

As expected, this does not work straight forward. Openstack requires us to install OVS manually as it is not Officially released by Ubuntu.

Download OVS 1.10 from here on compute nodes and Network node. 

Here are the installation instructions

./configure --prefix=/usr --localstatedir=/var  --with-linux=/lib/modules/`uname -r`/build
make
make install
sudo rmmod openvswitch
sudo  insmod datapath/linux/openvswitch.ko
sudo mkdir -p /usr/etc/openvswitch
sudo pkill ovsdb-tool
sudo pkill ovsdb-server
sudo pkill ovs-vswitchd
sudo rm -rf /usr/etc/openvswitch/conf.db
sudo  ovsdb-tool create /usr/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
sudo ovsdb-server --remote=punix:/var/run/openvswitch/db.sock \
                     --remote=db:Open_vSwitch,manager_options \
                     --private-key=db:SSL,private_key \
                     --certificate=db:SSL,certificate \
                     --bootstrap-ca-cert=db:SSL,ca_cert \
                     --pidfile --detach

sudo ovs-vsctl --no-wait init
sudo  ovs-vswitchd --pidfile --detach
sudo ovs-vsctl add-br br-int

One more issues I faced is the VNC console. Devstack multi node scripts seems to have problem. They dont generate proper config for Compute Nodes. The listen address is set to 127.0.0.1 and few variables not set. I changed the nova.conf and restarted nova-compute.

novnc_enabled=True
novncproxy_base_url=http://192.168.1.121:6080/vnc_auto.html
xvpvncproxy_base_url=http://192.168.1.121:6081/console
novncproxy_port=6080
vncserver_proxyclient_address=192.168.1.112
vncserver_listen=0.0.0.0

Hope, you find this post useful. Let me know in comments if you need more information.