I wanted to build a Openstack setup with VXLAN and try it out myself. But there was no proper documentation put together. Hope this post helps folks who want to build VXLAN overlay network based Openstack setup.
Setup:
I have two Ubuntu Servers and a Windows desktop. I used Ubuntu Servers as compute nodes , twos VMs on virtual box in Windows. One VM as a Controller and second VM as a Network node.
Since VXLAN support in OVS is in master branch, I chose to use devstack to setup Openstack.
For simplicity I will not mention second Compute Node in the config section as the settings are same as other. And each system has only one NIC.
Controller: 192.168.1.121
Compute Node: 192.168.1.112
Network Node: 192.168.1.123
localrc for Controller:
#SCHEDULER=nova.scheduler.simple.SimpleScheduler
SCHEDULER=nova.scheduler.filter_scheduler.FilterScheduler
LOGFILE=/opt/stack/data/stack.log
SCREEN_LOGDIR=/opt/stack/data/log
RECLONE=yes
#disable_service n-net, n-cpu
#enable_service q-svc, q-agt, q-l3, q-meta, q-dhcp, neutron
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,horizon,rabbit,mysql,neutron,q-svc
Q_SRV_EXTRA_OPTS=(tenant_network_type=vxlan)
Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_types=vxlan vxlan_udp_port=8472)
ENABLE_TENANT_TUNNELS=True
localrc for compute
ENABLED_SERVICES=n-cpu,rabbit,neutron,q-agt
LOGFILE=/opt/stack/data/stack.log
SCREEN_LOGDIR=/opt/stack/data/log
RECLONE=yes
# Openstack services running on controller node
SERVICE_HOST=192.168.1.121 # replace this with the IP address of the controller node
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
Q_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_types=vxlan vxlan_udp_port=8472)
Q_SRV_EXTRA_OPTS=(tenant_network_type=vxlan)
ENABLE_TENANT_TUNNELS=True
localrc for network node
SERVICE_HOST=192.168.1.121
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
Q_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
#SCHEDULER=nova.scheduler.simple.SimpleScheduler
SCHEDULER=nova.scheduler.filter_scheduler.FilterScheduler
LOGFILE=/opt/stack/data/stack.log
SCREEN_LOGDIR=/opt/stack/data/log
RECLONE=yes
ENABLED_SERVICES=q-agt,q-l3,q-dhcp,q-meta,rabbit
Q_SRV_EXTRA_OPTS=(tenant_network_type=vxlan)
Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_types=vxlan vxlan_udp_port=8472)
ENABLE_TENANT_TUNNELS=True
Other Changes
As expected, this does not work straight forward. Openstack requires us to install OVS manually as it is not Officially released by Ubuntu.
Download OVS 1.10 from here on compute nodes and Network node.
Here are the installation instructions
./configure --prefix=/usr --localstatedir=/var --with-linux=/lib/modules/`uname -r`/build
make
make install
sudo rmmod openvswitch
sudo insmod datapath/linux/openvswitch.ko
sudo mkdir -p /usr/etc/openvswitch
sudo pkill ovsdb-tool
sudo pkill ovsdb-server
sudo pkill ovs-vswitchd
sudo rm -rf /usr/etc/openvswitch/conf.db
sudo ovsdb-tool create /usr/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
sudo ovsdb-server --remote=punix:/var/run/openvswitch/db.sock \
--remote=db:Open_vSwitch,manager_options \
--private-key=db:SSL,private_key \
--certificate=db:SSL,certificate \
--bootstrap-ca-cert=db:SSL,ca_cert \
--pidfile --detach
sudo ovs-vsctl --no-wait init
sudo ovs-vswitchd --pidfile --detach
sudo ovs-vsctl add-br br-int
One more issues I faced is the VNC console. Devstack multi node scripts seems to have problem. They dont generate proper config for Compute Nodes. The listen address is set to 127.0.0.1 and few variables not set. I changed the nova.conf and restarted nova-compute.
novnc_enabled=True novncproxy_base_url=http://192.168.1.121:6080/vnc_auto.html xvpvncproxy_base_url=http://192.168.1.121:6081/console novncproxy_port=6080 vncserver_proxyclient_address=192.168.1.112 vncserver_listen=0.0.0.0
Hope, you find this post useful. Let me know in comments if you need more information.
