Openstack and Virtual Private Cloud

A virtual Private Cloud (VPC) is a virtual Cloud over the cloud. The resources and infrastructure is shared with other VPC users but they feel as if the resources are private to them.

As per wikipedia, the resources are allocated among the VPC users like VLAN or a set of encrypted communication channels etc.,

Is Project equivalent to VPC?

IMO, its  equivalent in a limited way. Project/Tenant Owner can create network resources specific to them. But the following are limitations

• Cannot manage other users within VPC
• Cannot isolate user resources. They are all shared across other users of that Project

In addition to the above - Ask the following questions and we get a solution

Why would any enterprise want to share private user list with Public Cloud Service Provider!!!?

Why would any Cloud Service Provider want to manage minor administrative things of a VPC user??

It must be one time effort for a Cloud Service Provider. Create a VPC, allocate resources and create a VPC Admin and hand off.

What is needed?

Keystone in Openstack need some changes. A new level of admin user for VPC must be created.

Public cloud is managed by Service Provider(SP) Admin. 

As a SP Admin, he can create new VPCs, manage VPC Admins, allocate resources for it.

As a VPC Admin, he can create new projects with in VPC, Manage resources allocated to the VPC and manage users of VPC.

Thus, a VPC would have full set of Cloud features under the control of VPC Admin. And SP Admin need not  intervene into management of VPC Cloud

Keystone needs to define VPC Admin as 

• Admin to all resources with in VPC
• Admin who can manage users with in VPC
• Admin who can manage(create/delete/update) new projects in the VPC

All openstack Services must create new set of policies for the VPC Admin.

Comments

I welcome comments and suggestions to define VPC model for Openstack. 

Openstack RabbitMQ issues and solution

Symptoms

In our deployment, we observed that when ever there is a network interruption, any Openstack operation that requires MQs get struck for long time. For example, VM creation will be struck in "Build State"

Observations

From our observations, it took 12 to 18 minutes to recover from that state. We could see lot of messages unacknowledged and Ready state.
Also, we used to see lot of Consumers per Queue. It means there are lot of TCP connections from the consumer to the RabbitMQ broker which does not make any sense.

We did check the TCP network connections on Rabbit Server and there are indeed several of them in ESTABLISHED state while on the consumer(say in compute) there is only one network connection.

It means problematic connections were closed on the consumer  but those connections are still there in rabbit server. The health check between rabbit client and server implementations is not implemented inOpenstack code.

This state would recover but takes lot of time depending on number of consumers( for example many number of nova-computes)

Solution

We introduced load balancer and placed rabbit servers behind the Virtual service. Load Balancer implements a sort of proxy where it maintains states on each side of connection. When ever there is a problem on client side (Say) it closes connection on the server end. 

With this in place, network interruptions like switch reboot etc., did not have any affect on our Openstack deployment.

We configured load balancer to have idle inactive timeout of 90 seconds as our periodic updates from compute happens every 60 seconds. Thus, we do not close our rabbit connections un-necessarily.

Update:

There are other advantages with loadbalancer. The distribution of load from Openstack rabbit client is not so good. It takes in a list of rabbit servers and picks the first active rabbit server blindly. It does not really understand the actual load of rabbit servers. With LB in place, we can distribute the consumers on all rabbit servers.

It is from our observations, this indeed improved overall performance of Openstack with this deployment.

My notes for Git

This is my random notes to bookmark git commands for future reference.

Push/Pull a local repository to a remote git location

Lets take an example. I have my local repo pointing to openstack/neutron.git. I made some changes in my local repo and want to take backup before I send the code for review. I just need to add 'remote' location and push the code there.

git remote add githubbackup <url>

You can verify by looking into .git/config file if remote is added as expected.

Now push the code from master 

git push githubbackup master

If you want to pull the code from actual repo

git pull origin master

Similarly to pull from githubbackup 

git pull githubbackup master

For generating the patches from the topmost commits from a specific sha1 hash:

git format-patch -<n> <SHA1>

The last 10 patches from head in a single patch file:

git format-patch -10 HEAD --stdout > 0001-last-10-commits.patch

Openstack devstack setup with VXLAN as overlay

Good News is that OVS now supports VXLAN tunnel protocol. This makes opensource Openflow Controllers equipped with power to get ready enter into new markets where Overlay networks are preferred. Here is the release notes for Openvswitch release 1.10.

I wanted to build a Openstack setup with VXLAN and try it out myself. But there was no proper documentation put together. Hope this post helps folks who want to build VXLAN overlay network based Openstack setup.

Setup:

I have two Ubuntu Servers and a Windows desktop. I used Ubuntu Servers as compute nodes , twos VMs on virtual box in Windows. One VM as a Controller and second VM as a Network node.

Since VXLAN support in OVS is in master branch, I chose to use devstack to setup Openstack.

For simplicity I will not mention second Compute Node in the config section as the settings are same as other. And each system has only one NIC.

Controller: 192.168.1.121
Compute Node: 192.168.1.112
Network Node: 192.168.1.123

localrc for Controller:

#SCHEDULER=nova.scheduler.simple.SimpleScheduler
SCHEDULER=nova.scheduler.filter_scheduler.FilterScheduler
LOGFILE=/opt/stack/data/stack.log
SCREEN_LOGDIR=/opt/stack/data/log
RECLONE=yes
#disable_service n-net, n-cpu
#enable_service q-svc, q-agt, q-l3, q-meta, q-dhcp, neutron
ENABLED_SERVICES=g-api,g-reg,key,n-api,n-crt,n-obj,n-cond,cinder,c-sch,c-api,c-vol,n-sch,n-novnc,n-xvnc,n-cauth,horizon,rabbit,mysql,neutron,q-svc
Q_SRV_EXTRA_OPTS=(tenant_network_type=vxlan)
Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_types=vxlan vxlan_udp_port=8472)
ENABLE_TENANT_TUNNELS=True

localrc for compute

ENABLED_SERVICES=n-cpu,rabbit,neutron,q-agt
LOGFILE=/opt/stack/data/stack.log
SCREEN_LOGDIR=/opt/stack/data/log
RECLONE=yes
# Openstack services running on controller node
SERVICE_HOST=192.168.1.121 # replace this with the IP address of the controller node
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
Q_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_types=vxlan vxlan_udp_port=8472)
Q_SRV_EXTRA_OPTS=(tenant_network_type=vxlan)
ENABLE_TENANT_TUNNELS=True 

localrc for network node

SERVICE_HOST=192.168.1.121
MYSQL_HOST=$SERVICE_HOST
RABBIT_HOST=$SERVICE_HOST
Q_HOST=$SERVICE_HOST
GLANCE_HOSTPORT=$SERVICE_HOST:9292
#SCHEDULER=nova.scheduler.simple.SimpleScheduler
SCHEDULER=nova.scheduler.filter_scheduler.FilterScheduler
LOGFILE=/opt/stack/data/stack.log
SCREEN_LOGDIR=/opt/stack/data/log
RECLONE=yes
ENABLED_SERVICES=q-agt,q-l3,q-dhcp,q-meta,rabbit
Q_SRV_EXTRA_OPTS=(tenant_network_type=vxlan)
Q_AGENT_EXTRA_AGENT_OPTS=(tunnel_types=vxlan vxlan_udp_port=8472)
ENABLE_TENANT_TUNNELS=True

Other Changes

As expected, this does not work straight forward. Openstack requires us to install OVS manually as it is not Officially released by Ubuntu.

Download OVS 1.10 from here on compute nodes and Network node. 

Here are the installation instructions

./configure --prefix=/usr --localstatedir=/var  --with-linux=/lib/modules/`uname -r`/build
make
make install
sudo rmmod openvswitch
sudo  insmod datapath/linux/openvswitch.ko
sudo mkdir -p /usr/etc/openvswitch
sudo pkill ovsdb-tool
sudo pkill ovsdb-server
sudo pkill ovs-vswitchd
sudo rm -rf /usr/etc/openvswitch/conf.db
sudo  ovsdb-tool create /usr/etc/openvswitch/conf.db vswitchd/vswitch.ovsschema
sudo ovsdb-server --remote=punix:/var/run/openvswitch/db.sock \
                     --remote=db:Open_vSwitch,manager_options \
                     --private-key=db:SSL,private_key \
                     --certificate=db:SSL,certificate \
                     --bootstrap-ca-cert=db:SSL,ca_cert \
                     --pidfile --detach

sudo ovs-vsctl --no-wait init
sudo  ovs-vswitchd --pidfile --detach
sudo ovs-vsctl add-br br-int

One more issues I faced is the VNC console. Devstack multi node scripts seems to have problem. They dont generate proper config for Compute Nodes. The listen address is set to 127.0.0.1 and few variables not set. I changed the nova.conf and restarted nova-compute.

novnc_enabled=True
novncproxy_base_url=http://192.168.1.121:6080/vnc_auto.html
xvpvncproxy_base_url=http://192.168.1.121:6081/console
novncproxy_port=6080
vncserver_proxyclient_address=192.168.1.112
vncserver_listen=0.0.0.0

Hope, you find this post useful. Let me know in comments if you need more information.