Openstack and Virtual Private Cloud

A virtual Private Cloud (VPC) is a virtual Cloud over the cloud. The resources and infrastructure is shared with other VPC users but they feel as if the resources are private to them.

As per wikipedia, the resources are allocated among the VPC users like VLAN or a set of encrypted communication channels etc.,

Is Project equivalent to VPC?

IMO, its  equivalent in a limited way. Project/Tenant Owner can create network resources specific to them. But the following are limitations

• Cannot manage other users within VPC
• Cannot isolate user resources. They are all shared across other users of that Project

In addition to the above - Ask the following questions and we get a solution

Why would any enterprise want to share private user list with Public Cloud Service Provider!!!?

Why would any Cloud Service Provider want to manage minor administrative things of a VPC user??

It must be one time effort for a Cloud Service Provider. Create a VPC, allocate resources and create a VPC Admin and hand off.

What is needed?

Keystone in Openstack need some changes. A new level of admin user for VPC must be created.

Public cloud is managed by Service Provider(SP) Admin. 

As a SP Admin, he can create new VPCs, manage VPC Admins, allocate resources for it.

As a VPC Admin, he can create new projects with in VPC, Manage resources allocated to the VPC and manage users of VPC.

Thus, a VPC would have full set of Cloud features under the control of VPC Admin. And SP Admin need not  intervene into management of VPC Cloud

Keystone needs to define VPC Admin as 

• Admin to all resources with in VPC
• Admin who can manage users with in VPC
• Admin who can manage(create/delete/update) new projects in the VPC

All openstack Services must create new set of policies for the VPC Admin.

Comments

I welcome comments and suggestions to define VPC model for Openstack. 

Openstack RabbitMQ issues and solution

Symptoms

In our deployment, we observed that when ever there is a network interruption, any Openstack operation that requires MQs get struck for long time. For example, VM creation will be struck in "Build State"

Observations

From our observations, it took 12 to 18 minutes to recover from that state. We could see lot of messages unacknowledged and Ready state.
Also, we used to see lot of Consumers per Queue. It means there are lot of TCP connections from the consumer to the RabbitMQ broker which does not make any sense.

We did check the TCP network connections on Rabbit Server and there are indeed several of them in ESTABLISHED state while on the consumer(say in compute) there is only one network connection.

It means problematic connections were closed on the consumer  but those connections are still there in rabbit server. The health check between rabbit client and server implementations is not implemented inOpenstack code.

This state would recover but takes lot of time depending on number of consumers( for example many number of nova-computes)

Solution

We introduced load balancer and placed rabbit servers behind the Virtual service. Load Balancer implements a sort of proxy where it maintains states on each side of connection. When ever there is a problem on client side (Say) it closes connection on the server end. 

With this in place, network interruptions like switch reboot etc., did not have any affect on our Openstack deployment.

We configured load balancer to have idle inactive timeout of 90 seconds as our periodic updates from compute happens every 60 seconds. Thus, we do not close our rabbit connections un-necessarily.

Update:

There are other advantages with loadbalancer. The distribution of load from Openstack rabbit client is not so good. It takes in a list of rabbit servers and picks the first active rabbit server blindly. It does not really understand the actual load of rabbit servers. With LB in place, we can distribute the consumers on all rabbit servers.

It is from our observations, this indeed improved overall performance of Openstack with this deployment.