Installing OpenStack, Quantum problems

During the following weeks we plan to expand more on the subject of setting up an OpenStack cloud using Quantum.
For now we have been experimenting with different Quantum functionality and settings.
At first Quantum might look like a black box, not due to its complexity, but because it deals with several different plugins and protocols that if a person is not very familiar with them it becomes hard to understand why Quantum is there in the first place.

In a nutshell Quantum has the role to provide an interface to configure the network of multiple VMs in a cluster.

In the last few years the lines between a system, network and virtualization admin have become really blury.
The classical unix admin is pretty much non existent now a days since most services are offered in the cloud in virtualized environments.
And since everything seems to be migrating over to the cloud some network principles that were applied into physical networks in the past some times don’t translate very well to virtualized networks.

Later we’ll have some posts explaining what technologies and techniques underlie the network configuration of a cloud, in our case focusing specifically on OpenStack and Quantum.

With that being said, below are a few errors that came up during the configuration of Quantum:

1. ERROR [quantum.agent.dhcp_agent] Unable to sync network state.

This is error is most likely caused due a misconfiguration of the rabbitmq server.
A few ways to debug the issue is to:
Check if the file /etc/quantum/quantum.conf in the controller node(where the quantum server is installed) has the proper rabbit credentials

By default rabbitmq runs on port 5672, so run:

netstat -an | grep 5672

and check if the rabbitmq server is up an running

On the network node(where the quantum agents are installed) also check if the /etc/quantum/quantum.conf have the proper rabbit credentials:

If you are running a multihost setup make sure the rabbit_host var points to the ip where the rabbit server is located.

Just to be safe check if you have a connection on the management networking by pinging all the hosts in the cluster and restart both the quantum and rabbitmq server as well the quantum agents.

2. ERROR [quantum.agent.l3_agent] Error running l3_nat daemon_loop

This error requires a very simple fix, however, it was very difficult to find information about the problem online.
Luckily, I found one thread on the mailing list of the fedora project explaining in more details the problem.

This is error is due to the fact that keystone authentication is not working.
A quick explanation – the l3 agent makes use of the quantum http client to interface with the quantum service.
This requires keystone authentication. If this fails then the l3 agent will not be able to communicate with the service.

To debug this problem check if the quantum server is up and running.
By default the server runs on port 9696

root@folsom-controller:/home/senecacd# netstat -an | grep 9696
tcp        0      0 0.0.0.0:9696            0.0.0.0:*               LISTEN
tcp        0      0 192.168.0.11:9696       192.168.0.12:40887      ESTABLISHED

If nothing shows up is because the quantum server is down, try restarting the service to see if the problems goes away:

quantum-server restart

You can also try to ping the quantum server from the network node(in a multihost scenario):

root@folsom-network:/home/senecacd# nmap -p 9696 192.168.0.11

Starting Nmap 5.21 ( http://nmap.org ) at 2013-01-28 08:07 PST
Nmap scan report for folsom-controller (192.168.0.11)
Host is up (0.00038s latency).
PORT     STATE SERVICE
9696/tcp open  unknown
MAC Address: 00:0C:29:0C:F0:8C (VMware)

Nmap done: 1 IP address (1 host up) scanned in 0.04 seconds

3.ERROR [quantum.agent.l3_agent] Error running l3_nat daemon_loop – rootwrap error

I didn’t come across this bug, but I found a few people running into this issue.
Kieran already wrote a good blog post explaining the problem and how to fix it

You can check the bug discussion here

4. Bad floating ip request: Cannot create floating IP and bind it to Port , since that port is owned by a different tenant.

This is just a problem of mixed credentials.
Kieran documented the solution for the issue here

There is also a post on the OpenStack wiki talking about the problem.

Conclusion

This should help fixing the problems that might arise with a Quantum installation.
If anybody knows about any other issues with Quantum or has any suggestions about the problems listed above please let us know!

Also check the official guide for other common errors and fixes

Advertisements

Bugfix: OpenStack Quantum L3 Agent Rootwrap Error

When trying to set up my Network Node (see this tutorial) my /var/log/quantum/l3_agent.log shows this error:

2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils] Running command: sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf /sbin/iptables-save -t filter
2012-10-22 09:00:48 DEBUG [quantum.agent.linux.utils]
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
Exit code: 99
Stdout: 'Unauthorized command: /sbin/iptables-save -t filtern'
Stderr: ''
2012-10-22 09:00:48 ERROR [quantum.agent.l3_agent] Error running l3_nat daemon_loop
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 170, in daemon_loop
    self.do_single_loop()
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 227, in do_single_loop
    self.process_router(ri)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 300, in process_router
    self.external_gateway_added(ri, ex_gw_port, internal_cidrs)
  File "/usr/lib/python2.7/dist-packages/quantum/agent/l3_agent.py", line 398, in external_gateway_added
    ri.iptables_manager.apply()
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py", line 282, in apply
    root_helper=self.root_helper))
  File "/usr/lib/python2.7/dist-packages/quantum/agent/linux/utils.py", line 55, in execute
    raise RuntimeError(m)
RuntimeError:
Command: ['sudo', '/usr/bin/quantum-rootwrap', '/etc/quantum/rootwrap.conf', '/sbin/iptables-save', '-t', 'filter']
Exit code: 99
Stdout: 'Unauthorized command: /sbin/iptables-save -t filtern'
Stderr: ''

This error has been well documented, but there hasn’t been a step by step guide to fixing it.

Luckily, there really is only one step!

Step 1: Edit quantum/agent/linux/iptables_manager.py

The problem is that the command that causes the error, sudo /usr/bin/quantum-rootwrap /etc/quantum/rootwrap.conf /sbin/iptables-save -t filter, cannot be an absolute path or the rootwrap won’t work. Specifically, /sbin/iptables-save -t filter cannot be absolute. For more details on the nature of the issue, check the bug report here. In any case, it’s a simple fix.

Change line 272 of /usr/lib/python2.7/dist-packages/quantum/agent/linux/iptables_manager.py from:

s = [('/sbin/iptables', self.ipv4)]

to

s = [('iptables', self.ipv4)]

And that’s it!