In part 1 I talked about the files that are required in the base of your new role repository. I'll go into information on setting up the required directories that are not specific to Ansible itself.

/doc directory

This directory contains all the files required to build the Sphinx documentation. In terms of a base there isn't too much - so depending on the role you can fill in more details.

Makefile - This is the Sphinx Makefile. You can copy it over from the Sphinx repo and or another OSA role.

requirements.txt - The pip requirements for building the docs, I'd recommend just copying this from an existing role. The OpenStack requirements project handles automatic requirements updates, so once your new role is official it will receive updates to the requirements file automatically.

source/conf.py - The documentation configuration file, a sample can be seen here. It's easier to copy this from an existing role - although the file is quite well commented so take a look inside. There is a reference to the role name that you should fix up.

source/index.rst - The index file. This is specified inside your conf.py, but if you copied it over the default is index. This file defines the initial index when going to the role documentation. You can put a placeholder in there, but writing your docs whilst creating the role is super important! Take a look at some other roles for examples on how to structure that.

/examples directory

One standard that exists across the majority of roles in OSA is the examples directory with an example playbook. This can be pretty simple, and isn't technically required. Additionally, make sure to include any files you do create in the examples directory into your doc/source/index.rst file. This is a good way to document what a playbook should look like to include the new role.

/meta directory

This is an Ansible directory, but the setup of this is more scaffolding, and is done in a pretty common way in OSA. These are pretty standard, so copy over the meta/main.yml and meta/openstack-ansible.yml from an existing mature role. The fields are pretty self explanatory in both, but you will probably want to change the supported platforms:, categories: at a minimum, inside meta/main.yml, as well as any descriptions. The meta/openstack-ansible.yml file is used to define maturity and role creation time, so adjust those fields to fit. There is documentation around that process here, so you can pick the right status, and understand that process.

/releasenotes directory

Release notes are another arrow in the documentation quiver (yes even I cringed after that sentence). The release notes use the reno tool to allow projects to create release notes to track changes that would be important to operators and deployers - alerting them of changes, new features, upgrade concerns and deprecations (amongst others).

There are 2 sub directories, the notes directory, you can more or less ignore this for now, although creating the directory is useful. This directory contains the individual release notes that you would add as you add features.

The source directory contains the configuration bits for the release notes. Inside there are several files that are required:

• conf.py - this is another Sphinx conf.py file - I'd recommend copying this over from an existing project, but again this is quite well commented.
• index.rst - the index file, you can again copy from an existing project, but make sure to adjust the included files removing any releases that don't exist - that would be all of them except unreleased at this point.
• unreleased.rst - the index file for current branches release notes. Copy this over, its a pretty minimal file though.
• _static/.placeholder & _templates/.placeholder - These are placeholder files that are blank, you can touch these to create them.

That's more or less it! We still need to cover testing and then create the role but now we should be pretty much ready to rumble, and start adding content to the documentation, and tasks/handlers/templates/default vars etc to the Ansible role itself!

The number of new roles in OSA has dropped over time, as roles are added and the number of new projects being created within the OpenStack ecosystem slows up. This has meant the need to create new roles has dropped quite a bit. That said, I've been working through the process to add a role for the Manila project. I'll walk through the process, starting with the first step, setting up the base repository.

Setting up your repository

There are a few files in the base directory that are required for various reasons, and can easily be copied from the existing repositories that are up to date. I picked the os_cinder role, but it's a good idea to pick a role that is well maintained and use that as a base. Always keep in mind, the review process should help you catch anything you miss out - reviews are great, and while it doesn't mean you can be careless, it does mean that you're not alone!

Some base files

There are a few files that are common to all repos and you can pretty much just copy and edit.

.gitignore - this is common list of files that can easily be ignored by git, for example the testing repository that gets cloned during testing. You don't need to change anything there.

.gitreview - this file handles the git-review location, change the project field to match your new role's path.

CONTRIBUTING.rst - Contribution guidelines, this has a few references to the project in the title - other than that this is the same for each OSA role.

LICENSE - All OpenStack projects use the Apache 2.0 License, so this would be the same across all OSA roles.

README.rst - The README file really just links to the documentation for the role. Given this will be a new role it won't exist yet - but documentation is super important (I'll be covering that later!), in this case I'd edit the paths to match the new role, the links won't work at first but once your documentation is created it should.

bindep.txt - This file handles the packages that will get installed during gating. As a start copy the existing bindep.txt from another project. NB This is not packages that are required by the role specifically, but more packages that are required for gating to work successfully, prior to Ansible running. If the package is required by the role itself, it should be installed as part of the role.

setup.cfg setup.py - These handle the sphinx doc builds, as well as pbr and a few other things. You can copy these over directly. setup.cfg containers a few references to the role name, which should be changed.

tox.ini - This file determines how testing is setup and run, and which scenarios exist. This is a bit trickier because you don't want to copy all the existing scenarios. I would remove any scenarios (listed inside [ ]) that call the test-ansible-functional.sh script, apart from the base [functional] job, as well as an [upgrade] job if it exists. This will remove any upgrade tests, as well as any additional scenarios (for example, the Keystone role carries a test to use Apache with uWSGI instead of the default NGINX with uWSGI), but won't remove any lint, pep8 or other generally good style tests. You can read more about tox here.

That's it for the main files in the base repository.
I'll cover over the next 4 main directories involved in a base repository in the following post.
Namely:

• doc directory
• examples directory
• meta directory
• releasenotes directory

Aside from that, the only other non-ansible bits are the zuul.d and tests directories which we will discuss later.

Recently in ceph-ansible I've been playing around with adding an entry to an existing line, in a configuration file that isn't managed by Ansible.

In general you would want to use the template module to manage a configuration file, which would mean you can completely control the contents of the file.

This isn't suitable for some configuration files, for example /etc/ssh/sshd_config, for which you don't want to carry a template file that can get out of date, and for which you generally don't want to make too many changes, and use the package defaults as set out by your distribution.

A simple example

There are some easy options, for example the PermitEmptyPasswords setting in /etc/ssh/sshd_config can only have 1 value, and be specified once.

You could adjust this by doing the following:

- name: Adjust PermitEmptyPasswords setting in /etc/ssh/sshd_config
  lineinfile:
    dest: /etc/ssh/sshd_config
    regexp: "^PermitEmptyPasswords"
    line: "PermitEmptyPasswords no"

This will look for a line starting with PermitEmptyPasswords and replace it with PermitEmtpyPasswords no.

Note that this would add a new line if #PermitEmptyPasswords existed as a line, since it's commented out and the line doesn't start with PermitEmptyPasswords.

We could adjust this to include looking for lines starting with a # by doing the following:

- name: Adjust PermitEmptyPasswords setting in /etc/ssh/sshd_config
  lineinfile:
    dest: /etc/ssh/sshd_config
    regexp: "^(#)?PermitEmptyPasswords"
    line: "PermitEmptyPasswords no"

Using Regular Expressions

At this point it gets a bit more complicated unless you are familiar with Regular Expressions! Ansible uses Python Regular Expresions and you can find out more by clicking the link.

The above regular expression says the line will match if the it starts with (^) 0 or 1 repetitions (?) of the # character, followed by PermitEmptyPasswords. In other words, if it starts with a # followed by PermitEmptyPasswords or just PermitEmptyPasswords without a #.

If that regular expression is matched, replace the line with PermitEmptyPasswords no. If the regular expression isn't matched the entry will be added to the end of the file.

If the line is exactly the same as what you are trying to add (PermitEmptyPasswords no) then the line won't be added because the lineinfile Ansible module will see it as no change.

How about multiple entries of the same value?

This is actually even easier! Since we don't care if other values of the setting have been specified, we can just ensure the line exists, for example:

- name: Add HostKey value to /etc/ssh/sshd_config
  lineinfile:
    dest: /etc/ssh/sshd_config
    line: "HostKey /path/to/my/host.key"

This will add the line entry if it doesn't exist.

This has a downside, your /etc/ssh/sshd_config will have a weird ordering, and you could get options all over the show, that aren't easy to view. We can fix that though, using the insertbefore or insertafter setting.

- name: Add HostKey value to /etc/ssh/sshd_config in order
  lineinfile:
    dest: /etc/ssh/sshd_config
    line: "HostKey /path/to/my/host.key"
    insertafter: "^(#)?HostKey"

You'll recognize the regular expression from the last section, in this case we are using it with the insertafter setting, which will mean our HostKey line is inserted into the file after the last entry that matches the regular expression. If the regular expression isn't matched it'll be added at the end of the file.

How about a setting that takes multiple values?

This is a bit trickier, and the crux of the article. I was playing around with it, and it occurred to me that this isn't that straight forward, especially if you are just starting out.

If you have a setting that takes multiple values in one line, for example the PRUNEPATHS setting in /etc/updatedb.conf, which will contain a list of paths that are to be skipped when running updatedb. When we update this line we don't want to replace the whole line, but what we want to do is ensure a specific entry is there, and update the line to include the entry if not (but not remove any existing entries!).

This is a bit trickier but can be done in 2 ways, using either the lineinfile module as before, or the replace module.

- name: add /var/lib/ceph to PRUNEPATHS in /etc/updatedb.conf
  replace:
    dest: /etc/updatedb.conf
    regexp: '^(PRUNEPATHS(?!.*/var/lib/ceph).*)"$'
    replace: '\1 /var/lib/ceph"'

The regular expression here is a bit more complex, but what it says is, look for a line that starts with(^) PRUNEPATHS, and isn't followed by (?!) any character (.) any number of times (*), followed by /var/lib/ceph, but is followed by any character (.) any number of times (*), followed by a " character and the end of line ($).

This means if /var/lib/ceph exists in the PRUNEPATHS line we won't replace the line, but if PRUNEPATHS exists and doesn't have /var/lib/ceph in it, it will match and initiate the replace.

The replace line says to take the matched text (\1) and add /var/lib/ceph" to it, and it's that simple!

Ok great, but what are the parenthesis for?

So what are all the parenthesis ( and ) for, and is their placement important?

The parenthesis denote what will be matched - this is really important, in this example we don't want to match on the final " otherwise we will end up with /some/path" /var/lib/ceph" at the end of the line, which would cause problems.

So that means the line starts with (^) and begins a match (() of PRUNEPATHS that is NOT followed (?!) by another match (() that consists of any character (.) any number of times (*), followed by /var/lib/ceph, which is the end of the match ()) for what can't follow PRUNEPATHS. The initial match, which started with PRUNEPATHS will match if any character (.) appears any number of times (*), which ends the text to be matched ()), and the line ends ($) with a " character, which isn't included in the match.

We are using the parenthesis to manipulate what is and isn't matched by the regular expression.

If we use the lineinfile module to do that there is one extra thing we need to do, and that's enable backrefs, as follows:

- name: add /var/lib/ceph to PRUNEPATHS in /etc/updatedb.conf using lineinfile
  lineinfile:
    dest: /etc/updatedb.conf
    regexp: '^(PRUNEPATHS(?!.*/var/lib/ceph).*)"$'
    line: '\1 /var/lib/ceph"'
    backrefs: true

Conclusion

That's pretty much it! I personally like replace, since it has backrefs enabled by default, I think lineinfile is more useful for multiple entries, and ensuring a specific line exists, but if you know the line exists and you want to change it, replace works well!

Knowing your Regular Expressions is pretty much the only way to use lineinfile and replace to their full potential, but once you get the hang of reading them and understanding what is going on, it isn't so frightening. Hopefully that helped!

We've been running into issues in the ceph-ansible project around restarting services in serial, in a way that is consistent and doesn't do unnecessary restarts. This goes back further than the above linked PR, and it's a pretty tricky thing to do! I've come up with a solution, and I'd like to run through the problem and changes with you.

Simple service restarts

In the simplest case, you would restart a service like this:

- name: Some change happens to host
  debug:
    msg: "Some change happens to host"
  notify:
    - restart my service

And then the handler would look like this:

- name: restart my service
  service:
    name: my_service
    state: restarted

This is fine for:

• One host
• Many hosts running in serial
• Many hosts where you don't care if my_service is restarted at the same time on all the hosts.

Restarting in serial

Ok so, we need to ensure my_service restarts in serial, and doesn't restart at the same time on every host, how can we do that? A pretty common, and simple way would be to do this with the handler instead:

- name: restart my service
  service:
    name: my_service
    state: restarted
  with_items "{{ ansible_play_batch }}"
  run_once: True
  delegate_to: "{{ item }}"

This will take the first host that calls the handler, and use it to initiate a restart of my_service on each other host in the current play_batch. That would mean any hosts included in the play. If run in serial, it will only be the current host.

If you were to use ansible_play_hosts instead of ansible_play_batch it would ignore the state of serial and cause a restart on all hosts, so for this ansible_play_batch would make more sense.

This solves the key problem from before, and keeps the same benefits:

• We can run this without restarting my_service on all hosts at the same time.
• This will still work in serial, and only restart the host in the current serial run.

However, we still have one case that is causing us issues...

What happens if changes don't all happen to all hosts?

There are a few cases that can cause a situation where a change only happens to one of the hosts in the group:

• host_vars change on one specific host initiates the handler on only one host.
• A host is in multiple groups, and the change has already happened on the host, as part of the original group's run, meaning it has already initiated the handler and had the service restarted.

Essentially any situation where a change is not being applied to all hosts in the group, for whatever reason.

In this situation, the handler will be skipped on the hosts that have not initiated the change, but run_once will ensure that the handler runs at least once, so even if only the last host in the run has a change, that host will initiate a restart on ALL other hosts in the group - even though they do not need it, and haven't asked for it.

The reason is, we are relying on one host's call of the handler to mean that all hosts should be restarted. In a run where a change happens uniformly this is great. But that isn't always the case! We have no way to tell if a host has called the handler itself.

The delegate_to fix works fine for a lot of situations, as long as you can agree to the following assumptions:

• You don't care that services are restarted unnecessarily.
• Changes will always happen to ALL hosts in a group.
• There is no situation where a change will be initiated on hosts in separate groups, in different orders. E.g. the same change happens to groupA and groupB, but groupA gets run first. This causes a situation where hosts in both groups will get restarted on both occasions.

Unfortunately, there is no fix for this in Ansible, ideally a serial mode on tasks would do the trick because then only hosts calling the handler would restart the service, and in serial. That doesn't exist though, there is a long issue report about it on github.

So what can we do about it?

How to solve this

As I mentioned the problem with delegate_to is that we are using one host to determine the requirement of a restart on other hosts based only on it's own state. We have no idea what the state of the other hosts is, or whether they called the handler or not.

Fortunately, we can work around this - Ansible allows the listen: directive to cause a set of tasks to be run in order, when a handler is called. Using this we can set and then test the required state of the hosts themselves manually, by doing the following:

- name: Set _restart_required var on before restart
  set_fact:
    _restart_required: True
  listen: "restart my service"

- name: Restart the service in serial if it needs it
  service:
    name: my_service
    state: restarted
  when: hostvars[item]['_restart_required'] | default(False)
  with_items: "{{ ansible_play_batch }}"
  delegate_to: "{{ item }}"
  run_once: True
  listen: "restart my service"

- name: Set _restart_required var off after restart
  set_fact:
    _restart_required: False
  listen: "restart my service"

This will execute the 3 tasks in order, starting with setting a fact to say it needs a restart, this will only be set if the host has called the handler and requires a restart. Next we restart my_service if the host has the fact set to True, implying it needs a restart. Finally, to ensure we don't restart services multiple times, we set the fact back to False so that if the host calls the handler again it will be set to True, and the service will restart.

Conclusion

I personally believe this should be a capability in Ansible itself, which is something I'd like to work on at some point!

I believe the above solution works well, and is about as good as I can think of since it will determine the restart based on whether the restart handler was called by the individual hosts.

If anybody knows a better and simpler way, please do let me know, would love to try it out.

I've been playing around with Ceph and Ceph RADOS Gateway recently. Mostly using the excellent ceph-ansible project.

I was trying to swap the ip address that was being used for the RadosGW service by setting the radosgw_address_block variable, and ran into a bug, for which I've put in a Pull Request. The bug meant that while the ceph.conf had been updated properly, the service hadn't been restarted, and so I began restarting the service manually.

This lead to a massive yak shaving exercise because I had the wrong service name, it will give you a confusing error message that makes it seem something has gone wrong that you can fix.

root@rgw1:/# service ceph-radosgw@rgw1 status
● ceph-radosgw@rgw1.service - Ceph rados gateway
   Loaded: loaded (/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/ceph-radosgw@.service.d
           └─ceph-radosgw-systemd-overrides.conf
   Active: inactive (dead)

Dec 20 04:03:11 rgw1 radosgw[8513]: 2017-12-20 04:03:11.233269 7f94c825ae80 -1 auth: unable to find a keyring on /var/lib/ceph/radosgw/ceph-rgw1/keyring: (2) No such file or directory
Dec 20 04:03:11 rgw1 radosgw[8513]: 2017-12-20 04:03:11.235727 7f94c825ae80 -1 monclient: authenticate NOTE: no keyring found; disabled cephx authentication
Dec 20 04:03:11 rgw1 radosgw[8513]: 2017-12-20 04:03:11.236865 7f94c825ae80 -1 Couldn't init storage provider (RADOS)
Dec 20 04:03:11 rgw1 systemd[1]: ceph-radosgw@rgw1.service: Main process exited, code=exited, status=5/NOTINSTALLED
Dec 20 04:03:11 rgw1 systemd[1]: ceph-radosgw@rgw1.service: Unit entered failed state.
Dec 20 04:03:11 rgw1 systemd[1]: ceph-radosgw@rgw1.service: Failed with result 'exit-code'.
Dec 20 04:03:11 rgw1 systemd[1]: ceph-radosgw@rgw1.service: Service hold-off time over, scheduling restart.
Dec 20 04:03:11 rgw1 systemd[1]: Stopped Ceph rados gateway.
Dec 20 04:03:11 rgw1 systemd[1]: ceph-radosgw@rgw1.service: Start request repeated too quickly.
Dec 20 04:03:11 rgw1 systemd[1]: Failed to start Ceph rados gateway.

At this point I found the keyring and linked it over to the /var/lib/ceph/radosgw/ceph-rgw1/keyring location. This results in another error.

root@rgw1:/# service ceph-radosgw@rgw1 status
● ceph-radosgw@rgw1.service - Ceph rados gateway
   Loaded: loaded (/lib/systemd/system/ceph-radosgw@.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/ceph-radosgw@.service.d
           └─ceph-radosgw-systemd-overrides.conf
   Active: inactive (dead)

Dec 20 04:01:59 rgw1 systemd[1]: Stopped Ceph rados gateway.
Dec 20 04:01:59 rgw1 systemd[1]: Started Ceph rados gateway.
Dec 20 04:01:59 rgw1 radosgw[8404]: 2017-12-20 04:01:59.241222 7faf4c8aae80 -1 Couldn't init storage provider (RADOS)
Dec 20 04:01:59 rgw1 systemd[1]: ceph-radosgw@rgw1.service: Main process exited, code=exited, status=5/NOTINSTALLED
Dec 20 04:01:59 rgw1 systemd[1]: ceph-radosgw@rgw1.service: Unit entered failed state.
Dec 20 04:01:59 rgw1 systemd[1]: ceph-radosgw@rgw1.service: Failed with result 'exit-code'.
Dec 20 04:01:59 rgw1 systemd[1]: ceph-radosgw@rgw1.service: Service hold-off time over, scheduling restart.
Dec 20 04:01:59 rgw1 systemd[1]: Stopped Ceph rados gateway.
Dec 20 04:01:59 rgw1 systemd[1]: ceph-radosgw@rgw1.service: Start request repeated too quickly.
Dec 20 04:01:59 rgw1 systemd[1]: Failed to start Ceph rados gateway.

This error Couldn't init storage provider (RADOS) is often due to permissions issues, so at that point I re-considered what I was doing and figured out the real issue.

TL;DR: When restarting ceph-radosgw use ceph-radosgw@rgw.hostname and NOT ceph-radosgw@hostname.

A couple of weeks ago I wrote a blog about using OpenStack-Ansible (OSA) as your development environment for OpenStack.

Adding this development path was a key goal for the OSA team, but there was another key goal that came about as a result of the development environment work, and that was the ability to incorporate OSA's testing jobs into your project.

Why use OSA to test your project?

The OpenStack-Ansible team uses source to build our pip packages. These packages deploy the various OpenStack services, making it an ideal way to test code changes and bug fixes. OSA utilizes containers to separate out components, meaning we can test scenarios on a single host that you would otherwise need multiple hosts for. Examples of this are:

1. "Zero down-time" upgrade tests, which require redundant services.
2. Multi-region scenarios requiring multiple hosts with potentially segregated networking.

OSA has the benefit of having certain tested scenarios, such as functional testing for the nova-lxd driver, that is currently not tested in Devstack:

• For Keystone, "zero downtime" upgrades are tested.
• Multi-region Swift deployments utilizing containers.
• Upgrade jobs for Nova, Swift, Glance, Cinder, Neutron (the aim is to transition these to "zero downtime" upgrade tests as we integrate the project specific upgrade paths).
• Nova scenario for nova-lxd.

How to test one off bug-fixes

If you want to use OSA to test a bug fix against the OSA gate, you can be done right now, without any additional configuration setup.

The developer mode structure change has meant that you can use the "Depends-On:" tag to test patches from upstream projects. For example, we ran into a Nova bug (that impacts a few deployment projects) and Artom from the Nova team is working on a fix. The Nova gate isn't failing on this issue, but the OSA gate is, we can use the "Depends-On:" tag and reference the patch Artom is working on. This will deploy Nova from the OSA role, and run the OSA functional tests, using the code included in Artom's patch. Here is the Test Patch to show you how the integration works in practice, and how you would test Artom's patch.

What is currently being tested?

At present we have 2 upstream projects using OSA's testing jobs for their project:

• The Ansible Runtime Analysis (ARA) project - maintained by the excellent dmsimard, currently utilizes a base OSA test job to test ARA is working with a consumer.
• Keystone currently has an experimental job that was added just this week, that utilizes the "zero-downtime" upgrade job in OSA.

Ok that sounds great, how do I do this?

It does sound great, doesn't it?!

If you wanted to integrate an OSA test with your repository, for example a "zero-downtime" upgrade test, you could:

• Write a job that runs on a multi-node gate (including a Load-Balancer, such as HAProxy)
• Ensure there is capability in Devstack to set up and run the multi-node test.
• Add test configuration to OpenStack project-infra.
• Fix Devstack to make sure it works.
• Run the tests.

Alternatively you could configure the OSA "upgrade" job to run on commit.

We've updated (as of the 15th September 2017) the process this cycle, and put a bit of work into ensuring we're doing this in the most simple way. It turns out we weren't, and it's now very simple.

All you have to do is adjust project-config in the following way:

• Edit jenkins/jobs/projects.yaml to ensure the job is a valid job - this is only required if, for example, you want a non-voting job (-nv) when one doesn't exist yet.
• Adjust zuul/layout.yml to include the OSA job in your project's jobs.

A great example is this patch to enable this for the nova-lxd project.

Why does OpenStack-Ansible care about testing other projects?

We're pretty proud of the work we've done, and we think that we have a lot of value we can give back to the community.

However, nothing is perfect, and there are a lot of upstream projects and teams. This does make it hard to keep all the projects in a state that would be considered "following best practices". That isn't to say we don't try!

By helping developers and projects to use OSA for testing and development, we hope that it will encourage developers and projects to integrate with OSA and as a result, we'll see features and fixes happen as the real project experts get more involved. Improving the quality of OSA deployments, and in turn helping the developers and projects who are using OSA.

We've seen this with the Keystone team. We have worked closely with the Keystone team to get the OSA upgrade path for Keystone to mirror the intentions of the Keystone developers, and we hope that Keystone will now benefit from the test jobs we have setup for the upgrades. This is something we'd like to reproduce.

We strongly believe that working with the community, as more than just a consumer, is the best way to improve the quality and standards of the community as a whole.

Help us, to help you, to help us!

As always, reach out to me or the team on #openstack-ansible on Freenode IRC, or on Twitter/mailing-lists etc.

In this blog post I aim to discuss how you could go about using OpenStack-Ansible as a development environment for OpenStack. Since all OpenStack services are built from source within OpenStack-Ansible, this has been a consistently requested feature for quite some time.

At the recent OpenStack Summit in Boston, I spoke a bit about the developer mode work we've been working on within the OpenStack-Ansible project.

To begin, we had a singular way to achieve this. But it was manual, and long-winded:

1. Setup your own private git repository for your project or service.
2. Commit your changes to your private git_repo.
3. Adjust the repos specific git_repo variable (e.g. nova_git_repo: https://github.com/andymcc/nova)
4. Adjust the repos specific git_install_branch variable, if on a specific branch (e.g. nova_git_install_branch: my_test_branch)
5. Run the setup plays.

This works, and is fine, but it's not particularly agile. It also means you have to continually commit and push to a remote location, re-run the install, rinse and repeat. Additionally, it was limited to projects with a "*_git_repo" variable already setup. For example, if you wanted to test a change you are writing to python-memcached, you would not be able to, since we installed this directly from PyPi (in developer mode).

Version 1: Keystone Proof of Concept for local developer mode.

Before the summit, we had proposed a Proof of Concept developer mode test for Keystone, which would allow you to clone the Keystone repository locally, write some code, and then push the changes from your local directory to the containers.

This worked fine, but had a few limitations.

1. It works for Keystone only, and is not generic.
2. This still requires code to be committed. For example, you can write some patch locally, but if you don't do a "git add" and a "git commit" it won't be included in the build on the containers.
3. Only works with Keystone.

Since then, we've spent some time improving that and making it more generic as a whole, and we've finally got that merged and working. So, without further ado!

Drum roll

All NEW developer mode for OpenStack-Ansible

The new approach is simpler and far more generic, and allows you to test local patches to any upstream repositories for projects that are installed via pip.

"That sounds great, but how do I do that?", I hear you say! Well... I'm glad you asked!

To use this, all you need to do is the following:

1. Get yourself an Ubuntu 16.04 or CentOS 7 testing server (for example a suitable VM or Cloud Instance).
2. Select and use git to clone the role for which you want to test, for example nova would be the openstack-ansible-os_nova role, which you could get clone from "https://github.com/openstack/openstack-ansible-os_nova"
3. Clone the upstream repository you want to test or develop on, into ~/git/openstack/.
4. Make your changes and develop away.
5. Run the scenario using tox, or the run_tests.sh in the openstack-ansible-os_nova role directory.

If the ~/git/openstack/ directory doesn't seem suitable for your purposes you can set the directory by setting the development_repo_directory variable inside the os-rolename_overrides.yml file inside the role repositories tests directory. For example, adding development_repo_directory: /path/to/my/repo_dir to the openstack-ansible-os_nova\tests\os-nova_overrides.yml file.

Note: You need to do this before your first run, and ideally don't change it afterwards (unless you plan on doing a full run, which will delete and recreate containers).

This will work for any pip package that are installed, for example, if you want to develop on libvirt-python, ensure that you clone it as the ~/git/openstack/libvirt-python directory, and develop there. This will also work with multiple packages, as long as they are all cloned into ~/git/openstack/. This also means you can develop on Keystone with your Nova deployment, by cloning Keystone into the ~/git/openstack/- directory.

It's also important to note, once you have a setup configured, for example you are testing Nova, you don't need to do a full re-deploy, you can just run the specific playbook for deploying nova, and skip redeploying all containers.

How does it work?

OpenStack-Ansible already has the functionality to install pip packages with constraints files. Using constraints files we can specify the location of a package as being file based, for example: file:///my/path/keystone#egg=keystone. This will ensure that if the constraints file is applied, and the Keystone package is installed, it will be installed from the file path specified instead of from an upstream git repository.

At this point, we loop through all the directories we find in the development_repo_directory directory, and generate a constraints file based on the directories we find. In this way, when you go to install any package that has a directory in development_repo_directory with the same name as the package, it will use the local file location as the location to install the package from.

This constraints file is then applied as the first constraints file (which will mean it takes precedence over any other constraints or options) when doing any pip installs and as a result, any packages listed in the constraints file will be installed from the file location.

The only other step is to mount the development_repo_directory into the containers at the same location. In other words, if it's /opt/repos on your host, it'll be /opt/repo in the containers - this is to ensure that services such as Nova (which will deploy compute onto localhost) use the same location as services in containers. The aim is that you don't have to log into the container.

What's next?

All the OpenStack-Ansible roles are now ready to go, with the exception of the Monasca role (which is currently being reworked).

We'll be adding some documentation to the official OpenStack-Ansible docs to explain how to use the developer mode.

But most importantly - give it a go! If you have any questions, or need feedback jump into the #openstack-ansible IRC channel on freenode, or reach out to andymccr (me) on Freenode and let me know your thoughts and feedback!

Boston is a wrap, and the OpenStack-Ansible (OSA) crew were there representing in force.

OpenStack-Ansible Deployments on Show

GE Keynote

The week started off, as usual, with keynotes. In particular, GE gave an impressive keynote about their usage of OpenStack in production. GE uses OpenStack-Ansible to deploy their production environment. Including my (easily) favourite quote of the week in their slide deck:

American Chemical Society (CAS)

CAS gave a "war stories" talk about getting to grips with OpenStack. They had some issues relating to Networking, Compute, and storage which they run through in the talk. What I loved was that this was another story of an OpenStack-Ansible deployment in production. CAS showcased the flexibility that OSA provides, allowing deployers and operators to overcome issues in their deployments in a managed and repeatable way.

It was really great to see large scale deployments of OpenStack-Ansible, in production, with happy operators and deployers.

OpenStack-Ansible Talks

There were several OpenStack-Ansible specific talks on show:

• Project Update - OpenStack-Ansible
  • The project update for OSA had a full house. We had a lot of great feedback, including specific commendations for our documentation and security roles, which have been well received. Credit to the team, the hard work is definitely showing.
• Making OpenStack-Ansible work for you!
  • This was my talk about helping deployers, developers, and new contributors get involved in OpenStack-Ansible. I had some really great and helpful feedback, which I'll be taking on board - look out for v2 at OpenStack days Germany on the 26th June in Munich.
• Securing OpenStack Clouds and Beyond with Ansible
  
  • Major Hayden gave a talk about the openstack-ansible-security role, which can run against non-OSA hosts, but is maintained and curated by the team.
• Using Ansible to automate the entire IT stack
  • Ricardo gives a run down of various projects, including OpenStack-Ansible, as well as various options for managing your whole stack.
• Extending OpenStack-Ansible with Automated Operational Management
  • William walks through using OSA, integrating OSA with Nagios, Logstash, and Kibana.
• The COA and why you should take it
  • Amy chaired this panel about the Certified OpenStack Administrator exam. Proud to say that the COA exam is setup using OpenStack-Ansible!
• Maximizing Hardware- Server Simulator
  • Kevin talks about using a "multi-node-aio" and testing OpenStack using OpenStack-Ansible.
• Grow Your Community- Inspire an Impostor
  • I personally can't get enough of Major's dulcet tones. This isn't an OSA related talk really, but I recommend it none-the-less!

OpenStack-Ansible Project Onboarding

This was the first summit where project-specific onboarding had been available, and it was a great opportunity to meet some interested contributors and help to get them involved! We were fortunate to have one of these slots, and I'm glad we did - we had another full house, and we were able to cover a lot of topics and information all driven by what the people in the room wanted to learn about.

You can view the etherpad we used to get new contributors more familiar with etherpad and to document and paste links to various topics as we discussed them.

Directly after the session, Chris made his first commit. Which is a great way to showcase the usefulness of the onboarding session. Congratulations Chris!

OpenStack-Ansible Operators Feedback

Operators feedback is really important to us as a deployment project. We had another great turnout, and a lot of really useful feedback.

Some of the key themes:

• Offline installs, or installs via proxy hosts are hard.
  • Improvements to the documentation and process to achieve these scenarios was requested.
• Trusty -> Xenial upgrades concerns.
  • We had already started an etherpad which covers this and can be added to. However, we will be looking into converting this into more official documentation.
• Leapfrog upgrades.
• Role integration for upstream roles.
  • Better documentation is required here.
• Being able to utilize patch-sets from upstream patches.

More details can be found on the session etherpad.

I18n check-site

Working with the Horizon and I18n teams, a translations check-site is being setup utilizing OpenStack-Ansible to allow translators to run through translations throughout the cycle. This is a great showcase of the strength of the OpenStack community. A great deal of progress was made, and we're looking to have something up and running soon!

Summary

Another successful summit for OSA, with a lot of buzz and interest in the project. New contributors coming onboard and a lot of great hallway-track discussions. There are a few pointers and places that we can focus on, but overall a great deal of positive feedback - especially around our documentation, the hard work is paying off!