Sneak preview: Upcoming Ceph Management Features

Despite the number of disruptive changes that we went through in the past few weeks, e.g. moving our code base from Mercurial to git, relocating our infrastructure to a new data center, refactoring our code base for version 3.0, our developers have been busy working on expanding the Ceph management capabilities in openATTIC.

I'd like to highlight two of them that are nearing completion and should land in the master branch shortly.

Read more…

openATTIC 2.0.20 has been released

It is our great pleasure to announce the release of openATTIC version 2.0.20. This is a minor bugfix release, which also provides a number of small selected improvements, e.g. in the WebUI (styling, usability), installation and logging (now adds PID and process name to logs). Furthermore, we updated our documentation - especially the installation instructions as well as the developer documentation.

Read more…

Clean up and split your branch with git

There are several reasons why you may need completely refactor your working branch. The most common one is that you stumbled upon some things you fixed along your way resolving an issue. Your branch will grow over time until you are finished. Now you want to submit your code. You have to split it up into digestible pieces and maybe have to rewrite some WIP commits you made. Luckily we are using git! With git you can do it with a few commands with a safety belt on!

Read more…

Demo currently unavailable

You might have noticed already that our live demo on demo.openattic.org is down and not reachable at the moment.

This issue is caused by our hardware move to a new and more secure datacenter. Right now, we're trying to figure out what's the best approach to make the demo accessible again. This isn't as easy as before because we now have a dedicated firewall and a dmz for external services.

Therefore the demo.openattic.org URL will be redirected to openattic.org as until we have a running demo again.

Implementing a more scalable storage management framework in openATTIC 3.0

Over the course of the last years, we've been working on expanding and enhancing both our "traditional" local storage management functionality (NFS/CIFS/iSCSI on top of local attached disks) as well as the Ceph management features in openATTIC.

Along the way, it became more and more clear to us that our current approach for managing storage locally does not scale well, as it requires openATTIC to be installed on every node.

When openATTIC was originally started, this wasn't so much of a concern, but today's IT infrastructures are evolving and demand more flexibility and scalability. Also, our goal of making it possible for an administrator to make changes on the command line outside of openATTIC is difficult to achieve in the current local storage implementation, in which the Django models are considered to be the "single source of truth" of a server's storage configuration.

The ongoing development of additional Ceph management functionality based on DeepSea and Salt allowed us to gather a lot of experience in implementing a more scalable approach using these frameworks and make it possible to decouple openATTIC from the node delivering the actual service. Communicating with a Salt master via the Salt REST API also enables us to separate the management UI (openATTIC) from the admin node (the Salt master).

Based on these findings, we wanted to create a playground for our developers to apply the lessons learned to the openATTIC code base. We therefore moved the current openATTIC 2.0 implementation into a separate 2.x git branch and have started working on version 3.x in the current master branch. Note that this will not be a complete rewrite of openATTIC, but rather an adaption/refinement of the existing code base.

In addition to the already existing Ceph management functionality based on librados (e.g. Ceph Pool management, RBD management), we're currently working on adding more Ceph-based storage management functionality e.g. managing iSCSI targets as well as NFS volume management via NFS Ganesha.

The focus in this 3.0 branch will be on completing the Ceph-related management functionality first, while aiming at being able to implement the "traditional" storage management functionality using this framework (e.g. providing storage services based on node-local disks) at a later step. Salt already includes a large number of modules for these purposes.

As usual, we welcome your feedback and comments! If you have any ideas or if you can help with implementing some of these features, please get involved!

openATTIC 2.0.19 beta has been released

openATTIC 2.0.19 is now available. This is a minor release, backwards compatible, where we deliver on our promise to make openATTIC easy to use, faster and with a great GUI.

On the frontend, we improved the feedback given to the user with a better error handling, useful toasty notifications, and loading spinners. We also added the DRBD support to the graphical user interface.

Since it's spring time, we also did a bit of house cleaning! On the backend side, we removed some obsolete modules such as peering, IPMI and mdraid. We also extracted the XML RPC daemon and its related API because they have been replaced by our REST API some time ago.

Besides that, the backend offered even more room for improvement and fixes like the creation of erasure coded Ceph pools.

We believe that documentation is as important as code. That's why we made many documentation improvements for this release. For instance, we restructured and improved the format of our CHANGELOG inspired by the Keep a Changelog project.

This release also contains two external contributions submitted by Uros Lates and David Díaz. Thank you, much appreciated!

We're happy to announce that this is also the last release built from our Mercurial repository, because we have moved to... GIT! We also updated the "developer" documentation section to reflect the mercurial to git workflow changes.

During this release we moved our hardware to a new infrastructure which delayed the announcement of this version, we apologize for the inconvenience.

Read more…

Branch Name Reorganisation

Now that we've concluded our migration to git, we also made some changes to our branch naming scheme.

In the Mercurial age, we had two main branches:

  • development: the active development branch, used by developers to derive their feature branches from and which was used as the target branch for pull requests.
  • default: the branch that nightly release builds and official releases were built from. This branch was updated by merging development into default by Jenkins jobs that performed a wide range of automatic tests against the development branch. Only if these tests had passed, an automatic merge to the default was performed.

During the conversion from Mercurial to git, the default branch was automatically renamed to master, which is the canonical name for the "main" repository branch in git. For most open source projects, the master branch resembles the current line of development, similar to how we used the development branch in Mercurial.

Read more…

openATTIC code repository migrated to git

From the very beginning, the openATTIC source code repository was managed using the Mercurial distributed source control management tool.

Our code is hosted on BitBucket which provides for a tight integration with our public Jira issue tracker.

Unfortunately, Mercurial is somewhat less flexible than git when it comes to using branches to separate ongoing development work (which is a workflow encouraged by using Jira/BitBucket) - there is a tight relationship between branch names across repositories and it's impossible to delete branches once they have been created or merged. Sure, we could have probably used Mercurial bookmarks for this, but they are not well supported by the Jira and BitBucket workflows we are using (and are more designed to be used locally, not across multiple repositories).

In addition to that, we have a growing developer/contributor base that simply is more familiar with git than Mercurial nowadays. Switching to git could potentially help us attracting a bigger number of developers.

These were the main reasons for our decision to convert. Thankfully, with the help of the fast-export utility, the actual conversion was straightforward and painless. It automatically renamed the former default branch to master, to be conforming to established git practices. We will revisit the branch naming conventions in a following step.

There was some cleanup work involved afterwards, to remove all the obsolete branches that were created by Mercurial when merging pull requests.

Additionally, all pending pull requests from the previous Mercurial repo had to be re-created, but that provided us with a good opportunity to clean up the commit history before submitting them. Going forward, we intend to also make use of the more advanced features of git like rebasing or squashing change sets, to have a cleaner revision history with less noise.

We're now in the process of updating our documentation and build scripts to support this change, but this should not take us too long to resolve.