Implementing a more scalable storage management framework in openATTIC 3.0

Over the course of the last years, we've been working on expanding and enhancing both our "traditional" local storage management functionality (NFS/CIFS/iSCSI on top of local attached disks) as well as the Ceph management features in openATTIC.

Along the way, it became more and more clear to us that our current approach for managing storage locally does not scale well, as it requires openATTIC to be installed on every node.

When openATTIC was originally started, this wasn't so much of a concern, but today's IT infrastructures are evolving and demand more flexibility and scalability. Also, our goal of making it possible for an administrator to make changes on the command line outside of openATTIC is difficult to achieve in the current local storage implementation, in which the Django models are considered to be the "single source of truth" of a server's storage configuration.

The ongoing development of additional Ceph management functionality based on DeepSea and Salt allowed us to gather a lot of experience in implementing a more scalable approach using these frameworks and make it possible to decouple openATTIC from the node delivering the actual service. Communicating with a Salt master via the Salt REST API also enables us to separate the management UI (openATTIC) from the admin node (the Salt master).

Based on these findings, we wanted to create a playground for our developers to apply the lessons learned to the openATTIC code base. We therefore moved the current openATTIC 2.0 implementation into a separate 2.x git branch and have started working on version 3.x in the current master branch. Note that this will not be a complete rewrite of openATTIC, but rather an adaption/refinement of the existing code base.

In addition to the already existing Ceph management functionality based on librados (e.g. Ceph Pool management, RBD management), we're currently working on adding more Ceph-based storage management functionality e.g. managing iSCSI targets as well as NFS volume management via NFS Ganesha.

The focus in this 3.0 branch will be on completing the Ceph-related management functionality first, while aiming at being able to implement the "traditional" storage management functionality using this framework (e.g. providing storage services based on node-local disks) at a later step. Salt already includes a large number of modules for these purposes.

As usual, we welcome your feedback and comments! If you have any ideas or if you can help with implementing some of these features, please get involved!

openATTIC 2.0.19 beta has been released

openATTIC 2.0.19 is now available. This is a minor release, backwards compatible, where we deliver on our promise to make openATTIC easy to use, faster and with a great GUI.

On the frontend, we improved the feedback given to the user with a better error handling, useful toasty notifications, and loading spinners. We also added the DRBD support to the graphical user interface.

Since it's spring time, we also did a bit of house cleaning! On the backend side, we removed some obsolete modules such as peering, IPMI and mdraid. We also extracted the XML RPC daemon and its related API because they have been replaced by our REST API some time ago.

Besides that, the backend offered even more room for improvement and fixes like the creation of erasure coded Ceph pools.

We believe that documentation is as important as code. That's why we made many documentation improvements for this release. For instance, we restructured and improved the format of our CHANGELOG inspired by the Keep a Changelog project.

This release also contains two external contributions submitted by Uros Lates and David Díaz. Thank you, much appreciated!

We're happy to announce that this is also the last release built from our Mercurial repository, because we have moved to... GIT! We also updated the "developer" documentation section to reflect the mercurial to git workflow changes.

During this release we moved our hardware to a new infrastructure which delayed the announcement of this version, we apologize for the inconvenience.

Read more…

Branch Name Reorganisation

Now that we've concluded our migration to git, we also made some changes to our branch naming scheme.

In the Mercurial age, we had two main branches:

  • development: the active development branch, used by developers to derive their feature branches from and which was used as the target branch for pull requests.
  • default: the branch that nightly release builds and official releases were built from. This branch was updated by merging development into default by Jenkins jobs that performed a wide range of automatic tests against the development branch. Only if these tests had passed, an automatic merge to the default was performed.

During the conversion from Mercurial to git, the default branch was automatically renamed to master, which is the canonical name for the "main" repository branch in git. For most open source projects, the master branch resembles the current line of development, similar to how we used the development branch in Mercurial.

Read more…

openATTIC code repository migrated to git

From the very beginning, the openATTIC source code repository was managed using the Mercurial distributed source control management tool.

Our code is hosted on BitBucket which provides for a tight integration with our public Jira issue tracker.

Unfortunately, Mercurial is somewhat less flexible than git when it comes to using branches to separate ongoing development work (which is a workflow encouraged by using Jira/BitBucket) - there is a tight relationship between branch names across repositories and it's impossible to delete branches once they have been created or merged. Sure, we could have probably used Mercurial bookmarks for this, but they are not well supported by the Jira and BitBucket workflows we are using (and are more designed to be used locally, not across multiple repositories).

In addition to that, we have a growing developer/contributor base that simply is more familiar with git than Mercurial nowadays. Switching to git could potentially help us attracting a bigger number of developers.

These were the main reasons for our decision to convert. Thankfully, with the help of the fast-export utility, the actual conversion was straightforward and painless. It automatically renamed the former default branch to master, to be conforming to established git practices. We will revisit the branch naming conventions in a following step.

There was some cleanup work involved afterwards, to remove all the obsolete branches that were created by Mercurial when merging pull requests.

Additionally, all pending pull requests from the previous Mercurial repo had to be re-created, but that provided us with a good opportunity to clean up the commit history before submitting them. Going forward, we intend to also make use of the more advanced features of git like rebasing or squashing change sets, to have a cleaner revision history with less noise.

We're now in the process of updating our documentation and build scripts to support this change, but this should not take us too long to resolve.

Introducing a new CHANGELOG format

A few days ago, I stumbled over an article on Opensource.com about "How to make release notes count".

The article is worth a read - it contains a number of useful suggestions on how to structure the documentation that summarizes the changes that have taken place in a software project's latest release.

Which was excellent timing, as we just merged a major overhaul of our own project's CHANGELOG file (see OP-1911 for details).

We adapted our CHANGELOG based on suggestions made by the Keep a Changelog project, with the exception of using reStructuredText for markup instead of Markdown.

To describe their impact on the project, changes are now grouped as follows:

  • Added for new features.
  • Changed for changes in existing functionality.
  • Deprecated for once-stable features removed in upcoming releases.
  • Removed for deprecated features removed in this release.
  • Fixed for any bug fixes.
  • Security to invite users to upgrade in case of vulnerabilities.

This will hopefully make it easier for our users to get an impression on the changes to expect when updating to a newer version.

How do you like the new structure? Please let us know. Thanks!

Meet the openATTIC Team at Chemnitzer Linuxtage 2017

https://chemnitzer.linux-tage.de/2017/static/img/logo_en.png

This weekend (March 11-12), the Chemnitzer Linuxtage take place again, and it's our great pleasure to attend and represent our project there, this time as a proud member of the growing openSUSE ecosystem.

We'll be showcasing the latest development version of openATTIC, including the latest Ceph management features and a preview of the DRBD mirroring functionality.

In addition to myself, you will be able to meet and talk the following members of the openATTIC team: Kai Wagner (one of the project founders), Volker Theile (maintainer of the OpenMediaVault project and also well-known as one of the early contributors to the FreeNAS project) and Stephan Müller (one of our WebUI developers).

We look forward to the event and good conversations with the general Linux and OSS community. If you are attending CLT, please stop by and say hello!

IMAP: Sync all Folders in Thunderbird

I could be very annoying to use Thunderbird with multiple folders. Everytime you'll create a new subfolder you have to change the properties and click the checkbox to download all emails to this folder without the need to click on this specific folder every time.

I found a global option which made my life much easier. Just use the config editor (about:config) and change the following entry:

mail.server.default.check_all_folders_for_new -> true

That's it.