How to do a Ceph cluster maintenance/shutdown

Last week someone asked on the ceph-users ML how to shutdown a Ceph cluster and I would like to summarize the steps that are neccessary to do that.

  1. Stop the clients from using your Cluster (this step is only neccessary if you want to shutdown your whole cluster)

  2. Important - Make sure that your cluster is in a healthy state before proceeding

  3. Now you have to set some OSD flags:

    # ceph osd set noout
    # ceph osd set nobackfill
    # ceph osd set norecover
    
    Those flags should be totally suffiecient to safely powerdown your cluster but you
    could also set the following flags on top if you would like to pause your cluster completely::
    
    # ceph osd norebalance
    # ceph osd nodown
    # ceph osd pause
    
    ## Pausing the cluster means that you can't see when OSDs come
    back up again and no map update will happen
    
  4. Shutdown your service nodes one by one

  5. Shutdown your OSD nodes one by one

  6. Shutdown your monitor nodes one by one

  7. Shutdown your admin node

After maintenance just do everything mentioned above in reverse order.

Ceph Manager Dashboard v2

The original Ceph Manager Dashboard that was introduced in Ceph "Luminous" started out as a simple, read-only view into various run-time information and performance data of a Ceph cluster, without authentication or any administrative functionality.

However, as it turns out, there is a growing demand for adding more web-based management capabilities, to make it easier for administrators that prefer a WebUI to manage Ceph over the command line. Sage Weil also touched upon this topic in the Ceph Developer monthly call in December and created an etherpad with some ideas for improvement.

/galleries/ceph-dashboard-v2-screenshots-2018-02-02/dashboard-v2-health.png

A preliminary screen shot of the Ceph health dashboard

After learning about this, we approached Sage and John Spray from the Ceph project and offered our help to implement the missing functionality. Based on our experiences in developing the Ceph support in openATTIC, we think we have a lot to offer in the form of code and experience in creating a Ceph administration and monitoring UI.

Read more…