Add the concept of reconcile Procedures

[JohnStrunk]

Describe what this PR does
In #51 reconcile Actions were introduced. This PR brings in the concept of a Procedure as a collection of Actions. A single Procedure should fully define the steps necessary to reconcile the operator state.

Multiple Procedures may exist within the operator to handle upgrades. Each Procedure has a minVersion from which it can upgrade.

Is there anything that requires special attention?
There is currently just pseudo-code to plug this into the actual sdk reconcile function.

Related issues:
Follow-on to #51

[rohantmp]

Do we want to redo actions when they're listed as in both Procedure.actions and Procedure.actions[i].prereqs[j] or are we okay parsing the Procedure to generate a list of actions to be executed in the correct order?

I can see this is a safety/perf tradeoff, and that it's good to at the very minimum ensure that the prereqs are expressed, but does it really help to actually run them each time?

Edit: NVM, I guess we could just not include actions that are in Procedure.actions[i].prereqs[j] in Procedure.actions :P What about repeated prereqs?

[JohnStrunk]

Do we want to redo actions...?

I think I clarified over chat, but just to log the answer here (and to make sure I did actually understand/answer the question)...

Each Action has a cache:

anthill/pkg/reconcileaction/reconcileaction.go

Line 35 in fb6d2c6

lastResult *Result

...that records the result of execution, and unless that cache is cleared (action.Clear()), the Action.action func will not be re-run.
The Procedure's Execute() starts by clearing that cache for all actions and their prereqs before evaluating the component Actions. This has the effect of permitting all Actions (including prereq actions) to execute at most once per invocation of Procedure.Execute() (aka, a controller reconcile cycle).

The benefits of the above are:

• Prereqs are evaluated consistently. If I have 2 Actions that have a common prereq, I am assured that the prereq will return the same result in both cases
• Expensive checks are cached. We may have to probe gd2 or other systems for checking, and repeatedly doing this could cause unnecessary load and slow down the reconcile loop.

[rohantmp]

I see now :) LGTM!

[humblec]

Inside "glustercluster" we could have "controller.go" instead of "glustercluster_controller.go"

[JohnStrunk]

It's sdk-chosen... not ideal, but 🤷‍♂️

[humblec]

somehow I dont like reconcileaction as the package name. :(

[JohnStrunk]

I'll rename to reconciler. That would give reconciler.Action and reconciler.Procedure which seem pretty good. The caveat is that controller-runtime has a reconcile package that gets imported already, so we'll have reconcile.foo and reconciler.foo running around.

[humblec]

reconsiler looks good to me. Our reconsiler can be imported by a different name , an ex: greconsiler

[humblec]

Instead of minVersion , why dont we define a dict of "compatible versions" ?

[JohnStrunk]

The "version" here is meant to be just numeric procedure versions, numbered independently of the operator release version (i.e., an operator release may not require a new version at all, or it may introduce multiple depending on upgrade requirements). As such, I really expect the simple >= compatibility test to be sufficient for this. Since it's encapsulated from the rest of the code, we could switch it later w/ little difficulty should it become necessary. Sound reasonable?

[humblec]

@JohnStrunk hmmm.. compatible version array or dict would sustain better compared to >= , but >= could be a start or first version. so, lets keep it as this for now.

[humblec]

@JohnStrunk few comments from my side.

[rohantmp]

@JohnStrunk I want to clarify some things:

1. When the next version of operator-sdk comes out We'll have to generate the scaffolding for the new version and manually apply our changes? with diff,etc?

2. We will create a new procedure version when GCS changes enough that the same actions have to be implemented differently?

3. If both 1) and 2) are true, then how do we handle the versioned actions? subpackages like v0? with an exported procedure and exported actions?

[humblec]

LGTM .. Merging ..