Crufd
· 10 min
To significantly reduce the required boilerplate for new user flows and greatly enhance Developer Experience, I designed and built a framework called CRUFD.
🧚♀️ Long long time ago…
It’s 2018, A little over half a year after I joined Rabobank. The transition from monolith to microservices is bringing a lot of new work to the teams. My team and I spend the better part of the next 2 years building (and rebuilding) a landscape to support our future endeavors, while slowly transforming existing applications and creating new customer configuration flows.
A configuration flow is a user journey in which they configures something, usually a product or set of products. Data is presented to guide the user to make choices. In the end the configuration is transformed and sent to downstream systems.
Frustrated with the amount of time needed to basically reinvent (large parts of) the wheel for every single configuration flow, I set out to find a way to reduce the boilerplace and focus on what was important: The business logic.
🛒 Shopping cart
To not reinvent the wheel on not reinventing the wheel, I looked at how this problem is tackled in other parts of our industry. The best parallel I found was the shopping cart. To explain why, I often use an analogy around shoe shopping:
When you are shopping online for shoes you select a type, configure the size and (if you are feeling frisky) select a fancy colors for the laces. When you are happy with your choices, you put it in your cart. Perhaps you also add some socks? Afterwards the cart gets checked out and the order is sent onwards.
If somewhere along the line you decide to leave and come back later, the cart content will need to be revalidated (are those socks still in stock?) after which you will be able to continue where you left off.
What we tried to achieve at Rabobank was a similar pipeline system, where we assemble a request for the next systems to process. The only big difference is that for our flows the configuration is usually much more complex and the number of items to land in the cart is often just one. The process steps however, are very much the same!
In software engineering, a pipeline consists of a chain of processing elements (processes, threads, coroutines, functions, etc.), arranged so that the output of each element is the input of the next. Wikipedia
The cart system
The cart system was implemented as a service initally called sales-cart. It tracked items which it used to point to and route requests for the right configuration-services. It was later renamed to crufd-router before eventually being decomissioned.
Because of this I’m not going into detail about how it worked.
To cart or not to cart
After some active development using and improving the framework it turned out that multiple items in a cart was functionality we were never really going to use. Because all configuration flows resulted in just 1 item in the cart. While there are product packages (multiple products that belonged together) we found that a single configuration flow for the whole package works best. The seemingly re-usable parts were mostly accidental duplication and even though pieces looked very similar, they almost always had slightly different business rules which ended up diverging as often as converging.
So after half a year of usage we got rid of the shopping cart completely and directly connect the frontend to the configuration services. This significantly reduced the complexity and while it might have been useful to be able to have multiple configurations at some point in the future. This decision payed back in a few months time due to the increased maintainability. To this day I believe that even though we learned a lot from the road taken this was one of the most important decisions made surrounding this project.
🎛️ Configuration services
Initially configuration services were plugins to the cart-system, but later they ran as standalone services. A configuration-service usually implements a single configuration flow, with some exceptions for a few very similar flows where re-use made more sense than duplication.
the CRUFD interface
A configuration service is the implementation of the CRUFD Interface: Create, Read, Update, Finalize and Delete actions and the model of the configuration or CrufdConfiguration to execute these actions on.
CRUDF CRUFD against Cruft
There was a very short time the name CRUDF was used, but it was soon changed to CRUFD. Moslty because it can be spoken as word, but also because of the link to the word Cruft.
Cruft is a jargon word for anything that is left over, redundant and getting in the way. It is used particularly for defective, superseded, useless, superfluous, or dysfunctional elements in computer software. Wikipedia
You can use CRUFD to fight against the Cruft!
CRUD
At the start of a user session the configuration flow starts by creating a new, or if there is an existing one, reading a configuration. If a configuration is re-loaded its content is verified and cleared if it is invalid.
Reading a configuration can be done at any time after creation. Validation happens if significant time has passed since the last update or a new sessionId is sent.
During the flow the configuration gets updated and information, (often based on decisions made by the user), gets appended as read-only data. This information can in turn be used to populate the views and direct the next choices of the user.
If a user wants to discard the whole configuration it can be deleted.
Finalizing
At the end of the flow the configuration is finalized.
Finalizing can mean many things, but in pricinple it means that (if successful) the configuration becomes fixed (read only) and an aggregate of the configuration is sent to subsequent systems for further processing.
As an example: When creating a new savings account, you have to choose if you want a fixed or variable interest rate. If fixed is selected, what the fixed duration will be. On finalizing the bank account gets created with the configured settings.
CrufdConfiguration
All the above actions take place on a CrufdConfiguration which is the model that holds all data for a specific configuration flow. The interface itself is simple:
public interface CrufdConfiguration {
boolean readyForFinalize();
}
The only assumption made by the CRUFD framework is that a configuration should be able to determine (from its data) if it is ready to finalize or not. With that we can determine if the finalize step is allowed to run or not.
The implementation of a CrufdConfiguration is a usually a data object with both read-only and writable fields. Some read-only fields are populated during initial creation of the configuration, for example the customer name. The writable fields get set during the configuration flow which during create or update are used to determine and set more read-only fields. Read-only fields are used by the frontend to display information required to fill in the next writable fields.
At the end of a configuration flow all required writable fields should be set. This is validated using readyForFinalize(). As an example:
@Value
public class SimpleConfiguration implements CrufdConfiguration {
@ConsumerWritable
@ResetOnReentry
String requiredValue;
String readOnlyValue;
public boolean readyForFinalize(){
return StringUtils.isNotBlank(requiredValue);
}
}
🧑💻 Crufd.js
Crufd.js is a JavaScript library created to connect to the CRUFD system and avoid duplicating boilerplate code in frontend applications. Initally it was called cart.js, but once the cart system was taken out it was renamed to fit the system better.
Crufd.js implements almost the same CRUFD interface from a consumers perspective. From here we can:
- readOrCreate to either reload an existing or create a new configuration of a specific type.
- read to read the current state of a write or readOnly field.
- update one or more fields and send the update to the backend for processing.
- delete to remove the configuration.
When a configuration flow is loaded we start by calling readOrCreate to initialize an existing or new configuration. This configuration is downloaded and put into a cache. Subsequent reads are done on the cache and no further calls are done to the backend unless:
- The cache expires, this happens after a configurable amount of time (default 10 minutes).
- An update is executed and flushed. This sends the new values to the backend and overwrites the cache with the response.
This means that components using the crufd-configuration don’t need to know or care about when the backend is being called or even that it exists.
JSONPath
Read and update actions are done on the root crufd-configuration directly. To avoid that every component knows the whole structure of the configuration (and reduce coupling) JSONPath is used to locate the fields/object to read/write and only require the knowledge of those exact fields in the consuming component.
🎬 CRUFD actions
Initially, creating a new CRUFD configuration service meant directly implementing the CRUFD interface in an imperative way. This usually boiled down to creating a service that calls other services that implement business logic etc. The usual layered aproach, but with the CRUFD interface replacing the controller and persistence layers.
As an extension to the CRUFD framework I created crufd-actions, a library to create all the required steps in a declarative way.
Using the Action interface and annotations to define if the Action had to run @OnCreate,@OnUpdate,@OnFinalize or @OnDelete (custom actions on read were prohibited) you could extend the functionality in an open-closed maner. Initially actions had an order, but this was later replaced by @DependsOn and @RunBefore annotations.
On startup 4 action-graphs are created (for @OnCreate,@OnUpdate,@OnFinalize and @OnDelete) to determine the action dependencies, which actions might run in parallel and which have to wait a for other steps to finish.
Once a call comes in on the CRUFD interface this graph is run by first checking conditions against the CrufdConfiguration in shouldRun() and if the condition are met the run() method is executed.
@RequiredArgsConstrutor
@OnCreate
@OnUpdate
@RunBefore(ValidateSavingsAccountAction.class)
class LoadSavingsAccountsAction implements Action<RtbNewCrufdConfiguration>{
private final SavingsAccountGateway savingsAccountGateway;
public void run(RtbNewCrufdConfiguration configuration){
var accounts = savingsAccountGateway.findAllSavingsAccounts();
configuration.setSavingsAccounts(accounts);
}
public boolean shouldRun(RtbNewCrufdConfiguration configuration) {
return CollectionUtils.isEmpty(configuration.getSavingsAccounts());
}
}
🚀 Lift-off
The CRUFD framework is one of a few mechanisms that enabled my team at Rabobank to go much faster. I will talk about other complementary improvements in future posts, but CRUFD definitely had impact by itself.
Especially after introducing crufd-actions and Crufd.js we could focus on writing business logic and forget about state management and even communication between front and backend. The only thing that mattered was to connect pieces together, implement business rules and decide on the order of execution of these rules. There was the obvious learning curve involved, but after that the Developer experience improved a lot, which ended up in some really awesome compliments from my team.
When the carts system was first introduced it still took a few months to fully implement a configuration flow, but the last flow I implemented before leaving Rabobank only took 2 weeks.
It always depends
One might expect that I’m still using CRUFD to this day, but so often in our industry one size fits nothing. I’m still working on a part of what could be called a configuration flow today, but since it’s not part of a pipeline system, CRUFD would never work. What I am still using is a lot of the lessons learned while building this. Especially about the worth of investing in developer friendly environments to eventually be able to get to and keep delivering at high velocity 🚀