An approach to real-time CRUD

Recently, I wrote an article about using REST over WebSockets which generated quite a bit of interest—So I would like to use this opportunity to follow up with a practical example.

A while ago, I created a basic sample app to demonstrate an approach to building single-page real-time apps; see sc-sample-inventory. The sample app demonstrates several key features that one might like to have in a real-time single page app including:

• Authentication (via JWT tokens)
• Access control using back-end middleware
• Reactive data binding
• Real-time REST-like (CRUD) interface
• Real-time updates with support for lists (pagination, sorting, filtering)

The sample app was written with Google’s Polymer framework, SocketCluster and RethinkDB but it could have been written with any other framework/database (I’m planning to do something similar for React/Redux). It borrows ideas from REST but takes the level of granularity one step further by applying modifications to individual fields within a resource instead of the entire resource at once — for the purpose of this article, we’ll call this technique “real-time CRUD”.

If you don’t want to read about all the reasoning behind this project, feel free to skip the “Background” section below and go straight to the “How real-time CRUD works” section.

Background

One of the core features of REST is that it provides front-end clients with an interface to interact with data in a consistent, granular way. The finest level of granularity typically offered by REST is a ‘resource’ which in database-land is equivalent to document (in NoSQL) or a table row (in relational DBs).

Resource-level granularity has served us pretty well for HTTP APIs in the past because the typical workflow on the front-end has mostly consisted of submitting entire forms (with all fields of a resource at the same time). Essentially, what this means is that if two users try to modify the same resource at the “same time” (e.g. using the same form), then the last HTTP POST request to hit the server will completely overwrite the previous one — So far, this has been the standard, accepted practice when building web apps because most users have become accustomed to the idea of dealing with outdated data (I.e. they know that they have to “refresh the page” to see the latest data).

When it comes to building modern single-page apps, however, this approach has started to lose its appeal and a lot of UX-sensitive developers have started using third-party real-time data-sync services like Firebase and Parse to synchronise their data between their front-end and back-end. This allows multiple users to collaboratively modify the same resource at the same time while being aware of each other’s changes. Unfortunately, as I’ve mentioned in other articles in the past, the data-sync approach is often an unnecessary third-wheel for your system and it does not mitigate potential data conflicts entirely; it reduces flexibility and adds complexity to your app.

Instead of doing expensive diffing and implementing conflict resolution handlers for everything, why not solve this problem by side-stepping the potential for conflicts entirely?

In the same way that REST provides isolation between individual resources, we can take the idea one step further by isolating fields/properties from each other within the same resource. To make sense of this; imagine that you have a form like this which represents a resource in your database:

In standard (not real-time) REST/HTTP, as a user, you simply wouldn’t know if someone else was trying to edit the same resource at the same time— You might find out later if you come back to that resource and notice that your changes have been overwritten with old data.

With real-time CRUD, because changes happen in real-time one field at a time, other concurrent users can see your changes in their browsers as they are made and therefore won’t unintentionally overwrite them. This approach doesn’t support Google Docs-style collaborative text editing by default (E.g. for large text-areas) but it’s suitable for 99% of cases (if you consider the form above, you wouldn’t benefit at all from allowing multiple users to edit individual input fields at the same time). For those rare cases that do require collaborative text-editing on a large body of text, you can use a transport-agnostic plugin like OT.js (or one of its more recent forks).

How real-time CRUD works

On the front-end, data can be represented either as a collection ( sc-collection ) or a field ( sc-field ). A collection is essentially an array of empty placeholder resources with only an ID field (e.g. [{id:”14443ea2-b553-4195-9220-bce456c8281b"}, ...]) and a field holds values for an individual property of a resource (associated with a resource ID). You can automatically attach sc-field objects to each placeholder within an sc-collection and they will automatically populate the placeholder with the appropriate field values from the server (more on that later).

On the back-end, the current implementation is built with SocketCluster/Node.js and RethinkDB, to use it, you can simply attach the sc-crud-rethink module like this:

var crud = scCrudRethink.attach(worker, crudOptions);

You will need to pass a crudOptions object which defines the schema of your CRUD data like this.

Fields

As a user, if another user makes a change to a specific field of a resource that you are also using, it will be updated in real-time on your front-end as well. Conversely, if you make a change to your sc-field model by invoking a save operation on it:

targetModel.save();

other interested users will see the change in real-time.

To construct an sc-field on the front end, you need to do it like this:

<sc-fieldid="product-name"resource-type="Product"resource-id="{{productId}}"resource-field="name"resource-value="{{productName}}"></sc-field>

<sc-fieldid="product-qty"resource-type="Product"resource-id="{{productId}}"resource-field="qty"resource-value="{{productQty}}"></sc-field>

<sc-fieldid="product-price"resource-type="Product"resource-id="{{productId}}"resource-field="price"resource-value="{{productPrice}}"></sc-field>

<sc-fieldid="product-desc"resource-type="Product"resource-id="{{productId}}"resource-field="desc"resource-value="{{productDesc}}"></sc-field>

In Polymer, by default, data can flow either in or out of components so tag attributes can be used either as an input to the component or to capture output from it. In this case, the resource-value attribute causes data to flow out into the specified property of the parent component (productName, productQty, productPrice, productDesc) — This resource-value output property can then be used as an input to other front-end component and it will update those other components in real-time. All other attributes of the sc-field model component are used as input to the sc-field and tell it which resource and field to hook up to on the back end.

Collections

You can add a new resource to an sc-collection by invoking the create operation on it:

var newProduct = {name: this.newProductName,category: this.categoryId};

// productListModel is our sc-collection model.productListModel.create(newProduct, function(err, newId) {if (err) {// TODO: Handle error} else {self.newProductName = '';}});

Just like with sc-field, all model updates will be propagated to the database and to all interested users in real-time.

As mentioned before, a collection is just a list of empty placeholder objects with an ID property. To fill-out the collection with data on the front end, you need to attach sc-field components to it. To do this, you just need to iterate over each item in the sc-collection and bind a property of the item/placeholder to an sc-field for the same resource. See this:

<sc-collectionid="category-products"disable-realtime="{{!realtime}}"resource-type="Product"resource-value="{{categoryProducts}}"resource-view="categoryView"resource-view-params="{{categoryViewParams}}"resource-page-offset="{{pageOffsetStart}}"resource-page-size="{{pageSize}}"resource-count="{{itemCount}}"></sc-collection>

<template is="dom-repeat" items="{{categoryProducts}}" filter="hasIdFilter" observe="id">

<sc-field  
    id$="{{item.id}}-qty"  
    resource-type="Product"  
    resource-id="{{item.id}}"  
    resource-field="qty"  
    resource-value="{{item.qty}}">  
</sc-field>

<sc-fieldid$="{{item.id}}-price"resource-type="Product"resource-id="{{item.id}}"resource-field="price"resource-value="{{item.price}}"></sc-field>

<sc-field  
    id$="{{item.id}}-name"  
    resource-type="Product"  
    resource-id="{{item.id}}"  
    resource-field="name"  
    resource-value="{{item.name}}">  
</sc-field> 

</template>

^ Here we’re iterating over each placeholder ( item ) in the sc-collection and binding each sc-field to a matching property on that item (see the resource-value attribute on sc-field components above — Remember, the resource-value is an output attribute and will set item.xxxx to whatever value the sc-field gets from the server).

Transformations of collections

Transformations (or sorting and filtering in REST-speak) are done using views that are defined on the back end (as part of your crudOptions object)—Views are defined under resource definitions like this (in the following snippet, we are only exposing one view alphabeticalView which provides a sorted list of Category resources):

var crudOptions = {defaultPageSize: 5,schema: {Category: {fields: {id: type.string(),name: type.string(),desc: type.string().optional()},

  // --- BEGIN VIEW DECLARATION ---  
  // Here we define a view for the 'Category' sc-collection  
  // ordered alphabetically.  
  // We use RethinkDB's ReQL to do the transformation.  
  views: {  
    **alphabeticalView**: {  
      transform: function(fullTableQuery, r) {  
        return fullTableQuery.orderBy(r.asc('name'));  
      }  
    }  
  },  
  // --- END VIEW DECLARATION ---

  // This is an auth filter for the 'Category' resource to  
  // decide which CRUD read/write operations to block/allow.  
  // DO NOT CONFUSE THIS WITH COLLECTION TRANSFORMATION.  
  filters: {  
    pre: mustBeLoggedIn  
  }  
},  
// ...  

},// ...}

Then, on the front end, you can specify what view of the Category collection to use with the resource-view attribute of the sc-collection component like this:

<sc-collectionid="categories"resource-type="Category"resource-value="{{categories}}"resource-view="alphabeticalView"resource-view-params="null"></sc-collection>

^ If you don’t provide a resource-view, sc-crud-rethink will fetch the raw (non-transformed) collection.

Sometimes sorting isn’t enough, you also want to filter/exclude specific items from your views (e.g. maybe you only want to show products that belong to a specific category), you can do this with your transform functions by providing a more advanced database query like this (note that we are using RethinkDB’s ReQL query language):

Product: {fields: {id: type.string(),name: type.string(),qty: type.number().integer().optional(),price: type.number().optional(),desc: type.string().optional(),// The category ID field is used as a parameter to transform// the collection for the categoryView defined below.category: type.string()},views: {categoryView: {// Declare the fields from the Product model which// are required by the transform function.paramFields: ['category'],transform: function(fullTableQuery, r, productFields) {// Because we declared the category field above, it is// available in here inside productFields.// This allows us to tranform the Product collection// based on a specific category ID.// Only include products that belong to the category ID// provided by the client as part of this view.return fullTableQuery.filter(r.row('category').eq(productFields.category)).orderBy(r.asc('qty'))}}},filters: {pre: mustBeLoggedIn,post: postFilter}}

^ The productFields.category property from the code above will hold the category ID which was provided by the resource-view-params attribute on the front end like this:

<sc-collectionid="category-products"resource-type="Product"resource-value="{{categoryProducts}}"resource-view="categoryView"resource-view-params="{{categoryViewParams}}"resource-page-offset="{{pageOffsetStart}}"resource-page-size="{{pageSize}}"resource-count="{{itemCount}}"></sc-collection>

The value of the resource-view-params attribute (categoryViewParams in our case) must be a plain JavaScript Object whose properties exactly match those that were specified in paramFields in the categoryView definition on the back end, so on the front end, we’re computing it like this:

computeCategoryViewParams: function (categoryId) {return {category: categoryId};}

Whatever ID we provide as category on the front end will be used as the value in productFields inside the transform function on the back end (which we use as an argument to our ReQL query). This part (as shown earlier):

return fullTableQuery.filter(r.row('category').eq(productFields.category))

The fullTableQuery is the ReQL query which fetches the entire table/collection — ReQL allows us to easily extend this query with additional filters and sorting rules so we can filter our products based on their category IDs as shown above.

Authorization

We don’t want to allow just any user to read/write from/to any resource using our real-time CRUD interface so we need to add some access control rules.

With SocketCluster, you can do this via custom middleware like this.

Note that in the example above, we are being very broad with our rules, any logged in user can read or modify any data in our database. You can read the values from the auth token to make more fine-grained decisions.

Notes

While sc-crud-rethink, sc-field and sc-collection modules/components have been well tested during development, I can’t guarantee that they have been used in production so proceed with caution.

If anything is not clear, feel free to mention it in the SocketCluster channel on Gitter. I will keep updating this article over time to improve it.