Object-relational data on Locomote.sh

A particularly powerful feature of Locomote is the ability to publish not just files, but also relational data which can be replicated to a relational database on the client and then made available for offline querying. The main purpose of this feature is to allow meta data published in a post's front matter to be made available to the client, but the mechanism can be easily extended to allow more complicated data sets to be published.

File database

Locomote is a file-based CMS, and is at its heart just a file database, i.e. a list of files which are active in the system at any one time. Pulling updates from the server is first and foremost about pulling updates to the file DB, and only afterwards are actual updates to file contents downloaded. On the client, the file DB is stored in a SQLite database instance, and this provides the basis for Locomote's relational data capabilities.

File based object-relational mapping

Locomote has its own object-relational mapping (ORM) layer, which on the server is used to map source data - typically provided as JSON files - into a relational data structure, and which is then used on the client to map the relational data back to its original form. Locomote's ORM assumes that the entire data set can be broken down into separate, file-sized chunks, which can then be saved to individual files as JSON and stored in a content repository. Each of these files can be thought of a single record of data, but whilst traditional DB records have a flat structure - i.e. are a tuple of field values - records in Locomote can have complex structure, containing sub-objects or collections of values. Locomote's ORM can then be configured with information describing what DB tables different parts of an object graph are to be stored in, and how the different tables relate to each other.

JSON source files

Locomote models relational data as a uni-directional graph, and all data subsets in Locomote are modelled on a source record in a source table. This table is always the main files table of the file database, and the source record is always the file record for the JSON file which the data subset was read from.

Using files in this way to publish data allows progressive, incremental updates of the data without imposing full downloads of the entire data set on the client. Instead, as individual record files are added or modified in the content repository, the client will only download data associated with the updated files.

Filesets and ORM

The ORM layer in Locomote is built on the fileset layer. The ORM is configured by defining filesets composed of source JSON files, and then defining relational mappings for the fileset. Additionally, a fileset processor function can be defined which rewrites the source data before it is mapped to the database.

Access control

Because the ORM layer is built on the fileset layer, the access control mechanism (ACM) can be used to control access to relational data. This can operate on the file level, preventing access to entire data subsets. It can also be operated at a more fine grained level, by defining ACM filter functions for a fileset which filter the contents of a data subset according to a user's ACM group membership.

Limitations and future developments

As mentioned above, one limitation with the Locomote ACM is that data must be capable of being modelled in discrete subsets. This is necessary to allow incremental updates of the data, and so needs to be seen as a limitation of the type of system that Locomote is, rather than as a problem with the ORM design.

Another limitation is that the current Locomote server implementation needs to load the entire relational data set into memory when performing OR mapping, even for small changes to the source data. This places a practical limit on the amount of data that can be modelled this way. Again, it needs to be kept in mind the kind of system that Locomote is - it is a system for publishing finite amounts of data to mobile clients, so there will always be limits on the total size of data that can be published, imposed by the need to transfer the data to the client, and by storage and memory limitations on the client. Even so, there are plans to upgrade the OR process on the Locomote server to an O(1) memory implementation in the near future.