Options
All
  • Public
  • Public/Protected
  • All
Menu

🚧 This documentation is for the developer preview of m-ld.

github licence npm (tag)

m-ld Javascript clone engine

The Javascript engine can be used in a modern browser or a server engine like Node.js.

The Javascript clone engine conforms to the m-ld specification. Its support for transaction pattern complexity is detailed below. Its concurrency model is based on immutable states.

Getting Started

npm install @m-ld/m-ld

There are two starter projects available:

  • The Node.js project uses Node processes to initialise two clones, and an MQTT broker for messaging.
  • The Web project shows one way to build a multi-collaborator forms application for browsers, using Socket.io for messaging.

Data Persistence

m-ld uses levelup to interface with a LevelDB-compatible storage backend.

  • For the fastest responses use memdown (memory storage).
  • In the browser, use level-js (browser-local storage).
  • In a service or native application, use leveldown (file system storage).

Connecting to Other Clones

A m-ld clone uses a 'remotes' object to communicate with other clones.

  • With an MQTT broker, use MqttRemotes.
  • For a scalable global managed service, use AblyRemotes.
  • If you have a live web server (not just CDN or serverless), you can use IoRemotes (🚀new!).

🚧 If your architecture includes some other publish/subscribe service like AMQP, or you would like to use a fully peer-to-peer protocol, please contact us to discuss your use-case.

Initialisation

The clone function initialises the m-ld engine with a leveldb back-end and the clone configuration.

import MemDown from 'memdown';
import { clone, uuid } from '@m-ld/m-ld';
import { MqttRemotes, MeldMqttConfig } from '@m-ld/m-ld/dist/mqtt';

const config: MeldMqttConfig = {
  '@id': uuid(),
  '@domain': 'test.example.org',
  genesis: true,
  mqtt: { hostname: 'mqtt.example.org' }
};

const meld = await clone(new MemDown, MqttRemotes, config);

The clone function returns control as soon as it is safe to start making data transactions against the domain. If this clone has has been re-started from persisted state, it may still be receiving updates from the domain. This can cause a UI to start showing these updates. If instead, you want to wait until the clone has the most recent data, you can add:

await meld.status.becomes({ online: true, outdated: false });

Remotes

MQTT Remotes

MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. It is convenient to use it for local development or if the deployment environment has an MQTT broker available. See below for specific broker requirements.

The MqttRemotes class and its companion configuration class MeldMqttConfig can be imported or required from '@m-ld/m-ld/dist/mqtt'. You must also install the async-mqtt package as a peer of @m-ld/m-ld.

The configuration interface adds an mqtt key to the base MeldConfig. The content of this key is a client options object for MQTT.js. It must not include the will and clientId options, as these are set internally. It must include a hostname or a host and port.

MQTT Broker Requirements

MqttRemotes requires broker support for:

  1. MQTT 3.1
  2. QoS 0 and 1
  3. Retained messages
  4. Last Will and Testament (LWT) messages

A good choice for local development is Aedes.

Ably Remotes

Ably provides infrastructure and APIs to power realtime experiences at scale. It is a managed service, and includes pay-as-you-go developer pricing. It is also convenient to use for global deployments without the need to self-manage a broker.

The AblyRemotes class and its companion configuration class MeldAblyConfig can be imported or required from '@m-ld/m-ld/dist/ably'. You must also install the ably package as a peer of @m-ld/m-ld.

The configuration interface adds an ably key to the base MeldConfig. The content of this key is an Ably client options object. It must not include the echoMessages and clientId options, as these are set internally.

If using token authentication, ensure that the clientId the token is generated for corresponds to the @id given in the MeldConfig.

Socket.io Remotes

Socket.IO enables real-time, bidirectional and event-based communication. It works on every platform, browser or device, focusing equally on reliability and speed. It is convenient to use when the app architecture has a live web server or app server, using HTTP.

The IoRemotes class and its companion configuration class MeldIoConfig can be imported or required from '@m-ld/m-ld/dist/socket.io'. You must also install the socket.io-client package as a peer of @m-ld/m-ld.

The configuration interface adds an io key to the base MeldConfig. The value is an optional object having:

  • uri: The server URL (defaults to the browser URL with no path)
  • opts: A Socket.io factory options object, which can be used to customise the server connection

Socket.io Server

When using Socket.io, the server must correctly route m-ld protocol operations to their intended recipients. The Javascript engine package bundles a class for Node.js servers, IoRemotesService, which can be imported from '@m-ld/m-ld/dist/socket.io/server'.

To use, initialise the Socket.io server as normal, and then construct an IoRemotesService, passing the namespace you want to make available to m-ld. To use the global namespace, pass the sockets member of the Server class. For example:

const socket = require('socket.io');
const httpServer = require('http').createServer();
// Start the Socket.io server, and attach the m-ld message-passing service
const io = new socket.Server(httpServer);
new IoRemotesService(io.sockets);

For a complete example, see the web starter project .

For other server types, contact us.

Transactions

A m-ld transaction is a json-rql pattern, which represents a data read or a data write. See the m-ld specification for a walk-through of the syntax.

Supported pattern types for this engine are (follow the links for examples):

🚧 If you have a requirement for an unsupported pattern, please contact us to discuss your use-case. You can browse the full json-rql syntax at json-rql.org.

Concurrency

A m-ld clone contains realtime domain data in principle. This means that any clone operation may be occurring concurrently with operations on other clones, and these operations combine to realise the final convergent state of every clone.

The Javascript clone engine API supports bounded procedures on immutable state, for situations where a query or other data operation may want to operate on a state that is unaffected by concurrent operations. In general this means that in order to guarantee data consistency, it is not necessary for the app to use the clone's local clock ticks (which nevertheless appear in places in the API for consistency with other engines).

An immutable state can be obtained using the read and write methods. The state is passed to a procedure which operates on it. In both cases, the state remains immutable until the procedure's returned Promise resolves or rejects.

In the case of write, the state can be transitioned to a new state from within the procedure using its own write method, which returns a new immutable state.

In the case of read, changes to the state following the procedure can be listened to using the second parameter, a handler for new states. As well as each update in turn, this handler also receives an immutable state following the given update.

Example: Initialising an App

await clone.read(async (state: MeldReadState) => {
  // State in a procedure is locked until sync complete or returned promise resolves
  let currentData = await state.read(someQuery);
  populateTheUi(currentData);
}, async (update: MeldUpdate, state: MeldReadState) => {
  // The handler happens for every update after the proc
  // State in a handler is locked until sync complete or returned promise resolves
  updateTheUi(update); // See §Handling Updates, below
});

Example: UI Action Handler

ui.on('action', async () => {
  clone.write(async (state: MeldState) => {
    let currentData = await state.read(something);
    let externalStuff = await doExternals(currentData);
    let todo = decideWhatToDo(externalStuff);
    // Writing to the current state creates a new live state
    state = await state.write(todo);
    await checkStuff(state);
  });
});

Example: UI Show Handler

ui.on('show', async () => {
  clone.read((state: MeldReadState) => {
    let currentData = await state.read(something);
    showTheNewUi(currentData);
  });
});

Handling Updates

Clone updates obtained from a read handler specify the exact Subject property values that have been deleted or inserted during the update. Utilities are provided to help update app views of data based on updates notified via the read method:

Index

Type aliases

Atom

Atom: jrql.Atom

ConstraintConfig

ConstraintConfig: SingleValuedConfig

Configuration of the clone data constraint. The supported constraints are:

  • single-valued: the given property should have only one value. The property can be given in unexpanded form, as it appears in JSON subjects when using the API, or as its full IRI reference.

🚧 Please contact us to discuss data constraints required for your use-case.

Container

Container: List | Set

Used to express an ordered or unordered container of data.

see

json-rql container

Context

Context: Context

A JSON-LD context for some JSON content such as a Subject. m-ld does not require the use of a context, as plain JSON data will be stored in the context of the domain. However in advanced usage, such as for integration with existing systems, it may be useful to provide other context for shared data.

see

json-rql context

ExpandedTermDef

ExpandedTermDef: ExpandedTermDef

An JSON-LD expanded term definition, as part of a domain Context.

see

json-rql expandedtermdef

Expression

Expression: jrql.Atom | Reference | Constraint

A stand-in for a Value used as a basis for filtering.

see

json-rql expression

GraphSubject

GraphSubject: Readonly<Subject & Reference>

A fully-identified Subject from the backend.

LiveStatus

LiveStatus: LiveStatus
see

m-ld specification

MeldStatus

MeldStatus: MeldStatus
see

m-ld specification

Pattern

Pattern: Pattern

A m-ld transaction is a json-rql pattern, which represents a data read or a data write. Supported pattern types are:

see

json-rql pattern

ReadResult

ReadResult: Observable<GraphSubject> & PromiseLike<GraphSubjects>

Convenience return type for reading data from a clone. Use as a Promise with .then or await to obtain the results as an array of identified Subjects. Use as an Observable with .subscribe (or other RxJS methods) to be notified of individual Subjects as they arrive.

Note that all reads operate on a data snapshot, so results will not be affected by concurrent writes, even outside the scope of a read procedure.

see

MeldStateMachine.read

Reference

Reference: jrql.Reference

A reference to a Subject. Used to disambiguate an IRI from a plain string. Unless a custom Context is used for the clone, all references will use this format.

see

json-rql reference

Result

Result: "*" | Variable | Variable[]

Result declaration of a Select query. Use of '*' specifies that all variables in the query should be returned.

StateProc

StateProc<S>: (state: S) => PromiseLike<unknown> | void

A function type specifying a 'procedure' during which a clone state is available as immutable. Strictly, the immutable state is guaranteed to remain 'live' until the procedure's return Promise resolves or rejects.

Type parameters

Type declaration

    • (state: S): PromiseLike<unknown> | void
    • Parameters

      • state: S

      Returns PromiseLike<unknown> | void

SubjectProperty

SubjectProperty: Iri | Variable | "@item" | "@index" | "@type" | ["@list", number | string, undefined | number]

'Properties' of a Subject, including from List and Slot. Strictly, these are possible paths to a SubjectPropertyObject aggregated by the Subject. An @list contains numeric indexes (which may be numeric strings or variables). The second optional index is used for multiple items being inserted at the first index, using an array.

SubjectPropertyObject

SubjectPropertyObject: Value | Container | SubjectPropertyObject[]

The allowable types for a Subject property value, named awkwardly to avoid overloading Object. Represents the "object" of a property, in the sense of the object of discourse.

see

json-rql SubjectPropertyObject

SubjectUpdate

SubjectUpdate: DeleteInsert<GraphSubject | undefined>

An update to a single graph Subject.

SubjectUpdates

SubjectUpdates: {}

A m-ld update notification, indexed by graph Subject ID.

Type declaration

UpdateProc

UpdateProc: (update: MeldUpdate, state: MeldReadState) => PromiseLike<unknown> | void

A function type specifying a 'procedure' during which a clone state is available as immutable following an update. Strictly, the immutable state is guaranteed to remain 'live' until the procedure's return Promise resolves or rejects.

Type declaration

Value

Variable

Variable: jrql.Variable

A query variable, prefixed with "?", used as a placeholder for some value in a query, for example:

{
  "@select": "?name",
  "@where": { "employeeNo": 7, "name": "?name" }
}
see

json-rql variable

Write

Write: Subject | Group | Update

A query pattern that writes data to the domain. A write can be:

  • A Subject (any JSON object not a Read, Group or Update). Interpreted as data to be inserted.
  • A Group containing only a @graph key. Interpreted as containing the data to be inserted.
  • An explicit Update with either an @insert, @delete, or both.

Note that this type does not fully capture the details above. Use isWrite to inspect a candidate pattern.

Variables

EventEmitter

EventEmitter: EventEmitter

Functions

Const any

  • A utility to generate a variable with a unique Id. Convenient to use when generating query patterns in code.

    Returns Variable

array

  • array<T>(value?: T | T[] | null): T[]
  • Utility to normalise a property value according to m-ld data semantics, from a missing value (null or undefined), a single value, or an array of values, to an array of values (empty for missing values). This can simplify processing of property values in common cases.

    Type parameters

    • T

    Parameters

    • Optional value: T | T[] | null

      the value to normalise to an array

    Returns T[]

asSubjectUpdates

  • Provides an alternate view of the update deletes and inserts, by Subject.

    An update is presented with arrays of inserted and deleted subjects:

    {
      "@delete": [{ "@id": "foo", "severity": 3 }],
      "@insert": [
        { "@id": "foo", "severity": 5 },
        { "@id": "bar", "severity": 1 }
      ]
    }
    

    In many cases it is preferable to apply inserted and deleted properties to app data views on a subject-by-subject basis. This property views the above as:

    {
      "foo": {
        "@delete": { "@id": "foo", "severity": 3 },
        "@insert": { "@id": "foo", "severity": 5 }
      },
      "bar": {
        "@delete": {},
        "@insert": { "@id": "bar", "severity": 1 }
      }
    }
    

    Javascript references to other Subjects in a Subject's properties will always be collapsed to json-rql Reference objects (e.g. { '@id': '<iri>' }).

    Parameters

    Returns SubjectUpdates

Const blank

  • blank(): string
  • A utility to generate a unique blank node.

    Returns string

clone

  • Create or initialise a local clone, depending on whether the given LevelDB database already exists. This function returns as soon as it is safe to begin transactions against the clone; this may be before the clone has received all updates from the domain. You can wait until the clone is up-to-date using the MeldClone.status property.

    Parameters

    • backend: AbstractLevelDOWN

      an instance of a leveldb backend

    • remotes: MeldRemotes | {}

      a driver for connecting to remote m-ld clones on the domain. This can be a configured object (e.g. new MqttRemotes(config)) or just the class (MqttRemotes).

    • config: MeldConfig

      the clone configuration

    • Optional options: undefined | { backendEvents?: EventEmitter; constraints?: MeldConstraint[] }

      runtime options

    Returns Promise<MeldClone>

includeValues

  • includeValues(subject: Subject, property: string, ...values: Value[]): void
  • Includes the given value in the Subject property, respecting m-ld data semantics by expanding the property to an array, if necessary.

    Parameters

    • subject: Subject

      the subject to add the value to.

    • property: string

      the property that relates the value to the subject.

    • Rest ...values: Value[]

      the value to add.

    Returns void

includesValue

  • includesValue(subject: Subject, property: string, value?: Value): boolean
  • Determines whether the given set of values contains the given value. This method accounts for the identity semantics of References and Subjects.

    Parameters

    • subject: Subject

      the subject to inspect

    • property: string

      the property to inspect

    • Optional value: Value

      the value to find in the set. If undefined, then wildcard checks for any value at all.

    Returns boolean

isPropertyObject

  • isPropertyObject(property: string, object: Subject["any"]): object is SubjectPropertyObject
  • Determines whether the given property object from a well-formed Subject is a graph edge; i.e. not a @context or the Subject @id.

    Parameters

    • property: string

      the Subject property in question

    • object: Subject["any"]

      the object (value) of the property

    Returns object is SubjectPropertyObject

isRead

  • isRead(p: Pattern): p is Read
  • Determines if the given pattern will read data from the domain.

    Parameters

    • p: Pattern

    Returns p is Read

isWrite

  • isWrite(p: Pattern): p is Write
  • Determines if the given pattern can probably be interpreted as a logical write of data to the domain.

    This function is not exhaustive, and a pattern identified as a write can still turn out to be illogical, for example if it contains an @insert with embedded variables and no @where clause to bind them.

    Returns true if the logical write is a trivial no-op, such as {}, { "@insert": {} } or { "@graph": [] }.

    see

    Write

    Parameters

    • p: Pattern

    Returns p is Write

shortId

  • shortId(spec?: number | string): string
  • Utility to generate a short Id according to the given spec.

    Parameters

    • Default value spec: number | string = 8

      If a number, a random Id will be generated with the given length. If a string, a stable obfuscated Id will be generated for the string with a fast hash.

    Returns string

    a string identifier that is safe to use as an HTML (& XML) element Id

updateSubject

  • Applies an update to the given subject in-place. This method will correctly apply the deleted and inserted properties from the update, accounting for m-ld data semantics.

    Referenced Subjects will also be updated if they have been affected by the given update, deeply. If a reference property has changed to a different object (whether or not that object is present in the update), it will be updated to a json-rql Reference (e.g. { '@id': '<iri>' }).

    Changes are applied to non-@list properties using only L-value assignment, so the given Subject can be safely implemented with property setters, such as using set in a class, or by using defineProperty, or using a Proxy; for example to trigger side-effect behaviour. Removed properties are set to an empty array ([]) to signal emptiness, and then deleted.

    Changes to @list items are enacted in reverse index order, by calling splice. If the @list value is a hash, it will have a length property added. To intercept these calls, re-implement splice on the @list property value.

    CAUTION: If this function is called independently on subjects which reference each other via Javascript references, or share referenced subjects, then the referenced subjects may be updated more than once, with unexpected results. To avoid this, use a SubjectUpdater to process the whole update.

    see

    m-ld data semantics

    Type parameters

    Parameters

    Returns T

uuid

  • uuid(): SUUID
  • Utility to generate a unique UUID for use in a MeldConfig

    Returns SUUID

Legend

  • Property
  • Method
  • Constructor

Generated using TypeDoc. Delivered by Vercel. @m-ld/m-ld - v0.7.0 Source code licensed MIT. Privacy policy