Options
All
  • Public
  • Public/Protected
  • All
Menu

🚧 This documentation is for the developer preview of m-ld.

github licence npm (tag)

m-ld Javascript clone engine

The Javascript engine can be used in a modern browser or a server engine like Node.js.

The Javascript clone engine conforms to the m-ld specification. Its support for transaction pattern complexity is detailed below. Its concurrency model is based on immutable states.

Getting Started

npm install @m-ld/m-ld

To see some executable code, have a look at the Node.js starter project.

Data Persistence

m-ld uses levelup to interface with a LevelDB-compatible storage backend.

  • For the fastest responses use memdown (memory storage).
  • In the browser, use level-js (browser-local storage).
  • In a service, use leveldown (file system storage).

Connecting to Other Clones

A m-ld clone uses a 'remotes' object to communicate with other clones.

  • If you have an MQTT broker available, use MqttRemotes.
  • For a scalable global managed service, use AblyRemotes.

🚧 If your architecture includes some other publish/subscribe service like AMQP, or you would like to use a fully peer-to-peer protocol, please contact us to discuss your use-case.

Initialisation

The clone function initialises the m-ld engine with a leveldb back-end and the clone configuration.

import MemDown from 'memdown';
import { clone, uuid } from '@m-ld/m-ld';
import { MqttRemotes, MeldMqttConfig } from '@m-ld/m-ld/dist/mqtt';

const config: MeldMqttConfig = {
  '@id': uuid(),
  '@domain': 'test.example.org',
  genesis: true,
  mqtt: { hostname: 'mqtt.example.org' }
};

const meld = await clone(new MemDown, MqttRemotes, config);

The clone function returns control as soon as it is safe to start making data transactions against the domain. If this clone has has been re-started from persisted state, it may still be receiving updates from the domain. This can cause a UI to start showing these updates. If instead, you want to wait until the clone has the most recent data, you can add:

await meld.status.becomes({ online: true, outdated: false });

MQTT Remotes

MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. It is convenient to use it for local development or if the deployment environment has an MQTT broker available. See below for specific broker requirements.

The MqttRemotes class and its companion configuration class MeldMqttConfig can be imported or required from '@m-ld/m-ld/dist/mqtt'.

The configuration interface adds an mqtt key to the base MeldConfig. The content of this key is a client options object for MQTT.js. It must not include the will and clientId options, as these are set internally. It must include a hostname or a host and port.

MQTT Broker Requirements

MqttRemotes requires broker support for:

  1. MQTT 3.1
  2. QoS 0 and 1
  3. Retained messages
  4. Last Will and Testament (LWT) messages

A good choice for local development is Aedes.

Ably Remotes

Ably provides infrastructure and APIs to power realtime experiences at scale. It is a managed service, and includes pay-as-you-go developer pricing. It is also convenient to use for global deployments without the need to self-manage a broker.

The AblyRemotes class and its companion configuration class MeldAblyConfig can be imported or required from '@m-ld/m-ld/dist/ably'.

The configuration interface adds an ably key to the base MeldConfig. The content of this key is an Ably client options object. It must not include the echoMessages and clientId options, as these are set internally.

If using token authentication, ensure that the clientId the token is generated for corresponds to the @id given in the MeldConfig.

Transactions

A m-ld transaction is a json-rql pattern, which represents a data read or a data write. See the m-ld specification for a walk-through of the syntax.

Supported pattern types for this engine are (follow the links for examples):

🚧 If you have a requirement for an unsupported pattern, please contact us to discuss your use-case. You can browse the full json-rql syntax at json-rql.org.

Concurrency

A m-ld clone contains realtime domain data in principle. This means that any clone operation may be occurring concurrently with operations on other clones, and these operations combine to realise the final convergent state of every clone.

The Javascript clone engine API supports bounded procedures on immutable state, for situations where a query or other data operation may want to operate on a state that is unaffected by concurrent operations. In general this means that in order to guarantee data consistency, it is not necessary for the app to use the clone's local clock ticks (which nevertheless appear in places in the API for consistency with other engines).

An immutable state can be obtained using the read and write methods. The state is passed to a procedure which operates on it. In both cases, the state remains immutable until the procedure's returned Promise resolves or rejects.

In the case of write, the state can be transitioned to a new state from within the procedure using its own write method, which returns a new immutable state.

In the case of read, changes to the state following the procedure can be listened to using the second parameter, a handler for new states. As well as each update in turn, this handler also receives an immutable state following the given update.

Example: Initialising an App

await clone.read(async (state: MeldReadState) => {
  // State in a procedure is locked until sync complete or returned promise resolves
  let currentData = await state.read(someQuery);
  populateTheUi(currentData);
}, async (update: MeldUpdate, state: MeldReadState) => {
  // The handler happens for every update after the proc
  // State in a handler is locked until sync complete or returned promise resolves
  updateTheUi(update); // See §Handling Updates, below
});

Example: UI Action Handler

ui.on('action', async () => {
  clone.write(async (state: MeldState) => {
    let currentData = await state.read(something);
    let externalStuff = await doExternals(currentData);
    let todo = decideWhatToDo(externalStuff);
    // Writing to the current state creates a new live state
    state = await state.write(todo);
    await checkStuff(state);
  });
});

Example: UI Show Handler

ui.on('show', async () => {
  clone.read((state: MeldReadState) => {
    let currentData = await state.read(something);
    showTheNewUi(currentData);
  });
});

Handling Updates

Clone updates obtained from a read handler specify the exact Subject property values that have been deleted or inserted during the update. Utility methods are provided to help update app views of data based on updates notified via the read method:

Index

Type aliases

ConstraintConfig

ConstraintConfig: SingleValuedConfig

Configuration of the clone data constraint. The supported constraints are:

  • single-valued: the given property should have only one value. The property can be given in unexpanded form, as it appears in JSON subjects when using the API, or as its full IRI reference.

🚧 Please contact us to discuss data constraints required for your use-case.

Container

Container: List | Set

Used to express an ordered or unordered container of data.

see

https://json-rql.org/interfaces/container.html

Context

Context: Context

A JSON-LD context for some JSON content such as a Subject. m-ld does not require the use of a context, as plain JSON data will be stored in the context of the domain. However in advanced usage, such as for integration with existing systems, it may be useful to provide other context for shared data.

see

https://json-rql.org/interfaces/context.html

ExpandedTermDef

ExpandedTermDef: ExpandedTermDef

An JSON-LD expanded term definition, as part of a domain Context.

see

https://json-rql.org/interfaces/expandedtermdef.html

LiveStatus

LiveStatus: LiveStatus
see

m-ld specification

MeldStatus

MeldStatus: MeldStatus
see

m-ld specification

Pattern

Pattern: Pattern

A m-ld transaction is a json-rql pattern, which represents a data read or a data write. Supported pattern types are:

see

https://json-rql.org/interfaces/pattern.html

ReadResult

ReadResult<T>: Observable<T> & PromiseLike<T[]>

Convenience return type for reading data from a clone. Use as a Promise with .then or await to obtain the results as an array of Subjects. Use as an Observable with .subscribe (or other RxJS methods) to be notified of individual Subjects as they arrive.

Note that all reads operate on a data snapshot, so results will not be affected by concurrent writes, even outside the scope of a read procedure.

see

MeldStateMachine.read

Type parameters

  • T

Reference

Reference: jrql.Reference

A reference to a Subject. Used to disambiguate an IRI from a plain string. Unless a custom Context is used for the clone, all references will use this format.

see

https://json-rql.org/#reference

Resource

Resource<T>: Subject & Reference & {}

Captures m-ld data semantics as applied to an app-specific subject type T. Applies the following changes to T:

  • Always includes an @id property, as the subject identity
  • Non-array properties are redefined to allow arrays (note this is irrespective of any single-valued constraint, which is applied at runtime)
  • Required properties are redefined to allow undefined

Since any property can have zero to many values, it may be convenient to combine use of this type with the array utility when processing updates.

Note that a Resource always contains concrete data, unlike Subject, which can include variables and filters as required for its role in query specification.

Type parameters

  • T

    the app-specific subject type of interest

Result

Result: "*" | Variable | Variable[]

Result declaration of a Select query. Use of '*' specifies that all variables in the query should be returned.

StateProc

StateProc<S>: (state: S) => PromiseLike<unknown> | void

A function type specifying a 'procedure' during which a clone state is available as immutable. Strictly, the immutable state is guaranteed to remain 'live' until the procedure's return Promise resolves or rejects.

Type parameters

Type declaration

    • (state: S): PromiseLike<unknown> | void
    • Parameters

      • state: S

      Returns PromiseLike<unknown> | void

SubjectPropertyObject

SubjectPropertyObject: Value | Container | SubjectPropertyObject[]

The allowable types for a Subject property value, named awkwardly to avoid overloading Object. Represents the "object" of a property, in the sense of the object of discourse.

see

https://json-rql.org/#SubjectPropertyObject

SubjectUpdates

SubjectUpdates: {}

A m-ld update notification, indexed by Subject.

see

asSubjectUpdates

Type declaration

UpdateProc

UpdateProc: (update: MeldUpdate, state: MeldReadState) => PromiseLike<unknown> | void

A function type specifying a 'procedure' during which a clone state is available as immutable following an update. Strictly, the immutable state is guaranteed to remain 'live' until the procedure's return Promise resolves or rejects.

Type declaration

Value

Value: jrql.Atom | Subject | Reference

Variable

Variable: jrql.Variable

A query variable, prefixed with "?", used as a placeholder for some value in a query, for example:

{
  "@select": "?name",
  "@where": { "employeeNo": 7, "name": "?name" }
}
see

https://json-rql.org/#variable

Write

Write: Subject | Group | Update

A query pattern that writes data to the domain. A write can be:

  • A Subject (any JSON object not a Read, Group or Update). Interpreted as data to be inserted.
  • A Group containing only a @graph key. Interpreted as containing the data to be inserted.
  • An explicit Update with either an @insert, @delete, or both.

Note that this type does not fully capture the details above. Use isWrite to inspect a candidate pattern.

Functions

any

  • A utility to generate a variable with a unique Id. Convenient to use when generating query patterns in code.

    Returns Variable

array

  • array<T>(value?: T | T[]): T[]
  • Utility to normalise a property value according to m-ld data semantics, from a missing value (null or undefined), a single value, or an array of values, to an array of values (empty for missing values). This can simplify processing of property values in common cases.

    Type parameters

    • T

    Parameters

    • Optional value: T | T[]

      the value to normalise to an array

    Returns T[]

asSubjectUpdates

  • Indexes a m-ld update notification by Subject.

    By default, updates are presented with arrays of inserted and deleted subjects:

    {
      "@delete": [{ "@id": "foo", "severity": 3 }],
      "@insert": [
        { "@id": "foo", "severity": 5 },
        { "@id": "bar", "severity": 1 }
      ]
    }

    In many cases it is preferable to apply inserted and deleted properties to app data views on a subject-by-subject basis. This method transforms the above into:

    {
      "foo": {
        "@delete": { "@id": "foo", "severity": 3 },
        "@insert": { "@id": "foo", "severity": 5 }
      },
      "bar": {
        "@delete": {},
        "@insert": { "@id": "bar", "severity": 1 }
      }
    }

    Parameters

    Returns SubjectUpdates

clone

  • Create or initialise a local clone, depending on whether the given LevelDB database already exists. This function returns as soon as it is safe to begin transactions against the clone; this may be before the clone has received all updates from the domain. You can wait until the clone is up-to-date using the MeldClone.status property.

    Parameters

    • backend: AbstractLevelDOWN

      an instance of a leveldb backend

    • remotes: MqttRemotes | AblyRemotes

      a driver for connecting to remote m-ld clones on the domain. This can be a configured object (e.g. new MqttRemotes(config)) or just the class (MqttRemotes).

    • config: MeldConfig

      the clone configuration

    • Optional constraints: MeldConstraint[]

      constraints in addition to those in the configuration. 🚧 Experimental: use with caution.

    Returns Promise<MeldClone>

includeValue

  • includeValue(subject: Subject, property: string, value: Value): void
  • Includes the given value in the Subject property, respecting m-ld data semantics by expanding the property to an array, if necessary.

    Parameters

    • subject: Subject

      the subject to add the value to.

    • property: string

      the property that relates the value to the subject.

    • value: Value

      the value to add.

    Returns void

includesValue

  • includesValue(subject: Subject, property: string, value?: Value): boolean
  • Determines whether the given set of values contains the given value. This method accounts for the identity semantics of References and Subjects.

    Parameters

    • subject: Subject
    • property: string
    • Optional value: Value

      the value to find in the set. If undefined, then wildcard checks for any value at all.

    Returns boolean

isPropertyObject

  • isPropertyObject(property: string, object: Subject["any"]): object is SubjectPropertyObject
  • Determines whether the given property object from a well-formed Subject is a graph edge; i.e. not a @context or the Subject @id.

    Parameters

    • property: string

      the Subject property in question

    • object: Subject["any"]

      the object (value) of the property

    Returns object is SubjectPropertyObject

isRead

  • isRead(p: Pattern): p is Read
  • Determines if the given pattern will read data from the domain.

    Parameters

    • p: Pattern

    Returns p is Read

isWrite

  • isWrite(p: Pattern): p is Write
  • Determines if the given pattern can probably be interpreted as a logical write of data to the domain.

    This function is not exhaustive, and a pattern identified as a write can still turn out to be illogical, for example if it contains an @insert with embedded variables and no @where clause to bind them.

    Returns true if the logical write is a trivial no-op, such as {}, { "@insert": {} } or { "@graph": [] }.

    see

    Write

    Parameters

    • p: Pattern

    Returns p is Write

shortId

  • shortId(spec?: number | string): string
  • Utility to generate a short Id according to the given spec.

    Parameters

    • Default value spec: number | string = 8

      If a number, a random Id will be generated with the given length. If a string, a stable obfuscated Id will be generated for the string with a fast hash.

    Returns string

    a string identifier that is safe to use as an HTML (& XML) element Id

updateSubject

  • Applies a subject update to the given subject, expressed as a Resource. This method will correctly apply the deleted and inserted properties from the update, accounting for m-ld data semantics.

    Type parameters

    • T

      the app-specific subject type of interest

    Parameters

    Returns Resource<T>

uuid

  • uuid(): string
  • Utility to generate a unique UUID for use in a MeldConfig

    Returns string

Legend

  • Property
  • Method

Generated using TypeDoc. Delivered by Vercel. @m-ld/m-ld - v0.5.0 Source code licensed MIT. Privacy policy