Options
All
  • Public
  • Public/Protected
  • All
Menu

🚧 This documentation is for the developer preview of m-ld.

github licence npm (tag)

m-ld Javascript clone engine

The Javascript engine can be used in a modern browser or a server engine like Node.js.

The Javascript clone engine conforms to the m-ld specification. Its support for transaction pattern complexity is detailed below. Its concurrency model is based on immutable states.

Getting Started

npm install @m-ld/m-ld

There are two starter projects available:

  • The Node.js project uses Node processes to initialise two clones, and an MQTT broker for messaging.
  • The Web project shows one way to build a multi-collaborator forms application for browsers, using Socket.io for messaging.

Data Persistence

m-ld uses levelup to interface with a LevelDB-compatible storage backend.

For the fastest in-memory responses, this library provides MeldMemDown, an optimised variant of memdown, which can be imported from '@m-ld/m-ld/dist/memdown' as shown in the example below. To use, you must also install the memdown package as a peer of @m-ld/m-ld.

In a service or native application, use leveldown (file system storage).

In a browser, you can use level-js (browser-local storage).

Connecting to Other Clones

A m-ld clone uses a 'remotes' object to communicate with other clones.

  • With an MQTT broker, use MqttRemotes.
  • For a scalable global managed service, use AblyRemotes.
  • If you have a live web server (not just CDN or serverless), you can use IoRemotes.

🚧 If your architecture includes some other publish/subscribe service like AMQP or Apache Kafka, or you would like to use a fully peer-to-peer protocol, please contact us to discuss your use-case. Remotes can even utilise multiple transport protocols, for example WebRTC with a suitable signalling service.

Initialisation

The clone function initialises the m-ld engine with a leveldb back-end and the clone configuration.

import { clone, uuid } from '@m-ld/m-ld';
import { MeldMemDown } from '@m-ld/m-ld/dist/memdown';
import { MqttRemotes, MeldMqttConfig } from '@m-ld/m-ld/dist/mqtt';

const config: MeldMqttConfig = {
  '@id': uuid(),
  '@domain': 'test.example.org',
  genesis: true,
  mqtt: { hostname: 'mqtt.example.org' }
};

const meld = await clone(new MemDown, MqttRemotes, config);

The clone function returns control as soon as it is safe to start making data transactions against the domain. If this clone has has been re-started from persisted state, it may still be receiving updates from the domain. This can cause a UI to start showing these updates. If instead, you want to wait until the clone has the most recent data, you can add:

await meld.status.becomes({ online: true, outdated: false });

Remotes

MQTT Remotes

MQTT is a machine-to-machine (M2M)/"Internet of Things" connectivity protocol. It is convenient to use it for local development or if the deployment environment has an MQTT broker available. See below for specific broker requirements.

The MqttRemotes class and its companion configuration class MeldMqttConfig can be imported or required from '@m-ld/m-ld/dist/mqtt'. You must also install the async-mqtt package as a peer of @m-ld/m-ld.

The configuration interface adds an mqtt key to the base MeldConfig. The content of this key is a client options object for MQTT.js. It must not include the will and clientId options, as these are set internally. It must include a hostname or a host and port, e.g.

const config = {
  '@id': uuid(), '@domain': 'test.example.org', genesis: true,
  mqtt: { host: 'localhost', port: 1883 }
};

MQTT Broker Requirements

MqttRemotes requires broker support for:

  1. MQTT 3.1
  2. QoS 0 and 1
  3. Retained messages
  4. Last Will and Testament (LWT) messages

A good choice for local development is Aedes.

MQTT in the Browser

MQTT remotes supports websockets for use in a browser environment. To configure, add protocol: 'ws' (or 'wss') to the mqtt configuration value. (Note that All the MQTT configuration goes through the mqtt key, even if it's actually using websockets for transport.) This requires the MQTT broker to support websocket connections, for example see the Aedes documentation.

Ably Remotes

Ably provides infrastructure and APIs to power realtime experiences at scale. It is a managed service, and includes pay-as-you-go developer pricing. It is also convenient to use for global deployments without the need to self-manage a broker.

The AblyRemotes class and its companion configuration class MeldAblyConfig can be imported or required from '@m-ld/m-ld/dist/ably'. You must also install the ably package as a peer of @m-ld/m-ld.

The configuration interface adds an ably key to the base MeldConfig. The content of this key is an Ably client options object. It must not include the echoMessages and clientId options, as these are set internally.

If using token authentication, ensure that the clientId the token is generated for corresponds to the @id given in the MeldConfig.

Socket.io Remotes

Socket.IO enables real-time, bidirectional and event-based communication. It works on every platform, browser or device, focusing equally on reliability and speed. It is convenient to use when the app architecture has a live web server or app server, using HTTP.

The IoRemotes class and its companion configuration class MeldIoConfig can be imported or required from '@m-ld/m-ld/dist/socket.io'. You must also install the socket.io-client package as a peer of @m-ld/m-ld.

The configuration interface adds an io key to the base MeldConfig. The value is an optional object having:

  • uri: The server URL (defaults to the browser URL with no path)
  • opts: A Socket.io factory options object, which can be used to customise the server connection

Socket.io Server

When using Socket.io, the server must correctly route m-ld protocol operations to their intended recipients. The Javascript engine package bundles a class for Node.js servers, IoRemotesService, which can be imported from '@m-ld/m-ld/dist/socket.io/server'.

To use, initialise the Socket.io server as normal, and then construct an IoRemotesService, passing the namespace you want to make available to m-ld. To use the global namespace, pass the sockets member of the Server class. For example:

const socket = require('socket.io');
const httpServer = require('http').createServer();
// Start the Socket.io server, and attach the m-ld message-passing service
const io = new socket.Server(httpServer);
new IoRemotesService(io.sockets);

For a complete example, see the web starter project .

For other server types, contact us.

Transactions

A m-ld transaction is a json-rql pattern, which represents a data read or a data write. See the m-ld specification for a walk-through of the syntax.

Supported pattern types for this engine are (follow the links for examples):

🚧 If you have a requirement for an unsupported pattern, please contact us to discuss your use-case. You can browse the full json-rql syntax at json-rql.org.

Subjects

Subjects in the Javascript engine are accepted and presented as plain Javascript objects whose content is JSON-LD (see the m-ld Specification). Utilities are provided to help the app use and produce valid subjects.

  • includeValues includes one or more given value in a subject property.
  • includesValue determines whether the given subject property contains the given value.
  • propertyValue gets a property value from the given subject, casted as a strict type.

Handling Updates

Clone updates obtained from a read handler specify the exact Subject property values that have been deleted or inserted during the update. Since apps often maintain subjects in memory, for example in a user interface, utilities are provided to help update these in-memory subjects based on updates:

  • updateSubject applies an update to the given subject in-place.
  • SubjectUpdater applies an update to more than one subject.
  • asSubjectUpdates provides an alternate view of the update deletes and inserts, by Subject, for custom processing.

Concurrency

A m-ld clone contains realtime domain data in principle. This means that any clone operation may be occurring concurrently with operations on other clones, and these operations combine to realise the final convergent state of every clone.

The Javascript clone engine API supports bounded procedures on immutable state, for situations where a query or other data operation may want to operate on a state that is unaffected by concurrent operations. In general this means that in order to guarantee data consistency, it is not necessary for the app to use the clone's local clock ticks (which nevertheless appear in places in the API for consistency with other engines).

An immutable state can be obtained using the read and write methods. The state is passed to a procedure which operates on it. In both cases, the state remains immutable until the procedure's returned Promise resolves or rejects.

In the case of write, the state can be transitioned to a new state from within the procedure using its own write method, which returns a new immutable state.

In the case of read, changes to the state following the procedure can be listened to using the second parameter, a handler for new states. As well as each update in turn, this handler also receives an immutable state following the given update.

Example: Initialising an App

await clone.read(async (state: MeldReadState) => {
  // State in a procedure is locked until sync complete or returned promise resolves
  let currentData = await state.read(someQuery);
  populateTheUi(currentData);
}, async (update: MeldUpdate, state: MeldReadState) => {
  // The handler happens for every update after the proc
  // State in a handler is locked until sync complete or returned promise resolves
  updateTheUi(update); // See §Handling Updates, below
});

Example: UI Action Handler

ui.on('action', async () => {
  clone.write(async (state: MeldState) => {
    let currentData = await state.read(something);
    let externalStuff = await doExternals(currentData);
    let todo = decideWhatToDo(externalStuff);
    // Writing to the current state creates a new live state
    state = await state.write(todo);
    await checkStuff(state);
  });
});

Example: UI Show Handler

ui.on('show', () => {
  clone.read(async (state: MeldReadState) => {
    let currentData = await state.read(something);
    showTheNewUi(currentData);
  });
});

Security

To mitigate integrity, confidentiality and availability threats to m-ld domain data, we recommend the following baseline security controls for your app.

  1. Apply secure application development practices, for example as recommended in the UK National Cyber Security Centre Guidelines.
  2. Apply operating-system access control to the storage location used for persistence, restricting access to authorised users of the app and administrators.
  3. Apply authorisation controls to remote connections, restricting access to authorised users of the app. This typically requires users to be authenticated in the app prior to connecting the m-ld clone. Authentication credentials can be passed to the remotes implementation via the configuration object.
  4. Encrypt remote connection transports, for example by applying transport layer security (TLS) to server connections, if applicable.

A more general discussion of security considerations in m-ld can be found on the website.

🚧 This library additionally includes an experimental extension for controlling access based on an Access Control List (ACL) in the domain data. Please see our Security Project for more details of our ongoing security research, and contact us to discuss your security requirements!

Index

Configuration Type aliases

ConstraintConfig

ConstraintConfig: SingleValuedConfig

Configuration of the clone data constraint. The supported constraints are:

  • single-valued: the given property should have only one value. The property can be given in unexpanded form, as it appears in JSON subjects when using the API, or as its full IRI reference.

🚧 Please contact us to discuss data constraints required for your use-case.

InitialApp

InitialApp: MeldApp & Partial<MeldExtensions>

Initial definition of a m-ld app. Extensions provided will be used for bootstrapping, prior to the clone joining the domain. After that, different extensions may come into effect if so declared in the data.

API Type aliases

GraphSubject

GraphSubject: Readonly<Subject & Reference>

A fully-identified Subject from the backend.

json-rql Type aliases

Context

Context: Context

A JSON-LD context for some JSON content such as a Subject. m-ld does not require the use of a context, as plain JSON data will be stored in the context of the domain. However in advanced usage, such as for integration with existing systems, it may be useful to provide other context for shared data.

see

json-rql context

Reference

Reference: jrql.Reference

A reference to a Subject. Used to disambiguate an IRI from a plain string. Unless a custom Context is used for the clone, all references will use this format.

This type is also used to distinguish identified subjects (with an @id field) from anonymous ones (without an @id field).

see

json-rql reference

VocabReference

VocabReference: { @vocab: Iri }

Like a Reference, but used for "vocabulary" references. These are relevant to:

  • Subject properties: the property name is a vocabulary reference
  • Subject @type: the type value is a vocabulary reference
  • Any value for a property that has been defined as @vocab in the Context
see

https://www.w3.org/TR/json-ld/#default-vocabulary

Type declaration

  • @vocab: Iri

Other Type aliases

Atom

Atom: jrql.Atom

BaseStream

BaseStream<T>: EventEmitter & { read: () => T | null }

Abstract stream of any type; implicit supertype of an RDFJS Stream

Type parameters

  • T

Container

Container: List | Set

Used to express an ordered or unordered container of data.

see

json-rql container

ExpandedTermDef

ExpandedTermDef: ExpandedTermDef

An JSON-LD expanded term definition, as part of a domain Context.

see

json-rql expandedtermdef

Expression

Expression: jrql.Atom | Constraint

A stand-in for a Value used as a basis for filtering.

see

json-rql expression

LiveStatus

LiveStatus: LiveStatus
see

m-ld specification

MeldStatus

MeldStatus: MeldStatus
see

m-ld specification

NativeAtomConstructor

NativeAtomConstructor: typeof Number | typeof String | typeof Boolean | typeof Date | typeof Object | typeof Uint8Array

NativeContainerConstructor

NativeContainerConstructor: typeof Array | typeof Set

NativeValueConstructor

Pattern

Pattern: Pattern

A m-ld transaction is a json-rql pattern, which represents a data read or a data write. Supported pattern types are:

see

json-rql pattern

Result

Result: "*" | Variable | Variable[]

Result declaration of a Select query. Use of '*' specifies that all variables in the query should be returned.

StateProc

StateProc<S, T>: (state: S) => PromiseLike<T> | T

A function type specifying a 'procedure' during which a clone state is available as immutable. Strictly, the immutable state is guaranteed to remain 'live' until the procedure's return Promise resolves or rejects.

Type parameters

Type declaration

    • (state: S): PromiseLike<T> | T
    • Parameters

      • state: S

      Returns PromiseLike<T> | T

SubjectProperty

SubjectProperty: Iri | Variable | "@item" | "@index" | "@type" | ["@list", number | string, undefined | number]

'Properties' of a Subject, including from List and Slot. Strictly, these are possible paths to a SubjectPropertyObject aggregated by the Subject. An @list contains numeric indexes (which may be numeric strings or variables). The second optional index is used for multiple items being inserted at the first index, using an array.

SubjectPropertyObject

SubjectPropertyObject: Value | Container | SubjectPropertyObject[]

The allowable types for a Subject property value, named awkwardly to avoid overloading Object. Represents the "object" of a property, in the sense of the object of discourse.

see

json-rql SubjectPropertyObject

SubjectUpdate

SubjectUpdate: DeleteInsert<GraphSubject | undefined>

An update to a single graph Subject.

SubjectUpdates

SubjectUpdates: {}

A m-ld update notification, indexed by graph Subject ID.

Type declaration

UpdateProc

UpdateProc: (update: MeldUpdate, state: MeldReadState) => PromiseLike<unknown> | void

A function type specifying a 'procedure' during which a clone state is available as immutable following an update. Strictly, the immutable state is guaranteed to remain 'live' until the procedure's return Promise resolves or rejects.

Type declaration

Value

ValueObject

ValueObject: ValueObject

Variable

Variable: jrql.Variable

A query variable, prefixed with "?", used as a placeholder for some value in a query, for example:

{
  "@select": "?name",
  "@where": { "employeeNo": 7, "name": "?name" }
}
see

json-rql variable

Write

Write: Subject | Group | Update

A query pattern that writes data to the domain. A write can be:

  • A Subject (any JSON object not a Read, Group or Update). Interpreted as data to be inserted.
  • A Group containing only a @graph key. Interpreted as containing the data to be inserted.
  • An explicit Update with either an @insert, @delete, or both.

Note that this type does not fully capture the details above. Use isWrite to inspect a candidate pattern.

API Functions

clone

  • Create or initialise a local clone, depending on whether the given LevelDB database already exists. This function returns as soon as it is safe to begin transactions against the clone; this may be before the clone has received all updates from the domain. You can wait until the clone is up-to-date using the MeldClone.status property.

    Parameters

    • backend: AbstractLevelDOWN

      an instance of a leveldb backend

    • constructRemotes: ConstructRemotes

      remotes constructor

    • config: MeldConfig

      the clone configuration

    • Default value app: InitialApp = {}

    Returns Promise<MeldClone>

json-rql Functions

isPropertyObject

  • isPropertyObject(property: string, object: Subject["any"]): object is SubjectPropertyObject
  • Determines whether the given property object from a well-formed Subject is a graph edge; i.e. not a @context or the Subject @id.

    Parameters

    • property: string

      the Subject property in question

    • object: Subject["any"]

      the object (value) of the property

    Returns object is SubjectPropertyObject

isRead

  • isRead(p: Pattern): p is Read
  • Determines if the given pattern will read data from the domain.

    Parameters

    • p: Pattern

    Returns p is Read

isWrite

  • isWrite(p: Pattern): p is Write
  • Determines if the given pattern can probably be interpreted as a logical write of data to the domain.

    This function is not exhaustive, and a pattern identified as a write can still turn out to be illogical, for example if it contains an @insert with embedded variables and no @where clause to bind them.

    Returns true if the logical write is a trivial no-op, such as {}, { "@insert": {} } or { "@graph": [] }.

    see

    Write

    Parameters

    • p: Pattern

    Returns p is Write

Utility Functions

Const any

  • A utility to generate a variable with a unique Id. Convenient to use when generating query patterns in code.

    Returns Variable

array

  • array<T>(value?: T | T[] | null): T[]
  • Utility to normalise a property value according to m-ld data semantics, from a missing value (null or undefined), a single value, or an array of values, to an array of values (empty for missing values). This can simplify processing of property values in common cases.

    Type parameters

    • T

    Parameters

    • Optional value: T | T[] | null

      the value to normalise to an array

    Returns T[]

asSubjectUpdates

  • Provides an alternate view of the update deletes and inserts, by Subject.

    An update is presented with arrays of inserted and deleted subjects:

    {
      "@delete": [{ "@id": "foo", "severity": 3 }],
      "@insert": [
        { "@id": "foo", "severity": 5 },
        { "@id": "bar", "severity": 1 }
      ]
    }
    

    In many cases it is preferable to apply inserted and deleted properties to app data views on a subject-by-subject basis. This property views the above as:

    {
      "foo": {
        "@delete": { "@id": "foo", "severity": 3 },
        "@insert": { "@id": "foo", "severity": 5 }
      },
      "bar": {
        "@delete": {},
        "@insert": { "@id": "bar", "severity": 1 }
      }
    }
    

    Javascript references to other Subjects in a Subject's properties will always be collapsed to json-rql Reference objects (e.g. { '@id': '<iri>' }).

    Parameters

    Returns SubjectUpdates

Const blank

  • blank(): string
  • A utility to generate a unique blank node.

    Returns string

castPropertyValue

  • Casts a property value to the given type. This is a typesafe cast which will not perform type coercion e.g. strings to numbers.

    throws

    TypeError if the given property does not have the correct type

    Type parameters

    • T

    Parameters

    Returns T

includeValues

  • includeValues(subject: Subject, property: string, ...values: Value[]): void
  • Includes the given value in the Subject property, respecting m-ld data semantics by expanding the property to an array, if necessary.

    Parameters

    • subject: Subject

      the subject to add the value to.

    • property: string

      the property that relates the value to the subject.

    • Rest ...values: Value[]

      the value to add.

    Returns void

includesValue

  • includesValue(subject: Subject, property: string, value?: Value): boolean
  • Determines whether the given set of subject property has the given value. This method accounts for the identity semantics of References and Subjects.

    Parameters

    • subject: Subject

      the subject to inspect

    • property: string

      the property to inspect

    • Optional value: Value

      the value to find in the set. If undefined, then wildcard checks for any value at all.

    Returns boolean

propertyValue

  • Extracts a property value from the given subject with the given type. This is a typesafe cast which will not perform type coercion e.g. strings to numbers.

    Per m-ld data semantics, a single value in a field is equivalent to a singleton set (see example), and will also cast successfully to a singleton array.

    Examples:

    propertyValue({ name: 'Fred' }, 'name', String); // => 'Fred'
    propertyValue({ name: 'Fred' }, 'name', Number); // => throws TypeError
    propertyValue({ name: 'Fred' }, 'name', Set, String); // => Set(['Fred'])
    propertyValue({ name: 'Fred' }, 'age', Set); // => Set([])
    propertyValue({
      shopping: { '@list': ['Bread', 'Milk'] }
    }, 'shopping', Array, String); // => ['Bread', 'Milk']
    propertyValue({
      birthday: {
        '@value': '2022-01-08T16:49:43.572Z',
        '@type': 'http://www.w3.org/2001/XMLSchema#dateTime'
      }
    }, 'birthday', Date); // => Javascript Date
    
    throws

    TypeError if the given property does not have the correct type

    Type parameters

    • T

    Parameters

    • subject: Subject

      the subject to inspect

    • property: string

      the property to inspect

    • type: NativeValueConstructor & {}

      the expected type for the returned value

    • Optional subType: NativeAtomConstructor

      if type is Array or Set, the expected item type. If not provided, values in a multi-valued property will not be cast

    Returns T

shortId

  • shortId(spec?: number | string): string
  • Utility to generate a short Id according to the given spec.

    Parameters

    • Default value spec: number | string = 8

      If a number, a random Id will be generated with the given length. If a string, a stable obfuscated Id will be generated for the string with a fast hash.

    Returns string

    a string identifier that is safe to use as an HTML (& XML) element Id

updateSubject

  • Applies an update to the given subject in-place. This method will correctly apply the deleted and inserted properties from the update, accounting for m-ld data semantics.

    Referenced Subjects will also be updated if they have been affected by the given update, deeply. If a reference property has changed to a different object (whether or not that object is present in the update), it will be updated to a json-rql Reference (e.g. { '@id': '<iri>' }).

    Changes are applied to non-@list properties using only L-value assignment, so the given Subject can be safely implemented with property setters, such as using set in a class, or by using defineProperty, or using a Proxy; for example to trigger side-effect behaviour. Removed properties are set to an empty array ([]) to signal emptiness, and then deleted.

    Changes to @list items are enacted in reverse index order, by calling splice. If the @list value is a hash, it will have a length property added. To intercept these calls, re-implement splice on the @list property value.

    CAUTION: If this function is called independently on subjects which reference each other via Javascript references, or share referenced subjects, then the referenced subjects may be updated more than once, with unexpected results. To avoid this, use a SubjectUpdater to process the whole update.

    see

    m-ld data semantics

    Type parameters

    Parameters

    Returns T

uuid

  • uuid(from?: Buffer): SUUID
  • Utility to generate a unique short UUID for use in a MeldConfig

    Parameters

    • Optional from: Buffer

    Returns SUUID

Legend

  • Property
  • Method
  • Constructor

Generated using TypeDoc. Delivered by Vercel. @m-ld/m-ld - v0.8.2 Source code licensed MIT. Privacy policy