Holistic Application Platform

Holistic means:

  • Customizable - Made for applications with service/customer specific deployments
  • Deployable - JVM + single JAR + static web apps + automated bootstrap process on *nix and Windows servers
  • Extendable - Meta model to represent any domain and extensible backend to plug-in domain specific services
  • Integrated - One build system for all artifacts + SCM + build server + software repository
  • Open - Built on the giant shoulders of free and open source software
  • Productive - From setting up the developer environment to provisioning and monitoring servers in production
  • Testable - Write and run automated unit, UI, integration and deployment tests
  • Web-based - JSON based WebSocket and REST-like API with Java and JavaScript clients

Enables developers to efficiently and rapidly:

  • Setup development environments for their team
  • Build domain specific applications based on a solid, extensible foundation
  • Thoroughly test the applications in automated ways
  • Build modules for the automated deployment of applications
  • Create customer specific (or otherwise customized) versions of modules
  • Provision hosted, on-premise or cloud servers with modules
  • Monitor application runtimes including metrics, healthchecks and alerts

Contact:

Content:

Guidelines

Built for:

  • Monoliths, Microservices, Sphagetti, Lasagna and Ravioli architectures
  • Near real time data collection, processing and display
  • Problems and applications that are based on or interact with physical objects
  • Simple, object oriented domain models without complex relations
  • Small and medium sized business problems

Not built for:

  • The Next Big-Tech Web-Scale Twooglebookflix

Principles:

  • Automate what you can. Don’t create or realy on automagic. Use automation as a form of documentation
  • Keep it simple, sometimes boring is better
  • Minimize abstractions, dependencies and lockin
  • Monorepos
  • Move fast and try to not break too many things
  • Remove cruft and YAGNI
  • Update frequently, to the bleeding edge if needed/required

Code

  • Alwayse use curly brackets for control flow statements
if (...) {
    // We don't like it simple
}
  • Only use < and <=
    • if (0 < limit && limit < 10)
  • Don’t use preincrement/postincrement operators
    • i = i + 1
  • Don’t use negation !
    • if (hasEnded === false)
  • For JavaScript, the rules of JSLint (and its author) stand above all others

Formatting

Setup formatters/extensions according to setup guide.

  • Java/Xtend: Ctrl+Shift+F and Ctrl+Shift+O in Eclipse
  • CSS: Shift+Alt+F in Visual Studio Code
  • HTML: Shift+Alt+F in Visual Studio Code
  • JavaScript: Manual formatting that conforms to JSLint
  • JSON: Shift+Alt+F in Visual Studio Code
  • XML: Shift+Alt+F in Visual Studio Code
  • Markdown: Manual formatting

Components

Central framework components and modules that are part of the platform source repository. It’s possible to use just parts of the framework and build applications using any build system in the Java/JavaScript ecosystem. But the easiest way is to stick to the same tools and principles the platform is built on.

See the Try and Extend chapters on how to get started.

Technical basics:

  • All binary artifacts are built with Maven, or at least Maven triggers the specific build systems
    • Server side Xtend code is transpiled to Java and then packaged as JAR or Shaded JAR
    • Client side HTML, CSS and JavaScript resources are bundled using Node.js with r.js and packaged into static ZIP archives and published to the Maven repository
    • Android apps are built using Gradle and published to the Maven repository
  • Java
    • Code is structured into different Maven modules and Java packages as usual
    • Relies on Dagger for dependency injection
  • JavaScript
    • Code is structured into AMD modules
    • Relies on alameda for dependency injection

Top level source repository structure:

- android
- bootstrap
- framework
  - bootstrap
  - ide
  - meta
  - util
  - web

Meta

meta is based around a simple meta model to represent any kind of domain specific model. It’s a complete client/server framework built around a collection of central backend services and a WebSocket based API for the frontend. It includes clients in Java and JavaScript and a web-based admin application.

meta-core

Contains the common classes that form the meta model and the protocol between server and client. All of them are implemented in Java and JavaScript. Also contains some meta specific helper classes and utility functions.

The meta model consists of the following entities:

  • MetaType - Defines a type (aka class) using a JSON schema that can then be used to validate objects of this type. Example of a type could be Book with a property Title and Author.
{
    "meta": "type",
    "id": "bfcf07388afd45faa06e259548e39ecd",
    "name": "Book",
    "schema": {
        "$schema": "https://json-schema.org/draft/2019-09/schema",
        "type": "object",
        "additionalProperties": false,
        "required": [
            "title",
            "author"
        ],
        "properties": {
            "title": {
                "title": "Title",
                "description": "Title of this book",
                "type": "string"
            },
            "author": {
                "title": "Author",
                "description": "Author of this book",
                "type": "string"
            },
            "new": {
                "type": "boolean"
            }
        },
        "definitions": {}
    }
}
  • MetaObject - Represents the concrete object of a certain meta type in the form of a JSON object. Example of an object could be of type Book with Title: The Lord of the Rings and Author: J. R. R. Tolkien.
{
    "meta": "object",
    "id": "...",
    "name": "LotR",
    "typeId": "bfcf07388afd45faa06e259548e39ecd",
    "value": {
        "title": "The Lord of the Rings",
        "author": "J. R. R. Tolkien",
        "new": false
    },
    "position": null
}
  • MetaAction - Application specific functions that are outside of normal CRUD operations, or that otherwise not possible using the meta API, can be modeled as actions using JSON schemas to validate their parameter and results. The schemas are optional if no validation should take place before handling the action. Example of an action could be Order Book with a parameter Book ID and Quantity.
{
    "meta": "action",
    "id": "...",
    "name": "Order Book",
    "paramSchema": {
        "$schema": "https://json-schema.org/draft/2019-09/schema",
        "type": "object",
        "additionalProperties": false,
        "required": [
            "bookId",
            "quantity"
        ],
        "properties": {
            "bookId": {
                "type": "string"
            },
            "quantity": {
                "type": "number"
            }
        },
        "definitions": {}
    },
    "resultSchema": {
        "$schema": "https://json-schema.org/draft/2019-09/schema",
        "type": "object",
        "additionalProperties": false,
        "required": [],
        "properties": {
            "orderId": {
                "type": "string"
            }
        },
        "definitions": {}
    }
}
  • MetaLogEntry - Changes to types, objects or actions over time are recorded in log entries. Some log entries are automatically generated (eg. for CRUD operations) but they can also be created manually by the application to record noteworthy events for specific meta entities
{
    "meta": "log",
    "id": "...",
    "name": null,
    "metaType": "object",
    "metaId": "...",
    "metaName": "LotR",
    "timestamp": "1954-08-29T11:51:48.314Z",
    "type": "UPDATE",
    "message": "write",
    "data": {
        "value": {
            "new": "false"
        },
        "change": [
            "new"
        ],
        "typeId": "bfcf07388afd45faa06e259548e39ecd"
    }
}

The result is a simple meta model with JSON schema and JSON based types and objects that can represent arbitrary domain specific entities. By recording all changes that happen over time and with the built-in geo spactial functions, the meta model is a good fit to represent and connect physical objects to the digital world in a Auto-ID/IoT context.

In addition to entities, the meta model also defines user, user groups and user configurations. These can be used to authenticate and authorize users in the application and grant fine grained access to APIs, types, objects, actions and so on.

  • MetaUser - Contains the credentials and grants of a single user. If any of the two factor methods are used, the user has to provide an OTP to get authenticated
{
    "name": "pw",
    "password": "pw",
    "phone": null,
    "email": null,
    "webhook": null,
    "totp": null,
    "webAuth": null,
    "grants": { // See MetaGrantPermissions / MetaGrantKeys
        "ac": 1,
        "cl": 1,
        "cln": 1,
        "dl": 1,
        "fs": 1,
        "li": 1,
        "men": 1,
        "nc": 1,
        "se": 1,
        "sy": 1,
        "tk": 1,
        "up": 1,
        "wr": 1,
        "wsst": 15
    }
}
  • MetaUserGroup - Group contains a number of users that share the same grants. Grants are first checked on a user level and then on the group level. There is a special group called admin that cannot be deleted
{
    "name": "admin",
    "users": [
        "pw"
    ],
    "grants": {
        "egn": 3
    }
}
  • MetaUserConfig - A configuration is assigned to a user, group or a combination of both. It can contain an arbitrary JSON object as the configuration. All matching configurations for the current user are automatically returned with a successful login
{
    "user": null,
    "group": "admin",
    "config": {
        "groot": true
    }
}
Client.xtend / client.js

TODO

Cst.xtend / cst.js

TODO

Model.xtend / model.js

TODO

Protocol.xtend / protocol.js

TODO

Tokenhelper.xtend / tokenhelper.js

TODO

meta-client

Client implementations in Java and JavaScript of the WebSocket API provided by the meta-engine server. The server requires a client to connect, login and authenticate before executing all other API requests. The only exception are logging messages, if allowed by the server.

In addition to the WebSocket API, a small subset of functions (currently only upload and download of files) can also be accessed with a REST-like API that is secured by JWT, see TokenHelper.xtend/tokenhelper.js. Applications are free to define additional REST-like APIs that can also be secured by tokens and more complex rules using UserService.

Java Client API
def ListenableFuture<Void> connect() 

def void disconnect()

def void destroy()

def boolean isConnected()

def boolean isReconnecting()

def boolean isAuth()

def ListenableFuture<LoginReply> login(LoginRequest it, RequestOption... options)

def ListenableFuture<AuthReply> auth(AuthRequest it, RequestOption... options)

def ListenableFuture<TokenReply> token(TokenRequest it, RequestOption... options)

def ListenableFuture<SubscriptionReply> subscribe(SubscriptionRequest it, RequestOption... options)

def ListenableFuture<LogoutReply> logout(LogoutRequest it, RequestOption... options)

def void postClientNotification(ClientNotification it) {

def ListenableFuture<ClientListReply> getClientList(ClientListRequest it, RequestOption... options)

def ListenableFuture<NotificationCacheReply> getNotificationCache(NotificationCacheRequest it, RequestOption... options)

def ListenableFuture<WriteReply> write(WriteRequest it, RequestOption... options)

def ListenableFuture<SearchReply> search(SearchRequest it, RequestOption... options)

def ListenableFuture<ActionReply> execute(ActionRequest it, RequestOption... options)

def void log(LoggingMessage it)

def ListenableFuture<LoggingEntrySearchReply> loggingEntrySearch(LoggingEntrySearchRequest it, RequestOption... options)

def ListenableFuture<LoggingMetricSearchReply> loggingMetricSearch(LoggingMetricSearchRequest it, RequestOption... options)

def ListenableFuture<LoggingDeleteReply> loggingDelete(LoggingDeleteRequest it, RequestOption... options)

def ListenableFuture<UserWriteReply> userWrite(UserWriteRequest it, RequestOption... options)

def ListenableFuture<UserSearchReply> userSearch(UserSearchRequest it, RequestOption... options)

def ListenableFuture<UserGroupWriteReply> groupWrite(UserGroupWriteRequest it, RequestOption... options)

def ListenableFuture<UserGroupSearchReply> groupSearch(UserGroupSearchRequest it, RequestOption... options)

def ListenableFuture<UserConfigWriteReply> configWrite(UserConfigWriteRequest it, RequestOption... options)

def ListenableFuture<UserConfigSearchReply> configSearch(UserConfigSearchRequest it, RequestOption... options)

def ListenableFuture<UploadReply> upload(UploadRequest it, RequestOption... options)

def ListenableFuture<UploadReply> uploadAndWaitDone(UploadRequest it, Path file)

def ListenableFuture<DownloadReply> download(DownloadRequest it, RequestOption... options)

def ListenableFuture<DownloadReply> downloadAndWaitDone(DownloadRequest it, Path file)

def ListenableFuture<FileDeleteReply> deleteFile(FileDeleteRequest it, RequestOption... options)

def ListenableFuture<FileSearchReply> fileSearch(FileSearchRequest it, RequestOption... options)

def ListenableFuture<BackupReply> backup(BackupRequest it, RequestOption... options)

def ListenableFuture<BackupRestoreReply> backupRestore(BackupRestoreRequest it, RequestOption... options)

def ListenableFuture<WatchdogUpdateReply> watchdogUpdate(WatchdogUpdateRequest it, RequestOption... options)

def ListenableFuture<SyncStatusReply> syncStatus(SyncStatusRequest it, RequestOption... options)

def ListenableFuture<SyncReplicateReply> syncReplicate(SyncReplicateRequest it, RequestOption... options)

def ClientSubscription registerOnEvent(Procedure1<Event> it)

def void unregister(ClientSubscription it)

def void registerDownloadListener(DownloadListener downloadListener)
JavaScript Client API
that.connect = function () {};

that.disconnect = function () {};

that.destroy = function () {}; // Not implemented

that.isConnected = ko.observable();

that.isReconnecting = ko.observable();

that.isAuth = ko.observable();

that.login = function (request, options) {};

that.auth = function (request, options) {};

that.token = function (request, options) {};

that.subscribe = function (request, options) {};

that.logout = function (request, options) {};

that.postClientNotification = function (clientNotification) {};

that.getClientList = function (request, options) {};

that.getNotificationCache = function (request, options) {};

that.write = function (request, options) {};

that.search = function (request, options) {};

that.execute = function (request, options) {};

that.log = function (loggingMessage) {};

that.loggingEntrySearch = function (request, options) {};

that.loggingMetricSearch = function (request, options) {};

that.loggingDelete = function (request, options) {};

that.userWrite = function (request, options) {};

that.userSearch = function (request, options) {};

that.groupWrite = function (request, options) {};

that.groupSearch = function (request, options) {};

that.configWrite = function (request, options) {};

that.configSearch = function (request, options) {};

that.upload = function (request, options) {};

that.uploadAndWaitDone = function (request, buffer) {};

that.download = function (request, options) {};

that.downloadAndWaitDone = function (request) {};

that.fileDelete = function (request, options) {};

that.fileSearch = function (request, options) {};

that.backup = function (request, options) {};

that.backupRestore = function (request, options) {};

that.watchdogUpdate = function (request, options) {};

that.syncStatus = function (request, options) {};

that.syncReplicate = function (request, options) {};

that.registerOnEvent = function (subscription) {};

that.unregister = function (clientSubscription) {};

that.registerDownloadListener = function (listener) {}; // Not implemented

Most of the API uses a request/reply pattern. ClientNotification and MetaNotification are notifications that allow clients to react to changes in near real time. LoggingMessage is the only other API method that does not follow request/reply.

Login component
  1. In HTML
<div class="row justify-content-md-center">
    <div class="col-md-6 col-xl-4">
        <login params="
            client: client,
            clientName: 'my-client',
            clientVersion: config.version,
            hasLoggedIn: hasLoggedIn,
            logout: doLogout,
            onAuth: onAuth,
            onConnectionFailed: onConnectionFailed,
            enableEditUser: true
        "></login>
    </div>
</div>
  1. In JavaScript
const hasLoggedIn = ko.observable();
const doLogout = ko.observable();
const isOffline = ko.observable();

const onAuth = function () {
    return Promise.resolve(...); // We are connected and authenticated, let's go
};

const onConnectionFailed = function () {
    hasLoggedIn(true);
    isOffline(true);
    return Promise.resolve(...);
};

const logout = function () {
    doLogout(true); // Notify login component that the user has decided to logout
    isOffline(false);
};
Searcher component
  1. In HTML
<form data-bind="submit: search">
    <button type="submit" class="btn btn-primary" data-bind="busyButton: mySearcher.isSearching">
        <span data-bind="i18n: 'search'"></span>
    </button>
</form>
<searcher params="objects: mySearcher, loadMore: loadMore, loadAll: loadAll, i18nPrefix: 'my_searcher'"></searcher>
  1. In JavaScript
const mySearcher = objects({}); // meta-client/objects
const searchInternal = function (limit, clearSearch) {
    return mySearcher.processSearch(...).then(function () {
        // ...
    });
};
const search = function () {
    return searchInternal(500, true);
};
const loadMore = function () {
    return searchInternal(500, false);
};
const loadAll = function () {
    return searchInternal(0, false);
};
objects.js

TODO

objectscache.js

TODO

meta-engine

Is the backend and server runtime of any application based on meta.


Various Services contain the the vital functions. Each service is one of:

  • AbstractEngineIdleService - Similar to AbstractIdleService
  • AbstractEngineExecutionThreadService - Similar to AbstractExecutionThreadService
  • AbstractEngineWorkQueueService - Custom version of AbstractEngineExecutionThreadService with a work queue
ActionService.xtend

Receives and dispatches MetaAction to services that can handle them. Does validation of parameters and results, in case schemas are defined.

Interface: None

Configuration: None

Usage:

  1. Define a MetaAction in Java and JavaScript
// Model.xtend

val public static ECHO_ACTION_ID = "echo"
val public static META_ECHO_ACTION = new MetaAction(ECHO_ACTION_ID, "Echo Action", EMPTY_OBJECT_SCHEMA, EMPTY_OBJECT_SCHEMA)

@Jsonized @Data class SimpleServiceEchoActionParam {
    String ping
}

@Jsonized @Data class SimpleServiceEchoActionResult {
    String pong
}
// model.js

that.ECHO_ACTION_ID = "echo";

factory.simpleServiceEchoActionParam = function (spec) {
    const param = {};

    param.ping = spec.ping;

    return param;
};

factory.simpleServiceEchoActionResult = function (spec) {
    const result = {};

    result.pong = spec.pong;

    return result;
};

// Web

factory.simpleServiceEchoActionRequest = function (params) {
    return protocol.factory.actionRequest({
        actionId: that.ECHO_ACTION_ID,
        params: factory.simpleServiceEchoActionParam(params)
    });
};

factory.simpleServiceEchoActionReply = function (result) {
    return protocol.factory.actionReply({
        result: factory.simpleServiceEchoActionResult(result)
    });
};
  1. Service to handle the actions
@Singleton class SimpleService extends AbstractEngineIdleService<SimpleServiceConfig> {
    @Inject new(EngineRuntime runtime, NotificationService notificationService, SimpleServiceConfig config) {
        super(runtime, notificationService, config)
    }

    override protected startUp() {
        if(runtime.awaitDependentServices(notificationService) === false) {
            return
        }

        LOG.info("startUp, config={}", config)
        super.startUp()
    }

    @Subscribe def void onActionExecutionRequestNotification(ActionExecutionRequestNotification it) {
        switch (action.id) {
            case ECHO_ACTION_ID: {
                runActionExecutionRequest(runtime, notificationService, it, SimpleServiceEchoActionParam, [echo], SYSTEM_ERROR)
            }
        }
    }

    def ListenableFuture<SimpleServiceEchoActionResult> echo(SimpleServiceEchoActionParam it) {
        checkRunning
        LOG.info("echo")
        return Futures.immediateFuture(new SimpleServiceEchoActionResult(ping))
    }
}
  1. Execute action from Java client
val params = new SimpleServiceEchoActionParam("Hello")
val reply = client.execute(ActionRequest.forgeFromObject(ECHO_ACTION_ID, params)).get(2, TimeUnit.SECONDS)
val result = reply.meltResult(SimpleServiceEchoActionResult)
  1. Execute action from JavaScript client
return client.execute(protocol.factory.simpleServiceEchoActionRequest({
    ping: "Hello"
})).then(function (actionReply) {
    LOG.debug("pong=" + actionReply.result.pong);
});
DatabaseService.xtend

Allows to connect to any configured SQL databases using c3p0. Executing queries against the databases is handled by JDBI.

Interface:

def Jdbi lookupDefaultDatabase()

def Jdbi lookupDatabase(String dbName)

Configuration:

{
    "dataSources": [
        {
                                                                               // This would be the default database, since no name is provided
            "properties": {                                                    // Properties are set using reflection, see https://www.mchange.com/projects/c3p0/#javabeans-style-properties for options
                "driverClass": "com.microsoft.sqlserver.jdbc.SQLServerDriver",
                "jdbcUrl": "jdbc:sqlserver://...",
                "user": "admin",
                "password": "UeDgud7x"
            }
        },
        {
            "name": "mariadb",
            "properties": {
                "driverClass": "org.mariadb.jdbc.Driver",
                "jdbcUrl": "jdbc:mariadb://..."
            }
        }
    ]
}

Make sure to include the JAR that contains the driver class in the Maven POM at compile time or on the classpath at runtime.

Usage:

  1. From another service
@Singleton class SimpleService extends AbstractEngineIdleService<SimpleServiceConfig> {
    DatabaseService databaseService

    @Inject new(EngineRuntime runtime, DatabaseService databaseService, SimpleServiceConfig config) {
        super(runtime, null, config)
        this.databaseService = databaseService
    }

    override protected startUp() {
        if(runtime.awaitDependentServices(databaseService) === false) {
            return
        }

        LOG.info("startUp, config={}", config)
        super.startUp()
    }

    def void sayHelloWithDao(String name) {
        val dao = databaseService.lookupDefaultDatabase.onDemand(SimpleServiceHelloDao)
        dao.insertName(name)
    }

    def void sayHelloWithHandler(String name) {
        databaseService.lookupDatabase("mariadb").withHandle [
            execute(SimpleServiceHelloDao.CREATE_TABLE_QUERY)
            execute(SimpleServiceHelloDao.INSERT_QUERY, name)
        ]
    }
}

interface SimpleServiceHelloDao {
    val static CREATE_TABLE_QUERY = '''CREATE TABLE Hello (
        Name VARCHAR(20) NOT NULL
    )'''
    val static INSERT_QUERY = "INSERT INTO Hello (Name) VALUES (?)"

    @SqlUpdate(CREATE_TABLE_QUERY)
    def void createHelloTable()

    @SqlUpdate(INSERT_QUERY)
    def void insertName(String name)
}
FileService.xtend

Upload, download, search and delete files. Provides backup and restore functionality to meta-engine services. Uses a LuceneService instance to store file meta data. File contents are stored on disk.

Interface:

def ListenableFuture<Pair<FileInfo, Path>> localDownload(DownloadRequest it, @Nullable Path existingTarget)

def ListenableFuture<UploadReply> localUpload(UploadRequest it, Path file)

def ListenableFuture<FileDeleteReply> fileDelete(FileDeleteRequest it)

def ListenableFuture<FileSearchReply> fileSearch(FileSearchRequest it)

def ListenableFuture<BackupReply> backup(BackupRequest it)

def ListenableFuture<BackupRestoreReply> backupRestore(BackupRestoreRequest it)

interface FileServiceBackupClient {
    def void onFileServiceBackupNotification(FileServiceBackupNotification it)

    def void onFileServiceRestoreNotification(FileServiceRestoreNotification it)
}

Configuration:

{
    "storageDir": "C:/data/engine/file/store", // Path to the directory where files should be stored in
    "cleanupSchedule": 300,                    // Interval in seconds after which incomplete uploads are cleand up
    "bufferSize": 0,                           // Size in bytes of buffer that is used to send chunks of data to clients. Optional
    "lucene": {}                               // See LuceneService
}

Usage:

  1. From another service
val uploadFuture = fileService.localUpload(uploadRequest, path)
val downloadFuture = fileService.localDownload(downloadRequest, null)
Futures.transform(downloadFuture, [ fileInfoAndPath |
    val info = fileInfoAndPath.key
    val path = fileInfoAndPath.value
    ...
    deleteIfExists(path)
], runtime)
  1. From Java client
val uploadDoneFuture = client.uploadAndWaitDone(uploadRequet, path)
val downloadDoneFuture = client.downloadAndWaitDone(downloadRequest, path)
  1. Upload in JavaScript client
// HTML
<input type="file" class="custom-file-input" data-bind="event: {change: loadCsvFile}">
that.loadCsvFile = function (ignore, event) {
    LOG.debug("loadCsvFile");
    return ext.loadFileContent(event, "buffer", null, [".csv"]).then(function (loadedFile) {
        return client.uploadAndWaitDone(protocol.factory.uploadRequest({
            name: loadedFile.name,
            tags: ["csv"]
        }), loadedFile.content);
    }).then(function (uploadReply) {
        LOG.debug("fileKey=" + uploadReply.key);
    });
};
  1. Download in JavaScript client
return client.downloadAndWaitDone(protocol.factory.downloadRequest({
    key: fileKey
})).then(function (downloadReply) {
    LOG.debug("fileInfo=" + downloadReply.file);
    // Do something with downloadReply.buffer
});
GatewayService.xtend

Sends SMS, Emails and is able to invoke webhooks.

SMS delivery depends on an external provider: iNetWorx SMS-Gateway.

Interface:

def ListenableFuture<GatewayServiceDeliveryReply> sendSms(GatewayServiceSmsRequest it)

def ListenableFuture<GatewayServiceDeliveryReply> sendEmail(GatewayServiceEmailRequest it)

def ListenableFuture<GatewayServiceDeliveryReply> invokeWebhook(GatewayServiceWebhookRequest it)

Configuration:

{
    "senderName": "My Server", // Name used as SMS and Email sender
    "smsUser": "",             // User for INetWorx SMS-Gateway
    "smsHttpPassword": "",     // HTTP password for INetWorx SMS-Gateway
    "smsGatewayPassword": "",  // Gateway password for INetWorx SMS-Gateway
    "emailServer": "",         // URL of SMTP server
    "emailPort": 465,          // Port of SMTP server
    "emailSsl": true,          // True if SMTP server uses SSL
    "emailTls": false,         // True if SMTP server uses TLS
    "emailUser": "",           // User for SMTP server. Optional
    "emailPassword": "",       // Password for SMTP server. Optional
    "emailSender": "",         // Address of email sender
    "emailReplyTo": null       // Address to reply-to. Optional
}

Usage:

  1. From another serivce
val file = new GatewayServiceEmailFileRequest(true, "/9j/4AAQSkZJRgABAQAA...", "image/jpg", "picture.jpg", null)
val emailFuture = gatewayService.sendEmail(new GatewayServiceEmailRequest("garbage@thedumpster.com", "Hello World", null, #[file]))
LoggingService.xtend

Centralized store and search for logs and metrics. Uses a LuceneService instance to store logging data.

Internally attatches to SLF4J root logger to log everything from the server runtime. Clients are able to log to this, as well as use server side metrics.

All messages that contain parameters in the format arg1=value, arg2=value, ... are automatically indexed and can be searched. All field names (arg1) are sanitized, see sanitizeStringFieldValue.

Interface:

def ListenableFuture<LoggingEntrySearchReply> loggingEntrySearch(LoggingEntrySearchRequest it)

def ListenableFuture<LoggingMetricSearchReply> loggingMetricSearch(LoggingMetricSearchRequest it)

def ListenableFuture<LoggingDeleteReply> loggingDelete(LoggingDeleteRequest it)

Configuration:

{
    "monitorSchedule": 30,                      // Interval in seconds for healthchecks and metrics. Optional
    "retention": 7,                             // Time in days after which logging entries are removed. Optional
    "logPath": "${logDir}",                     // Path to directory where log files are stored. If retention is enabled, all files here will also be removed after the set period. Optional
    "unhealthyFiles": [                         // Path to files that should not exist and indicate some problem with the system. Optional
        "${logDir}/jvm_error*",                 // File name with wildcard ending
        "${logDir}/*.mdmp",                     // File name with wildcard beginning
        "${logDir}/error.log"                   // Exact file name
    ],
    "loggers": {                                // Overwrite log levels of specific loggers. Optional
        "ch.mycompany.UnstableService": "DEBUG"
    },
    "alert": {                                  // If smsReceiver is set, SMS will be generated. If emailReceiver is set, emails will be generated. Optional
        "smsReceiver": "123 45 67",             // Phone number to send alert SMS to. Optional
        "smsMessage": "My Server has alerts!",  // SMS message if server has alerts
        "emailReceiver": "alerts@mycompany.ch", // Email address to send alert emails to. Optional
        "emailSubject": "My Server Alert"       // Subject for alert email
    },
    "lucene": {}                                // See LuceneService
}

Usage:

  1. From another service
import static com.codahale.metrics.MetricRegistry.name

@Singleton class SimpleService extends AbstractEngineIdleService<SimpleServiceConfig> {
    val static LOG = LoggerFactory.getLogger(SimpleService)

    Meter helloMeter

    @Inject new(EngineRuntime runtime, SimpleServiceConfig config) {
        super(runtime, null, config)
    }

    override protected startUp() {
        LOG.info("startUp, config={}", config)
        helloMeter = runtime.metricRegistry.meter(name(this.class, "hello"))
        super.startUp()
    }

    def void sayHello() {
        LOG.debug("sayHello")
        helloMeter.mark
    }
}
  1. Logging and metrics from Java client
class ApplicationClient {
    val static LOG = LoggerFactory.getLogger(ApplicationClient)

    def static void main(String... args) {
        val clientConfig = new ClientConfig("app.mycompany.ch", null, 0, 0, 0, 0)
        val client = new Client(clientConfig)

        LOG.info("connecting client")
        client.connect.get

        val appender = Logging.installEngineAppender(LOG, client) // Everything logged after this point is also sent to the server
        val loginTimer = Logging.forgeMetricTimer(client, "ApplicationClient-login")
        val timer = loginTimer.time

        val loginRequest = new LoginRequest("pw", "pw", "ApplicationClient", Cst.VERSION)
        val loginReply = client.login(loginRequest).get
        LOG.info("done with connect and login")
        val authRequest = new AuthRequest(loginReply.otp, null)
        client.auth(authRequest).get
        LOG.info("done with auth")
        timer.stop

        LOG.info("shutting down")
        client.logout(new LogoutRequest).get
        appender.stop
        client.destroy
    }
}
  1. Logging and metrics from JavaScript client
define([
    "logger",
    "meta-client/client",
    "meta-client/logging",
    "meta-core/protocol",
    "myapp-web/config",
    "domReady!"
], function (
    Logger,
    metaClient,
    logging,
    protocol,
    config
) {
    "use strict";

    const LOG = Logger.get("myapp-web/index");
    const that = Object.create(null);
    let timer;

    that.client = metaClient({
        workerPath: config.workerPath,
        engine: config.engine
    });
    logging.installEngineLogger(that.client, (
        config.debug
        ? Logger.DEBUG
        : Logger.INFO
    )); // Everything logged after this point is also sent to the server

    that.loginTimer = logging.forgeMetricTimer(that.client, "myapp-web/login");

    that.client.connect().then(function () {
        timer = that.loginTimer.time();
        return that.client.login(protocol.factory.loginRequest({
            user: "pw",
            password: "pw",
            clientName: "myapp-web",
            clientVersion: config.version
        }));
    }).then(function (loginReply) {
        LOG.info("done with connect and login");
        return that.client.auth(protocol.factory.authRequest({
            otp: loginReply.otp
        }));
    }).then(function () {
        LOG.info("done with auth");
        timer.stop();
        LOG.info("shutting down");
        return that.client.logout(protocol.factory.logoutRequest());
    }).then(function () {
        return client.disconnect();
    });
});
  1. Search from JavaScript client
const query = `clientId:${client.clientId()}`;
const from = metaCst.now().subtract(1, "hour");
const to = metaCst.now();
const timestampRangeQuery = ` AND timestamp:{${metaCst.toDateTimeNumber(from)} TO ${metaCst.toDateTimeNumber(to)}}`;
const searchRequest = protocol.factory.loggingEntrySearchRequest({
    query: query + dateRangeQuery,
    limit: 1000
});
client.loggingEntrySearch(searchRequest).then(function (searchReply) {
    console.log(searchReply);
});
LuceneService.xtend

Wrapper for working with a Lucene index to persist JSON entities. Mapping JSON entities to Lucene documents happens in DocumentForge.

For some entities like MetaObject and MetaLogEntry the generated document contains dynamic fields generated from the specific contents. The document for a MetaObject contains (among others) the following fields that are always present:

  • id - The objects UUID
  • typeId - The UUID from the MetaType that this object is an instance of

For a MetaObject with the following value:

{
    "owner": "My Company Inc.",
    "flagged": true
}

DocumentForge will generate additional fields with forgeAndAddCustomFields: owner and flagged. Using the JSON schema of the MetaType we:

  • Define what fields are sortable
  • What fields are indexed as numeric values. We need to know this ahead of time, so we can configure the QueryParser (for 8.5.1) accordingly
{
    "$schema": "https://json-schema.org/draft/2019-09/schema",
    "type": "object",
    "additionalProperties": false,
    "required": [],
    "sortable": ["myNumber"],
    "properties": {
        "myNumber": {
            "title": "A Number",
            "description": "This field is going to be sortable, since we added it to the 'sortable' list of fields. It is also indexed as a number, since we defined its type as number.",
            "type": "number"
        }
    },
    "definitions": {}
}

The fields in the document can be further customized by using the AbstractDocumentForgeFieldCustomizer to:

  • Generate additional fields
  • Remove fields that would be automatically generated
  • Register additional fields as numeric

Customizer to add an additional numeric field:

class ArraySizeField extends AbstractDocumentForgeFieldCustomizer {
    val static SIZE_FIELD = "arraySize"

    override forgeFields(String meta, String name, JsonNode node, boolean isSortable) {
        if(META_TYPE_OBJECT == meta && node.array) {
            return #{DocumentForge.forgeNumberField(SIZE_FIELD, root.size)} -> false
        }
    }

    override forgePointFieldNames(String name, JsonNode node, String typeId) {
        return #{SIZE_FIELD}
    }
}

Customizer to ignore certain fields:

class IgnoreObjectFields extends AbstractDocumentForgeFieldCustomizer {
    val static IGNORED_FIELDS = #{"field1", "field2"}

    override forgeFields(String meta, String name, JsonNode node, boolean isSortable) {
        if(IGNORED_FIELDS.contains(name)) {
            return #{} -> true
        }
    }
}

Multiple instances of LuceneService are used for FileService, MetaService, LoggingService and UserService. Services that need their own Lucene index must create their own LuceneService instance. Never interface with another LuceneService directly, use the interface given by its owner. See the mentioned meta-engine services as reference.

Interface:

def <T> ListenableFuture<T> search(Function1<LuceneServiceSearchContext, T> search)

def <T> ListenableFuture<T> write(Function1<LuceneServiceWriteContext, T> write)

def <T> ListenableFuture<T> searchAndWrite(Function1<LuceneServiceSearchAndWriteContext, T> searchAndWrite

public ConcurrentHashMap<String, String> getExtendedCommitData()

Configuration:

{
    "indexDir": "/var/tmp/lucene",                             // Path to working directory
    "analyzer": "org.apache.lucene.analysis.nl.DutchAnalyzer", // Class of analyzer to use. Optional. Defaults to StandardAnalyzer
    "commitSchedule": 600,                                     // Interval in seconds between automatic commits. Optional
    "workQueue": {                                             // AbstractEngineWorkQueueService configuration
        "capacity": 1000,
        "offerTimeout": 2000
    }
}

Usage:

See existing services that use a LuceneService.

MetaService.xtend

CRUD service for all meta entities:

  • MetaLogEntry
  • MetaType
  • MetaObject
  • MetaAction

Uses a LuceneService instance to store them.

Interface:

def ListenableFuture<SearchReply> search(SearchRequest it)

def ListenableFuture<WriteReply> write(WriteRequest it)

Configuration: None

Usage:

  1. From another service
val request = SearchRequest.forge("myNumber:[0 TO 100]", 0)
return Futures.transform(metaService.search(request), [ searchReply |
    val numberOfObjects = searchReply.metas.size;
    LOG.debug("numberOfObjects={}", numberOfObjects)
    return numberOfObjects
], runtime)
  1. From Java cient
val request = SearchRequest.forge(META_OBJECT_QUERY, 0)
val reply = client.search(request).get
  1. From JavaScript client
return client.search(protocol.factory.searchRequest({
    query: META_OBJECT_QUERY,
    limit: 0,
    sortField: "timestamp",
    sortReverse: true
})).then(function (searchReply) {
    const numberOfObjects = searchReply.metas.length;
    LOG.debug(`numberOfObjects=${numberOfObjects}`);
    return numberOfObjects;
});
NotificationService.xtend

Wrapper for EventBus to publish and subscribe to notifications. This is used to send ClientNotification and MetaNotification to clients. It can also be used as an event bus between engine services. Keeps a local cache to allow clients that were offline to catch up.

Watches the server JSON config files and notifies services via AbstractNotificationServiceConfigWatcher if anything changes. See EngineConfigWatcher.

Clients must subscribe to certain types of notifications, properties of a notification and channnels. They do not receive any notifications by default. See SubscriptionRequest and receivesNotification in EngineCore.

Interface:

def void register(Object it)

def void unregister(Object it)

def void post(Object it)

def ListenableFuture<NotificationCacheReply> getNotificationCache(NotificationCacheRequest it)

Configuration:

 {
    "cacheCapacity": 1000 // Size of notification cache to keep in memory. Optional
}

Usage:

  1. Subscribe and send from a service
import com.google.common.eventbus.Subscribe

@Singleton class SimpleService extends AbstractEngineIdleService<SimpleServiceConfig> {
    @Inject new(EngineRuntime runtime, NotificationService notificationService, SimpleServiceConfig config) {
        super(runtime, notificationService, config) // Will register this service with notificationService
    }

    override protected startUp() {
        if(runtime.awaitDependentServices(notificationService) === false) {
            return
        }

        LOG.info("startUp, config={}", config)
        super.startUp()
    }

    @Subscribe def void onSimpleServiceNotification(SimpleServiceNotification it) {
        // Invoked when ever someone posts a SimpleServiceNotification
    }

    def void postNotification() {
        notificationService.post(new SimpleServiceNotification("Hello"))
    }
}

@Data class SimpleServiceNotification {
    String greeting
}
  1. From Java client
client.registerOnEvent[
    switch (it) {
        ClientNotification: {
            // We received a ClientNotification
        }
    }
]
client.subscribe(new SubscriptionRequest(true, false, null, #{"mychannel"})).get(2,TimeUnit.SECONDS)
client.postClientNotification(ClientNotification.newNow(null, null, "mychannel", "INFO", "Hello"))
  1. From JavaScript client
client.registerOnEvent(function () {
    const subscription = Object.create(null);
    subscription[protocol.CLIENT_NOTIFICATION] = function (clientNotification) {
        // We received a ClientNotification
    };
    return subscription;
}());
client.subscribe(protocol.factory.subscriptionRequest({
    client: true,
    channels: ["mychannel"]
}));
client.postClientNotification(protocol.factory.clientNotification({
    channel: "mychannel",
    action: "INFO",
    message: "Hello"
}));
SyncService.xtend

Experimental service to synchronize files, meta entities, notifications and users across a cluster of meta-engine runtimes. etcd is used as the distributed key-value store and at least one node is required to synchronize multiple meta-engine runtimes.

Services are registered to watch for changes on the last locally commited revision of their key. Since etcd does not store revisions for a key indefinitely, services must be able to catch up to the quorum by replicating the complete state from another node by means outside of etcd. After replication, it can try to watch the key from the revision that was last commited.

Services must register their delegates with SyncService on startup. The delegate will either route the request through the etcd cluster or do a simple local invocation, depending on the SyncService configuration.

Interface:

def void registerServiceDelegate(AbstractSyncServiceDelegate serviceDelegate)

def ListenableFuture<SyncStatusReply> syncStatus(SyncStatusRequest it)

def ListenableFuture<SyncReplicateReply> syncReplicate(SyncReplicateRequest it)

interface SyncServiceDelegate<T, R> {
    def long getLatestRevision()

    def ListenableFuture<List<String>> prepareReplicate()

    def ListenableFuture<Void> restoreReplicate(List<Path> files)

    def ListenableFuture<R> write(T request, @Nullable Map<String, String> context)

    def ListenableFuture<R> onNext(T request, @Nullable Map<String, String> context, long revision)
}

Configuration:

{
    "engineId": "peer1",                    // Unique Id of this engine in the cluster
    "peers": {                              // Optional
        "peer2": {                          // Meta client configuration for peers
            "engine": "peer2.mycompany.ch",
            "user": "peer1",
            "password": "qHY2mmC9PXpLnM8TWK7vJrA6"
        }
    },
    "etcd": {                               // Optional
        "endpoints": [                      // URLs of etcd client endpoints
            "https://etcd1.mycompany.ch:2379",
            "https://etcd2.mycompany.ch:2379"
        ],
        "maxMessageSize": 0,                // Max size of messages in bytes. Optional
        "user": "sync",                     // User for etcd client auth. Optional
        "password": "xXU5bUkj",             // Password for etcd client auth. Optional
        "authority": "mycompanyca",         // Name of etcd SSL authority. Optional
        "trustManager": "C:/data/peer1.pem" // Path to etcd SSL certificates. Optional
    }
}

Usage:

  1. From another service
class MyService {
    override protected startUp() {
        (...)
        serviceDelegate = new MyServiceSyncDelegate(this)
        syncService.registerServiceDelegate(serviceDelegate)
        (...)
    }

    def ListenableFuture<MyServiceResponse> doIt(MyServiceRequest request) {
        return serviceDelegate.write(it, syncContext)
    }

    def protected ListenableFuture<MyServiceResponse> processDoIt(MyServiceRequest it, @Nullable Map<String, String> syncContext, long revision) {
        return Futures.immediateFuture(new MyServiceResponse)
    }
}

class MyServiceSyncDelegate extends AbstractSyncServiceDelegate<MyServiceRequest, MyServiceResponse> {
    val static KEY = "myservice"
    val static REQUEST_READER = OM.readerFor(MyServiceRequest)
    val static REQUEST_WRITER = OM.writerFor(MyServiceResponse)

    MyService myService

    new(MyService myService) {
        super(KEY, REQUEST_READER, REQUEST_WRITER)
        this.myService = myService
    }

    override getLatestRevision() {
        throw new UnsupportedOperationException("TODO: auto-generated method stub")
    }

    override prepareReplicate() {
        throw new UnsupportedOperationException("TODO: auto-generated method stub")
    }

    override restoreReplicate(List<Path> files) {
        throw new UnsupportedOperationException("TODO: auto-generated method stub")
    }

    override onNext(WriteRequest request, @Nullable Map<String, String> context, long revision) {
        return myService.processDoIt(request, context, revision)
    }
}
UserService.xtend

CRUD service for all user entities:

  • MetaUser
  • MetaUserGroup
  • MetaUserConfig

Provides login, authentication, authorization, verification and other access control functionality. Uses a LuceneService to store them.

Field name of MetaUser is sanitized to letters and digits. Valid names can be pw, Hello123, but not super_user.

Passwords are stored using SCrypt (implemented by wg/scrypt). Passwords are optional, since it’s possible to use OTP or WebAuthn. OTPs are 6 digit numbers like 123 456. They are sanitized before validation, so ASD123ASD456, 1 2 3!456 would be valid OTPs for 123 456. OTPs can be SMS, Email, webhook or TOTP.

verification of a MetaUser can be done with SMS, Email or a webhook. This can be used to temporarily grant a user certain permissions, if he can verify that he owns the specific phone number, email or endpoint.

Interface:

def ListenableFuture<TokenReply> token(TokenRequest it, String clientId)

def Pair<Boolean, Jws<Claims>> verifyToken(String token, @Nullable String user, @Nullable Map<String, Integer> requiredGrants)

def ListenableFuture<String> verifyUser(MetaUser it, String clientId)

def ListenableFuture<String> verifyUserOtp(String otp, String clientId, int tokenExpiration, @Nullable String user, @Nullable Map<String, Integer> grants)

def ListenableFuture<String> webhookAuth(String authRef)

def ListenableFuture<ClientListReply> getClientList(ClientListRequest it)

def ListenableFuture<UserWriteReply> userWrite(UserWriteRequest it)

def ListenableFuture<UserSearchReply> userSearch(UserSearchRequest it)

def ListenableFuture<UserGroupWriteReply> groupWrite(UserGroupWriteRequest it)

def ListenableFuture<UserGroupSearchReply> groupSearch(UserGroupSearchRequest it)

def ListenableFuture<Boolean> hasAnyPermission(String userName, int permission, String... keys)

def ListenableFuture<Boolean> isMemberOf(String userName, String groupName)

def ListenableFuture<UserConfigWriteReply> configWrite(UserConfigWriteRequest it)

def ListenableFuture<UserConfigSearchReply> configSearch(UserConfigSearchRequest it)

Configuration:

{
    "otpEmailSubject": "My Company OTP",      // Email subject for OTP messages
    "authTimeout": 60,                        // Duration in seconds after which OTP, challenges, etc. expire
    "attempts": 5,                            // Number of allowed attempts to fail login, authentication, challenges, etc. after which the client/user is kicked
    "scryptCpu": 1024,                        // SCrypt CPU cost parameter
    "scryptMemory": 1,                        // SCrypt memory cost parameter
    "token": {                                // Optional
        "expiration": 172800,                 // Duration in seconds after which JWT tokens expire
        "passphrase": "bURs343GsBBnRN2ZdtvG", // Passphrase to derive token key from
        "salt": "997XhfKT7w6yT8BkJX72",       // Salt to derive token key from
        "iterations": 10240,                  // PBEKeySpec iteration count parameter
        "keySize": 512                        // PBEKeySpec key size parameter
    },
    "lucene": {}                              // See LuceneService
}

Usage:

  1. Verification from another service
def void sendOtpForVerification(String phone, String clientId, String userName) {
    val user = new MetaUser(userName, null, phone, null, null, null, null)
    userService.verifyUser(user, clientId)
}

def void verifyUserOtp(String otp, String clientId, String userName) {
    val expiration = 60 * 60
    val verifyFuture = userService.verifyUserOtp(otp, clientId, expiration, userName, null)
    runtime.addCallback(verifyFuture, new FunctionalFutureCallback([ exc |
        LOG.warn("verifyFuture failed", exc)
    ], [ meta |
        // Do something with the token (can be null if rejected)
    ]))
}
  1. Webhook from another service
val client = new Client(clientConfig)
client.connect.get

// On the remote engine, 'myuser' must be registered with a webhook and the url has to conform 
// to UserService.forgeWebhookAuthPath("MyServiceAuth")

val loginRequest = new LoginRequest("myuser", null, "MyService", Cst.VERSION)
val loginReply = client.login(loginRequest).get

val otp = userService.webhookAuth("MyServiceAuth").get // This will return otp passed to the webhook

val authRequest = new AuthRequest(otp, null)
client.auth(authRequest).get
WatchdogService.xtend

Can report runtime health/metrics to another meta-engine runtime, check if other runtimes are healthy and schedule automated updates.

Interface: None

Configuration:

{
    "report": {                                       // Report health of this engine to a watchdog. Optional
        "schedule": 300,                              // Interval in seconds to report status to watchdog
        "engineId": "peer1",                          // Id of this engine for status reports
        "unsafe": false,                              // True if status report HTTP client should use unsafe option. Optional
        "engine": "watchdog.mycompany.ch",            // Watchdog url
        "user": "peer1",                              // Watchdog user
        "password": "RH5nbFaRPJhcGtqbNBNKJqQA"        // Watchdog password
    },
    "update": {                                       // Update configuration of this engine. Optional
        "workspace": "/tmp/watchdog",                 // Path to directory of watchdog update workspace
        "archivePassword": null                       // Password for update archives. Optional
    },
    "check": {                                        // Check health of remote engines that report to this watchdog. Optional
        "schedule": 600,                              // Interval in seconds between checks for the engine health
        "retention": 7,                               // Time in days after which received status and metrics are deleted. Optional
        "engineIds": [                                // Engine ids that are expected to report to this watchdog
            "peer1",
            "peer2"
        ],
        "alertHealthChecks": [                        // Names of healthchecks that should raise an alert if unhealthy
            "jvm-DiskSpace"
        ],
        "alertSmsReceiver": "123 45 67",              // Phone number to send alert SMS to. Optional
        "alertEmailReceiver": "alerts@mycompany.ch",  // Email address to send alert emails to. Optional
        "updateSmsReceiver": "123 45 67",             // Phone number to send update status SMS to. Optional
        "updateEmailReceiver": "devteam@mycompany.ch" // Email address to send update status emails to. Optional
    }
}

In addition to these vital services, there are rgw WebSocket and REST-like server APIs for clients:

EngineCore.xtend

Service facade for the WebSocket API. Delegates access control for service calls and returned entities to UserService.

Configuration:

{
    "allowUnauthorizedLog": true // True to allow clients to send logging messages to server, before login and authentication is completed
}
EngineResource.xtend

Provides the REST-like API.

  • EngineAuthResource - OTP webhook authentication, delegates to UserService
  • EngineTokenResource - Issues tokens, delegates to UserService
  • EngineUploadResource - File upload, delegates to FileService
  • EngineDownloadResource - File download, delegates to FileService
EngineWebSocket.xtend

Provides the WebSocket API in the form of a WebSocketListener.

Sends/receives Event objects serialized to JSON over sendString/onWebSocketText.

Sends/receives byte[] file content for DownloadRequest/UploadRequest over sendBytes/onWebSocketBinary.


The actual runtime and lifecycle functions are provided by:

Engine.xtend

Launcher and dependency injection module/components.

EngineRuntime.xtend

Provides the runtime with executor services, ServiceManager for service lifecycle and centralized Metric/Health-Check/Alert registries.

Implements Executor interface and also provides convenience methods to submit and handle async tasks, see ListenableFuture.

Interface:

def boolean awaitDependentServices(Service... services)

def <T> ListenableFuture<T> submit(Function0<T> func)

def void submitVoid(Procedure0 proc)

def ListenableScheduledFuture<?> schedule(Procedure0 proc, long delay, TimeUnit unit)

def ListenableScheduledFuture<?> scheduleWithFixedDelay(Procedure0 proc, long initialDelay, long delay, TimeUnit unit)

def <V> void addCallback(ListenableFuture<V> future, FunctionalFutureCallback<? super V> callback)

def boolean isHealthy()

def boolean isServiceManagerHealthy()

def List<String> getUnhealthyServices()

public MetricRegistry getMetricRegistry()

public HealthCheckRegistry getHealthCheckRegistry()

public AlertRegistry getAlertRegistry()

public boolean isRunning()

Configuration:

{
    "startUpTimeout": 10, // Time in seconds that is given to services to get healthy on startup
    "shutDownTimeout": 10 // Time in seconds that is given to services to stop on shut down
}

Usage:

  1. Async tasks from another service
val request = SearchRequest.forge(metaId, 1)
val metaFuture = Futures.transform(metaService.search(request), [ searchReply |
    if(searchReply.metas.empty === false) {
        return searchReply.metas.get(0) as MetaObject
    }
    return null
], runtime)
runtime.addCallback(metaFuture, new FunctionalFutureCallback([ exc |
    LOG.warn("metaFuture failed", exc)
], [ meta |
    LOG.info("meta={}", meta)
]))
  1. Alerts from another service
TODO
EngineUpdater.xtend

Execute tasks against two meta-engine instances for importing/exporting and migrating data. The current instance acts as the source and a new, pristine instance acts as the target. After one or more tasks run without error, all data directories of the first, existing instance are replaced with the data directories of the new instance.

Predefined tasks:

  • CloneTask - Clones all data (FileService, MetaService, UserService) to the new instance. This means a complete re-index of the underlying Lucene indexes. Useful in case of schema changes or new Lucene versions
  • EngineImportTask - Imports selected data from a remote engine to the new instance. See EngineImportTaskConfig for configuration options
  • JsonImportTask - Imports all data of the input JSON file to the new instance. See JsonImportTaskFile for file structure
  • JsonExportTask - Exports selected data from the existing instance to a JSON file. The JSON file is in the format of JsonImportTaskFile. See JsonExportTaskConfig for configuration options

Important notes:

  • Tasks are executed in the order that they were passed in
  • If any of the tasks returns true for skipIndexReplacement, the data directories of the existing instance are not replaced with the new ones
  • Replacement of data directories always takes place if all services returns false for skipIndexReplacement. If no task copies or imports any data into the new instance, you end up with an empty instance!
  • To import new data into an existing instance, run clone followed by jsonimport (or other way around if the imported data takes precedence)
  • There is no automatic backup before running the tasks. Use meta-admin to manually create backups

Interface: None

Configuration: None

Usage:

  1. Define a new tasks extending AbstractEngineUpdaterTask
class SimpleEchoTask extends AbstractEngineUpdaterTask {
    val static LOG = LoggerFactory.getLogger(SimpleEchoTask)
    val public static TASK_ID = "echo"

    override getId() {
        return TASK_ID
    }

    override update(EngineUpdaterServices oldServices, EngineUpdaterServices newServices) {
        val searchRequest = SearchRequest.forge('''«Model.ECHO_ACTION_ID»''', 1)
        val searchReply = newServices.metaService.search(searchRequest).get
        if (searchReply.metas.empty === false) {
            LOG.info("echo action exists in new index!")
        }
    }
}
  1. Update entities after changes to the schema with a EngineUpdaterEntityHandler:
class SimpleFieldUpdater implements EngineUpdaterEntityHandler {
    val static LOG = LoggerFactory.getLogger(SimpleFieldUpdater)
    val static OLD_FIELD = "\"oldField\":"
    val static NEW_FIELD = "\"newField\":"

    override handles(Object entity) {
        if(entity instanceof MetaObject) {
            return Model.SIMPLE_TYPE_ID == entity.typeId
        }
        return false
    }

    override <T> update(T entity) {
        val meta = entity as MetaObject
        val valueJson = Cst.OM.writeValueAsString(meta.value)
        val newValueJson = valueJson.replaceAll(OLD_FIELD, NEW_FIELD)
        LOG.info("update, metaId={}, newValueJson={}", meta.id, newValueJson)
        return MetaObject.build(meta, [
            value = Optional.of(JsonizedExtensions.forgeFromString(Cst.OM, newValueJson))
        ]) as T
    }
}
  1. Custom tasks and entity handlers have to be registered in the EngineUpdaterModule:
@Data class MyAppLauncher extends AbstractEngineLauncher<MyAppConfig> {
    override launchUpdater() {
        return DaggerFactory.forgeEngineUpdater(new MyAppEngineUpdaterModule(config.loadEngineConfig))
    }

    override launchRuntime() {
        return DaggerFactory.forgeEngineRuntime(new MyAppConfigModule(config, configPath))
    }
}

@Module class MyAppEngineUpdaterModule extends EngineUpdaterModule {
    new(EngineConfig config) {
        super(config)
    }

    @Provides @IntoSet def EngineUpdaterTask getSimpleEchoTask() {
        return new SimpleEchoTask
    }

    @Provides @IntoSet def EngineUpdaterEntityHandler getSimpleFieldHandler() {
        return new SimpleFieldUpdater
    }
}
HttpService.xtend

Wrapper around a Jetty Server instance with options for SSL, Caching, CORS, CSP, black-/whitelists, rate limits, etc.

Interface: None

Configuration:

{
    "maxThreads": 50,                                       // Number of threads to use for thread pool
    "httpPort": 0,                                          // Port to use for HTTP connector. Optional
    "httpsPort": 443,                                       // Port to use for HTTPS connector. Optional
    "ssl": {                                                // Optional
        "keystore": "C:/data/engine/bin/keystore.pfx",      // Path to keystore file
        "keystorePw": "GjMkytwtSb3z8tBvatwfaPwb",           // Keystore password
        "protocols": [                                      // Accepted protocols
            "TLSv1.2"
        ],
        "cipherSuites": [                                   // Accepted cipher suites
            "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
            ...
        ]
    },
    "logPath": "C:/data/logs",                              // Path to directory where access logs are stored
    "resourceBase": "C:/data/www",                          // Path to directory to use as root. Optional
    "cacheControl": "max-age=120",                          // Value for cache-control header. Optional. Defaults to no-cache
    "mimeTypes": {                                          // Mapping of file extensions to mime types. Optional
        "text": "text/plain"
    },
    "header": {                                             // Optional
        "cors": {                                           // Optional
            "origins": "https://remoteserver.mycompany.ch", // Allowed origins
            "methods": "PUT",                               // Allowed methods. Optional
            "headers": "X-Custom-Header"                    // Allowed headers. Optional
        },
        "csp": "..."                                        // CSP header value
    },
    "access": {                                             // InetAccessHandler configuration. Optional
        "exclude": [
            ...
        ],
        "include": [
            ...
        ]
    },
    "maxTextMessageSize": 0,                                // Optional
    "maxTextMessageBufferSize": 0,                          // Optional
    "maxBinaryMessageSize": 0,                              // Optional
    "maxBinaryMessageBufferSize": 0,                        // Optional
    "webSocketRateLimit": 0                                 // QPS allowed per websocket
}

Usage:

  1. Implementing a REST resource:
@Path("greeting")
@Produces(MediaType.TEXT_PLAIN)
class GreetingResource {
    val static LOG = LoggerFactory.getLogger(GreetingResource)

    EngineRuntime runtime
    GreetingResourceConfig config

    @Inject new(EngineRuntime runtime, GreetingResourceConfig config) {
        this.runtime = runtime
        this.config = config
    }

    @Path("/hello") @GET
    def void getGreeting(@QueryParam("name") String name) {
        LOG.info("getGreeting, name={}", name)
        async.setTimeout(10, TimeUnit.SECONDS)
        runtime.execute[
            async.resume('''«config.greeting» «name»!''')
        ]
    }
}

@Jsonized @Data class GreetingResourceConfig {
    String greeting
}

@Module class GreetingResourceModule {
    @Provides @IntoSet def HttpServiceRestComponentConfig getGreetingComponent(EngineRuntime runtime, GreetingResourceConfig config) {
        val binder = new AbstractBinder {
            override protected configure() {
                bind(runtime).to(EngineRuntime)
                bind(config).to(GreetingResourceConfig)
            }
        }
        return new HttpServiceRestComponentConfig(#[GreetingResource], binder)
    }
}
Main.xtend

Generic main entry point for all meta-engine based applications. Applications define their own configuration and launcher classes that they pass as arguments.

Interface:

def static void main(String[] args)

Configuration: None

Usage:

  1. Start the instance
java -cp engine.jar ch.rswk.meta.engine.Main -configcls ch.rswk.meta.engine.EngineConfig -config engine.json -launchcls ch.rswk.meta.engine.EngineLauncher -start
  1. Run update tasks
java -cp engine.jar ch.rswk.meta.engine.Main -configcls ch.rswk.meta.engine.EngineConfig -config engine.json -launchcls ch.rswk.meta.engine.EngineLauncher -update clone,jsonimport fixture.json

For more command line examples see:

  • platform/bootstrap/bootstrap-common/src/main/resources/bootstrap/definitions/engine-clone.json
  • platform/bootstrap/bootstrap-common/src/main/resources/bootstrap/definitions/engine-import.json
  • platform/bootstrap/bootstrap-common/src/main/resources/bootstrap/definitions/engine-install-service.json

meta-engine-test

Test module for meta-core, meta-client, meta-engine.

GatewayServiceTest.xtend

Requires custom VM arguments in Eclipse run configuration:

-Dgatewayservicetest.sendername=
-Dgatewayservicetest.sms.user=
-Dgatewayservicetest.sms.httppassword=
-Dgatewayservicetest.sms.gatewaypassword=
-Dgatewayservicetest.email.server=
-Dgatewayservicetest.email.user=
-Dgatewayservicetest.email.password=
-Dgatewayservicetest.email.sender=

meta-admin

Management and monitoring application for meta-engine.

  • Editor - CRUD operations for meta model entities
  • Clients - List all currently connected WebSocket clients
  • Notifications - List and send notifications
  • Users - CRUD operations for all user related entities
  • Logging - Centralied logging and metrics
  • File - CRUD operations for files
  • Watchdog - Check on other engines, schedule local and remote updates
  • Map - Show geospatial data and entities on OSM based map layers

Can be used as reference on how to structure and build web application based on meta.

Editor

Editor

Editor Logs

Editor Logs

Editor Types

Editor Types

Editor Objects

Editor Objects

Editor Objects Update

Editor Objects Update

Editor Actions

Editor Actions

Clients

Clients

Notifications

Notifications

Users

Users

Logging

Logging

File

File

Watchdog

Watchdog

Watchdog Update

Watchdog Update

meta-admin-test

Test module for meta-admin.

Util / Web

The most basic utility modules for meta, bootstrap and other, more complex components. Mostly based around utility classes/modules that group some common functionality. util is for the server/backend what web is for the client/frontend.

util-macro

Due to technical restrictions in Xtend, util-macro contains all Xtend Active Annotations, so we can use them in util-core.

Defines the basic Maven dependencies that are inherited by all Maven modules.

Jsonized.xtend

TODO

MutableBuilder.xtend

TODO

Observable.xtend

TODO

Strings.xtend

TODO

util-macro-test

Test module for util-macro.

Derfines the basic Maven dependencies that are inherited by all Maven test modules.

Important Note: Test modules are just regular Maven modules and seperated from the modules under test. This does not conform to the normal way to setup tests with Maven, but we feel like the benefits from transitive dependencies for test modules make up for this.

util-core

Utility functions for server/backend modules.

Env.xtend

TODO

Files.xtend

TODO

FutureCallback.xtend

TODO

P7zip.xtend

TODO

VariableResolver.xtend

TODO

WeakCrypto.xtend

TODO

Wget.xtend

TODO

util-core-test

Test module for util-core.

web-core

Utility functions for client/frontend modules.

Defines the build process for web clients:

  • web-build.bat or web-build.sh is called from Maven build
  • myapp-web/src/main/resources/web-build.json
    • Configuration for web-build.js build script
  • myapp-web/src/main/resources/web-dist.js
    • Configuration for r.js bundler/optimizer
  • myapp-web/src/main/resources/web-custom.js
    • Any custom Node script for module specific needs
buffers.js

TODO

checker.js

Check if current browser supports a feature/API and generate log entries if not.

checker.require("navigator.serviceWorker", navigator.serviceWorker);
console.log(checker.isMissingFeatures());
ConfigLoader.xtend / configloader.js

RequireJS plugin to overwrite specific properties from a config object. The config overwrite is passed as a query string.

  1. In Java test
val configOverwrite = forgeFromString(OM, '''{"debug":true}''')
val path = '''myapp-web/src/main/resources/?«ConfigLoader.forgeQueryString(OM, configOverwrite)»'''
val uri = forgePlatformUri(HTTP_SERVICE_TEST_PORT, path).toString
  1. In JavaScript
define([
    "web-core/configloader!myapp-web/config",
    "domReady!"
], function (
    config
) {
    // Do something with config object
});
crypto.js

Possibly unsafe crypto functions using crypto and crypto.subtle.

let id = crypto.generateRandomString();

Can be used to generate, export, import keys and then sign, verify, encrypt and decrypt (see crypto-test.js).

Device.xtend / device.js

TODO

Implementations for browser and Android

  • device-android.js
  • device-browser.js
ext.js

Various extension and helper functions.

  1. Key-Value Store
ext.db.set("key", {value: 1}).then(function () {
    return ext.db.get("key");
}).then(function (obj) {
    console.log(obj.value);
    return ext.db.delete("key");
});
i18n.js

Wrapper for i18next, so it can be used in a Knockout application.

  1. In i18nResourceStore.js
define(function () {
    "use strict";

    const resourceStore = {};

    resourceStore.de = {
        translation: {
            "cancel": "Abbrechen",
        }
    };

    resourceStore.en = {
        translation: {
            "cancel": "Cancel",
        }
    };

    return resourceStore;
});
  1. In i18nResourceStore.js, if you want to inherit translations from an existing resource store
define([
    "meta-client/i18nResourceStore",
    "myapp-web/config",
    "web-core/i18n"
], function (
    clientResourceStore,
    config,
    i18n
) {
    "use strict";

    return {
        de: i18n.forgeExtendedResourceStore("de", clientResourceStore, config, {
            greeting: "Hallo {{name}}",
            title: "Begrüssung"
        }),
        en: i18n.forgeExtendedResourceStore("en", clientResourceStore, config, {
            greeting: "Hallo {{name}}",
            title: "Greeting"
        })
    };
});
  1. Initializing on application startup in JavaScript
define([
    "myapp-web/i18nResourceStore",
    "web-core/i18n"
], function (
    resourceStore,
    i18n
) {
    "use strict";

    i18n.init(resourceStore).then(function () {
        const title = i18n.t("title");
        const greetingComputed = i18n.pureComputedT("greeting", {name: "World"});
    });
});
  1. In HTML with Knockout binding handler
<h1 data-bind="i18n: 'title'"></h1>
<span data-bind="i18n: {key: 'greeting', options: {name: 'World'}}"></span>
keyval.js

Key-value store wrapper around indexedDB based on idb-keyval.

Should not be used directly, but through the exposed db object in ext.js (see examples above).

nav.js

Navigation helper to pass/store UI state for a SPA.

// Register with nav
nav.register("mycomponent", function onUpdate(optionalParam) {
    console.log(`optionalParam=${optionalParam}`);
}, function onRemove() {
    console.log("Bye!");
});

// Navigate to component
const params = nav.newParams(); // or nav.currentParams() to only overwrite certain parameters
params.set("active_component", "mycomponent");
nav.pushState(params);

// or nav.pushState(params, {...}) to pass more application state
// or nav.replaceState(...) to replace the current history entry
ui.js

UI extensions, automatically includes Bootstrap JavaScript modules.

  • init

Helper to register Knockout components and bind the view model:

const viewModel = Object.create(null);
viewModel.hasLoaded = ko.observable();

ui.init(viewModel, [
    {name: "login", module: "meta-client/component/login"}
]).then(function () {
    that.hasLoaded(true);
});

Custom Knockout binding handlers:

  • busyButton

Shows a spinner/disables button when the value is true.

  1. In HTML
<form data-bind="submit: load">
    <button type="submit" class="btn btn-primary" data-bind="busyButton: isLoading">
        <span data-bind="i18n: 'search'"></span>
    </button>
</form>

Optional second binding buttonDisabled that will disable the button if value is true and will play nice with the busy indicator.

  1. In JavaScript
that.isLoading = ko.observable();
that.load = function () {
    that.isLoading(true);
    return doTheLoading().then(
        that.isLoading(false);
    });
};
  • fadeIn
  • fadeOut
  • fixHeight

Sets the elements height according to the current window height. Updates when ever the value changes or the window is resized.

  1. In HTML
<div class="row" data-bind="fixHeight: isActive">
    ...
</div>
  • modal

Shows a modal if the value is true and hides it if the value is false.

  1. In HTML
<div class="modal" tabindex="-1" role="dialog" data-bind="modal: activeObject">
    <div class="modal-dialog" role="document" data-bind="with: activeObject">
        ...
    </div>
</div>

Optional second binding confirmClose that will show a confirmation dialog (text of dialog must be set with confirmCloseI18n) if the modal is about to be closed and the value of the binding returns true.

  1. In JavaScript
that.activeObject = ko.observable();
that.editObject = function (object) {
    that.activeObject(object);
};
worker.js

Helper to create requirejs based service workers. See meta-admin implementation as example.

  1. In JavaScript define a worker.js
/*!
    myapp-web/worker
*/
/*jslint
    browser, long
*/
/*global
    define
*/
define([
    "myapp-web/config",
    "web-core/worker"
], function (
    config,
    worker
) {
    "use strict";

    return worker({
        name: "myapp-web",
        version: config.version,
        debug: config.debug
    });
});
  1. In JavaScript define a worker-init.js
/*!
    myapp-web/worker-init
*/
/*jslint
    browser, long
*/
/*global
    addEventListener, console, importScripts, requirejs, skipWaiting
*/
importScripts("../lib/alameda.js");
importScripts("require-config.js");

// Version: ${project.version}

const resolveWorker = new Promise(function (resolve, reject) {
    "use strict";

    requirejs.onError = function (err) {
        console.error(err);
        reject(err);
    };
    requirejs({
        baseUrl: "."
    }, ["myapp-web/worker"], function (worker) {
        resolve(worker);
    });
});

addEventListener("install", function (event) {
    "use strict";

    skipWaiting();
    event.waitUntil(resolveWorker.then(function (worker) {
        return worker.onInstall(event);
    }));
});

addEventListener("activate", function (event) {
    "use strict";

    event.waitUntil(resolveWorker.then(function (worker) {
        return worker.onActivate(event);
    }));
});

addEventListener("fetch", function (event) {
    "use strict";

    event.waitUntil(resolveWorker.then(function (worker) {
        return worker.onFetch(event);
    }));
});

addEventListener("notificationclick", function (event) {
    "use strict";

    event.waitUntil(resolveWorker.then(function (worker) {
        return worker.onNotificationClick(event);
    }));
});

addEventListener("notificationclose", function (event) {
    "use strict";

    event.waitUntil(resolveWorker.then(function (worker) {
        return worker.onNotificationClose(event);
    }));
});
  1. In JavaScript register the service worker in index.js using registerServiceWorker
ext.registerServiceWorker("myapp-web", config.version, config.scope, function () {
    // New version is available
});

web-core-test

Test module for web-core.

Defines AbstractWebDriver that is used for all Selenium based UI tests, including running and asserting all QUnit client side tests.

Android

Android WebView based wrapper to enhance wep applications with access to hardware specific functions of an Android based smartphone, like Bluetooth and NFC.

This wrapper becomes obsolete once Web-APIs provide the remaining functions that are currently only possible with native functions.

Web applications must use web-core/device module, where the API is defined. See index.js, device.js/device.html in web-core-test for examples on how to use the functionality, including opening up a web app in the Android wrapper from the browser via a pleb:// link.

IDE

Custom Eclipse based IDE for platform developers. Is skipped in the regular Maven build due to the time it takes. Can be run manually (eg. to update to latest Eclipse version) and creates *nix and Windows based products.

The ZIP files containing the built IDEs can be found in platform/ide/ch.rswk.ide.product/target/products.

Bootstrap (Framework)

Framework that works on *nix and Windows to:

  • Define simple automated installers, called bootstrap modules
  • Build customer/service specific versions of bootstrap modules
  • Execute bootstrap modules to install/configure software
  • Provision servers with bootstrap modules

bootstrap-core

Main components to build, provision and execute bootstrap modules.

Build.xtend

Parameters:

  • module - Path to bootstrap module
  • customer - Path to customer specific module (optional)
  • version - Maven version to set on the module before build (optional)
  • offline - If set, executes all wget actions after build and stores the downloaded files in the built module
  • debug - If set, skips executing the Maven commands

Configuration:

{
    "targetEnv": "WIN",
    "dataDir": "C:/data",
    "logDir": "C:/data/logs",
    "toolDir": "C:/tool",
    "startDefinition": "{moduleDir}/definitions/watchdog-server.json",
    "properties": {
        "keystoreFile": "keystore.pfx",
        "mydir": "{dataDir}/mydir"
    },
    "mvn": {
        "repoDir": "{WORKSPACE}/.repository",
        "settingsFile": "{PLATFORM_DATA_DIR}/settings.xml",
        "args": []
    },
    "merge": [
        {
            "source": "{moduleDir}/config/engine/engine.json",
            "purge": [],
            "with": "{customerDir}/config/engine/engine.json"
        }
    ],
    "copy": [
        {
            "source": "{PLATFORM_DATA_DIR}/mykeystore.pfx",
            "target": "{binDir}/{keystoreFile}"
        }
    ],
    "validate": [
        {
            "jar": "{moduleDir}/bin/meta-engine-{VERSION}-shaded.jar",
            "config": "{moduleDir}/config/engine/engine.json",
            "configClass": "ch.rswk.meta.engine.EngineConfig"
        }
    ],
    "deploy": {
        "groupId": "ch.rswk.platform.bootstrap",
        "artifactId": "bootstrap-watchdog-server-platform-win",
        "repositoryId": "dev-server",
        "url": "https://dev.platform.rswk.ch/nexus/repository/bootstrap/"
    }
}
Provision.xtend

Parameters:

  • module - Path to the module to provision
  • await - Number of seconds to wait to receive an update status from watchdog (optional)

Configuration:

{
    "targetEnv": "WIN",
    "watchdog": {
        "id": "customerone",
        "engine": "watchdog.mycompany.ch",
        "user": "watchdog",
        "password": "{WATCHDOG_PW}"
    },
    "startDefinition": "{moduleDir}/definitions/myapp-server.json",
    "command": "-flags certbot",
    "schedule": "2020-02-20T02:00:00.000Z",
    "data": {
        "wincmd.script.user": "Administrator",
        "wincmd.script.password": "{SERVER_PW}"
    },
    "server": {
        "vultr": {
            "apiEndpoint": "https://api.vultr.com",
            "apiKey": "{API_KEY}",
            "serverPassword": "{SERVER_PW}",
            "dataCenterId": null,
            "planId": null,
            "osId": null,
            "reservedIp": null,
            "sshKey": "{SSH_KEY}",
            "firewallGroup": "{FIREWALL_GROUP}"
        }
    }
}
Execute.xtend

Built in actions:

  • CmdActionExecutor.xtend - Execute shell command
  • CopyActionExecutor.xtend - Copy file or folders
  • EnvvarActionExecutor.xtend - Add, append or remove environment variables
  • IncludeActionExecutor.xtend - Include other bootstrap definitions
  • UnpackActionExecutor.xtend - Unpack archives
  • WgetActionExecutor.xtend - Download files via HTTP

Every action supports conditions for its execution:

  • env - Only execute if running on particular OS
  • exists - Only execute if this file/folder exists or if prefixed with ! not exists
  • flag - Only execute if this flag is present or if prefixed with ! not present

Parameters:

  • start - Path to root of extracted bootstrap module
  • flags - Sets any flags to use for execution. Format: -flags 'flag1,flag2' (optional)
  • setowner - Set owner of configured data, log and tool directories to this user (optional)
  • webhook - URL of webhook to invoke after execution (optional)
  • webhookid - ID to set in ExecuteWebhookStatus that is posted to webhook (optional)

Configuration:

{
    "dataDir": "C:/data",
    "logDir": "C:/data/logs",
    "toolDir": "C:/tool",
    "startDefinition": "{moduleDir}/definitions/myapp-server.json",
    "properties": {
        "keystoreFile": "keystore.pfx",
        "mydir": "{dataDir}/mydir"
    }
}
Main.xtend

Main class to run the three phases with commands:

  • -build
  • -execute
  • -provision

See the specific components above for required parameters.

bootstrap-module

CLI scripts to bootstrap and run bootstrap-core on a PC/server. Defines the Maven assembly for the specific file structure of a bootstrap module:

- bootstrap-my-server
    - src/main/bootstrap
        - bin // Binary dependencies. Do not put stuff here manually
        - config // Configuration files. Maven filtering is applied to these during build
            - engine.json
        - definitions // JSON bootstrap definitions. Maven filtering is applied here as well
        - fixture // Any file that should not be touched by Maven filterig
    - pom.xml

Binary dependencies for the bin directory can be defined in the Maven build using maven-dependency-plugin:

<build>
    <plugins>
        <plugin>
            <artifactId>maven-dependency-plugin</artifactId>
            <executions>
                <execution>
                    <phase>prepare-package</phase>
                    <goals>
                        <goal>copy</goal>
                    </goals>
                    <configuration>
                        <artifactItems>
                            <!-- Web -->
                            <artifactItem>
                                <groupId>ch.rswk.platform</groupId>
                                <artifactId>meta-admin</artifactId>
                                <version>${project.version}</version>
                                <type>zip</type>
                                <overWrite>true</overWrite>
                                <outputDirectory>${bootstrapBinDir}</outputDirectory>
                            </artifactItem>
                            <!-- Engine -->
                            <artifactItem>
                                <groupId>ch.rswk.platform</groupId>
                                <artifactId>meta-engine</artifactId>
                                <version>${project.version}</version>
                                <type>jar</type>
                                <classifier>shaded</classifier>
                                <overWrite>true</overWrite>
                                <outputDirectory>${bootstrapBinDir}</outputDirectory>
                            </artifactItem>
                        </artifactItems>
                    </configuration>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

The contents of other bootstrap modules can be included by adding them as a Maven dependency:

<dependencies>
    <dependency>
        <groupId>ch.rswk.platform.bootstrap</groupId>
        <artifactId>bootstrap-common</artifactId>
        <version>${project.version}</version>
        <type>zip</type>
    </dependency>
</dependencies>

This general folder structure is present in the regular source repository, eg. platform, where the base version of a bootstrap module is defined. Any number of customer/service specific versions can be defined in a seperate bootstrap source repositor, eg. platform-bootstrap. This is also where the build or provision specific config files are usually stored.

- customerone
    - bootstrap-watchdog-server
        - config
            - engine.json // Overwrite options from base module engine.json
        - definitions     // Additional definitions for the bootstrap process or overwrite existing ones
        - build.json      // Configuration for bootstrap-core build step
        - provision.json  // Configuration for bootstrap-core provision step

For example see platform/bootstrap/bootstrap-watchdog-server for a basic module and platform-bootstrap/platform/bootstrap-watchdog-server for a customer specific module that overwrites a couple of files and config options. The modules are merged during the build process:

SET CUSTOMER_DIR=platform-bootstrap\platform\bootstrap-watchdog-server
java -jar bootstrap.jar -build %CUSTOMER_DIR%\build.json -module platform\bootstrap\bootstrap-watchdog-server -customer %CUSTOMER_DIR% -version %VERSION%

Variables for the build step can be defined with:

  • properties in the JSON configuration build.json of the build step (1)
    • Two predefined variables are also available during build
      • moduleDir and customerDir
  • Java System Properties form the command line, eg. with -DmyProperty=123 (2)
  • Environment variables, depending on the OS (3)

All variables for the build step can be used:

  • Inside the JSON configuration build.json with {myProperty}
  • Inside any files in config and definitions folder with Maven Filtering
    • (1)/(2) with ${myProperty}
    • (3) with ${env.myProperty}

The build step generates the bootstrap.json that is the configuration file for the execution step of bootstrap-core.

Variables for the execution step can be defined with:

  • properties in the JSON configuration bootstrap.json of the execution step (1)
    • All properties of JSON configuration of the build step are added automatically
    • A couple of predefined variables are also available during execution
      • {moduleDir}
      • {dataDir}, {logDir}, {toolDir}
        • Including a Windows specific version, eg. {dataDirWin}
      • {JAVA_HOME}
  • Java System Properties from the comman dline, eg. with -DmyProperty=123 (2)
  • Environment variables, depending on the OS (3)

All variables for the execution step can be used in the bootstrap JSON definitions with {myProperty}.

{
    "actions": [
        {
            "wget": {
                "uri": "https://mycompany.ch/myfile.zip",
                "target": "{moduleDir}/download/myfile.zip"
            }
        },
        {
            "cmd": {
                "command": "systemctl enable {serviceName}"
            }
        }
    ]
}

bootstrap-test (Framework)

Test module for bootstrap-core.

Defines AbstractBootstrapTest that is used for testing bootstrap modules by building and provisioning servers for *nix and Windows.

ProvisionTest.xtend

Requires custom VM arguments in Eclipse run configuration:

-Dprovisiontest.watchdog.password=
-Dprovisiontest.vultr.apikey=
-Dprovisiontest.vultr.serverpassword=
-Dprovisiontest.vultr.reservedip=
-Dprovisiontest.vultr.firewallgroup=

Bootstrap

bootstrap-dev-vm

Bootstrap module to provision the development environment for working on the platform.

bootstrap-dev-server

Bootstrap module to provision the development server of the platform.

The central pillars of our server are:

  • Source control, powered by SCM-Manager
  • Build automation, powered by Jenkins
  • Software repository, powered by Nexus

bootstrap-watchdog-server

Bootstrap module to provision a watchdog server.

Can be used as reference to provision meta-engine based servers.

bootstrap-test

Test module for all bootstrap modules.

BootstrapTest.xtend

Requires custom VM arguments in Eclipse run configuration:

-Dprovisiontest.watchdog.password=
-Dprovisiontest.vultr.apikey=
-Dprovisiontest.vultr.serverpassword=
-Dprovisiontest.vultr.reservedip=
-Dprovisiontest.vultr.firewallgroup=

Try

Test drive meta-engine as a standalone server component. Create simple meta models using meta-admin and explore all the basic platform functionality.

The scripts below are made to run in Ubuntu and Windows. Should also work on other *nix and *BSD variants, but may need to be adapted. On Windows, run in PowerShell as Administrator.

If you already have a local JDK/JRE (Java 11 or newer), skip the first step.

1. Download and extract the JDK

Ubuntu:

cd /var/tmp
wget -O java14.tar.gz "https://cdn.azul.com/zulu/bin/zulu14.28.21-ca-jdk14.0.1-linux_x64.tar.gz"
tar -zxf java14.tar.gz
JRE=$PWD/zulu14.28.21-ca-jdk14.0.1-linux_x64/bin

Windows:

$global:ProgressPreference = 'SilentlyContinue'
mkdir C:\tmp
cd C:\tmp
Invoke-WebRequest -Uri https://cdn.azul.com/zulu/bin/zulu14.28.21-ca-jdk14.0.1-win_x64.zip -OutFile java14.zip
Expand-Archive -LiteralPath java14.zip -DestinationPath (Get-Location).tostring()
$JRE = (Get-Location).tostring() + "\zulu14.28.21-ca-jdk14.0.1-win_x64\bin"

2. Download server and client

Ubuntu:

sudo apt-get install unzip
VERSION='0.1180'
wget -O engine.jar "https://dev.platform.rswk.ch/nexus/repository/maven-releases/ch/rswk/platform/meta-engine/$VERSION/meta-engine-$VERSION-shaded.jar"
wget -O admin.zip "https://dev.platform.rswk.ch/nexus/repository/maven-releases/ch/rswk/platform/meta-admin/$VERSION/meta-admin-$VERSION.zip"
unzip admin.zip -d admin

Windows:

$VERSION = '0.1180'
Invoke-WebRequest -Uri "https://dev.platform.rswk.ch/nexus/repository/maven-releases/ch/rswk/platform/meta-engine/$VERSION/meta-engine-$VERSION-shaded.jar" -OutFile engine.jar
Invoke-WebRequest -Uri "https://dev.platform.rswk.ch/nexus/repository/maven-releases/ch/rswk/platform/meta-admin/$VERSION/meta-admin-$VERSION.zip" -OutFile admin.zip
Expand-Archive -LiteralPath admin.zip -DestinationPath admin

3. Setup localhost keystore

Ubuntu:

wget -O test.jar "https://dev.platform.rswk.ch/nexus/repository/maven-releases/ch/rswk/platform/web-core-test/$VERSION/web-core-test-$VERSION.jar"
"$JRE/jar" xf test.jar localhost.platform.rswk.ch.pfx

Windows:

Invoke-WebRequest -Uri "https://dev.platform.rswk.ch/nexus/repository/maven-releases/ch/rswk/platform/web-core-test/$VERSION/web-core-test-$VERSION.jar" -OutFile test.jar
& "$JRE\jar.exe" xf test.jar 'localhost.platform.rswk.ch.pfx'

4. Manual steps

  • Change admin/js/config.js
    • "workerPath: "js/"
  • Add local dev domain to hosts file
    • *nix /etc/hosts
    • Windows C:/Windows/System32/drivers/etc/hosts
    • 127.0.0.1 localhost.platform.rswk.ch
    • Or don’t and proceed with the browser security warning and possibly reduced functionality

5. Import test user and start server

Ubuntu:

CONFIG=$(cat <<EOF
{
    "action": {},
    "database": {
        "dataSources": null
    },
    "core": {
        "allowUnauthorizedLog": true
    },
    "runtime": {
        "startUpTimeout": 10,
        "shutDownTimeout": 10
    },
    "file": {
        "storageDir": "file/store",
        "cleanupSchedule": 300,
        "bufferSize": 0,
        "lucene": {
            "indexDir": "file/lucene",
            "analyzer": null,
            "commitSchedule": 600,
            "workQueue": {
                "capacity": 1000,
                "offerTimeout": 2000
            }
        }
    },
    "gateway": {
        "senderName": "",
        "smsUser": "",
        "smsHttpPassword": "",
        "smsGatewayPassword": "",
        "emailServer": "",
        "emailPort": 465,
        "emailSsl": true,
        "emailTls": false,
        "emailUser": "",
        "emailPassword": "",
        "emailSender": "",
        "emailReplyTo": null
    },
    "http": {
        "maxThreads": 50,
        "httpPort": 0,
        "httpsPort": 9090,
        "ssl": {
            "keystore": "localhost.platform.rswk.ch.pfx",
            "keystorePw": "platform",
            "protocols": [
                "TLSv1.2"
            ],
            "cipherSuites": [
                "TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256",
                "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256",
                "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384",
                "TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384",
                "TLS_DHE_RSA_WITH_AES_128_GCM_SHA256",
                "TLS_DHE_RSA_WITH_AES_256_GCM_SHA384",
                "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256",
                "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256",
                "TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA",
                "TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA",
                "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384",
                "TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384",
                "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA"
            ]
        },
        "logPath": "logs",
        "unhealthyFiles": null,
        "resourceBase": "admin",
        "cacheControl": null,
        "mimeTypes": null,
        "header": {
            "cors": null,
            "csp": "connect-src 'self' wss://localhost.platform.rswk.ch:9090 https://*.tile.openstreetmap.org; font-src 'self'; frame-src 'self'; img-src 'self' https://*.tile.openstreetmap.org data:; ​manifest-src 'self'; media-src 'self' mediastream: blob:; script-src 'self' 'unsafe-eval' 'unsafe-inline'; style-src 'unsafe-inline' 'self' data:; worker-src 'self' blob:"
        },
        "access": null,
        "maxTextMessageSize": 0,
        "maxTextMessageBufferSize": 0,
        "maxBinaryMessageSize": 0,
        "maxBinaryMessageBufferSize": 0,
        "webSocketRateLimit": 0
    },
    "logging": {
        "monitorSchedule": 30,
        "retention": 7,
        "loggers": null,
        "alert": null,
        "lucene": {
            "indexDir": "logging/lucene",
            "analyzer": null,
            "commitSchedule": 0,
            "workQueue": {
                "capacity": 1000,
                "offerTimeout": 2000
            }
        }
    },
    "meta": {
        "lucene": {
            "indexDir": "meta/lucene",
            "analyzer": null,
            "commitSchedule": 600,
            "workQueue": {
                "capacity": 1000,
                "offerTimeout": 2000
            }
        }
    },
    "notification": {
        "cacheCapacity": 1000
    },
    "sync": {
        "engineId": "localhost",
        "peers": null,
        "etcd": null
    },
    "user": {
        "otpEmailSubject": "YourOTP",
        "authTimeout": 60,
        "attempts": 5,
        "scryptCpu": 1024,
        "scryptMemory": 1,
        "token": null,
        "webAuth": null,
        "lucene": {
            "indexDir": "user/lucene",
            "analyzer": null,
            "commitSchedule": 600,
            "workQueue": {
                "capacity": 1000,
                "offerTimeout": 2000
            }
        }
    },
    "watchdog": {
        "report": null,
        "update": null,
        "check": null
    }
}
EOF
)
FIXTURE=$(cat <<EOF
{
    "users": [
        {
            "name": "test",
            "password": "test",
            "grants": {
                "cl": 1,
                "dl": 1,
                "ac": 1,
                "fs": 1,
                "li": 1,
                "nc": 1,
                "cln": 1,
                "men": 1,
                "se": 1,
                "sy": 1,
                "tk": 1,
                "up": 1,
                "wr": 1
            }
        }
    ],
    "groups": [
        {
            "name": "admin",
            "users": [
                "test"
            ],
            "grants": {
                "egn": 3,
                "wsst": 15
            }
        }
    ]
}
EOF
)
"$JRE/java" -Xms128m -Xmx1g -DPLATFORM_LOG_DIR=logs -cp "engine.jar" 'ch.rswk.meta.engine.Main' -configcls 'ch.rswk.meta.engine.EngineConfig' -config "$CONFIG" -launchcls 'ch.rswk.meta.engine.EngineLauncher' -update clone,jsonimport "$FIXTURE"
"$JRE/java" -Xms128m -Xmx1g -DPLATFORM_LOG_DIR=logs -cp "engine.jar" 'ch.rswk.meta.engine.Main' -configcls 'ch.rswk.meta.engine.EngineConfig' -config "$CONFIG" -launchcls 'ch.rswk.meta.engine.EngineLauncher' -start

Windows:

$CONFIG = @"
{
    \"action\": {},
    \"database\": {
        \"dataSources\": null
    },
    \"core\": {
        \"allowUnauthorizedLog\": true
    },
    \"runtime\": {
        \"startUpTimeout\": 10,
        \"shutDownTimeout\": 10
    },
    \"file\": {
        \"storageDir\": \"file/store\",
        \"cleanupSchedule\": 300,
        \"bufferSize\": 0,
        \"lucene\": {
            \"indexDir\": \"file/lucene\",
            \"analyzer\": null,
            \"commitSchedule\": 300,
            \"workQueue\": {
                \"capacity\": 1000,
                \"offerTimeout\": 2000
            }
        }
    },
    \"gateway\": {
        \"senderName\": \"\",
        \"smsUser\": \"\",
        \"smsHttpPassword\": \"\",
        \"smsGatewayPassword\": \"\",
        \"emailServer\": \"\",
        \"emailPort\": 465,
        \"emailSsl\": true,
        \"emailTls\": false,
        \"emailUser\": \"\",
        \"emailPassword\": \"\",
        \"emailSender\": \"\",
        \"emailReplyTo\": null
    },
    \"http\": {
        \"maxThreads\": 50,
        \"httpPort\": 0,
        \"httpsPort\": 9090,
        \"ssl\": {
            \"keystore\": \"localhost.platform.rswk.ch.pfx\",
            \"keystorePw\": \"platform\",
            \"protocols\": [
                \"TLSv1.2\"
            ],
            \"cipherSuites\": [
                \"TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256\",
                \"TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256\",
                \"TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384\",
                \"TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384\",
                \"TLS_DHE_RSA_WITH_AES_128_GCM_SHA256\",
                \"TLS_DHE_RSA_WITH_AES_256_GCM_SHA384\",
                \"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256\",
                \"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256\",
                \"TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA\",
                \"TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA\",
                \"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384\",
                \"TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384\",
                \"TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA\"
            ]
        },
        \"logPath\": \"logs\",
        \"resourceBase\": \"admin\",
        \"cacheControl\": null,
        \"mimeTypes\": null,
        \"header\": {
            \"cors\": null,
            \"csp\": \"connect-src 'self' wss://localhost.platform.rswk.ch:9090 https://*.tile.openstreetmap.org; font-src 'self'; frame-src 'self'; img-src 'self' https://*.tile.openstreetmap.org data:; ​manifest-src 'self'; media-src 'self' mediastream: blob:; script-src 'self' 'unsafe-eval' 'unsafe-inline'; style-src 'unsafe-inline' 'self' data:; worker-src 'self' blob:\"
        },
        \"access\": null,
        \"maxTextMessageSize\": 0,
        \"maxTextMessageBufferSize\": 0,
        \"maxBinaryMessageSize\": 0,
        \"maxBinaryMessageBufferSize\": 0,
        \"webSocketRateLimit\": 0
    },
    \"logging\": {
        \"monitorSchedule\": 30,
        \"retention\": 7,
        \"loggers\": null,
        \"unhealthyFiles\": null,
        \"alert\": null,
        \"lucene\": {
            \"indexDir\": \"logging/lucene\",
            \"analyzer\": null,
            \"commitSchedule\": 0,
            \"workQueue\": {
                \"capacity\": 1000,
                \"offerTimeout\": 2000
            }
        }
    },
    \"meta\": {
        \"lucene\": {
            \"indexDir\": \"meta/lucene\",
            \"analyzer\": null,
            \"commitSchedule\": 300,
            \"workQueue\": {
                \"capacity\": 1000,
                \"offerTimeout\": 2000
            }
        }
    },
    \"notification\": {
        \"cacheCapacity\": 1000
    },
    \"sync\": {
        \"engineId\": \"localhost\",
        \"peers\": null,
        \"etcd\": null
    },
    \"user\": {
        \"otpEmailSubject\": \"YourOTP\",
        \"authTimeout\": 60,
        \"attempts\": 5,
        \"scryptCpu\": 1024,
        \"scryptMemory\": 1,
        \"token\": null,
        \"webAuth\": null,
        \"lucene\": {
            \"indexDir\": \"user/lucene\",
            \"analyzer\": null,
            \"commitSchedule\": 300,
            \"workQueue\": {
                \"capacity\": 1000,
                \"offerTimeout\": 2000
            }
        }
    },
    \"watchdog\": {
        \"report\": null,
        \"update\": null,
        \"check\": null
    }
}
"@
$FIXTURE = @"
{
    \"users\": [
        {
            \"name\": \"test\",
            \"password\": \"test\",
            \"grants\": {
                \"cl\": 1,
                \"dl\": 1,
                \"ac\": 1,
                \"fs\": 1,
                \"li\": 1,
                \"nc\": 1,
                \"cln\": 1,
                \"men\": 1,
                \"se\": 1,
                \"sy\": 1,
                \"tk\": 1,
                \"up\": 1,
                \"wr\": 1
            }
        }
    ],
    \"groups\": [
        {
            \"name\": \"admin\",
            \"users\": [
                \"test\"
            ],
            \"grants\": {
                \"egn\": 3,
                \"wsst\": 15
            }
        }
    ]
}
"@
& "$JRE\java.exe" -Xms128m -Xmx1g -DPLATFORM_LOG_DIR=logs -cp "engine.jar" 'ch.rswk.meta.engine.Main' -configcls 'ch.rswk.meta.engine.EngineConfig' -config $CONFIG -launchcls 'ch.rswk.meta.engine.EngineLauncher' -update clone,jsonimport $FIXTURE
& "$JRE\java.exe" -Xms128m -Xmx1g -DPLATFORM_LOG_DIR=logs -cp "engine.jar" 'ch.rswk.meta.engine.Main' -configcls 'ch.rswk.meta.engine.EngineConfig' -config $CONFIG -launchcls 'ch.rswk.meta.engine.EngineLauncher' -start

Press Ctrl+C to shutdown the server at any time.

6. Explore meta-admin

  • Open browser (Firefox, Chromium/Chrome) https://localhost.platform.rswk.ch:9090/
    • User test
    • Password test
  • Create a MetaType
    • New > Type
    • Enter a name, eg. My Type
    • Press Save
    • Go to Users
    • Click on admin group to edit it
    • In grants text area, place cursor after {
    • In Insert Type Grants select My Type
    • Change inserted grant from permission level 1 to 15
    • Press Save
  • Create MetaObject
    • New > My Type
    • Enter a name
    • Copy value from below
    • Press Save
    • Create another object with the other value from below
    • Search for objects, eg. with query myNumber:{0 to 999}
  • Login a second client
    • Go to Editor > Meta Objects
    • Select My Type in Type selection
    • Press Search
    • Select one of the objects
    • Select the same object on the first client
  • Update the object
    • Press Update
    • Delete all properties from value except myNumber and change its value
    • Press Update
    • Observe the value change in first client

Object 1 Value

{
    "myDate": "1990-01-01T00:00:00.000Z",
    "myDateTime": "2020-02-02T23:59:59.123Z",
    "myBoolean": true,
    "myNumber": 123,
    "myText": "Hello World 1"
}

Object 2 Value

{
    "myDate": "1990-01-01T00:00:00.000Z",
    "myDateTime": "2020-02-01T23:59:59.123Z",
    "myBoolean": false,
    "myNumber": 567,
    "myText": "Hello World 2"
}

Extend

Extend meta and other parts of the platform to build more complex applications with custom backend services, web applications and deployments.

This is a short overview of the whole setup with organization specific development server, development workstation, watchdog and customer specific application servers.

Development environment

The platform uses the same setup for development as an organization that would like to use the platform itself: A central development server.

Using the bootstrap modules in the platform repository, examples from platform-archetype and platform-archetype-bootstrap repositories, an organization should be able to setup their own environment with a development server and developer VMs.

Development environment

Production environment

A central part of the production environment is one or more watchdog servers. They collect metrics from production servers and can alert about failures and problems. They are also used to distribute bootstrap modules to new and existing servers. The third use case is related to running automated test deployments. It’s up to the individual organization if the want to setup multiple watchdogs or use a single instance.

Production environment

Get started

For an organization to setup their own development server and so on, we use the existing platform repositories and then bootstrap the organization specific environment from there. This takes several steps:

  1. Setup a initial dev-vm
  2. Setup the organization dev-server
  3. Setup organization source repositories
  4. Setup organization integration build jobs
  5. Setup organization watchdog server
  6. Setup organization deployment build jobs

Step 1 is a good starting point to explore the platform and how applications based on the framework components of it are built. Steps 2 to 6 are more complicated and not documented in detail at this time.

Bootstrap dev-vm

Setting up our development environment consists of running a bootstrap module and then some manual configuration steps.

1. Setup and install VM

Setup a fresh Ubuntu or Windows VM, eg. using VirtualBox. Download an ISO:

Make sure to name the initial user account dev. If you are using another *nix or *BSD variant, some things may need to be adapted.

Use the scripts below to start the bootstrap process once you have finished the OS installation.

Ubuntu:

sudo apt-get install unzip
VERSION="0.1180"
cd /var/tmp
wget -O bootstrap.zip "https://dev.platform.rswk.ch/nexus/repository/bootstrap/ch/rswk/platform/bootstrap/bootstrap-dev-vm-platform-nix/$VERSION/bootstrap-dev-vm-platform-nix-$VERSION.zip"
unzip bootstrap.zip -d bootstrap
cd bootstrap
chmod +x fixture/bootstrap.sh
sudo ./fixture/bootstrap.sh -execute $PWD -setowner dev

Windows:

$global:ProgressPreference = 'SilentlyContinue'
$VERSION = '0.1180'
cd C:\tmp
Invoke-WebRequest -Uri https://dev.platform.rswk.ch/nexus/repository/bootstrap/ch/rswk/platform/bootstrap/bootstrap-dev-vm-platform-win/$VERSION/bootstrap-dev-vm-platform-win-$VERSION.zip -OutFile bootstrap.zip
Expand-Archive -LiteralPath bootstrap.zip -DestinationPath bootstrap
cd bootstrap
Set-ExecutionPolicy Unrestricted
.\fixture\bootstrap.ps1 -execute (Get-Location).tostring() -setowner dev

2. Manual steps dev-vm

After the bootstrap module is done, you find all required tools via start menu entries or in:

  • *nix /usr/local/platform/tool
  • Windows C:\tool

There are some manual steps required to finish the setup:

  • Linux specific
    • Log Out and log in back again to get profile variables set
  • Windows specific
    • Virus & threat protection
    • Add C:\data, C:\hg, C:\tool to exclusion settings
  • Start and setup Eclipse
    • Window > Preferences
      • General
        • Appearance > Theke > Dark
        • Editors > Text Editors
          • Insert spaces for tabs
          • Show line numbers
      • Java
        • Code Style > Formatter > New
          • Indentation > Tab policy > Spaces only
        • Installed JREs
          • *nix /usr/local/platform/tool/jdk14
          • Windows C:\tool\jdk14
      • Maven
        • Annotation Processing > Automatically configure
        • Installations > Add
          • *nix /usr/share/maven
          • Windows C:\tool\maven
      • Xtend
        • Compiler > Output folder for generated Java files > ../../target/generated-sources/xtend
        • Formatter > New > Initialize settings from “default”
        • Syntax Coloring
    • Use Package Explorer instead of Project Explorer view
      • Alt+Shift+Q > Q
      • Package Explorer > Open
      • In Package Explorer view > ... > Top Level Elements > Working Sets
    • Add more views
      • Console
      • JUnit
      • Tasks
      • Problems
        • ... > Show > Errors/Warnings on Selection
    • Import Maven modules
      • File > Import > Existing Maven Projects
        • *nix /home/dev/hg/platform
        • Windows C:\hg\platform
      • Deselect All
      • Select all child modules
        • Eg. for util: util-macro, util-macro-tests, util-core, util-core-tests)
        • Ignore
          • meta-admin (since it’s a pure web module)
          • bootstrap-module
          • All modules under the top level bootstrap
    • Import second bootstrap-test module
      • File > Import > Existing Maven Projects
      • Advanced > Name template [artifactId]-2
      • Select bootstrap-test under the top level bootstrap module
    • Enable Xtend
      • Select all imported projects in Package Explorer
      • Right click > Configure > Convert to Xtext Project
      • Wait for build, in case of errors Project > Clean > Clean all projects
  • Start and setup Visual Studio Code
    • Add SCM root to Workspace
      • *nix /home/dev/hg/
      • Windows C:\hg
    • Install extensions
      • CSS Formatter
      • Hg
      • XML Tools
  • Install and setup Android Studio
    • In downlad directory where you extracted the bootstrap module find
      • *nix android-studio.tar.gz
      • Windows android-studio.exe
    • Install
      • *nix unpack archive to /usr/local/platform/tool/android-studio and run ./bin/studio.sh
      • Windows run installer
    • Setup
      • Do not import settings
      • Custom
      • Dracula
      • Android SDK location
        • *nix /usr/local/platform/tool/android-sdk
        • Windows C:\tool\android-sdk
      • Finish
    • Open an existing Android Studio project
      • *nix /home/dev/hg/platform/android
      • Windows C:\hg\platform\android

3. Explore platform-archetype

With the local development environment ready, we can explore the platform-archetype repository. It’s the reference for a platform based repository and contains a typical application setup in myapp.

  • In Eclipse, import the platform-archetype modules
    • File > Import > Existing Maven Projects
      • *nix /home/dev/hg/platform-archetype
      • Windows C:\hg\platform-archetype
    • Deselect All
    • Select myapp-core, myapp-engine, myapp-test
    • Working set archetype
  • Convert imported modules to Xtext projects

Before we dive into running and making changes to the example application, let’s make sure everything is setup correctly by running meta-admin locally:

  • Start meta-admin in Eclipse
    • Project meta-admin-test
      • Open WebDriverTest.xtend
        • On the last line of setupRuntime, insert System.in.read
      • Right click on WebDriverTest.xtend > Run As > JUnit Test
      • Browser https://localhost.platform.rswk.ch:9090/meta/meta-admin/src/main/resources/
      • See Try for some examples
  • Stop test case in JUnit view when done

Back to the myapp example, we are going to make some changes and additions to the back- and frontend and see how things work together.

  • TODO: Create new backend service
  • TODO: Wire up backend service in client
  • TODO: Write a JUnit test, QUnit and Selenium test

Bootstrap dev-server

If you are serious about software development and creating applications based on the platform, the next step is to bootstrap your own environment, starting with the development server.

1. Build dev-server module

  • Build a local copy of the bootstrap-dev-server module, parameterized for your own development server
  • TODO: Explain how to parameterize build
  • Final ZIP is bootstrap/bootstrap-dev-server/target/bootstrap-dev-server-1.0-SNAPSHOT.zip

2. Upload and install dev-server

  • Setup a Ubuntu or Windows Server VM, eg. with a cloud computing provider like Vultr
  • Upload final ZIP to server
    • *nix /var/tmp
    • Windows C:/tmp
  • Install bootstrap module

Ubuntu:

find /var/tmp -name 'bootstrap-*.zip' -exec unzip {} -d /var/tmp/bootstrap \;
cd /var/tmp/bootstrap
chmod +x fixture/bootstrap.sh
sudo ./fixture/bootstrap.sh -execute $PWD

Windows:

$global:ProgressPreference = 'SilentlyContinue'
Expand-Archive -LiteralPath (Get-ChildItem C:\tmp | Where-Object {$_.Name -like 'bootstrap-*.zip'}).FullName -DestinationPath C:\tmp\bootstrap
cd C:\tmp\bootstrap
Set-ExecutionPolicy Unrestricted
.\fixture\bootstrap.ps1 -execute (Get-Location).tostring()

3. Manual steps dev-server

As with the dev-vm, some manual steps are required to finish our setup:

  • Install and setup Android Studio
    • In downlad directory where you extracted the bootstrap module find
      • *nix android-studio.tar.gz
      • Windows android-studio.exe
    • Install
      • *nix unpack archive to /usr/local/platform/tool/android-studio and run ./studio.sh
      • Windows run installer
    • Setup
      • Do not import settings
      • Custom
      • Android SDK location
        • *nix /usr/local/platform/tool/android-sdk
        • Windows C:\tool\android-sdk
      • Finish
  • Setup source control
    • Browser http://127.0.0.1:8082/scm
    • Change scmadmin password (default scmadmin)
    • Create platform user with password platform (skip if not platform)
    • Import repositories > Mercurial
  • Setup automation server
    • Browser http://127.0.0.1:8082/jenkins
    • Install suggested plugins
    • Manage plugins
      • Available
        • Mercurial
        • Maven Integration
    • Global tool configuration
      • JDK
        • *nix /usr/local/platform/tool/java14
        • Windows C:\tool\java14
      • Git
        • *nix TODO
        • Windows C:\tool\git\bin\git.exe
      • Mercurial
        • *nix TODO
        • Windows C:\tool\tortoisehg, INSTALLATION\hg.exe
      • Maven
        • *nix /usr/share/maven
        • Windows C:\tool\maven
    • Credentials > Jenkins > Global credentials
      • Add Credentials
        • scmadmin and password
          • platform and password platform
  • Setup software repository
    • Browser http://127.0.0.1:8081/nexus
    • Change password of admin user
    • Add hosted maven2 repos
      • bootstrap with release, allow redeploy policy
      • thirdparty with release policy
    • Add hosted raw repo site
    • Add proxy maven2 repo google
    • Add proxy maven2 repo jcenter
    • Add proxy maven2 repo platform
    • Add all to maven-public repo
    • Add cleanup policies
      • Repository > Cleanup Policies > Create
        • Format maven2
        • Last Downloaded Before 200
      • Repository > Repositories
        • bootstrap, maven-central, maven-release (and others if needed)
        • Set cleanup policy
    • Setup compacting task
      • System > Tasks > Create task
      • Compact Blob Store
      • Blob store default
      • Task frequency monthly
      • Time to run this task 2:00
      • Days to run this task Last

Bootstrap My Company

With a development VM and server running, we can setup the source repositories and build jobs.

1. Setup source repositories

On the development server:

  • Browser http://127.0.0.1:8082/scm
  • Login with scmadmin and password
  • Create mycompany and mycompany-bootstrap Mercurial repositories
    • Note: This is only required for the initial server installation, it will become part of your dev-server bootstrap definition
  • Create additional users and grant permissions for the created repos

2. Populate bootstrap source repository

On your development VM:

  • Open .hgrc
    • *nix /home/dev/.hgrc
    • Windows C:/Users/dev/.hgrc
  • Add your company server credentials to the auth section
mycompany.prefix = dev.mycompany.ch
mycompany.username = dev
mycompany.password = dev
  • Clone mycompany-bootstrap
    • hg clone https://dev.mycompany.ch/scm/hg/mycompany-bootstrap
  • Copy contents of platform-archetype-bootstrap to mycompany-bootstrap
  • Rename and adapt configs in jobs folder
    • bootstrap-dev-server-mycompany.xml
    • bootstrap-dev-vm-mycompany.xml
    • bootstrap-watchdog-server-mycompany.xml
    • mycompany.xml
  • TODO: Explain how to adapt these jobs
  • Commit changes in mycompany-bootstrap repository and push to server

3. Setup integration build jobs

On your development server:

  • Copy all the adapted job configs to a local folder
  • Import all jobs to Jenkins (adapt path to XML file and job name accordingly)
    • *nix curl --data-binary @mycompany.xml --request POST --header 'Content-Type: text/xml' 'http://127.0.0.1:8082/jenkins/createItem?name=mycompany'
    • Windows curl -body "file=$(get-content mycompany.xml -raw)" -method POST -ContentType 'text/xml' -uri 'http://127.0.0.1:8082/jenkins/createItem?name=mycompany'
  • Change build number if needed in Jekins script console
    • Jenkins.instance.getItemByFullName("mycompany").updateNextBuildNumber(123)

4. Populate software source repository

On your development VM:

  • Clone mycompany repository
    • hg clone https://dev.mycompany.ch/scm/hg/mycompany
  • Copy contents of platform-archetype repository into mycompany
    • bootstrap-dev-server
    • bootstrap-dev-vm
    • TODO: Explain what to adapt
  • Adapt bootstrap configs in mycompany-bootstrap/mycompany
    • bootstrap-dev-server
    • bootstrap-dev-vm
    • bootstrap-watchdog-server
    • TODO: Explain what to adapt
  • Commit and push mycompany, mycompany-bootstrap changes to server

5. Run build jobs

  • mycompany
  • bootstrap-dev-server-mycompany
    • TODO: Params
  • bootstrap-dev-vm-mycompany
    • TODO: Params
  • bootstrap-watchdog-server-mycompany
    • TODO: Params

Now you can loop back to the first chapter and setup a new development VM, but instead of using the platform Nexus, use your own Nexus to download the dev-vm-mycompany bootstrap module:

  • https://dev.mycompany.ch/nexus/repository/bootstrap/ch/mycompany/bootstrap/bootstrap-dev-vm-mycompany/$VERSION/bootstrap-dev-vm-mycompany-$VERSION.zip

With this new VM, all your company repositories should already be checked out and the Maven build should be using your own Nexus server.

6. Install watchdog server

  • Run bootstrap-watchdog-server-mycompany build job
    • TODO: Params
  • Setup a server VM with a cloud provider of your choice
  • Download bootstrap-watchdog-server-mycompany from your Nexus
    • https://dev.mycompany.ch/nexus/repository/bootstrap/ch/mycompany/bootstrap/bootstrap-watchdog-server-mycompany/$VERSION/bootstrap-watchdog-server-mycompany-$VERSION.zip
  • Upload bootstrap ZIP to server
    • *nix /var/tmp
    • Windows C:/tmp
  • Install bootstrap module

Ubuntu:

find /var/tmp -name 'bootstrap-*.zip' -exec unzip {} -d /var/tmp/bootstrap \;
cd /var/tmp/bootstrap
chmod +x fixture/bootstrap.sh
sudo ./fixture/bootstrap.sh -execute $PWD -flags 'certbot'

Windows:

$global:ProgressPreference = 'SilentlyContinue'
Expand-Archive -LiteralPath (Get-ChildItem C:\tmp | Where-Object {$_.Name -like 'bootstrap-*.zip'}).FullName -DestinationPath C:\tmp\bootstrap
cd C:\tmp\bootstrap
Set-ExecutionPolicy Unrestricted
.\fixture\bootstrap.ps1 -execute (Get-Location).tostring() -flags 'certbot'

7. Run local deployment test

  • Setup a vultr account
    • Account settings > API
    • Enable API and copy key
    • Setup Access Control
  • Eclipse
    • Create a new run configuration for BootstrapTest with VM arguments
    • TODO: Explain the details

8. Setup deployment build jobs

  • Setup build job for customerone
    • bootstrap-myapp-server-customerone.xml
    • build.json
    • provision.json
    • TODO: Explain the details

Free & Open Source

Bounties

Bounty of CHF 750 is awarded for each meta-client implementations in another language:

  • See Client.xtend/client.json in meta-client for existing implementations
  • Must implement model and protocol classes. See Model.xtend/Protocol.xtend and model.js/protocol.js in meta-core
  • Must implement client interface. See Client.xtend/client.json in meta-core
  • Must implement the same test cases as ClientTest.xtend/client-test.js in meta-engine-test
  • Bounties will be awarded for
    • C# 8
    • Python 3.x
    • Go 1.1x

A bounty of CHF 750 is awarded for reviewing security in meta-engine:

  • Must make recommendations on how to improve existing code to cover different scenarios and attacks
  • Password, OTP, WebAuthn and token handling in UserService.xtend
  • Authorization in EngineCore.xtend

Please get in contact before working on a bounty, so we can prevent the unlikely case that multiple people start working on the same thing.

You can receive bounties in:

  • PayPal
  • Wire transfer (must provide IBAN)
  • Gift cards (must be available in Switzerland)
  • TWINT (Switzerland only)

Development

Backend

Frontend

Cheatsheet

Bootstrap (Cheatsheet)

  • *nix
    • Bash with bootstrap-my-module-0.1.zip in directory /var/tmp
sudo apt-get install unzip
find /var/tmp -name 'bootstrap-*.zip' -exec unzip {} -d /var/tmp/bootstrap \;
cd /var/tmp/bootstrap
chmod +x fixture/bootstrap.sh
sudo ./fixture/bootstrap.sh -execute $PWD -start "{moduleDir}/definitions/my-module.json" -flags "clearLogging" -setowner dev
  • Windows
    • PowerShell as Administrator with bootstrap-my-module-0.1.zip in directory C:/tmp
$global:ProgressPreference = 'SilentlyContinue'
Expand-Archive -LiteralPath (Get-ChildItem C:\tmp | Where-Object {$_.Name -like 'bootstrap-*.zip'}).FullName -DestinationPath C:\tmp\bootstrap
cd C:\tmp\bootstrap
Set-ExecutionPolicy Unrestricted
.\fixture\bootstrap.ps1 -execute (Get-Location).tostring() -start '{moduleDir}/definitions/my-module.json' -flags 'clearLogging' -setowner dev
  • Misc
    • If a bootstrap definition is placed in the default directory {moduleDir}/definitions, the full path can be omitted, eg. -start my-module.json instead of -start "{moduleDir}/definitions/my-module.json"

Certificates/keystores

  • Java keystore with self signed
keytool -genkey -alias server -keystore keystore.pfx -storetype PKCS12 -keyalg RSA -validity 720
  • Remove certificate from Java keystore
keytool -delete -alias server -keystore keystore.pfx
  • Install OpenSSL, eg. together with HTTPD (See bootstrap definition platform/bootstrap/bootstrap-common/src/main/resources/bootstrap/definitions/httpd.json)
  • Configuration
set RANDFILE=C:\data\openssl.rnd
set OPENSSL_CONF=C:\tool\httpd\conf\openssl.cnf
  • Convert PEM certificate and private key to PFX Java keystore
openssl pkcs12 -export -in server.crt -inkey server.key -out server.pfx -name server -password pass:123
  • Self signed CA, server certificate with SAN
    1. Adapt configuration in openssl.cnf
    2. Generate certificate
[ req ]
req_extensions = v3_req

[ v3_req ]
basicConstraints = CA:FALSE
keyUsage = digitalSignature, keyEncipherment
subjectAltName = @alt_names

[ alt_names ]
DNS.1 = myapp.com

[ v3_ca ]
authorityKeyIdentifier = keyid,issuer
basicConstraints = critical,CA:true,pathlen:3
keyUsage = critical, cRLSign, keyCertSign
nsCertType = sslCA, emailCA
subjectKeyIdentifier = hash
openssl genrsa -out ca.key 2048
openssl req -new -x509 -extensions v3_ca -days 720 -key ca.key -sha256 -out ca.crt
openssl genrsa -out server.key 2048
openssl req -extensions v3_req -sha256 -new -key server.key -out server.csr
openssl x509 -req -extensions v3_req -days 720 -sha256 -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -extfile C:\tool\httpd\conf\openssl.cnf

device.js

  • Simulate device events in browser dev tools (make sure to change port if running a test with PlatformResourceServer)
const deviceFactory = require('web-core/device').factory;

// NFC read
window.postMessage(JSON.stringify(deviceFactory.deviceReadNfcTagEvent({writable: false, id: "1"})), "https://localhost.platform.rswk.ch:9090");

// NFC write
window.postMessage(JSON.stringify(deviceFactory.deviceWriteNfcTagEvent({id: "1"})), "https://localhost.platform.rswk.ch:9090");

// BTLE scan
window.postMessage(JSON.stringify(deviceFactory.deviceBtleScanEvent({devices: [
    webDeviceFactory.deviceBtleDevice({address: "A1:B2", name: "Test Beacon", rssi: 90, timestamp: "2007-01-01T23:58:59.999Z"})
]})), "https://localhost.platform.rswk.ch:9090");
  • Simulate device events in WebDriverTest
// NFC read
driver.windowPostDeviceWebViewEvent(new DeviceReadNfcTagEvent(false, "1", null))

// NFC write
driver.windowPostDeviceWebViewEvent(new DeviceWriteNfcTagEvent("1", null))

// BTLE scan
driver.windowPostDeviceWebViewEvent(new DeviceBtleScanEvent(#[
    new DeviceBtleDevice("A1:B2", "Test Beacon", 90, Cst.now, null)
]))
  • Make sure to implement getMessageOrigin when using windowPostDeviceWebViewEvent
override getMessageOrigin() {
    return forgePlatformUri(HTTP_SERVICE_TEST_PORT, null).toString
}

Dev VM

  • Setup WiFi Hotpost for local testing with mobile devices
    • Windows Settings > Network > Mobile hotspot
      • Set SSID and Password and turn on
      • Change adapter options > Microsoft WiFi Direct Virtual Adapter > Properties > Internet Protocol Version 4
        • Use the following IP: 192.168.1.1
        • Use the following DNS: 192.168.1.1
    • Install and setup Simple DNS Plus
      • In downlad directory where you extracted the bootstrap module find simpledns.exe
        • *nix 🤷‍♂️
        • Windows Run installer
    • Setup
      • Records > New Zone > Primary Zone > Forward Zone > rswk.ch
      • Right click rswk.ch > New A-Record > localhost.platform.rswk.ch with IP 192.168.1.1
    • If you are running in a VM, make sure to forward the necessary ports!
  • Setup em microelectronic Beacons tool kit
    • Product https://www.emmicroelectronic.com/catalog?title=&term_node_tid_depth=42
    • Tool kit required https://www.emmicroelectronic.com/product/microcontroller-tools-support/em6819-tool-kit
    • SDK
      • Sign up on https://forums.emdeveloper.com/index.php
      • Download latest https://forums.emdeveloper.com/index.php?resources/embc01-02-sdk.18/
      • Unpack SDK, eg. to C:/tool/EMBCxx
      • Download and install RubyInstaller from https://rubyinstaller.org/downloads/
    • Ride7
      • Download https://dev.platform.rswk.ch/dl/sx32w.dll.zip
        • Copy sx32w.dll to C:\tool\EMBCxx\util\bin
        • Add this folder to PATH
      • Download and insall Ride7 IDE from http://support.raisonance.com/content/ride

Links

Logging

  • Java tests in platform
    • See framework/util/util-macro-test/src/main/resources/logback-test.xml
  • Java in general
    • See logback.xml or logback-test.xml in src/main/resources
  • Java with meta-engine
    • In engine.json
"logging": {
    "loggers": {
        "ch.rswk.meta.engine": "DEBUG"
    }
}
  • JavaScript SPA
    • In config.js with debug: true
  • JavaScript QUnit tests in qunit.html
Logger.useDefaults({
    defaultLevel: Logger.DEBUG
});
  • JavaScript client with meta-admin
    • From Notifications, send a ClientNotification
      • Channel egn (or any channel you know the client is subscribed to)
      • Action INFO
      • Data {"logLevel": 'DEBUG'}

Lucene

  • Inspect index with Luke
    • *nix /usr/local/platform/tool/lucene/luke/luke.sh
    • Windows C:\tool\lucene\luke\luke.bat
    • Edit scripts and set correct path to Java directory
  • Re-index existing index, eg. after new Lucene version
    • Use clone task from EngineUpdater
  • Re-index existing index, eg. after changes to meta model
    • Implement a EngineUpdaterEntityHandler to migrate existing entities with the clone task
    • See EngineUpdaterSchemaTest.xtend
  • Test with existing index
    • Get copy of index you would like to run locally
      • Manually create a 7z-file from an existing work directory
      • Or backup and download index with meta-admin > File > Search backups
    • Upload in local meta-admin > File
      • Enter correct tags, eg. for MetaService: backup,mttluceneservicemeta
    • Use restore option on uploaded file
  • Alternative way to test with existing index
    • Extract/copy index to a local directory, eg. C:/tmp/index/work
    • In MetaServiceTest.xtend, assembleMetaServiceTestConfig, pass directory to assembleLuceneServiceTestConfig, eg. C:/tmp/index

JRE packaging

  • Download JDK
  • Generate list of all .jmod files in jmod directory, eg. with ls and a text editor
  • Generate JRE
    • Start shell in bin directory
    • Replace XXX below with comma separated list of all jmod files (without the .jmod endings)
    • *nix ./jlink --module-path ../jmods --add-modules "XXX" --output /home/dev/java14
    • Windows .\jlink.exe --module-path ..\ --add-modules "XXX" --output C:\data\java14
  • Create 7z from content of java14 directory
    • *nix java14-nix.7z
    • Windows java14-win.7z

Maven

  • Only build current module and not full reactor mvn -N
  • Skip tests mvn -DskipTests
  • Use parallel test feature mvn -T 1C (may break some tests)
  • Only build certain modules mvn -pl meta-core,meta-client
  • Only build certain module and all dependencies mvn -pl meta-engine -am

meta-admin

  • Run alongside another web module for testing
    • Build meta-admin or download latest release ZIP from Nexus
    • Extract to a folder inside the web module, eg. to myapp-web/target/admin
    • Adapt config.js
      • workerPath: "js/"
    • Run WebDriverTest of the application and go to https://localhost.platform.rswk.ch:9090/myapp-web/target/admin
  • Force client reload
    • Notification > Send client notification to a channel that the client is subscribed to
      • Data {"reload": true}
  • Schedule a remote server update
    • File > Upload file and copy key when done from the file table
    • Watchdog > Click on wrench icon of engine to update or use Update... for a local update
    • Insert *nix or Windows command and adapt accordingly, eg. set different definition, or additional flags, etc.
    • Paste file key
    • Enter file name, eg. bootstrap.zip
    • Set a scheduled time and date, if left blank update will run immediately
    • Next time the remote engine sends it status to the watchdog, it will start processing the update
    • Check watchdog update status with Editor > Objects > Type > Watchdog Status

VPN

To generate CA, server and a client cert (multiple client certs can be generated) with OpenVPN on Windows:

  • Download OpenVPN
  • Select EasyRSA Management Scripts in installer
  • Open command line as Administrator in easy-rsa directory
    • init-config.bat
      • Open vars.bat and copy adapted values from below
    • vars.bat
    • clean-all.bat
    • build-ca.bat
      • Common Name and Name: myca
    • build-key-server.bat myserver
    • build-key.bat myclient
      • Common Name and Name: myclient
    • build-dh.bat
  • Find the generated files in easy-rsa/keys folder
Rem vars.bat example
set KEY_COUNTRY=CH
set KEY_PROVINCE=ZH
set KEY_CITY=Zurich
set KEY_ORG=My Company
set KEY_EMAIL=ict@mycompany.ch
set KEY_CN=myserver.mycompany.ch
set KEY_NAME=myserver.mycompany.ch
set KEY_OU=ICT
set PKCS11_MODULE_PATH=changeme
set PKCS11_PIN=1234

Setup server and clients:

  • pfSense VPN server
    • System > Cert Manager
      • CAs > Add
        • Method > Import existing
        • ca.crt
      • Certificates > Add
        • Method > Import existing
        • myserver.crt
        • myserver.key
    • VPN > OpenVPN > Servers > Add
      • Peer Certificate Authority > myca
      • Server certificate > myserver
      • Encryption algorith > AES-256-GCM
      • Enable NCP > Uncheck
      • Auth digest algorithm > SHA512
      • Compression > LZ4 Compression v2
  • pfSense VPN client
    • System > Cert Manager
      • CAs > Add
        • Method > Import existing
        • ca.crt
      • Certificates > Add
        • Method > Import existing
        • myclient.crt
        • myclient.key
    • VPN > OpenVPN > Clients > Add
      • Server host or address > vpn.mycompany.ch
      • Use Automatically generate a TLS Key (and paste it into the VPN server config) if you don’t have one or paste the one already generated on the VPN server
      • Peer Certificate Authority > myca
      • Client certificate > myclient
      • Encryption algorith > AES-256-GCM
      • Enable NCP > Uncheck
      • Auth digest algorithm > SHA512
      • Compression > LZ4 Compression v2
  • OpenVPN server
    • Copy sample-config/server.ovpn to config/server.ovpn
    • Copy the generated files required below to config folder
    • Set/overwrite
      • ca ca.crt
      • cert myserver.crt
      • key myserver.key
      • dh dh2048.pem
      • tls-auth ta.key > Copy generated key
      • cipher AES-256-GCM
      • auth SHA512
      • compress lz4-v2
  • OpenVPN client
auth SHA512
ca [inline]
cert [inline]
cipher AES-256-GCM
client
compress lz4-v2
dev tun
keepalive 10 120
key [inline]
persist-key
persist-tun
port 1194
proto udp
remote vpn.mycompany.ch 1194
remote-cert-tls server
resolv-retry infinite
tls-auth [inline] 1
verb 1
<ca>
INSERT CONTENTS OF ca.crt HERE
</ca>
<cert>
INSERT CONTENTS OF myclient.crt HERE
</cert>
<key>
INSERT CONTENTS OF myclient.key HERE
</key>
<tls-auth>
INSERT CONTENTS OF ta.key (or from pfSense) HERE
</tls-auth>

Web

  • Development/testing
    • In WebDriverTest, at the end of @BeforeEach def void setupRuntime(), insert breakpoint or add System.in.read to halt the test execution. The server is now ready and you can open up the web module, see URLs above
  • Serviceworker
    • Force reload and refresh
      • In config.js change version
      • In worker-init.js change comment // Version: ${project.version}
  • URL for tests with StaticResourceServer/PlatformResourceServer
https://localhost.platform.rswk.ch:8088/web-core-test/src/main/resources/qunit.html
  • URL for tests with meta-engine/HttpService
https://localhost.platform.rswk.ch:9090/meta-admin/src/main/resources/
  • After updating dependencies in web-build.json or changing the platform version, make sure to run a Maven build on the web module. This will downloaded the latest files into target and lib directories

TODO

  • util-macro
    • Set JsonInclude.Include.NON_NULL on nullable properties
    • Generated forge/melt methods for JsonNode fields in Jsonized
  • util-core
    • Replace Env.exec with commons-exec
    • Real FreeBSD support?
  • web-core
    • In nav, store application state in local storage and restore it on page load/refresh
  • meta-engine
    • Don’t store JSON in index, so we can reindex at anytime?
    • Lucene soft-delete mechanism to retain full document history?
    • SyncService with etcd as backend
      • Integrate FileService
      • Catch up and replication tests
  • bootstrap-test
    • SyncService test with etcd node, at least two meta-engine nodes