Introduction to RATT 101

Thomas de Groot (thomas.de.groot@triply.cc)

Overview

  1. RATT
  2. Modelling
  3. Middelwares
  4. Installation
  5. Use case 1: Stolpersteine
  6. Use case 2: Ministery of Finance

Before RATT

RATT

Functions of RATT

  • Data Ingestion
  • Data Transformation
  • Data Linking
  • Data Publication

RATT vs RML vs COW

RATT RML COW
Language: Typescript Java/ttl Python/JSON-LD
works: Out of the box Needs interpreter Needs interpreter
linked data No Yes Yes
Standard: No Yes Yes
Input format: Everything csv, json, xml csv
expandable Yes Limited Limited

Prerequisites

For linked data project we follow the Triply workflow

Basic prefixes


                  prefix foaf: <http://xmlns.com/foaf/0.1/>
                  prefix graph: <https://triplydb.com/academy/stolpersteine/graph/>
                  prefix person: <https://triplydb.com/academy/stolpersteine/id/person/>
                  prefix rdf: <http://www.w3.org/1999/02/22-rdf-syntax-ns#>
                  prefix time: <http://www.w3.org/2006/time#>
                  prefix def: <https://triplydb.com/academy/stolpersteine/def/>
                  prefix xsd: <http://www.w3.org/2001/XMLSchema#>
                  prefix void: <http://rdfs.org/ns/void#>
In the following, we can assume these prefix declarations.

Modelling Tooling

middelwares

Application


              // Ratt application
              const configObject = {
                defaultGraph: NamedNode,
              };
              const app = new Ratt(configObject);

              // Middleware context, record and store
              class Context  {
                record: Record; // [key: string]: unknown;
                store: Store; // N3Store
              }
RATT Application

Read file types


                // read from array:
                app.use(fromArray(input: string[][], opts: {columns: boolean});
                // read from csv:
                app.use(fromCsv(files: string | string[], opts:{compressed: boolean, columns: boolean}))
                // read from shapeFile:
                app.use(fromShapeFile(files: string | string[]))
                // read from xml(Not generic yet):
                app.use(readXml(files: string | string[], opts: Config)))
Reading files

Record manipulations


                app.use(set(key: string; value: (ctx) => return "" ))
                app.use(copy(fromKey: string; toKey: string; change?: (val, ctx) => return `${val}` ))
                app.use(change(key: string; change: (val, ctx) => return val ? 0 : 1 ))
                app.use(remove(key: string; when: (val, ctx) => return ctx.record.Reason ))
Record manipulations

Add quads


                app.use(addQuad(toNamedNode("COLUMN-person"), rdf("type"), def("Person")))
                app.use([addQuad((ctx) => {return person(ctx.record.COLUMN-person);}, def("mother"), toNamedNode("COLUMN-mother")),
                         addQuad(toNamedNode("COLUMN-person"), rdfs("label"), toLiteral("T Riply", { datatype: xsd("string") }),
                         addQuad(toNamedNode("COLUMN-person"), foaf("name"), toLiteral("Tim", { language: "nl" })),
                         addQuad(toNamedNode("COLUMN-person", {prefix: person}), sdo("countryOfOrigin"), DataFactory.literal("Netherlands")),
                         addQuad(toNamedNode("COLUMN-person"), toNamedNode("adres"),toBnode([
                                                                                  [rdf("type"), sdo("Address")],
                                                                                  [foaf("name"), toLiteral("gemeenteNaam")]]))]);
                app.useWhen((ctx) => {return !!ctx.record.COLUMN-kind;},
                            addQuad(toNamedNode("COLUMN-person"), sdo("child"), person("COLUMN-kind")));

Write, execute and publish


                // Write to file middelware
                app.use(writeToFile(path.resolve(outputDir, "dummy.nq"), {compress: true}));
                // execute
                await app.run();
                //Upload to TriplyDB
                const account = await getAccount();
                const ds = await putDs(account, { name: `dummy` });
                await ds.importFromFiles(path.resolve(outputDir, "dummy.nq"));
Write to file, execute and publish

Installation

Install Node and follow the steps for your OS.

Install Yarn and follow the steps for your OS.

Installation

For installation we can use the boilerplates repository: https://git.triply.cc/triply/boilerplates 

                git clone ssh://git@git.triply.cc:10072/triply/boilerplates.git
                ./bootstrapBoilerplate --name myNewEtl --boilerplate etl ~/triply/myNewEtl

To install

set your tokens in `.envrc-local` and install the packages + build the repo. 

                yarn
                yarn run build

To run


                yarn run start

Use case 1: Stolpersteine

  • Prefixes and static variables
  • Generic ETL middelwares
  • ETL specific middelwares
  • Upload to TriplyDB

Use case 2: Ministery of Finance

  • Multi file ETL
  • CI/CD and docker
  • versioning and Deployen of container

Thank you for your attention!