Documentation

Tutorial

You can get quickly started with the tutorial in Google Colaboratory.

Installation

In the following we describe different ways in which you can install and use Morph-KGC.

PyPI

PyPI is the fastest way to install Morph-KGC:

pip install morph-kgc

Some data sources and output formats require additional dependencies. Check Advanced Setup for specific installation instructions or install all the dependencies:

pip install morph-kgc[all]

We recommend to use virtual environments to install Morph-KGC.

From Source

You can also grab the latest source code from the GitHub repository:

pip install git+https://github.com/morph-kgc/morph-kgc.git

Usage

Morph-KGC uses an INI file to configure the materialization process, see Configuration.

Command Line

Installing Morph-KGC puts the morph-kgc command on your path. To run the engine using the command line you just need to execute the following:

morph-kgc path/to/config.ini

Running the package as a module is equivalent, and is what the Docker image does:

python -m morph_kgc path/to/config.ini

Library

Morph-KGC can be used as a library, providing different methods to materialize the knowledge graph. It integrates with RDFLib and Oxigraph to easily create and work with knowledge graphs in Python.

The methods in the API accept the configuration as the path to an INI file, as an INI string, or as a Python dictionary.

import morph_kgc

# the path to an INI file
config = '/path/to/config.ini'

# an INI string
config = """
            [DataSource1]
            mappings: /path/to/mapping/mapping_file.rml.ttl
            db_url: mysql+pymysql://user:password@localhost:3306/db_name
         """

# a dictionary with one section per data source
config = {
    'CONFIGURATION': {'output_format': 'N-QUADS'},
    'DataSource1': {
        'mappings': '/path/to/mapping/mapping_file.rml.ttl',
        'db_url': 'mysql+pymysql://user:password@localhost:3306/db_name'
    }
}

# a flat dictionary, when there is a single data source
config = {
    'output_format': 'N-QUADS',
    'mappings': '/path/to/mapping/mapping_file.rml.ttl',
    'db_url': 'mysql+pymysql://user:password@localhost:3306/db_name'
}

Every method takes an optional second argument: a dictionary with the in-memory data structures the mapping reads (DataFrames, dictionaries, lists, tuples and JSON strings), keyed by the name the logical sources give them. See In-Memory Data.

import pandas as pd

films_df = pd.DataFrame({'id': [1, 2], 'title': ['Metropolis', 'M']})

graph = morph_kgc.materialize(config, {'films': films_df})

RDFLib

morph_kgc.materialize(config)

Materialize the knowledge graph to RDFLib.

# generate the triples and load them to an RDFLib graph

graph = morph_kgc.materialize(config)
# or
graph = morph_kgc.materialize('/path/to/config.ini')

# work with the RDFLib graph
q_res = graph.query(' SELECT DISTINCT ?classes WHERE { ?s a ?classes } ')

Note: RDFLib does not read RDF 1.2 triple terms or directional language-tagged strings, hence materialize does not support the RML 1.2 constructs that generate them. Use materialize_set or write the knowledge graph to a file instead.

Oxigraph

morph_kgc.materialize_oxigraph(config)

Materialize the knowledge graph to Oxigraph.

# generate the triples and load them to Oxigraph

graph = morph_kgc.materialize_oxigraph(config)
# or
graph = morph_kgc.materialize_oxigraph('/path/to/config.ini')

# work with Oxigraph
q_res = graph.query(' SELECT DISTINCT ?classes WHERE { ?s a ?classes } ')

Set of Triples

morph_kgc.materialize_set(config)

Materialize the knowledge graph to a Python Set of triples. Each element is a serialized statement without its trailing ., so this is the method to use for knowledge graphs that no RDF library reads back yet, such as those with RML 1.2 triple terms.

# create a Python Set with the triples

graph = morph_kgc.materialize_set(config)
# or
graph = morph_kgc.materialize_set('/path/to/config.ini')

# work with the Python set
print(len(graph))

Configuration

The configuration of Morph-KGC is done via an INI file. This configuration file can contain the following sections:

CONFIGURATION

• Contains the parameters that tune the execution of Morph-KGC, see Engine Configuration.

One section for each DATA SOURCE

• Each input data source has its own section, see Data Sources.

One RESOURCE:<name> section for each accessed resource

• Resources are not materialized, they are accessed while materializing: a vocabulary to reconcile against, a SPARQL endpoint to query. See Resources.

DEFAULT

• It is optional and it declares variables that can be used in all other sections for convenience. For instance, you can set main_dir: ../testing so that main_dir can be used in the rest of the sections.

Below is an example configuration file with one input relational source. In this case DataSource1 is the only data source section, but other data sources can be considered by including additional sections. Here you can find a configuration file which is more complete.

[DEFAULT]
main_dir: ../testing

[CONFIGURATION]
output_file: knowledge-graph

[DataSource1]
mappings: ${main_dir}/mapping_file.rml.ttl
db_url: mysql+pymysql://user:password@localhost:3306/db_name

The parameters of the sections in the INI file are explained below.

Engine Configuration

The execution of Morph-KGC can be tuned via the CONFIGURATION section in the INI file. This section can be empty, in which case Morph-KGC will use the default property values.

Property	Description	Values
output_file	File to write the resulting knowledge graph to. The file extension is derived from output_format, so it does not need to be given.	Default: knowledge-graph
output_dir	Directory to write the resulting knowledge graph to. If it is specified, output_file will be ignored and multiple output files will be generated, one for each mapping partition.	Default:
output_format	RDF serialization to use for the resulting knowledge graph.	Valid: N-TRIPLES, N-QUADS, JELLY Default: N-TRIPLES
na_values	Set of values to be interpreted as NULL when retrieving data from the input sources. The set of values must be separated by commas.	Default: ,nan
literal_escaping_chars	Additional set of characters to be escaped in the generation of literals. The set of characters must be separated by commas. The backslash, the double quote, the line feed and the carriage return are always escaped.	Default: ",\n,\r
safe_percent_encoding	Set of ASCII characters that should not be percent encoded. All characters are encoded by default.	Example: :/ Default:
udfs	File with Python user-defined functions to be called from RML-FNML.	Default:
state_dir	Directory the shared context of stateful functions is persisted to. If it is not provided, a temporary directory is created and removed for every run.	Default:
mapping_partitioning	Mapping partitioning algorithm to use. Mapping partitioning can also be disabled.	Valid: PARTIAL-AGGREGATIONS, MAXIMAL, no, false, off, 0 Default: PARTIAL-AGGREGATIONS
infer_sql_datatypes	Infer datatypes for relational databases. If a datatypeable term map has a rml:datatype property, then the datatype will not be inferred.	Valid: yes, no, true, false, on, off, 1, 0 Default: no
number_of_processes	The number of processes to use. If 1, Morph-KGC will use sequential processing (minimizing memory consumption), otherwise parallel processing is used (minimizing execution time).	Default: 2 * number of CPUs in the system
logging_level	Sets the level of the log messages to show.	Valid: DEBUG, INFO, WARNING, ERROR, CRITICAL, NOTSET Default: INFO
logging_file	If not provided, log messages will be redirected to stderr. If a file path is provided, log messages will be written to the file.	Default:

Note: there are some configuration properties that are ignored when using Morph-KGC as a library, such as output_file. Parallel processing is only available on Linux when running as a library, so number_of_processes is forced to 1 on macOS and Windows; use the command line for multi-process materialization there.

Data Sources

One data source section should be included in the INI file for each data source to be materialized. The properties in the data source section vary depending on the data source type (relational database or data file). Remote mapping files are supported.

Note: Morph-KGC is case sensitive regarding identifiers. This means that table, column and reference names in the mappings must be the same as those in the data sources (no matter if the mapping uses delimited identifiers).

Relational Databases

The properties to be specified for relational databases are listed below. mappings and db_url are required.

Property	Description	Values
mappings	Specifies the mapping file(s) or URL(s) for the relational database.	[REQUIRED] Valid: - The path to a mapping file or URL. - The paths to multiple mapping files or URLs separated by commas. - The path to a directory containing all the mapping files.
db_url	It is a URL that configures the database engine (username, password, hostname, database name). See here how to create the database URLs. {ENV_VAR} placeholders are replaced with environment variables, so credentials need not be written to the file.	[REQUIRED] Example: dialect+driver://username:password@host:port/db_name
connect_args	A dictionary string of options for SQLAlchemy. See here the SQLAlchemy documentation.	Example: {"http_path": ""}

Example db_url values (see here all the information):

• MySQL: mysql+pymysql://username:password@host:port/db_name
• PostgreSQL: postgresql+psycopg://username:password@host:port/db_name
• Microsoft SQL Server: mssql+pymssql://username:password@host:port/db_name
• MariaDB: mariadb+pymysql://username:password@host:port/db_name
• Oracle: oracle+oracledb://username:password@host:port[/db_name][?service_name=[&key=value&key=value...]]
• SQLite: sqlite:///db_path/db_name.db
• Databricks: databricks://token:@?http_path=&catalog=&schema=
• Snowflake: snowflake://:@
• Neo4j: neo4j://host:port@username:password/db_name

Neo4j is a property graph database rather than a relational one. It is configured in the very same way, with a db_url, and its logical sources use Cypher queries with the rml:Cypher reference formulation.

Data Files

The properties to be specified for data files are listed below. Remote data files are supported. The mappings property is required.

Property	Description	Values
mappings	Specifies the mapping file(s) or URL(s) for the data file.	[REQUIRED] Valid: - The path to a mapping file or URL. - The paths to multiple mapping files or URLs separated by commas. - The path to a directory containing all the mapping files.
file_path	Specifies the local path or URL of the data file. It is optional since it can be provided within the mapping file with rml:source. If it is provided it will override the local path or URL provided in the mapping files.	Default:

Note: CSV, TSV, Stata and SAS support compressed files (gzip, bz2, zip, xz, tar). Files are decompressed on-the-fly and compression format is automatically inferred.

Resources

A resource is something the engine accesses while materializing, but that is not itself materialized: the SKOS vocabulary a value is reconciled against, the SPARQL endpoint it is looked up in, the lookup table a stateful function of your own reads.

Each one is declared in a [RESOURCE:<name>] section. The mapping only names the resource, while its location, credentials and access options stay in the configuration file. The same mapping therefore runs unchanged against a local copy of a vocabulary, a staging server or production.

[RESOURCE:disease_vocabulary]
resource_type: SKOS_VOCABULARY
url: https://example.org/vocabulary/disease
username: {VOCABULARY_USER}
password: {VOCABULARY_PASSWORD}
matching: CASE-INSENSITIVE

The properties common to every resource type are listed below. {ENV_VAR} placeholders in url, username and password are replaced with environment variables, so credentials need not be written to the file at all.

Property	Description	Values
resource_type	The kind of resource. The built-in reconciliation functions understand SKOS_VOCABULARY and SPARQL_ENDPOINT; any other value is accepted for resource types declared by your own stateful functions.	Example: SKOS_VOCABULARY
url	Where the resource is downloaded from, or the endpoint queried. A local path is read from disk.	Example: https://example.org/vocabulary/disease
iri	The IRI identifying the resource, when it differs from url. A mapping may name the resource by it.	Default:
username, password	HTTP Basic Authentication credentials.	Default:
matching	How a value is matched against the indexed ones. CASE-INSENSITIVE also collapses whitespace.	Valid: EXACT, CASE-INSENSITIVE Default: EXACT
attributes	Comma-separated attributes to index when the mapping names none.	Default: every property with a literal value, for a vocabulary; the SKOS labelling properties, for an endpoint
timeout	Seconds to wait for the resource.	Default: 30

Only for SKOS_VOCABULARY:

Property	Description	Values
format	RDF serialization of the vocabulary. It is guessed from the response and the URL when omitted.	Valid: turtle, xml, nt, nquads, trig, json-ld, n3

Only for SPARQL_ENDPOINT:

Property	Description	Values
query	The SELECT query the index is built from. It defaults to a query over every concept of the endpoint, on the attributes the mapping matches against or, when it names none, on the SKOS labelling properties.	Default:
method	HTTP method used to query the endpoint.	Valid: GET, POST Default: GET
concept_variable, attribute_variable, value_variable	Names of the variables projected by the query.	Default: concept, attribute, value

Advanced Setup

Relational Databases

The supported DBMSs are MySQL, PostgreSQL, Oracle, Microsoft SQL Server, MariaDB and SQLite. To use relational databases it is necessary to additionally install DBAPI drivers. You can install them via:

• MySQL and MariaDB: pip install morph-kgc[mysql].
• PostgreSQL: pip install morph-kgc[postgresql].
• Microsoft SQL Server: pip install morph-kgc[mssql].
• Oracle: pip install morph-kgc[oracle].
• SQLite: pip install morph-kgc[sqlite].
• Neo4j: pip install morph-kgc[neo4j].
• Snowflake: pip install morph-kgc[snowflake].
• Databricks: pip install morph-kgc[databricks].

Tabular Files

The supported tabular files formats are CSV, TSV, Excel, Parquet, Feather, ORC, Stata, SAS, SPSS and ODS. To work with some of them it is necessary to install additional dependencies. You can install them via:

• Excel and ODS: pip install morph-kgc[excel].
• Parquet, Feather and ORC: pip install morph-kgc[tabular].
• SPSS: pip install morph-kgc[spss].

Geospatial Files

GeoParquet and Shapefiles are read with GeoPandas, which is installed via pip install morph-kgc[geoparquet]. Both are selected with the reference formulation of the logical source, rml:GeoParquet and rml:Shapefile respectively, and their geometries are handed to the mapping as WKT strings in a geometry reference.

<#TM1>
    rml:logicalSource [
        rml:source "data.parquet";
        rml:referenceFormulation rml:GeoParquet;
    ];
    rml:subjectMap [
        rml:template "http://example.com/{id}";
    ];
    rml:predicateObjectMap [
        rml:predicate geo:asWKT;
        rml:objectMap [
            rml:reference "geometry";
            rml:datatype geo:wktLiteral;
        ];
    ].

Hierarchical Files

The supported hierarchical files formats are XML and JSON.

Morph-KGC uses XPath 3.0 to query XML files and JSONPath to query JSON files. The specific JSONPath syntax supported by Morph-KGC can be consulted here.

Output Formats

N-Triples and N-Quads need no additional dependency. Jelly, a binary RDF serialization designed for large graphs and streaming, is written with pyjelly and is installed via pip install morph-kgc[jelly].

Note: Jelly is serialized in one shot rather than streamed partition by partition, so materializing to it holds the whole knowledge graph in memory.

Docker

You can also use Morph-KGC with the provided Dockerfile.

Image Building

Build the container as follows:

docker build -t morph-kgc .

To include optional dependencies, use the optional_dependencies option as follows:

docker build -t morph-kgc --build-arg optional_dependencies="sqlite,jelly" .

Execution

The container is designed to mount a local directory containing the required files. To run the container, use the following command, replacing $(pwd)/files with the path to the local directory containing your files:

docker run -v $(pwd)/files:/app/files morph-kgc files/config.ini

This will mount the local directory to /app/files within the container and execute the application using the provided configuration file.

Bootstrapping

Morph-KGC can bootstrap a mapping from the schema of a relational database, following the Direct Mapping of the database to RDF. It inspects the tables, primary keys and foreign keys, and writes a YARRRML mapping you can then edit to target your own ontology.

python -m morph_kgc.bootstrapping path/to/config.ini

The configuration file reuses the db_url of the data source section, which must be named DataSource1, and adds a BOOTSTRAPPING section:

[CONFIGURATION]
output_dir: bootstrapped

[BOOTSTRAPPING]
base_iri: http://example.org/
build_mappings: yes

[DataSource1]
db_url: sqlite:///path/to/db_name.db

Property	Description	Values
base_iri	Base IRI of the generated terms.	Default: http://example.org/
build_mappings	Also materialize the knowledge graph with the generated mapping, instead of only writing it.	Valid: yes, no, true, false, on, off, 1, 0 Default: no

The mapping is written to direct_mapping.yaml in output_dir.