Mu Python template

Template for mu.semte.ch-microservices written in Python3. Based on the Flask-framework.

Quickstart

Create a Dockerfile which extends the semtech/mu-python-template-image and set a maintainer.

FROM semtech/mu-python-template:2.0.0-beta.2
LABEL maintainer="[email protected]"

Create a web.py entrypoint-file. (naming of the entrypoint can be configured through APP_ENTRYPOINT)

@app.route("/hello")
def hello():
    return "Hello from the mu-python-template!"

Build the Docker-image for your service

docker build -t my-python-service .

Run your service

docker run -p 8080:80

You now should be able to access your service's endpoint

curl localhost:8080/hello

Developing a microservice using the template

Dependencies

If your service needs external libraries other than the ones already provided by the template (Flask, SPARQLWrapper and rdflib), you can specify those in a requirements.txt-file. The template will take care of installing them when you build your Docker image and when you boot the template in development mode for the first time.

Development mode

By leveraging Dockers' bind-mount, you can mount your application code into an existing service image. This spares you from building a new image to test each change. Just mount your services' folder to the containers' /app. On top of that, you can configure the environment variable MODE to development. That enables live-reloading of the server, so it immediately updates when you save a file.

example docker-compose parameters:

    environment:
      MODE: "development"
    volumes:
      - /home/my/code/my-python-service:/app

Helper methods

generate_uuid

def generate_uuid()

Generates a random unique user id (UUID) based on the host ID and current time

log

def log(msg, *args, **kwargs)

Write a log message to the log file.

Works exactly the same as the logging.info (https://docs.python.org/3/library/logging.html#logging.info) method from pythons' logging module. Logs are written to the /logs directory in the docker container.

Note that the helpers module also exposes logger, which is the logger instance (https://docs.python.org/3/library/logging.html#logger-objects) used by the template. The methods provided by this instance can be used for more fine-grained logging.

error

def error(msg, status=400, **kwargs)

Returns a Response object containing a JSONAPI compliant error response with the given status code (400 by default).

Response object documentation: https://flask.palletsprojects.com/en/1.1.x/api/#response-objects The kwargs can be any other key supported by JSONAPI error objects: https://jsonapi.org/format/#error-objects

session_id_header

def session_id_header(request)

Returns the MU-SESSION-ID header from the given requests' headers

rewrite_url_header

def rewrite_url_header(request)

Returns the X-REWRITE-URL header from the given requests' headers

validate_json_api_content_type

def validate_json_api_content_type(request)

Validate whether the request contains the JSONAPI content-type header (application/vnd.api+json). Returns a 404 otherwise

validate_resource_type

def validate_resource_type(expected_type, data)

Validate whether the type specified in the JSON data is equal to the expected type. Returns a 409 otherwise.

query

def query(the_query)

Execute the given SPARQL query (select/ask/construct) on the triplestore and returns the results in the given return Format (JSON by default).

update

def update(the_query)

Execute the given update SPARQL query on the triplestore. If the given query is not an update query, nothing happens.

update_modified

def update_modified(subject, modified=datetime.datetime.now())

(DEPRECATED) Executes a SPARQL query to update the modification date of the given subject URI (string). The default date is now.

sparql_escape_string

def sparql_escape_string(obj)

Converts the given string to a SPARQL-safe RDF object string with the right RDF-datatype.

sparql_escape_datetime

def sparql_escape_datetime(obj)

Converts the given datetime to a SPARQL-safe RDF object string with the right RDF-datatype.

sparql_escape_date

def sparql_escape_date(obj)

Converts the given date to a SPARQL-safe RDF object string with the right RDF-datatype.

sparql_escape_time

def sparql_escape_time(obj)

Converts the given time to a SPARQL-safe RDF object string with the right RDF-datatype.

sparql_escape_int

def sparql_escape_int(obj)

Converts the given int to a SPARQL-safe RDF object string with the right RDF-datatype.

sparql_escape_float

def sparql_escape_float(obj)

Converts the given float to a SPARQL-safe RDF object string with the right RDF-datatype.

sparql_escape_bool

def sparql_escape_bool(obj)

Converts the given bool to a SPARQL-safe RDF object string with the right RDF-datatype.

sparql_escape_uri

def sparql_escape_uri(obj)

Converts the given URI to a SPARQL-safe RDF object string with the right RDF-datatype.

sparql_escape

def sparql_escape(obj)

Converts the given object to a SPARQL-safe RDF object string with the right RDF-datatype.

These functions should be used especially when inserting user-input to avoid SPARQL-injection. Separate functions are available for different python datatypes. The sparql_escape function however can automatically select the right method to use, for the following Python datatypes:

• str
• int
• float
• datetime.datetime
• datetime.date
• datetime.time
• boolean

The sparql_escape_uri-function can be used for escaping URI's.

Writing SPARQL Queries

The template itself is unopinionated when it comes to constructing SPARQL-queries. However, since Python's most common string formatting methods aren't a great fit for SPARQL queries, we hereby want to provide an example on how to construct a query based on template strings while keeping things readable.

from string import Template
from helpers import query
from escape_helpers import sparql_escape_uri

my_person = "http://example.com/me"
query_template = Template("""
PREFIX mu: <http://mu.semte.ch/vocabularies/core/>
PREFIX foaf: <http://xmlns.com/foaf/0.1/>

SELECT ?name
WHERE {
    $person a foaf:Person ;
        foaf:firstName ?name .
}
""")
query_string = query_template.substitute(person=sparql_escape_uri(my_person))
query_result = query(query_string)

Deployment

Example snippet for adding a service to a docker-compose stack:

my-python:
  image: my-python-service
  environment:
    LOG_LEVEL: "debug"

Environment variables

• LOG_LEVEL takes the same options as defined in the Python logging module.

• MODE to specify the deployment mode. Can be development as well as production. Defaults to production

• MU_SPARQL_ENDPOINT is used to configure the SPARQL endpoint.

  • By default this is set to http://database:8890/sparql. In that case the triple store used in the backend should be linked to the microservice container as database.

• MU_APPLICATION_GRAPH specifies the graph in the triple store the microservice will work in.

  • By default this is set to http://mu.semte.ch/application. The graph name can be used in the service via settings.graph.

• MU_SPARQL_TIMEOUT is used to configure the timeout (in seconds) for SPARQL queries.

Since this template is based on the meinheld-gunicorn-docker image, all possible environment config for that image is also available for the template. See meinheld-gunicorn-docker#environment-variables for more info. The template configures WEB_CONCURRENCY in particular to 1 by default.

Production

For hosting the app in a production setting, the template depends on meinheld-gunicorn-docker. All environment variables used by meinheld-gunicorn can be used to configure your service as well.

Other

Reassigning app

In regular Flask applications (e.g. those not run within this template) you are required to define app by using app = Flask(__name__) or similar. This does not need to be done in your web.py, as this is handled by the microservice architecture/template. Redefining this may cause The requested URL was not found on the server. If you entered the URL manually please check your spelling and try again. to be thrown on your routes, which can be luckily be fixed by simply removing the previously mentioned app = ... line.

readme.py

To simplify documenting the helper functions, README.py can be used to import & render the docstrings into README.md. Usage:

python3 -m pip install pydoc-markdown
python3 README.py

You can customise the output through the API configuration! See README.py && the pydoc-markdown docs.