Phlex::RDF proposal (schema.org, ActivityStreams/ActivityPub, FOAF, etc.)

[ziggythehamster]

This is going to be a long post, mostly containing commented Ruby code.

@joeldrapper has been spitballing ways to support things besides HTML, and I've been thinking about specifically RDF. RDF has been around forever and can be serialized in a ton of different ways. And, in fact, it's possible to output RDF in many semantically identical but extremely different ways. In order to encourage people to use RDF-aware tooling, I think we should talk about how we might model RDF in Phlex.

Some things to think about:

• How do we indicate what serialization format we're going to use? The same Phlex view might be used for RDF/XML, JSON-LD, and Turtle. But we also want to avoid hiding this magic in Hanami or Rails. Possibly we could make the first argument to the constructor take the format?
• What's the best way to deal with nesting templates? I think it's reasonable someone might want to add a bunch of Dublin Core in addition to schema.org stuff, and do that in two different templates, but bring them together in a third one. You could repeat the subject block to do that so it's a matter of just having the two parallel templates. Maybe render does this already out of the box?
• What's the best way to model RDF literals? I just created one in my examples but if we have our proxy classes, perhaps schema.Date.new('value') would be good UX? (But how do we prevent you from doing schema.Thing.new, which isn't a literal?)
• The Ruby JSON-LD tooling is great, and is AFAIK what they built the spec from, but there are some fixups which are sometimes necessary when you interoperate with non-RDF-aware tooling. How do we empower users to make these fixups? The JSON-LD framing spec fixes the majority of problems, and we can basically automatically do it in Phlex. Two unresolved issues I am aware of in Mastodon/ActivityStreams/ActivityPub:
  • to should always be an array. This seems to be a bug in either the spec's definition file or in json-ld-preloaded's parser of the spec's definition file. This can be fixed by making the TermDefinition for to set container_mapping: "@set"
  • as:Public should be expanded to https://www.w3.org/ns/activitystreams#Public. This does not appear to be fixable inside ruby-rdf, so we could make the helper that builds RDF::Resources have an option like force_expand: true which causes us to prefix the resource URI with __phlex_force_expand__, and we simply remove that after we have a hash of JSON-LD but before we serialize the JSON.
• ruby-rdf and friends desperately want to fetch things on the network and don't have the capability of turning it off via configuration. with preloaded vocabs and JSON-LD contexts, network access should not be required and so Phlex::RDF should by default patch ruby-rdf/json-ld so network access raises. maybe add a Phlex::RDF.allow_rdf_network_access! that can be called to restore the default.

# An example using Friend of a Friend
class FOAFExample < Phlex::RDF
  # Ruby has very good RDF support, so Phlex itself just needs to be told which vocabularies
  # you are going to use in this "template". If you need a custom vocabulary, you should
  # register it like rdf-vocab does. The stock RDF vocabularies - RDF::OWL, RDF::RDFS, RDF::RDFV,
  # RDF::XSD - should be pre-registered in Phlex, with RDFV being available as either RDFV or RDF here.
  #
  # Basically this will +RDF::Vocabulary.from_sym(x)+, probably lazily so it doesn't matter what
  # order you register new vocabs and whatnot so long as it's before rendering. This would be pretty
  # easy to set up because this could set up a delegate and later the delegate actually does its
  # magic. See further below.
  register_vocab :FOAF

  # JSON-LD contexts are needed whenever you want to encode a RDF graph as JSON-LD. Context URIs
  # are rarely the same as the vocabulary URIs, so this macro exists to bridge that. Also, this
  # can take a Hash if you want. When there's one context, we use it, otherwise it's an array of
  # the strings and hashes.
  add_context 'http://xmlns.com/foaf/context'

  def initialize(person:)
    @person = person
  end

  # The RDF graph. This could maybe be +graph+ instead. An RDF graph is a collection of statements,
  # which need not refer to the same subjects, but should be related to one another. RDF statements
  # consist of a subject, predicate, and object. If you +yield+ inside a subject block, then
  # the inner template inherits that subject as you would expect.
  #
  # Since RDF is easy to screw up, we should disallow RDF predicates unless they are in a subject block.
  def template
    # RDF subjects that aren't anonymous BNodes are _always_ URLs (technically IRIs), so the subject method should support
    # several ways of trying to find a URL. I would propose:
    #
    # - if arg is a RDF::Resource or RDF::URI already, do nothing and use arg
    # - if arg is something Railsy (ActiveRecord/ActiveModel instance), use helpers.url_for
    # - if arg is a ::URI, make it a RDF::Resource
    # - if arg is a String, try to make it a RDF::Resource if it parses as a ::URI, otherwise raise
    # - if arg is a regular Ruby object, try +#to_rdf_resource+ followed by +#to_uri+, otherwise raise
    # - if arg is nil or missing, use RDF::Node.new
    subject(@person) do
      # Note that +foaf+ here is a delegate to the registered vocabulary RDF::Vocab::FOAF, but is some kind of
      # wrapper class that delegates because vocabulary terms aren't callable and ruby-rdf would normally have you
      # do something like graph << [subject, predicate, object] and that's less pretty.

      a foaf.Person # +a(foo)+ is a helper that does the same as +rdf.type(foo)+ - this is the same as the Turtle RDF syntax

      foaf.name     @person.name
      foaf.mbox     resource("mailto:#{@person.email}") # resource could be an alias for subject except disallowing a block
      foaf.homepage resource(@person.homepage)
      foaf.nick     @person.nickname
      foaf.interest resource('https://phlex.fun/')

      # an example of using an anonymous subject, so we can refer to abstract things which don't have IRIs (or we don't want
      # to include them). For JSON-LD, this will also cause this predicate to be framed (e.g., rather than having a +@graph+
      # and reference BNodes, the anonymous object is nested).
      foaf.interest do # this could perhaps be +foaf.interest subject do+
        a           foaf.Document
        foaf.topic 'Ruby'
      end

      # users can also deliberately frame something that isn't an anonymous object
      foaf.knows do
        resource('https://github.com/joeldrapper')
      end
    end

    # This just demonstrates that you can add statements to subjects without having to do it where you use them initially
    subject('https://phlex.fun') do
      rdfs.label 'Phlex'
    end
  end
end

# Another example, this time using the schema.org PodcastEpisode type representing a real episode
class AStaticPodcastEpisode < Phlex::RDF
  register_vocab :SCHEMA
  add_context 'http://schema.org/'

  def template
    subject do # schema.org objects typically do not have IRIs because they link together a bunch of different URLs
      a schema.PodcastEpisode

      schema.name          'Marathon'
      schema.datePublished RDF::Literal.new('2022-06-09', datatype: schema.Date) # schema.org uses a different date type than RDF usually uses
      schema.description   "This week the gang talks about the ins and outs of Marathon, one of Bungie's " \
                           "first breakout hits, and the granddaddy of Halo and Destiny! Also, we're gearing " \
                           "up for our Grand Reopening, Friday June 10th!"
      schema.timeRequired  "PT35M30S"
      schema.url           resource('https://art19.com/shows/the-madecast/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb')

      schema.about do
        a schema.VideoGame

        schema.name         'Marathon'
        schema.gamePlatform 'Macintosh'
        schema.publisher    'Bungie'
      end

      schema.associatedMedia do
        a schema.MediaObject

        schema.contentUrl     resource('https://rss.art19.com/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb.mp3')
        schema.embedUrl       resource('https://art19.com/shows/the-madecast/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb/embed')
        schema.encodingFormat 'audio/mpeg'
      end

      schema.partOfSeries do
        a schema.PodcastSeries

        schema.name    'The MADEcast'
        schema.url     resource('https://art19.com/shows/the-madecast')
        schema.webFeed resource('https://rss.art19.com/the-madecast')
      end
    end
  end
end

require 'json/ld'
require 'json/ld/preloaded'
require 'rdf'
require 'rdf/turtle'
require 'rdf/vocab'

SCHEMA = RDF::Vocab::SCHEMA

# our subjects
subject = RDF::Node.new
game    = RDF::Node.new
media   = RDF::Node.new
series  = RDF::Node.new

graph = RDF::Graph.new

# build our graph
graph << [subject, RDF.type, SCHEMA.PodcastEpisode]

graph << [subject, SCHEMA.name,            'Marathon']
graph << [subject, SCHEMA.datePublished,   RDF::Literal.new('2022-06-09', datatype: SCHEMA.Date)]
graph << [subject, SCHEMA.description,     "This week the gang talks about the ins and outs of Marathon, one of Bungie's " \
                                           "first breakout hits, and the granddaddy of Halo and Destiny! Also, we're gearing " \
                                           "up for our Grand Reopening, Friday June 10th!"]
graph << [subject, SCHEMA.timeRequired,    'PT35M30S']
graph << [subject, SCHEMA.url,             RDF::Resource.new('https://art19.com/shows/the-madecast/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb')]
graph << [subject, SCHEMA.about,           game]
graph << [subject, SCHEMA.associatedMedia, media]
graph << [subject, SCHEMA.partOfSeries,    series]

graph << [game, RDF.type,            SCHEMA.VideoGame]
graph << [game, SCHEMA.name,         'Marathon']
graph << [game, SCHEMA.gamePlatform, 'Macintosh']
graph << [game, SCHEMA.publisher,    'Bungie']

graph << [media, RDF.type,              SCHEMA.MediaObject]
graph << [media, SCHEMA.contentUrl,     RDF::Resource.new('https://rss.art19.com/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb.mp3')]
graph << [media, SCHEMA.embedUrl,       RDF::Resource.new('https://art19.com/shows/the-madecast/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb/embed')]
graph << [media, SCHEMA.encodingFormat, 'audio/mpeg']

graph << [series, RDF.type,       SCHEMA.PodcastSeries]
graph << [series, SCHEMA.name,    'The MADEcast']
graph << [series, SCHEMA.url,     RDF::Resource.new('https://art19.com/shows/the-madecast')]
graph << [series, SCHEMA.webFeed, RDF::Resource.new('https://rss.art19.com/the-madecast')]

# Our JSON-LD frame, which Phlex::RDF would need to keep track of if serializing JSON-LD
frame = {
  "@context" => "http://schema.org/",

  # +{}+ is a wildcard for the framing algorithm, it means to nest anything under schema:about here.
  # I guess technically you could nest things under about but only if they're a certain type and then
  # in other cases you create graph nodes, but I can't imagine anyone would want to do this if building
  # a schema from scratch (rather than using JSON-LD to annotate a regular JSON response). We can just
  # use +{}+ for the time being, and perhaps make it configurable if wanted. 
  #
  # oh, and I could just put +about+ here, without the prefix, but Phlex would want to build the frame with
  # the prefix to avoid ambiguity, so I'm doing so here.
  "schema:about" => {},
  "schema:associatedMedia" => {},
  "schema:partOfSeries" => {}
}

expanded = JSON::LD::API.fromRDF(graph, compactArrays: true)
framed   = JSON::LD::API.frame(expanded, frame)

# Turtle
puts graph.dump(:ttl, prefixes: { SCHEMA.__prefix__ => SCHEMA.to_s })

# JSON-LD
puts JSON.pretty_generate(framed)

For reference, the above prints this output:

@prefix schema: <http://schema.org/> .

 [
    a schema:PodcastEpisode;
    schema:about [
      a schema:VideoGame;
      schema:gamePlatform "Macintosh";
      schema:name "Marathon";
      schema:publisher "Bungie"
    ];
    schema:associatedMedia [
      a schema:MediaObject;
      schema:contentUrl <https://rss.art19.com/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb.mp3>;
      schema:embedUrl <https://art19.com/shows/the-madecast/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb/embed>;
      schema:encodingFormat "audio/mpeg"
    ];
    schema:datePublished "2022-06-09"^^schema:Date;
    schema:description "This week the gang talks about the ins and outs of Marathon, one of Bungie's first breakout hits, and the granddaddy of Halo and Destiny! Also, we're gearing up for our Grand Reopening, Friday June 10th!";
    schema:name "Marathon";
    schema:partOfSeries [
      a schema:PodcastSeries;
      schema:name "The MADEcast";
      schema:url <https://art19.com/shows/the-madecast>;
      schema:webFeed <https://rss.art19.com/the-madecast>
    ];
    schema:timeRequired "PT35M30S";
    schema:url <https://art19.com/shows/the-madecast/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb>
  ] .

{
  "@context": "http://schema.org/",
  "type": "PodcastEpisode",
  "name": "Marathon",
  "datePublished": "2022-06-09",
  "description": "This week the gang talks about the ins and outs of Marathon, one of Bungie's first breakout hits, and the granddaddy of Halo and Destiny! Also, we're gearing up for our Grand Reopening, Friday June 10th!",
  "timeRequired": "PT35M30S",
  "url": "https://art19.com/shows/the-madecast/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb",
  "about": {
    "type": "VideoGame",
    "name": "Marathon",
    "gamePlatform": "Macintosh",
    "publisher": "Bungie"
  },
  "associatedMedia": {
    "type": "MediaObject",
    "contentUrl": "https://rss.art19.com/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb.mp3",
    "embedUrl": "https://art19.com/shows/the-madecast/episodes/2f98892e-1855-4d34-ba83-a4764b7cd7cb/embed",
    "encodingFormat": "audio/mpeg"
  },
  "partOfSeries": {
    "type": "PodcastSeries",
    "name": "The MADEcast",
    "url": "https://art19.com/shows/the-madecast",
    "webFeed": "https://rss.art19.com/the-madecast"
  }
}

The compile phase would probably just have a graph it passes through successive templates and just have a bunch of RDF::Statement.new's and cache the RDF::Resources in a Hash (or use RDF::URI.intern).

Also I wanted to do an ActivityStreams example since that's what I'm most interested in, but it's a good test-bed for nesting templates, so I want to gut check my thoughts before I went down that rabbit hole :).

[joeldrapper]

Thanks for writing this up. There's lots to unpack here, we should pair sometime.