1   Abstract¶

LSST DocHub is a proposed solution to information discovery for LSST Data Management, and the LSST project in general. LSST documentation and information artifacts are published through a variety of platforms by virtue of the way information is created — from documents archived on DocuShare, to source code on GitHub, to conversations on community.lsst.org. Currently, staff and users must go to each platform to find information. This has an overall effect of slowing, and even preventing, knowledge sharing. LSST DocHub can solve this problem by decoupling information publication from information discovery. DocHub consists of a unified web front-end for documentation browsing, filtering, and search. The front-end is fed by a web API to centralized metadata and full-text databases. These databases are populated by adapters that monitor each of LSST’s information platforms for new and updated artifacts. DocHub stores metadata as JSON-LD, which is a community-standard, extensible, and self-describing schema. This technote establishes the basic design concept for DocHub, including its architecture and JSON-LD metadata patterns.

2   DocHub’s purpose¶

LSST continuously produces a vast number of information artifacts across numerous platforms. Software and associated documentation repositories are published through GitHub. Documents and presentations may also be archived in DocuShare, but also on Zenodo to enable scientific citation. Conversations about tasks may happen in JIRA, while more general conversations happen on the Community.lsst.org forum. Papers may be written on GitHub, but are ultimately made available through ADS and a publisher’s website. Ultimately, finding information is more difficult than it should be.

One response is to pare down the number of places where LSST’s information artifacts can exist. While laudable — and using platforms with identical use cases and feature sets simultaneously is certainly confusing — retreating to a single platform is not a solution. Each platform facilitates different types of work. Arbitrarily forcing work onto platforms that aren’t a good fit for that work reduces productivity.

Instead, our approach is to decouple information discovery from the platforms that information exists on. We will index LSST information artifacts, and make their metadata centrally available through a website and search API. Then, finding LSST information involves searching through a single website rather than iteratively trawling individual platforms. This system for indexing LSST information artifacts, storing metadata in a database, and making metadata available through an API and website is called DocHub. This technote describes LSST DocHub’s design.

3   Existing metadata systems¶

Information discovery is a common issue across large, distributed organizations like LSST. These are some implementations of DocHub-like systems by other open software and data organizations:

• 18F embeds .about.yml metadata files in their repositories. These metadata are indexed and published on 18F’s Dashboard.
• Code for America similarly implements a civic.json metadata format. These are used by Code for America’s project search page. Primarily it is used to denote the status of a project and to provide tags for search. Much of the information for the search page is also automatically obtained from GitHub metadata, like the project description.
• Code.gov embeds a code.json file in federal repositories. This metadata is tracked and published by the Code.gov website and API. Additional discussion about code.gov’s metadata schema is taking place on a GitHub issue. Earlier discussion also happened on a White House source-code-policy issue.
• CodeMeta is a minimal metadata schema, written in JSON-LD, that can describe scientific software repositories. CodeMeta’s schema is designed to be cross-walked to other metadata schemas including Dublin Core, schema.org, Python’s PyPI, and DataCite.
• Asset description metadata for software from the EU. See Issue #41 at codemeta as well.

These metadata systems share a common pattern: metadata is embedded with the information artifact, centrally indexed, and made available through a search API and website. This approach scales well since it federates metadata definition and maintenance to the source repositories themselves. DocHub builds upon this design pattern.

4   DocHub architecture¶

DocHub uses a microservice architecture (Figure 1) to gain flexibility. The components are:

• A metadata schema. DocHub uses JSON-LD since it is extensible, yet self describing. Like code.gov and similar implementations, this metadata embedded in source repositories whenever possible. The same metadata format is used in the database.
• A metadata database. DocHub uses a MongoDB database to store all metadata. MongoDB is a document database that works natively with JSON. The JSON-LD that’s embedded in source repositories is available through MongoDB.
• A full-text database. While MongoDB is well-suited to querying semi-structured data like JSON-LD, its full-text search capabilities are more limited. Where possible, the content of documents will be stored and made available through Elasticsearch.
• Ingest adapters. Each adapter is a microservice built to transform content and metadata for a particular type of resource into a JSON-LD record and full-text entry stored in the MongoDB and Elasticsearch databases. This adapter architecture helps DocHub scale: indexing a new arbitrary information source involves deploying a new adapter service. Adapters can either by pushed to (say, by a GitHub webhook), or can poll a platform for new and updated records. Each adapter handles the platform specific challenge of transforming either templated JSON-LD stored in a source repository or a platform’s native metadata into standardized DocHub JSON-LD.
• An API server. The web API server allows applications to query against DocHub’s metadata and full-text databases.
• A web front end. This front end is how people typically use DocHub. This website will allow users to browse and filter DocHub information artifacts, and also provide a generic search against the full-text and metadata databases. The website will be editorially designed to some extent. For example, the front page will show featured projects, papers and documents in addition to giving entry points to search and browse against usefully-selected categories. Generally the website (and API) will allow anonymous access. DocHub can be designed to facilitate authorization-based access to non-public documentation (private GitHub repositories) for example, though this will depend on a centralized user database that doesn’t exist in the needed form yet.

Figure 1 DocHub’s architecture. Adapter microservices pull metadata and content from information artifacts, which could be GitHub repositories, JIRA issues, forum topics, and ADS entries. The adapters build JSON-LD metadata documents and persist them into a MongoDB metadata database. An Elasticsearch cluster stores full text content from the adapters, where possible. The API server provides GraphQL and RESTful interfaces to the MongoDB and Elasticsearch databases.

5   DocHub’s JSON-LD metadata¶

All DocHub metadata records share a common JSON-LD (linked data) schema. Through a @context, JSON-LD documents map the names of fields to semantic definitions in http://schema.org and other vocabularies. Specifically, DocHub adopts and extends codemeta, which is a minimal schema of concepts needed to describe a scientific software repository. CodeMeta JSON-LD objects can be cross-walked to other repository metadata schemas to enable automatic submission pipelines from GitHub to a repository like Zenodo, for example.

DocHub metadata exists in two contexts: the metadata database, and in artifact repositories (such as GitHub repositories). Metadata at rest in DocHub’s database is intended to be complete and authoritative, while metadata embedded in repositories is templated. Metadata templates are transformed by ingest adapters into complete JSON-LD stored by DocHub. This section describes these DocHub metadata as it is authoritatively stored in the metadata database.

See also: 9   Appendix: JSON-LD reading list.

{
  "@context": "...",
  "version": "master"
  "relationships": [
    {
      "relationshipType": "wasRevisionOf",
      "namespace": "http://www.w3.org/ns/prov#",
      "relatedIdentifier": "https://github.com/lsst-sqre/sqr-013.git",
      "relatedIdentifierType": "URL"
    }
  ]
}

find({
  relationships: {$elemMatch: {relationshipType: "wasRevisionOf",
                               relatedIdentifier: "https://github.com/lsst-sqre/sqr-013.git"}}
})

find({
  version: "master",
  relationships: {$elemMatch: {relationshipType: "wasRevisionOf",
                               relatedIdentifier: "https://github.com/lsst-sqre/sqr-013.git"}}
})

{
  "@context": "...",
  "version": "v1",
  "relationships": [
    {
      "relationshipType": "compiles",
      "relatedIdentifier": "doi:10.5281/zenodo.153867",
      "relatedIdentifierType": "DOI"
    }
  ]
}

{
  "@context": "...",
  "version": "master"
  "relationships": [
    {
      "relationshipType": "isPartOf",
      "relatedIdentifier": "https://github.com/lsst/pipelines_docs.git",
      "relatedIdentifierType": "URL"
    }
  ]
}

{
   "@context": "...",
   "agents": [
     {
       "@id": "https://orcid.org/0000-0003-3001-676X",
       "@type": "person",
       "email": "jsick@lsst.org",
       "name": "Jonathan Sick",
       "affiliation": "AURA/LSST",
       "mustbeCited": true,
       "isMaintainer": true,
       "isRightsHolder": false,
     }
   ]
}

{
   "@context": "...",
   "agents": [
     {
       "@type": "organization",
       "name": "Association of Universities for Research in Astronomy",
       "isRightsHolder": true,
       "isMaintainer": false,
       "role": {
         "namespace": "http://www.ngdc.noaa.gov/metadata/published/xsd/schema/resources/Codelist/gmxCodelists.xml#CI_RoleCode",
         "roleCode": "rightsHolder"
       }
      },
   ]
}

{
   "@context": "...",
   "agents": [
     {
       "@type": "organization",
       "name": "Association of Universities for Research in Astronomy",
       "isRightsHolder": true,
       "isMaintainer": false,
       "role": {
         "namespace": "http://www.ngdc.noaa.gov/metadata/published/xsd/schema/resources/Codelist/gmxCodelists.xml#CI_RoleCode",
         "roleCode": "rightsHolder"
       }
      },
      {
        "@type": "organization",
        "name": "Large Synoptic Survey Telescope",
        "parentOrganization": "Association of Universities for Research in Astronomy",
        "isRightHolder": false,
        "isMaintainer": false
      },
      {
        "@type": "organization",
        "name": "Data Management",
        "parentOrganization": "Large Synoptic Survey Telescope",
        "isRightHolder": false,
        "isMaintainer": false
      },
      {
        "@type": "organization",
        "name": "Science Quality and Reliability Engineering Team",
        "parentOrganization": "Data Management",
        "isRightHolder": false,
        "isMaintainer": true
      }

   ]
}

{
  "@context":"https://raw.githubusercontent.com/codemeta/codemeta/master/codemeta.jsonld",
  "@type": "SoftwareSourceCode",
}

{
  "@context": "http://schema.org",
  "@graph": [
    {
        "@id": "#issue",
        "@type": "PublicationIssue",
        "issueNumber": "5",
        "datePublished": "2012",
        "isPartOf": {
            "@id": "#periodical",
            "@type": [
                "PublicationVolume",
                "Periodical"
            ],
            "name": "Cataloging & Classification Quarterly",
            "issn": [
                "1544-4554",
                "0163-9374"
            ],
            "volumeNumber": "50",
            "publisher": "Taylor & Francis Group"
        }
    },
    {
        "@type": "ScholarlyArticle",
        "isPartOf": "#issue",
        "description": "The library catalog as a catalog of works was an infectious idea, which together with research led to reconceptualization in the form of the FRBR conceptual model. Two categories of lacunae emerge--the expression entity, and gaps in the model such as aggregates and dynamic documents. Evidence needed to extend the FRBR model is available in contemporary research on instantiation. The challenge for the bibliographic community is to begin to think of FRBR as a form of knowledge organization system, adding a final dimension to classification. The articles in the present special issue offer a compendium of the promise of the FRBR model.",
        "sameAs": "http://dx.doi.org/10.1080/01639374.2012.682254",
        "about": [
            "Works",
            "Catalog"
        ],
        "pageEnd": "368",
        "pageStart": "360",
        "name": "Be Careful What You Wish For: FRBR, Some Lacunae, A Review",
        "author": "Smiraglia, Richard P."
    }
  ]
}

{
  "@context": "http://schema.org",
  "@graph": [
    {
      "@id": "#issue4",
      "@type": "PublicationIssue",
      "datePublished": "2006-10",
      "issueNumber": "4"
    },
    {
      "@id": "#volume50",
      "@type": "PublicationVolume",
      "volumeNumber": "50"
    },
    {
      "@id": "#periodical",
      "@type": "Periodical",
      "name": "Library Resources and Technical Services"
    },
    {
      "@id": "#article",
      "@type": "ScholarlyArticle",
      "author": "Carlyle, Allyson.",
      "isPartOf": [
        {
          "@id": "#periodical"
        },
        {
          "@id": "#volume50"
        },
        {
          "@id": "#issue4"
        }
      ],
      "name": "Understanding FRBR as a Conceptual Model: FRBR and the Bibliographic Universe",
      "pageEnd": "273",
      "pageStart": "264"
    }
  ]
}

{
  "@context": "...",
  "@type": "ScholarlyArticle",
  "relationships": [
    {
      "relationshipType": "isSupplementTo",
      "relatedIdentifier": "https://github.com/lsst/example_analysis_software.git",
      "relatedIdentifierType": "URL"
    },
    {
      "relationshipType" "isPartOf",
      "@id": "#issue",
      "@type": [
          "PublicationVolume",
          "Periodical",
          "PublicationIssue"
      ],
      "name": "Cataloging & Classification Quarterly",
      "volumeNumber": "50",
      "issueNumber": "5",
      "publisher": "Taylor & Francis Group"
      "pageEnd": "368",
      "partStart": "360",
    },
    {
      "relationshipType": "isIdenticalTo",
      "relatedIdentifier": "doi:...",
      "relatedIdentifierType": "DOI"
    },
  ],
  "name": "Article's Name",
  "description": "Article's abstract ..."
}

6   JSON-LD metadata templates¶

Although complete JSON-LD metadata documents can be embedded in GitHub (and similar) repositories, managing metadata this way may not be sustainable. First, some metadata changes with each commit, and the time of commit (such dateModified). Second, a lot of metadata is inherent to a repository and its content. Git commit trees contain information to build contributor metadata, the LICENSE file authoritatively defines the repository’s license, and the document’s text authoritatively describes its content. Repeating information inherent to the GitHub repository in a metadata file introduces fragility.

DocHub’s approach is to shift the responsibility of building a complete metadata record to the ingest adapter. To help the ingest adapter, and to store metadata that can be statically managed, we store metadata templates in the Git repository.

{
  "@context": "...",
  "licenseId": "MIT"
}

{
  "@context": "...",
  "licenseId": {"@template": "GitHubLicenseId"}
}

{
  "@context": "...",
  "agents": {"@template": "GitContributors",
             "additionalAgents": [
               {
                 "@type": "organization",
                 "name": "Science Quality and Reliability Engineering Team",
                 "parentOrganization": "Data Management",
                 "isRightHolder": false,
                 "isMaintainer": true
               }
             ]
}

7   Ingest Adapters¶

Ingest adapters are microservices that take an artifact in its native form, and index it in the DocHub databases. That is, it transforms the artifact’s native metadata into DocHub JSON-LD metadata. Each type of artifact has a dedicated ingest adapter microservice. This way all platform-specific logic is contained within individual ingest adapter code bases. The DocHub API server does not largely need to know about platforms; it only needs to interpret metadata in DocHub’s schema.

Ingest adapters can either be designed for pulling artifact updates, or being pushed update’s from the artifact’s platform. For example, GitHub repositories can emit webhook events that trigger ingest adapters. Alternatively, ingest adapters can poll for updates from platforms that do not support webhooks.

8   DocHub API server¶

GET https://dochub.lsst.codes/metadata/identifier.json

9   Appendix: JSON-LD reading list¶

• JSON-LD best practices.
• Building a better book in the browser.
• Linked Data Patterns
• Indexing bibliographic linked data with JSON-LD, ElasticSearch.
• JSON-LD: Building meaningful data APIs.
• BibJSON describes resources with JSON objects with fields defined in BibTeX. Being JSON, it’s also possible to describe these files with JSON-LD.