Fetchers

In DBnomics, data acquisition is done by fetchers, small programs that download data from data provider infrastructures, and convert it to a common data model and format.

There is one fetcher per data provider.

Diagram of the components of a fetcher

The source code of the fetchers is hosted here.

To write a new fetcher or maintain an existing one, see contributing.

Pipelines

Fetchers are scheduled on a regular basis in order to keep DBnomics data up to date.

They are executed by a pipeline running jobs:

Diagram of the pipeline of a fetcher

The dashboard shows the status of the latest pipeline executions for each fetcher.