Fetchers

What is a fetcher?

In DBnomics, data acquisition is done by fetchers, small programs that download data from provider infrastructures and convert it to a common data model and format.

There is one fetcher per data provider. For example, there is one fetcher for Eurostat, another one for IMF, etc.

Here is a diagram that shows the main steps of a fetcher:

flowchart LR
    provider@{ shape: cloud, label: "Data Provider<br/>(e.g. OECD)" }

    subgraph fetcher["Fetcher"]
        direction LR
        download["Download"]
        source_data[("Source<br/>data")]
        convert["Convert"]
        converted_data[("Converted<br/>data")]
    end

    style fetcher fill:#ffffff,stroke:#000000,stroke-dasharray: 3 3

    download -->|download| provider
    download -->|write| source_data
    convert -->|read| source_data
    convert -->|write| converted_data

The source code of the fetchers is hosted in a GitLab group: https://git.nomics.world/dbnomics-fetchers. Each repository is dedicated to a fetcher, for example https://git.nomics.world/dbnomics-fetchers/insee-fetcher.

To write a new fetcher or maintain an existing one, see the documentation of dbnomics-toolbox and the contributing page.

Incremental mode

The incremental mode is not supported by all fetchers. In incremental mode, fetchers only download and convert data that has changed since the last execution, making the process more efficient.

A fetcher running in this mode can be executed as often as needed: if no new data is available compared with the previous run, it simply does nothing. This avoids downloading and processing the same data repeatedly, resulting in significant savings in bandwidth, processing time and computing resources. Fetchers supporting this mode can therefore be run much more frequently than once a day.

For incremental updates to work reliably, the provider's infrastructure needs to expose a reliable update schedule, so that we can determine when new data is actually expected to be available.

Note

Please read the incremental mode documentation of dbnomics-toolbox for more details about how to implement this mode in a fetcher.

Fetcher pipeline

The fetcher pipeline is the part of the DBnomics infrastructure that runs fetchers and makes their output available on the DBnomics website and Web API.

The source code of the fetcher pipeline is available in the dbnomics-fetcher-pipeline repository.

Fetcher pipelines are scheduled regularly (daily by default) in order to keep DBnomics data up to date.

flowchart LR
    scheduler["GitLab scheduler"]

    subgraph fetcher["Fetcher domain"]
        direction LR
        download["Download job"]
        source_data[("Source data")]
        convert["Convert job"]
        converted_data[("Converted data")]
    end

    index["Index job"]
    validate["Data validation job"]
    deploy["Deploy job"]

    style fetcher fill:#ffffff,stroke:#000000,stroke-dasharray: 6 4

    scheduler --> download
    download --> convert
    convert --> index
    convert --> validate
    download -.->|push| source_data
    convert -.->|push| converted_data
    index --> deploy

Dashboard

The dashboard shows the status of the latest pipeline executions for each fetcher.

Its source code is available in the dbnomics-dashboard repository.