Skip to content

Frequently asked questions (FAQ)

How can I contact the DBnomics core team?

I like DBnomics, how can I contribute?

Well, glad you like it :)

See the contributing page.

How do you guarantee that data on DBnomics is correct?

In short, there is no formal guarantee, but...

Fetcher developers use a validation script in order to detect some errors automatically.

Of course, a program cannot verify the correctness of floating-point values. To do that, manual checks must be performed against the original source data on the provider website. The DBnomics core team includes economists who manually verify data produced by a fetcher once developers have finished writing it.

Finally, like many free software projects, DBnomics quality improves each time users contribute, and reporting problems with data is very helpful.

I've seen a problem with data on DBnomics, what should I do?

Please read: report problems with data.

When is data updated?

Updating data as soon as the provider publishes it is a major concern. Here are more explanations about how it works.

DBnomics data, distributed by its web API, is updated by fetchers. Under normal circumstances, newly published data should become available in DBnomics within 24 hours, because each fetcher is run by the DBnomics infrastructure once a day by default.

For sponsors, we can also grant permission to trigger a data acquisition pipeline themselves, which can reduce this delay considerably when more timely updates are needed.

There can be two types of delays:

  1. the delay between the publication of new data on the provider side and the next execution of the corresponding fetcher,
  2. and the delay due to a failure of the fetcher, requiring manual intervention.

The delay can be reduced by running fetchers in incremental mode, which is supported by some fetchers. A fetcher running in this mode can be executed as often as needed: if no new data is available compared with the previous run, it simply does nothing. This avoids downloading and processing the same data repeatedly, resulting in significant savings in bandwidth, processing time and computing resources, so fetchers supporting this mode can be run much more frequently than once a day.

For incremental updates to work reliably, however, the provider's infrastructure needs to expose a reliable update schedule, so that we can determine when new data is actually expected to be available. We are working on porting more fetchers to incremental mode.

DBnomics is maintained by a relatively small team, and we do our best to keep the large number of data providers up to date. As part of our support commitments, requests to repair or restore data providers from sponsors are given priority.

Where do I find the source code of a fetcher?

Look at that GitLab group: https://git.nomics.world/dbnomics-fetchers/.

Can I have private data on DBnomics?

DBnomics is designed to aggregate and redistribute publicly available economic data from external sources — it is not a storage service for private datasets. Data that a researcher keeps private, i.e. not published anywhere publicly, cannot be integrated into DBnomics.

However, if a researcher publishes their data in a publicly accessible location (e.g. a Git repository, an institutional website), DBnomics can treat them as a data provider and fetch the data from there. Their institution can serve as the provider code. In other words, DBnomics is not meant to be the primary storage for the data: it retrieves data from wherever it is already published.

In addition, DBnomics will not redistribute paid data or data that is not licensed for redistribution: doing so would violate the provider's terms of use. This is a legal constraint, not a technical one.

Please contact us if you have questions about a specific data source — we'd be happy to discuss your needs.

Why do fetchers break?

If you look at the dashboard, you will see that many fetchers are in error status with red icons.

Fetchers download and convert data from providers, but the provider website, web API, or data can evolve in an unpredictable way.

Fetcher stability is correlated with provider stability. This includes stable infrastructure, stable API URLs, stable responses, stable data models, etc. When a provider changes something, the fetcher breaks and needs to be fixed by its developers.

Which data sources are easier to process?

Data distributed in manually formatted Excel files is more difficult to process by fetchers, mainly because data is not produced systematically.

For example, having cells with a color background to add semantics to data is quite annoying to process, as well as handling merged cells, or data tables shifted below a header, or having a footer with notes.

On the other hand, structured data files with a stable schema, such as XML, JSON, or CSV, are easier to work with because there are almost no special cases to handle.

Another aspect is the transport layer: it is easier to download static files and process them locally than to follow the pagination of a web API or scrape a website.