Skip to content

How to access processed data

In this guide, you will use the ArchiveQuery Python client to search for NOMAD Entries, download selected parts of their processed archive data, and work with the results as schema-aware Python objects. You will also learn when to retrieve plain dictionaries, process results in batches, or use the asynchronous interface.

Use ArchiveQuery when you want to retrieve large collections of processed archives efficiently or work with fully deserialized, schema-aware Python objects. For direct control over REST requests and responses with minimal Python dependencies, use How-to guides > ... > Use the API.

Before you begin

Install the nomad-lab Python package as described in How-to guides > ... > Install > How to install the NOMAD Python Library.

The examples query public data and do not require authentication. By default, the client connects to NOMAD Central. To use another NOMAD deployment, set its API base URL before starting Python. For example, to test against the NOMAD develop deployment:

export NOMAD_CLIENT_URL="https://nomad-lab.eu/develop/backend/api"

Search and select archive data

An ArchiveQuery combines two selections:

  • query selects the Entries to retrieve using searchable metadata.
  • required selects the archive sections and quantities to download for each matching Entry.

The following query selects public simulation Entries produced with VASP for materials containing both titanium and oxygen. It downloads identifying metadata and the normalized material data, while limiting the result to five Entries:

from nomad.client import ArchiveQuery

archive_query = ArchiveQuery(
    owner='public',
    query={
        'results.method.simulation.program_name': 'VASP',
        'results.material.elements': {'all': ['Ti', 'O']},
    },
    required={
        'metadata': {
            'entry_id': '*',
            'mainfile': '*',
        },
        'results': {
            'material': {
                'chemical_formula_reduced': '*',
            },
        },
    },
    results_max=5,
    page_size=5,
)

Use the query syntax described in Reference > REST API > Entry queries. Request only the archive data that your analysis needs: a narrowly scoped required selection reduces both server work and transferred data.

Retrieve and inspect the results

Call download() to search for and retrieve the matching archives. If you have not called fetch() first, download() performs the search automatically:

archives = archive_query.download()

for archive in archives:
    print(
        archive.metadata.entry_id,
        archive.results.material.chemical_formula_reduced,
    )

By default, each result is an EntryArchive object. Its attributes follow the archive schema. Numerical arrays are represented as NumPy arrays, physical quantities carry Pint units, and archive references can be followed as Python attributes. Call archive.m_to_dict() to serialize a typed archive back to a JSON-compatible dictionary.

The built-in metadata and results sections used above are available with the nomad-lab package. To access a plugin-defined archive section as a typed object, install its matching schema package in the same Python environment. Otherwise, retrieve it as a plain dictionary as described below.

Retrieve results in batches

For a larger result set, separate the search from archive retrieval. fetch() adds matching (entry_id, upload_id) pairs to a local pending list, and download() removes successfully retrieved Entries from that list:

batch_query = ArchiveQuery(
    owner='public',
    query={'results.material.elements': 'Ti'},
    required={'metadata': '*'},
    results_max=20,
)

batch_query.fetch(20)
print(batch_query.entry_list())

while batch_query.entry_list():
    batch = batch_query.download(5)
    for archive in batch:
        print(archive.metadata.entry_id)

Calling download(number) without fetching first is also valid. The client fetches more matching Entries whenever its pending list contains fewer than number Entries. A failed download request remains pending so that a later download() call can retry it.

For large queries, tune request concurrency and rate limiting conservatively for the deployment you use. See Reference > ArchiveQuery client > Request and batching controls for the relevant options.

Retrieve plain dictionaries

Use as_plain_dict=True when you need JSON-like dictionaries instead of typed EntryArchive objects. This is also useful when your local environment does not have the plugin schema package used by an Entry:

plain_query = ArchiveQuery(
    owner='public',
    query={'results.material.elements': 'Ti'},
    required={'metadata': '*', 'data': '*'},
    results_max=5,
)

archives = plain_query.download(as_plain_dict=True)
print(archives[0]['metadata']['entry_id'])

Convert schema-aware results to a dataframe

After retrieving schema-aware EntryArchive objects with the synchronous download() method, flatten the results into a pandas dataframe:

dataframe = archive_query.entries_to_dataframe(
    keys_to_filter=['metadata', 'results.material'],
)
print(dataframe)

By default, the method converts the most recent synchronous download() result. It can instead combine all downloads or select particular download calls. See Reference > ArchiveQuery client > Dataframe conversion for the selection options and limitations.

Use the asynchronous interface

In asynchronous applications and notebooks, call the coroutine methods directly:

archive_query = ArchiveQuery(
    owner='public',
    query={'results.material.elements': 'Ti'},
    required={'metadata': '*'},
    results_max=5,
)

await archive_query.async_fetch(5)
archives = await archive_query.async_download(5)

The synchronous fetch() and download() methods also work when an event loop is already running, including in a Jupyter notebook. They do not require nest_asyncio.

Access non-public entries

Create a personal access token as described in How-to guides > ... > Authenticate programmatically > Create a PAT and set the NOMAD_CLIENT_ACCESS_TOKEN environment variable before starting Python:

export NOMAD_CLIENT_ACCESS_TOKEN="<personal-access-token>"

Do not put access tokens directly in source files. For each query, choose an ownership scope that includes the non-public entries you need, such as visible, user, shared, or staging.

Complete example

from nomad.client import ArchiveQuery

query = ArchiveQuery(
    owner='public',
    query={
        'results.method.simulation.program_name': 'VASP',
        'results.material.elements': {'all': ['Ti', 'O']},
    },
    required={
        'metadata': {
            'entry_id': '*',
            'mainfile': '*',
        },
        'results': {
            'material': {
                'chemical_formula_reduced': '*',
            },
        },
    },
    results_max=5,
    page_size=5,
)

for archive in query.download():
    print(
        archive.metadata.entry_id,
        archive.results.material.chemical_formula_reduced,
    )

# Convert the retrieved typed archives into a pandas dataframe.
dataframe = query.entries_to_dataframe(keys_to_filter=['metadata', 'results.material'])
print(dataframe)