Migrate to NOMAD 2.0¶

Elasticsearch Migration (v7.17 to v9.5)¶

Starting with NOMAD 2.0, the default Elasticsearch image has been changed from Elasticsearch 7 to Elasticsearch 9 (default container version 9.5.0).

Because Elasticsearch 9 cannot read indices created with Elasticsearch 7 due to major Lucene version changes, existing data cannot be read directly from an old Elasticsearch 7 volume.

There are two migration options:

Option 1: Reindex data directly from ES7 to ES9 (running two containers) — Runs ES9 alongside ES7 and migrates the existing index through Elasticsearch's _reindex API. This is usually faster because it copies indexed documents without reprocessing archive files.
Option 2: Start with a fresh ES9 index and reindex uploads — Discards the ES7 search index and rebuilds it from MongoDB and the archive files (.volumes/fs), which are NOMAD's source of truth. This is simpler but can be considerably slower because NOMAD must read and index every upload again.

Note

Previously, NOMAD maintained a separate materials index (nomad_materials_v1 / nomad_oasis_materials_v1). This index is deprecated and dropped in NOMAD 2.0. Only the entries index (nomad_oasis_entries_v1 or nomad_entries_v1) needs to be migrated.

Option 1: Reindex data directly from ES7 to ES9 (running two containers)¶

This approach runs a temporary Elasticsearch 9 container alongside your existing Elasticsearch 7 container, copies over the index mappings, and streams documents over the Docker network using Elasticsearch's remote _reindex API.

1. Add the Temporary `nomad_oasis_elastic_9` Service¶

Add the nomad_oasis_elastic_9 service and volume to your docker-compose.yaml (while keeping your existing elastic 7.x service running):

services:
  # ... existing services ...

  nomad_oasis_elastic_9:
    restart: "no"
    image: docker.elastic.co/elasticsearch/elasticsearch:9.5.0
    container_name: nomad_oasis_elastic_9
    environment:
      - ES_JAVA_OPTS=-Xms512m -Xmx512m
      - cluster.routing.allocation.disk.threshold_enabled=true
      - cluster.routing.allocation.disk.watermark.flood_stage=1gb
      - cluster.routing.allocation.disk.watermark.low=4gb
      - cluster.routing.allocation.disk.watermark.high=2gb
      - discovery.type=single-node
      - xpack.security.enabled=false
      - reindex.remote.whitelist=elastic:9200
    ports:
      - "9201:9200"
    volumes:
      - nomad_oasis_elastic_9:/usr/share/elasticsearch/data
    healthcheck:
      test: ["CMD", "curl", "-fsS", "http://localhost:9200/_cluster/health?wait_for_status=yellow&timeout=2s"]
      interval: 2s
      timeout: 5s
      retries: 60
      start_period: 10s

volumes:
  # ... existing volumes ...
  nomad_oasis_elastic_9:
    name: "nomad_oasis_elastic_9"

Start the temporary ES9 container and wait for its health check to pass:

docker compose up -d --wait --wait-timeout 180 nomad_oasis_elastic_9

2. Copy Mapping and Create Index in ES9¶

Export the settings and mappings from ES7 and create the new index in ES9 (port 9201).

The effective index name may come from configuration overrides rather than directly from configs/nomad.yaml. Read it from the running NOMAD app container:

set -o pipefail

# Read the effective entries index name from NOMAD's configuration
INDEX="$(docker compose exec -T app python -c \
  'from nomad.config import config; print(config.elastic.entries_index)')"
echo "Entries index: ${INDEX}"

# 1. Fetch settings and mapping from ES7 through its service container
docker compose exec -T elastic \
  curl -fsS "http://localhost:9200/${INDEX}/_settings?flat_settings=false" \
  > "/tmp/${INDEX}-settings.json"
docker compose exec -T elastic \
  curl -fsS "http://localhost:9200/${INDEX}/_mapping" \
  > "/tmp/${INDEX}-mapping.json"

# 2. Create the index on ES9 with replica count set to 0 and matching analysis settings
jq -n \
  --arg index "$INDEX" \
  --slurpfile settings "/tmp/${INDEX}-settings.json" \
  --slurpfile mapping "/tmp/${INDEX}-mapping.json" \
  '{
    settings: {
      number_of_replicas: 0,
      analysis: ($settings[0][$index].settings.index.analysis // {})
    },
    mappings: $mapping[0][$index].mappings
  }' | \
  curl -sS --fail-with-body -X PUT "http://localhost:9201/${INDEX}" \
    -H 'Content-Type: application/json' \
    --data-binary @- && \
  printf '\nCreated %s on ES9\n' "$INDEX"

3. Reindex from ES7 to ES9¶

Trigger the remote reindex from the ES7 container (http://elastic:9200) into ES9:

set -o pipefail

jq -n \
  --arg index "$INDEX" \
  '{
    source: {
      remote: {host: "http://elastic:9200"},
      index: $index,
      size: 100
    },
    dest: {index: $index}
  }' | \
  curl -fsS -X POST \
    "http://localhost:9201/_reindex?wait_for_completion=true&refresh=true" \
    -H 'Content-Type: application/json' \
    --data-binary @- | jq .

4. Verify Document Counts¶

Verify that the document count in ES9 matches ES7:

echo "ES7 count:"
docker compose exec -T elastic \
  curl -fsS "http://localhost:9200/${INDEX}/_count" | jq .

echo "ES9 count:"
curl -fsS "http://localhost:9201/${INDEX}/_count" | jq .

echo "ES9 mapping property count:"
curl -fsS "http://localhost:9201/${INDEX}/_mapping" | \
  jq 'to_entries[0].value.mappings.properties | length'

5. Finalize the Migration¶

Once verified:

Stop the running containers:
```
docker compose down
```
Update docker-compose.yaml:
- Set the main elastic service image to docker.elastic.co/elasticsearch/elasticsearch:9.5.0.
- Add xpack.security.enabled=false to the main elastic service's environment list to disable Elasticsearch's built-in TLS and authentication, matching NOMAD's existing plain-HTTP connection configuration.
- Change the volume mount for elastic to use the migrated volume nomad_oasis_elastic_9 (or replace the old volume).
- Ensure NOMAD_ELASTIC_VERSION: 9 is present in your NOMAD services (app, worker, etc.).
- Remove the temporary nomad_oasis_elastic_9 service definition from docker-compose.yaml.
Start your upgraded NOMAD Oasis:
```
docker compose up -d
```
Remove the old ES7 volume if no longer needed:
```
docker volume rm nomad_oasis_elastic
```

Option 2: Start with a fresh ES9 index and reindex uploads¶

In NOMAD, MongoDB and the archive files (.volumes/fs) are the single source of truth. This option wipes the Elasticsearch volume, starts a fresh Elasticsearch 9 container, and reindexes all uploads from MongoDB and archive storage.

1. Back Up MongoDB and File Storage¶

Before proceeding, run a MongoDB backup:

bash scripts/backup-mongo.sh

Ensure .volumes/fs is intact.

2. Update Compose File¶

In docker-compose.yaml:

Set image: docker.elastic.co/elasticsearch/elasticsearch:9.5.0 under the elastic service.
Add xpack.security.enabled=false to the elastic service's environment list to disable Elasticsearch's built-in TLS and authentication, matching NOMAD's existing plain-HTTP connection configuration.
Ensure NOMAD_ELASTIC_VERSION: 9 is set under the NOMAD container environment (app, worker, etc.).
Keep the elastic:/usr/share/elasticsearch/data service mount and change the top-level elastic.name to nomad_oasis_elastic_9:
```
volumes:
  # ... other volumes ...
  elastic:
    name: "nomad_oasis_elastic_9"
```

3. Create the Fresh ES9 Volume and Start Services¶

# Stop all containers
docker compose down

# Start the upgraded stack; Compose creates nomad_oasis_elastic_9
docker compose up -d

The old nomad_oasis_elastic volume remains available as a rollback until the migration is verified.

4. Run the Indexing CLI Command¶

Wait until app and elastic are healthy, then trigger re-indexing across all uploads:

docker compose exec app python -m nomad.cli admin uploads index --parallel 4

Tip

You can increase --parallel based on available CPU cores. For large deployments, add --print-progress 10 to monitor ongoing progress.

5. Remove the Old ES7 Volume¶

After verifying that the uploads are indexed correctly, remove the old ES7 volume:

docker volume rm nomad_oasis_elastic