acdh-oeaw/acdh-search-indexer: Release v0.1.0
Fuente:
Zenodo
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Recurso digital |
| Veröffentlicht: |
Zenodo
2025
|
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866902210008317952 |
|---|---|
| author | Elsner, Daniel |
| author_facet | Elsner, Daniel |
| contents | <h1>ACDH Search Indexer</h1> <p>This repository provides Python scripts to index data from TEI/XML sources into NoSketch Engine TSV Index files. It also contains a Dockerfile to build a NoSketch Engine Docker image with the ACDH Search Indexer scripts. Next to NoSketch it provides Python scripts to create an Index for Typesense Search.</p> <h2>Fetch Data</h2> <p>Update `shellscripts/fetch_data.sh` to fetch the data you want to index. The script will be executed in the Github [Actions Workflow](./.githbub/workflows/index.yml).</p> <h2>Create Noske Index</h2> <p>Update `noske/mk_verticals.py` constants to match your data. The script will be executed in the Github [Actions Workflow](./.githbub/workflows/index.yml). Or manually run:</p> <p><code>pip install pipenv</code><br><code>pipenv install</code><br><code>pipenv run python noske/mk_verticals.py</code></p> <h2>Create Typesense Index</h2> <p>Update `typesense/mk_index.py` constants to match your data. The script will be executed in the Github [Actions Workflow](./.githbub/workflows/index.yml). Or manually run:</p> <p><code>pip install pipenv</code><br><code>pipenv install</code><br><code>pipenv run python typesense/mk_index.py</code><br><br></p> <p>Make sure the returned data type matches the Schema. If not, you can adjust the `typesense/mk_index.py` script <br>`match case` statements to your schema. For example, the following code snippet creates a record for the Typesense index:<br> <br><code>def create_index_record(doc: TeiReader, blacklist: list) -> dict:</code><br><code> record = {}</code><br><code> for key, value in XPATHS.items():</code><br><code> items = extract_structure(doc, value)</code><br><code> match key:</code><br><code> case "date":</code><br><code> # returns an integer of an attribute value</code><br><code> record[key] = int("".join(items))</code><br><code> case "places":</code><br><code> # returns a list of strings from a list of elements</code><br><code> record[key] = [extract_text(place, blacklist) for place in items]</code><br><code> case "edition":</code><br><code> # returns a list of strings from a list of elements</code><br><code> record[key] = [extract_text(edition, blacklist) for edition in items]</code><br><code> case _:</code><br><code> # returns a string from a list of elements</code><br><code> record[key] = " ".join([extract_text(text, blacklist) for text in items])</code><br><code> return record</code></p> <h2>Noske Docker</h2> <p>The Image uses [acdh-oeaw/noske-ubi9](https://github.com/acdh-oeaw/noske-ubi9) as base image.<br>Build the Docker image with the ACDH Search Indexer scripts:</p> <p><code>docker build -f nosketchengine/Dockerfile -t acdh/noske .</code><br><br></p> <p>Run Noske Docker Image:</p> <p><br><code>docker run --rm -it -p 8080:8080 acdh/noske:latest</code><br><br></p> <p><strong>Full Changelog</strong>: https://github.com/acdh-oeaw/acdh-search-indexer/commits/v0.1.0</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_17150946 |
| institution | Zenodo |
| language | |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | acdh-oeaw/acdh-search-indexer: Release v0.1.0 Elsner, Daniel <h1>ACDH Search Indexer</h1> <p>This repository provides Python scripts to index data from TEI/XML sources into NoSketch Engine TSV Index files. It also contains a Dockerfile to build a NoSketch Engine Docker image with the ACDH Search Indexer scripts. Next to NoSketch it provides Python scripts to create an Index for Typesense Search.</p> <h2>Fetch Data</h2> <p>Update `shellscripts/fetch_data.sh` to fetch the data you want to index. The script will be executed in the Github [Actions Workflow](./.githbub/workflows/index.yml).</p> <h2>Create Noske Index</h2> <p>Update `noske/mk_verticals.py` constants to match your data. The script will be executed in the Github [Actions Workflow](./.githbub/workflows/index.yml). Or manually run:</p> <p><code>pip install pipenv</code><br><code>pipenv install</code><br><code>pipenv run python noske/mk_verticals.py</code></p> <h2>Create Typesense Index</h2> <p>Update `typesense/mk_index.py` constants to match your data. The script will be executed in the Github [Actions Workflow](./.githbub/workflows/index.yml). Or manually run:</p> <p><code>pip install pipenv</code><br><code>pipenv install</code><br><code>pipenv run python typesense/mk_index.py</code><br><br></p> <p>Make sure the returned data type matches the Schema. If not, you can adjust the `typesense/mk_index.py` script <br>`match case` statements to your schema. For example, the following code snippet creates a record for the Typesense index:<br> <br><code>def create_index_record(doc: TeiReader, blacklist: list) -> dict:</code><br><code> record = {}</code><br><code> for key, value in XPATHS.items():</code><br><code> items = extract_structure(doc, value)</code><br><code> match key:</code><br><code> case "date":</code><br><code> # returns an integer of an attribute value</code><br><code> record[key] = int("".join(items))</code><br><code> case "places":</code><br><code> # returns a list of strings from a list of elements</code><br><code> record[key] = [extract_text(place, blacklist) for place in items]</code><br><code> case "edition":</code><br><code> # returns a list of strings from a list of elements</code><br><code> record[key] = [extract_text(edition, blacklist) for edition in items]</code><br><code> case _:</code><br><code> # returns a string from a list of elements</code><br><code> record[key] = " ".join([extract_text(text, blacklist) for text in items])</code><br><code> return record</code></p> <h2>Noske Docker</h2> <p>The Image uses [acdh-oeaw/noske-ubi9](https://github.com/acdh-oeaw/noske-ubi9) as base image.<br>Build the Docker image with the ACDH Search Indexer scripts:</p> <p><code>docker build -f nosketchengine/Dockerfile -t acdh/noske .</code><br><br></p> <p>Run Noske Docker Image:</p> <p><br><code>docker run --rm -it -p 8080:8080 acdh/noske:latest</code><br><br></p> <p><strong>Full Changelog</strong>: https://github.com/acdh-oeaw/acdh-search-indexer/commits/v0.1.0</p> |
| title | acdh-oeaw/acdh-search-indexer: Release v0.1.0 |
| url | https://doi.org/10.5281/zenodo.17150946 |