WTHB TOC Language Pipeline

Overview

The linkenhancer-wthb-toc-lang package delivers the table of contents (TOC) of the German webtrees manual — and its translations — to the WTHB help feature. Its production is a round trip that lives entirely outside the webtrees install: it reads the manual from GenWiki, regenerates the TOC, writes the result back to GenWiki as a MediaWiki template, and — in parallel — ships the structured data to an external Codeberg repo that feeds Codeberg Pages, Weblate (translations) and, finally, the module. This page documents the production side; the consumer side (cache, CLI, rendering) is covered on the WTHB feature page and the CLI scripts page.

Ressources

• GenWiki German Webtrees Manual: https://wiki.genealogy.net/Webtrees_Handbuch
• Code Repositories:
  • Module: https://codeberg.org/bschwede/linkenhancer
  • TOC-Translation: https://codeberg.org/bschwede/linkenhancer-wthb-toc-lang
• Weblate translation component: https://translate.codeberg.org/projects/wt-modules/linkenhancer-wthb-toc
• GenWiki Bot project: https://gitlab.genealogy.net/genwiki/wikibots (gw-skripte / pywikibot documentation)

Pipeline

The round trip has two halves: the GenWiki round trip (read the manual → generate → write the TOC template back) and the delivery pipeline (structured data → external repo → Pages / Weblate → module).

The GenWiki round trip:

flowchart TB
    subgraph HOST["🖥️ Host · gw-skripte/"]
        ORCH["`🚦 **wt-maintain.sh**<br/>*Orchestration*`"]
        PREAD["`🤖 **pywikibot** · *wt-maintain.py*<br/>SectionsExportBot — *READ*`"]
        GEN["`⚙️ **wt-gen-toc.py** *(pandas)*`"]
        PREP["`📄 *pagefromfile file*<br/>vorlage_wt-verzeichnis-toc.txt`"]
        PWRITE["`🤖 **pywikibot** · *pagefromfile*<br/>*WRITE* — Bot-Summary`"]
    end

    GW["`📖 **GenWiki** · wiki.genealogy.net<br/>*webtrees handbook pages*`"]
    CSV["`📄 *3 CSV*<br/>pageinfo · sections · images`"]
    OUT["`📦 *Outputs*<br/>wthb-toc.wiki · .html · .json · de.json`"]
    TPL["`📖 **Vorlage:WT-Verzeichnis-TOC**<br/>*noinclude + includeonly TOC*`"]
    API["`🔍 *API action=parse* → HTML<br/>(curl + jq)`"]

    GW -->|"allpages prefix 'Webtrees Handbuch'<br/>API parse/sections + imagelinks"| PREAD
    ORCH -.->|"drives"| PREAD
    PREAD --> CSV
    CSV -->|"+ wthb-pageinfo_ext.csv (target)"| GEN
    GEN --> OUT
    OUT --> PREP
    PREP --> PWRITE
    PWRITE -->|"pagefromfile -force"| TPL
    TPL -->|"curl API parse"| API

    classDef srcStyle fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
    classDef coreStyle fill:#ede7f6,stroke:#5e35b1,stroke-width:2px
    classDef okStyle fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
    classDef pwbStyle fill:#fff8e1,stroke:#f57f17,stroke-width:2px
    classDef targetStyle fill:#fce4ec,stroke:#ad1457,stroke-width:2px

    class GW srcStyle
    class ORCH,GEN coreStyle
    class CSV,OUT,API okStyle
    class PREAD,PWRITE pwbStyle
    class TPL targetStyle

Orchestration — wt-maintain.sh

Everything runs from one host script (in gw-skripte/):

wt-maintain.sh <wthb-pageinfo_ext.csv> <DATADIR>

It depends on setenv.sh exporting PWB_PY, PWB_CONF and SCRIPTDIR, and drives these steps in order:

1. query the GenWiki data (pywikibot READ)
2. generate the TOC
3. build the pagefromfile wrapper file
4. write the template back to GenWiki (pywikibot WRITE)
5. parse-check the rendered template (curl + jq)
6. deploy the JSON to the external repo
7. downstream: repo CI, Weblate and the module pick the data up

GenWiki data query (pywikibot READ)

Step 1 is wt-maintain.py, a pywikibot bot (SectionsExportBot, extending ExistingPageBot). It calls site.login(), iterates allpages(prefix="Webtrees Handbuch") and, per page, queries the MediaWiki API (action=parse&prop=sections) plus imagelinks() (FilePage info).
The result is three CSV files:

1. wthb-pageinfo.csv,
2. wthb-sections.csv and
3. wthb-images.csv.

Two manual follow-ups remain (per the script header): (b) maintaining the image directory and (c) adding the images to the Kategorie:Webtrees category — done by hand (helper: wt-webtrees-cat-files.sh).

TOC generation — wt-gen-toc.py

Step 2 is a pandas script that combines the IST state (the three CSVs) with the SOLL target structure in wthb-pageinfo_ext.csv. classify_prefix() assigns chapter/section prefixes. It writes four outputs:

• wthb-toc.wiki — the MediaWiki TOC source
• wthb-toc.json — the structured data the module consumes
• de.json — the German language file for Weblate
• wthb-toc.html — a rendered HTML copy (legacy/parallel — the current consumer renders from JSON)

GenWiki template (pywikibot WRITE)

Steps 3–4 wrap the generated TOC in the MediaWiki template Vorlage:WT-Verzeichnis-TOC and write it back:

• the pagefromfile file combines a <noinclude> part ({{Dokumentation}} + [[Kategorie:Webtrees]]) with an <includeonly> part (Stand: <datum> + <div class="wthbtoc">…</div>).
• pywikibot pagefromfile -force -pt:1 -summary:"Bot: Webtrees Handbuch Inhaltsverzeichnis aktualisiert" performs the write.

Step 5 verifies the result by parsing it back through the GenWiki API (action=parse&format=json) and extracting the HTML with jq -r '.parse.text["*"]'.

Note: this is automatic writing to GenWiki via pywikibot — the opposite of the “produce review output, no bot” approach used for the GenWiki help-docs export. Vorlage:WT-Verzeichnis-TOC and Kategorie:Webtrees are GenWiki-specific German names and are kept verbatim.

External repo, CI & Pages

The second half of the pipeline (step 6 onward) ships the structured data out of the host:

flowchart TB
    OUT["`📦 *Outputs*<br/>wthb-toc.json + de.json`"]
    DEP["`🚀 **wt-deploy-toc-lang.sh**`"]

    subgraph EXTERNAL["🌐 External infrastructure"]
        REPO["`📦 **linkenhancer-wthb-toc-lang**<br/>(Codeberg)`"]
        META["`🔧 *tools/gen_meta.py*<br/>languages.json + version.json`"]
        CI["`⚙️ *CI regen-meta.yml*<br/>gen_toc_html.py`"]
        PG["`🌐 **Codeberg Pages**`"]
        WLB["`🌐 **Weblate** · linkenhancer/wthb-toc`"]
        TNG["🗝 en.json · fr.json · …"]
    end

    subgraph MODULE["📦 Module (linkenhancer)"]
        SVC["`**WthbTocService**`"]
        DS["`💾 *data/…/wthb-toc-lang/*<br/>2-Layer-Cache`"]
        RDR["`📄 **help-wthb-toc.phtml**<br/>TOC-Modal`"]
    end

    OUT -->|"cp wthb-toc.json + de.json"| DEP
    DEP -->|"git add / commit / push"| REPO
    REPO -->|"regen meta"| META
    REPO --> CI --> PG
    REPO -->|"de.json (Source)"| WLB
    WLB --> TNG
    TNG -->|"[lang].json"| REPO
    PG -->|"fetch JSON"| MODULE
    SVC --> DS --> RDR

    classDef coreStyle fill:#ede7f6,stroke:#5e35b1,stroke-width:2px
    classDef okStyle fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
    classDef queueStyle fill:#fff3e0,stroke:#e65100,stroke-width:2px

    class OUT okStyle
    class DEP,REPO,META,CI,SVC coreStyle
    class PG,TNG,DS,RDR okStyle
    class WLB queueStyle

• wt-deploy-toc-lang.sh copies wthb-toc.json and de.json into the linkenhancer-wthb-toc-lang repo, runs tools/gen_meta.py (regenerates languages.json + version.json) and does git add/commit/push. It is standalone-callable and soft-skips (warning + exit 0) when the repo or JSON is missing, so it never aborts the set -e orchestrator.
• Repo CI (regen-meta.yml) runs gen_toc_html.py and publishes to Codeberg Pages.

Weblate

The repo’s de.json is the source of the Weblate component wt-modules/linkenhancer-wthb-toc; the other [lang].json files are its translation targets. Translations flow back into the repo and are served via Codeberg Pages.

Consumer (module)

Briefly — see the WTHB feature page and the CLI scripts page for details:

• WthbTocService.php pulls the language JSON from Codeberg Pages and serves it through a two-layer cache.
• refresh-wthb-toc.php (and cli/validate-wthb-toc.php) refresh and verify that cache, typically on a cron schedule.
• help-wthb-toc.phtml renders the TOC in the help dialog.

Configuration & prerequisites

• setenv.sh — exports PWB_PY, PWB_CONF, SCRIPTDIR.
• pywikibot config — user-config-test.py / user-config-prod.py.
• wthb-toc-lang.env — TOC_LANG_REPO=$HOME/dev/linkenhancer-wthb-toc-lang, TOC_LANG_DATADIR=$HOME/dev/wthb-toc-datadir.
• Prerequisites: pywikibot, pandas, jq, python3.

Ops & caveats

• Manual image follow-up — the image directory and Kategorie:Webtrees must be maintained by hand.
• pagefromfile login — the script comment notes that site.login() may need to be inserted into pagefromfile.py.
• Soft-skip deploy — a missing repo or JSON is a warning, not a failure (keeps set -e safe).