# WTHB TOC Language Pipeline > How the German webtrees manual TOC is produced (GenWiki → pywikibot → generation → external repo / Pages / Weblate → module) --- ## Overview The **`linkenhancer-wthb-toc-lang`** package delivers the table of contents (TOC) of the German webtrees manual — and its translations — to the [WTHB help feature](../features/wthb). Its production is a **round trip** that lives entirely outside the webtrees install: it reads the manual from **GenWiki**, regenerates the TOC, writes the result back to GenWiki as a **MediaWiki template**, and — in parallel — ships the structured data to an **external Codeberg repo** that feeds **Codeberg Pages**, **Weblate** (translations) and, finally, the **module**. This page documents the *production* side; the *consumer* side (cache, CLI, rendering) is covered on the [WTHB feature page](../features/wthb) and the [CLI scripts](cli-scripts) page. ### Ressources - GenWiki German Webtrees Manual: - Code Repositories: - Module: - TOC-Translation: - Weblate translation component: - GenWiki Bot project: (gw-skripte / [pywikibot documentation](https://doc.wikimedia.org/pywikibot/stable/introduction.html)) ## Pipeline The round trip has two halves: the **GenWiki round trip** (read the manual → generate → write the TOC template back) and the **delivery pipeline** (structured data → external repo → Pages / Weblate → module). The GenWiki round trip: 
flowchart TB
    subgraph HOST["🖥️ Host · gw-skripte/"]
        ORCH["`🚦 **wt-maintain.sh**<br/>*Orchestration*`"]
        PREAD["`🤖 **pywikibot** · *wt-maintain.py*<br/>SectionsExportBot — *READ*`"]
        GEN["`⚙️ **wt-gen-toc.py** *(pandas)*`"]
        PREP["`📄 *pagefromfile file*<br/>vorlage_wt-verzeichnis-toc.txt`"]
        PWRITE["`🤖 **pywikibot** · *pagefromfile*<br/>*WRITE* — Bot-Summary`"]
    end

    GW["`📖 **GenWiki** · wiki.genealogy.net<br/>*webtrees handbook pages*`"]
    CSV["`📄 *3 CSV*<br/>pageinfo · sections · images`"]
    OUT["`📦 *Outputs*<br/>wthb-toc.wiki · .html · .json · de.json`"]
    TPL["`📖 **Vorlage:WT-Verzeichnis-TOC**<br/>*noinclude + includeonly TOC*`"]
    API["`🔍 *API action=parse* → HTML<br/>(curl + jq)`"]

    GW -->|"allpages prefix 'Webtrees Handbuch'<br/>API parse/sections + imagelinks"| PREAD
    ORCH -.->|"drives"| PREAD
    PREAD --> CSV
    CSV -->|"+ wthb-pageinfo_ext.csv (target)"| GEN
    GEN --> OUT
    OUT --> PREP
    PREP --> PWRITE
    PWRITE -->|"pagefromfile -force"| TPL
    TPL -->|"curl API parse"| API

    classDef srcStyle fill:#e3f2fd,stroke:#1565c0,stroke-width:2px
    classDef coreStyle fill:#ede7f6,stroke:#5e35b1,stroke-width:2px
    classDef okStyle fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
    classDef pwbStyle fill:#fff8e1,stroke:#f57f17,stroke-width:2px
    classDef targetStyle fill:#fce4ec,stroke:#ad1457,stroke-width:2px

    class GW srcStyle
    class ORCH,GEN coreStyle
    class CSV,OUT,API okStyle
    class PREAD,PWRITE pwbStyle
    class TPL targetStyle
## Orchestration — `wt-maintain.sh` Everything runs from one host script (in `gw-skripte/`): ```bash wt-maintain.sh ``` It depends on `setenv.sh` exporting `PWB_PY`, `PWB_CONF` and `SCRIPTDIR`, and drives these steps in order: 1. query the GenWiki data (pywikibot READ) 2. generate the TOC 3. build the [`pagefromfile`](https://doc.wikimedia.org/pywikibot/stable/scripts_ref/scripts.html#module-scripts.pagefromfile) wrapper file 4. write the template back to GenWiki (pywikibot WRITE) 5. parse-check the rendered template (curl + jq) 6. deploy the JSON to the external repo 7. downstream: repo CI, Weblate and the module pick the data up ## GenWiki data query (pywikibot READ) Step 1 is `wt-maintain.py`, a pywikibot bot (`SectionsExportBot`, extending `ExistingPageBot`). It calls `site.login()`, iterates `allpages(prefix="Webtrees Handbuch")` and, per page, queries the [MediaWiki API](https://www.mediawiki.org/wiki/API:Action_API) (`action=parse&prop=sections`) plus `imagelinks()` (FilePage info). \ The result is **three CSV files**: 1. `wthb-pageinfo.csv`, 2. `wthb-sections.csv` and 3. `wthb-images.csv`. Two manual follow-ups remain (per the script header): (b) maintaining the image directory and (c) adding the images to the `Kategorie:Webtrees` category — done by hand (helper: `wt-webtrees-cat-files.sh`). ## TOC generation — `wt-gen-toc.py` Step 2 is a pandas script that combines the **IST** state (the three CSVs) with the **SOLL** target structure in `wthb-pageinfo_ext.csv`. `classify_prefix()` assigns chapter/section prefixes. It writes four outputs: - `wthb-toc.wiki` — the MediaWiki TOC source - `wthb-toc.json` — the structured data the module consumes - `de.json` — the German language file for Weblate - `wthb-toc.html` — a rendered HTML copy (legacy/parallel — the current consumer renders from **JSON**) ## GenWiki template (pywikibot WRITE) Steps 3–4 wrap the generated TOC in the MediaWiki template [**`Vorlage:WT-Verzeichnis-TOC`**](https://wiki.genealogy.net/Vorlage:WT-Verzeichnis-TOC) and write it back: - the `pagefromfile` file combines a `` part (`{{Dokumentation}}` + `[[Kategorie:Webtrees]]`) with an `` part (`Stand: ` + `
…
`). - `pywikibot pagefromfile -force -pt:1 -summary:"Bot: Webtrees Handbuch Inhaltsverzeichnis aktualisiert"` performs the write. Step 5 verifies the result by parsing it back through the GenWiki API (`action=parse&format=json`) and extracting the HTML with `jq -r '.parse.text["*"]'`. > **Note:** this is *automatic* writing to GenWiki via pywikibot — the opposite of the "produce review output, no bot" approach used for the GenWiki help-docs export. `Vorlage:WT-Verzeichnis-TOC` and `Kategorie:Webtrees` are GenWiki-specific German names and are kept verbatim. ## External repo, CI & Pages The second half of the pipeline (step 6 onward) ships the structured data out of the host: 
flowchart TB
    OUT["`📦 *Outputs*<br/>wthb-toc.json + de.json`"]
    DEP["`🚀 **wt-deploy-toc-lang.sh**`"]

    subgraph EXTERNAL["🌐 External infrastructure"]
        REPO["`📦 **linkenhancer-wthb-toc-lang**<br/>(Codeberg)`"]
        META["`🔧 *tools/gen_meta.py*<br/>languages.json + version.json`"]
        CI["`⚙️ *CI regen-meta.yml*<br/>gen_toc_html.py`"]
        PG["`🌐 **Codeberg Pages**`"]
        WLB["`🌐 **Weblate** · linkenhancer/wthb-toc`"]
        TNG["🗝 en.json · fr.json · …"]
    end

    subgraph MODULE["📦 Module (linkenhancer)"]
        SVC["`**WthbTocService**`"]
        DS["`💾 *data/…/wthb-toc-lang/*<br/>2-Layer-Cache`"]
        RDR["`📄 **help-wthb-toc.phtml**<br/>TOC-Modal`"]
    end

    OUT -->|"cp wthb-toc.json + de.json"| DEP
    DEP -->|"git add / commit / push"| REPO
    REPO -->|"regen meta"| META
    REPO --> CI --> PG
    REPO -->|"de.json (Source)"| WLB
    WLB --> TNG
    TNG -->|"[lang].json"| REPO
    PG -->|"fetch JSON"| MODULE
    SVC --> DS --> RDR

    classDef coreStyle fill:#ede7f6,stroke:#5e35b1,stroke-width:2px
    classDef okStyle fill:#e8f5e9,stroke:#2e7d32,stroke-width:2px
    classDef queueStyle fill:#fff3e0,stroke:#e65100,stroke-width:2px

    class OUT okStyle
    class DEP,REPO,META,CI,SVC coreStyle
    class PG,TNG,DS,RDR okStyle
    class WLB queueStyle
- **`wt-deploy-toc-lang.sh`** copies `wthb-toc.json` and `de.json` into the `linkenhancer-wthb-toc-lang` repo, runs `tools/gen_meta.py` (regenerates `languages.json` + `version.json`) and does `git add/commit/push`. It is standalone-callable and **soft-skips** (warning + `exit 0`) when the repo or JSON is missing, so it never aborts the `set -e` orchestrator. - **Repo CI** (`regen-meta.yml`) runs `gen_toc_html.py` and publishes to **Codeberg Pages**. ## Weblate The repo's `de.json` is the **source** of the Weblate component **`wt-modules/linkenhancer-wthb-toc`**; the other `[lang].json` files are its translation targets. Translations flow back into the repo and are served via Codeberg Pages. ## Consumer (module) Briefly — see the [WTHB feature page](../features/wthb) and the [CLI scripts](cli-scripts) page for details: - WthbTocService.php pulls the language JSON from Codeberg Pages and serves it through a **two-layer cache**. - refresh-wthb-toc.php (and `cli/validate-wthb-toc.php`) refresh and verify that cache, typically on a cron schedule. - help-wthb-toc.phtml renders the TOC in the help dialog. ## Configuration & prerequisites - `setenv.sh` — exports `PWB_PY`, `PWB_CONF`, `SCRIPTDIR`. - pywikibot config — `user-config-test.py` / `user-config-prod.py`. - `wthb-toc-lang.env` — `TOC_LANG_REPO=$HOME/dev/linkenhancer-wthb-toc-lang`, `TOC_LANG_DATADIR=$HOME/dev/wthb-toc-datadir`. - Prerequisites: pywikibot, pandas, jq, python3. ## Ops & caveats - **Manual image follow-up** — the image directory and `Kategorie:Webtrees` must be maintained by hand. - **`pagefromfile` login** — the script comment notes that `site.login()` may need to be inserted into `pagefromfile.py`. - **Soft-skip deploy** — a missing repo or JSON is a warning, not a failure (keeps `set -e` safe).