# Holdings pipeline: how the daily data is produced The ingester turns the issuers' own holdings pages into dated JSON: ```bash node tools/ingest/holdings.mjs # fetch live, write data/holdings/ node tools/ingest/validate.mjs data/holdings node tools/ingest/render-pages.mjs # render the static HTML pages from the newest snapshot node tools/diff/changes.mjs # diff the newest snapshot against the one before it node --test tools/ingest/ # fixtures only, no network ``` Files it writes: `data/holdings/YYYY-MM-DD.json` plus `data/holdings/latest.json`, and then `holdings-full.html` with one `fund-.html` per fund that publishes rows. The differ writes one change document per comparison, `data/changes/YYYY-MM-DD.json`, plus the small `data/changes/index.json` the `/changes` page reads. Field-by-field schema, determinism contract and consumer notes: [holdings data schema](holdings-schema.md). How the rendered pages behave, and how to check them: [holdings page](holdings-page.md). Zero dependencies and no build step, Node 20 or newer (global `fetch`, `node:test`). Two HTTP requests per run, one per fund, with a descriptive User-Agent and two retries. | Path | What it is | |---|---| | `tools/ingest/holdings.mjs` | CLI: fetch, parse, build snapshot, write files | | `tools/ingest/validate.mjs` | Schema and semantic validation of published files | | `tools/ingest/render-pages.mjs` | Renders `holdings-full.html` and `fund-.html` from a snapshot, and keeps the sitemap block in step | | `tools/ingest/holdings.test.mjs` | Fixture-driven tests (parser, determinism, rights gate) | | `tools/ingest/render-pages.test.mjs` | Tests for the renderer (no rows from a link_only fund, protected strings, repeatable bytes) | | `tools/ingest/lib/sources.mjs` | Fund registry, rights posture, HLAL gate | | `tools/ingest/lib/parse.mjs` | Table parsing, cell normalisation, row building | | `tools/ingest/lib/ingest.mjs` | Records, snapshot assembly, rename detection, file writing | | `tools/ingest/lib/serialize.mjs` | Canonical JSON, hashing, rounding, ordering | | `tools/ingest/fixtures/` | Test inputs: an excerpt of the real SPUS table, plus synthetic edge cases | --- ## Status right now (2026-09-13) - The pipeline, schema, validator, tests, docs and the CI workflow are in place. The workflow is committed at `.github/workflows/holdings-daily.yml` (see "Turn on the schedule"). - **One dated snapshot is committed: `data/holdings/2026-09-12.json` (219 SPUS rows).** It came from the workflow's first run, on 2026-09-12, not from this environment, which has no Node runtime. - **The static page renderer is in the repo but has not run here for the same reason.** The first `holdings-full.html` and `fund-spus.html` land on the next run of the workflow (or any shell with Node 20), and the same run fills in the generated block of `sitemap.xml`. Until then those two URLs do not exist, and nothing links to a page that is not there. - **The differ now runs in CI, one step after the ingest.** It builds the change document for the newest snapshot and rebuilds `data/changes/index.json` from live comparisons only, and `data/changes` is committed in the same commit as `data/holdings`. See "Change log (the differ)" below. - The ingest task's "two consecutive daily runs are committed" still needs its second day: one more run (the next scheduled run, or **Actions, then Daily holdings ingest, then Run workflow**) writes `data/holdings/2026-09-13.json` beside it. That run is also the first with two live snapshots, so it is when `data/changes` gains a live document. The job log is where the schema validation and the byte-identity check are proven; neither can be run from a tooling-only environment. - **HLAL's source is checked, and unconfirmed as a daily feed.** The one URL the pipeline fetches is `https://www.wahed.com/hlal`, and on 2026-09-12 it carried no dated holdings table in three URL variants. Evidence and the remaining browser check: [HLAL source check](hlal-source-check.md). HLAL stays link-only, and the site no longer promises a daily HLAL alert. - Consequence for anything that reads the data: treat a missing `data/holdings/latest.json` as "not published yet" and show nothing rather than an empty table. ## Static pages `render-pages.mjs` reads the same `latest.json` and dated snapshot the browser table reads and writes plain HTML: `holdings-full.html` for every row, `fund-.html` for each fund that publishes rows. It keeps the per-row issuer link, the fund's own as-of date and the data-not-advice notice, and it never renders a row for a `link_only` fund. Two properties matter for CI: - **Nothing is written when the bytes would not change**, so an unchanged snapshot leaves the working tree clean and the commit step has nothing to add. - **There is no wall-clock timestamp in the output**: dates come from the snapshot, so a re-render is byte-identical and `--check` is a real staleness test rather than a diff generator. Detail, including the sitemap block and the exact CLI flags: [holdings page](holdings-page.md). ## Change log (the differ) `tools/diff/changes.mjs` reads two dated snapshots and writes what changed between them: entries, exits, weight moves, renames and unchanged rows. It picks the newest snapshot in `data/holdings/` as the target and the newest strictly older snapshot with the same data status as the baseline, so a fixture-mode snapshot is never diffed against a live one. Output is one document per comparison, `data/changes/YYYY-MM-DD.json`, plus the small `data/changes/index.json` that lists them newest first. Writes are idempotent, and a revision bump is recorded rather than a silent overwrite if a published comparison changes. Exit codes: `0` ok, `2` partial (a fund could not be compared, the rest still published), `1` failed or bad usage. The daily workflow runs it with `--prune-fixtures`, which rebuilds the index from live comparisons only, so the fixture-mode DEMO document in `tools/diff/fixtures/` never reaches the published index. The step masks the differ's exit code and the next step reads the run summary: `partial` prints a `::warning::` with the compared fund count, a missing snapshot or baseline prints a note, and anything else non-`ok` warns. A link-only fund counts as not comparable, so while HLAL is link-only every day's comparison is `partial`, and that warning is expected rather than a problem to chase. `data/changes` is committed in the same commit as `data/holdings`, so the log never falls behind the snapshots it describes. ## Rights gate (read this before changing a fund's `default_mode`) Per the internal review *SPUS/HLAL Holdings Data Rights Review (republishing check)*: | Fund | Posture | What the pipeline does | |---|---|---| | **SPUS** (`sp-funds.com/spus`) | `conditional_go`: no published terms of use, and the Trust's SEC filing states daily holdings are posted on the fund website. Attributed display and internal storage approved; paid bulk redistribution and API access wait for written confirmation. | `publishing_mode: rows`. Rows are parsed and published with `rights` metadata in the file. | | **HLAL** (`wahed.com/hlal`) | `no_go`: Wahed's terms prohibit copying, republishing or incorporating site content without written consent, with no carve-out for public data. Separately, the page showed no dated holdings table when checked on 2026-09-12, so daily HLAL freshness is unconfirmed. | `publishing_mode: link_only`. The page is fetched only to record reachability; **zero rows** are stored or published. The public page links out to Wahed and to the fund's SEC filings for history. | Lifting the HLAL gate requires *both* flags and a written permission first: ```bash node tools/ingest/holdings.mjs --hlal-mode=rows --hlal-permission-acknowledged ``` Without `--hlal-permission-acknowledged` the run refuses to start. The license-clean alternative, quarter-end HLAL holdings from SEC EDGAR (Form N-PORT data sets), is a public regulatory filing rather than scraped page content, and it is an open task (see "Next" below). Sources are also **not transferable**: IGDA, ISDW, SPSK and MNZL each need their own terms review before they are registered in `lib/sources.mjs`. The same gate holds for the rendered pages: a `link_only` fund contributes no row to `holdings-full.html`, gets no `fund-.html` of its own, and appears only in the per-fund summary and the link-out block. ### The exact URL the HLAL run fetches One URL, no fallbacks, no scraped mirrors: ``` https://www.wahed.com/hlal ``` It is `FUNDS[1].source_url` in `tools/ingest/lib/sources.mjs`, and `ingestOne()` in `tools/ingest/holdings.mjs` fetches `spec.source_url` and nothing else. The response URL is recorded as `source.resolved_url`, the HTTP status as `source.http_status`, and in `link_only` mode the row array stays empty (`parse.strategy: link_only`). `https://funds.wahedinvest.com/hlal` is not referenced anywhere in the code; it answered HTTP 403 `AccessDenied` when checked. What that page looked like on 2026-09-12: HTTP 200, a standalone "Holdings" label in its document list with no table under it, no ticker-level rows anywhere in the text, and the only holdings-shaped dates on the page belonging to semi-annual Schedule of Investments PDFs (newest "as of February 28, 2026") and a quarterly purification PDF ("as of 30th June 2026"). The static Fund Information block is stamped "Data as of 02-21-2024". Full evidence, including the one thing that still needs a real browser: [HLAL source check](hlal-source-check.md). Until a dated daily table is confirmed on the issuer page, HLAL history comes from the fund's SEC filings (Form N-PORT, quarter-end) and the daily alert is promised for SPUS only. We do not invent a source to keep a marketing line true. --- ## How parsing works (and how it fails) 1. `table_header`: find the row whose cells name the columns, then read the following rows *by column name*. Column order, added or removed columns, and renamed headers are tolerated. Recognised aliases include `StockTicker`/`Ticker`/`Symbol`, `SecurityName`/`Security`/`Description`, `Weightings`/`Weight (%)`/`% of net assets`, `NetAssets`, `SharesOutstanding`, `CreationUnits`. A merged "Date Account" header cell is corrected with a CUSIP anchor (recorded as `parse.offset`). 2. `column_order_fallback`: no usable header, so locate the date and the CUSIP-like cell and map the known SP Funds column order around them. Recorded in `parse.note`. 3. `none`: nothing usable, so the fund status is `parse_failed` with `rows: []`, and the run is `partial` or `failed`. We publish "we could not read it", never a guess. Pages with several tables (the SPUS page has a fund-details table, a short Top-10 block and the full holdings table) are parsed table by table and **the table with the most rows wins**, so a Top-10 block can never silently become our published holdings. Handling of the cases the task called out: | Case | Behaviour | |---|---| | Non-numeric cell (an em dash, `N/A`, blank, `Bad&Cell`) | value `null`, row flag `unparsed:`. Never 0, never invented. | | Missing cell or short row | If the CUSIP anchor gives a corrected offset, the row is read; otherwise the row is **skipped** as `misaligned_row` with a `raw` preview of at most 200 characters, because publishing a shifted row would corrupt ticker and number pairs. | | Summary/total rows (`colspan` "Total", ticker `CASH`) | skipped as `summary_row`, recorded in `skipped`. | | Ticker with an internal space or padding (` 601138 C1 `) | normalised to `601138 C1` and kept, because these are real holdings. Invalid tickers (for example `BAD&TICKER`) are skipped as `invalid_ticker`. | | Renamed ticker | Same CUSIP, new ticker vs. the previous snapshot gives `changed_ticker_from` on the row plus a `ticker_changed:::->` run warning, so the change log reports a rename instead of an exit plus an entry. | | Missing rows at the issuer | Row count drops by more than 20% (and the previous file had 20 rows or more), which records `row_count_dropped::->` as a run warning. The run still publishes. | | Mixed dates inside one table | `mixed_dates:…` warning; `as_of` is the latest date seen. | | Duplicate CUSIP under two tickers | `duplicate_cusip:…` warning; both rows are kept. | | Same fund page returns nothing | Fund status `empty_source` or `parse_failed`; a good file for the same date is **not** overwritten by a failed run (`--force` overrides). | `fund_totals` (net assets, shares outstanding, creation units) are taken as the majority value across the table's rows; rows carrying a different value are not published as totals. `weight_pct_sum` is recorded on every run, and only flagged when the table actually has 50 rows or more (a fixture or a partial table legitimately sums to far less than 100). --- ## CLI reference | Flag | Default | Meaning | |---|---|---| | `--out=DIR` | `/data/holdings` | Where snapshots are written. | | `--public-base=PATH` | `data/holdings` | Site-relative path recorded in `latest.json`. Change it if you change `--out`. | | `--date=YYYY-MM-DD` | today, `Asia/Kuala_Lumpur` | Snapshot date and file name. | | `--funds=SPUS,HLAL` | all registered | Restrict a run. | | `--previous-dir=DIR` | same as `--out` | Where the rename and diff baseline is read from. | | `--fixtures=DIR` | off | Read `.html` from DIR instead of fetching. Offline smoke test. | | `--hlal-mode=rows` | `link_only` | Needs `--hlal-permission-acknowledged`. | | `--timeout-ms=MS` | 30000 | Per-request timeout. | | `--strict` | off | Exit 1 instead of 2 on a partial run. | | `--force` | off | Allow a failed run to overwrite a good snapshot. | | `--json` | off | Machine-readable run summary on stdout (used by CI). | Exit codes: `0` ok, `2` partial (some fund failed, others published), `1` failed or bad usage. Idempotence is enforced in `writeSnapshot`: a file is written only when its bytes would change, so `node tools/ingest/holdings.mjs` twice in a row leaves the working tree clean. If an issuer really republishes the same date's data, `revision` becomes 2 with a `revised_on` date instead of a silent overwrite. The snapshot date is an input, not an accident of the clock, so the same `--date` on both runs is what makes the second run comparable to the first. The renderer takes its own flags: `--data-dir`, `--pointer`, `--out-dir`, `--base-url`, `--sitemap` or `--no-sitemap`, `--check` and `--json`. `--check` writes nothing and exits 1 when the pages or the sitemap block on disk no longer match the snapshot, which is the cheap way to catch a hand-edited generated page. The differ takes its own flags too: `--holdings-dir`, `--out`, `--public-base`, `--date`, `--from`, `--to`, `--min-delta`, `--prune-fixtures`, `--no-index`, `--dry-run`, `--stdout`, `--json`, `--strict` and `--help`. --- ## Turn on the schedule The workflow is committed at `.github/workflows/holdings-daily.yml`. It runs on weekdays at 21:00 Asia/Kuala_Lumpur, after the US trading day, with a 09:00 catch-up the following morning, and it can be started by hand with a date override. The YAML is not repeated here on purpose. GitHub Actions YAML is one of the file types the SafaStack site tooling refuses to write (`.html`, `.css`, `.js`, `.mjs`, `.json`, `.svg`, `.txt`, `.md`, `.xml`, `.webmanifest`), so the workflow file can only be created or edited from a repo shell. The copy to edit and paste back is parked as text at [`internal/holdings-daily.workflow.yml.txt`](../internal/holdings-daily.workflow.yml.txt), and that is the single copy to keep in step with the committed file. What the job does, in order: run the fixture tests, fetch both fund pages, check that a second run produces byte-identical files, validate against the schema, rebuild the change log from the newest snapshot (with `--prune-fixtures`, so the fixture-mode DEMO entry stays out of the index), render the static pages, then commit `data/holdings`, `data/changes` and the rendered pages in one commit. Both ingester steps take the same date. The dispatch `date` input reaches the ingest step through the `SNAPSHOT_DATE` env, and the byte-identity step uses the same env and the same `--date` argument, so the second run rewrites the date you asked for and `diff -rq` compares like with like. Before 2026-09-13 the idempotency step ran with no `--date`, so a manual dispatch for a past date let the second run write today's file: the check always warned about changed output and a stray dated snapshot could ride along in the commit. It needs no secrets, because both fund pages are public. Commits are made by the SafaStack bot with `info@safastack.com` as the identity, which is the company's single public contact address. ## Next - **A second dated snapshot.** The change log needs two live days to compare, so the next scheduled run (or a manual run: Actions, then Daily holdings ingest, then Run workflow) is what turns `data/changes` into a live document. Until it lands, the only published comparison is the fixture-mode one. - **Copy the workflow fix into the repo.** The idempotency date fix is in `internal/holdings-daily.workflow.yml.txt`; the committed `.github/workflows/holdings-daily.yml` still needs the copy, which also brings it the render and differ steps. It is a repo-shell step (task #42 fix, tracked in [GitHub, CI and hosting](github-and-deploy.md)). - **HLAL history from SEC EDGAR.** Quarter-end holdings from the Form N-PORT data sets are a public regulatory filing rather than scraped page content, which makes them the license-clean way to give HLAL a history instead of a link. - **Wider coverage.** IGDA, ISDW, SPSK and MNZL each need their own terms review before they are registered in `lib/sources.mjs`. Sources are not transferable, so one fund's position says nothing about the next. - **The developer feed.** Versioned, CORS-enabled JSON with a one-page schema doc and a status endpoint, which ships once the issuers' written permission covers redistribution.