# SafaStack holdings data schema (v1.0.0)

Two files are published by the ingester (`tools/ingest/holdings.mjs`). Both are plain static files, so
they can be linked, fetched from a page, or handed to a developer without a build step or a server.

| File | What it is | Schema |
|---|---|---|
| `data/holdings/YYYY-MM-DD.json` | One dated snapshot: every fund, every row, source URL, as-of date | [`data/holdings/schema.json`](../data/holdings/schema.json) |
| `data/holdings/latest.json` | Pointer at the newest snapshot: path, its byte hash, and a per-fund summary | [`data/holdings/schema-latest.json`](../data/holdings/schema-latest.json) |

Validate any file (or the whole directory) with:

```bash
node tools/ingest/validate.mjs data/holdings
node tools/ingest/validate.mjs data/holdings/latest.json
```

**This is data, not investment advice.** Every file carries its own `disclaimer` and `attribution`
strings so the source and the not-advice line travel with the data. We never claim a fund is
"certified" or "halal": we report what the issuer publishes as held. See
[Holdings pipeline](holdings-pipeline.md) for the per-fund rights posture.

---

## Determinism contract

Two runs over the same source bytes must produce byte-identical files. That is a test, not a hope
(`tools/ingest/holdings.test.mjs`, and the second-run check in the CI workflow).

1. **No wall-clock timestamps.** Only dates: `date`, `ingested_on`, `retrieved_on`. There is no
   `generated_at`. Two runs on the same day cannot differ because of the clock.
2. **Key order is inserted, never inherited.** Rows are built by `canonicalRow()`, snapshots by
   `buildSnapshot()`, sources by `makeSource()`. Nothing is spread from a parsed object.
3. **Fixed formatting:** 2-space indent, LF line endings, exactly one trailing newline
   (`canonicalJson()`).
4. **Numbers are normalised** before serialisation: shares to 3dp, price to 4dp, market value to
   2dp, weights to 4dp, through `roundTo()` (half away from zero, `toFixed`-based so float noise such
   as `0.30000000000000004` becomes `0.3`).
5. **Arrays are sorted** with code-unit comparisons, never `localeCompare`: rows by ticker, then
   CUSIP, then name. Warnings are sorted lexicographically and dates ascending.
6. **Writing is idempotent.** A file is only rewritten when its bytes would change, and the
   previous-day baseline used for rename detection explicitly excludes the date being written.
   A re-run on an unchanged source therefore rewrites nothing.
7. **Backstop:** if an issuer changes its page mid-day and the bytes *would* change, the file is
   rewritten with `revision` incremented and a `revised_on` date, never silently.

Honest caveat: number formatting relies on V8's `Number.prototype.toFixed`, so byte-identity is
guaranteed for a pinned Node major version. The workflow pins one version, 22 today.

---

## Dated snapshot: `data/holdings/YYYY-MM-DD.json`

| Field | Type | Notes |
|---|---|---|
| `schema_version` | string | `"1.0.0"`. Bumped only for breaking changes. |
| `dataset` | string | `"safastack.holdings.daily"`. |
| `date` | string | Snapshot date in `Asia/Kuala_Lumpur`, equals the file-name stem. |
| `ingested_on` | string | Date the run happened (a date, never a time). |
| `timezone` | string | `"Asia/Kuala_Lumpur"`. |
| `producer` | string | `safastack-ingest-holdings/1.0.0`. |
| `revision` | integer ≥ 1 | `1` on first write; increments only if the issuer changes the same date's data. |
| `revised_on` | string | Present only when `revision > 1`. |
| `run` | object | `status`, `funds_total`, `funds_ok`, `funds_link_only`, `funds_failed`, `warnings[]`. |
| `run.status` | enum | `ok` (every fund ok or link-only by design), `partial` (some fund failed), `link_only` (nothing publishable by design), `failed` (nothing usable). |
| `funds` | array | One entry per fund, in registry order. |
| `disclaimer` | string | Data-not-advice text, inside the file. |
| `attribution` | string | Issuer attribution, inside the file. |
| `integrity` | object | `{ "algorithm": "sha256", "content_sha256": … }`, the hash of the canonical document *without* the `integrity` block, so it recomputes. |

### `funds[]`

| Field | Type | Notes |
|---|---|---|
| `fund` | string | Ticker of the *fund*: `SPUS`, `HLAL`. |
| `fund_name` | string | Full fund name as the issuer states it. |
| `issuer` | string | `SP Funds`, `Wahed Invest LLC`, … |
| `status` | enum | `ok`, `link_only`, `source_unavailable`, `parse_failed`, `empty_source`. |
| `publishing_mode` | enum | `rows` (holdings published here), `link_only` (we only record reachability and link out). |
| `rights` | object | `rating` (`conditional_go`/`no_go`), `review_doc`, `note`, `attribution_required`: our rights posture, copied into the file. |
| `as_of` | string \| null | The holdings date the issuer published, parsed from the table (for example `2026-09-09`). Not the fetch date. |
| `source` | object | `url` (canonical page), `resolved_url` (after redirects), `type`, `retrieved_on`, `mode` (`live`/`fixture`), `http_status`, optional `note`. |
| `parse` | object | `strategy` (`table_header`, `column_order_fallback`, `link_only`, `none`), `offset`, `columns[]`, `rows_seen`, `rows_skipped`, `rows_kept`, optional `holdings_table_found`, `rows_detected`, `note`. |
| `fund_totals` | object \| null | `net_assets`, `shares_outstanding`, `creation_units` (the fund-level columns that repeat on every row). |
| `counts` | object | `rows`, `weight_pct_sum` (sum of published weights, 4dp, a sanity check that lands near 100 on a full table). |
| `warnings` | string[] | Machine-readable, prefixed: `duplicate_cusip:…`, `mixed_dates:…`, `account_mismatch:…`, `weight_sum_out_of_band:…`, `rows_withheld:…`, `fetch_or_parse_failed:…`. |
| `skipped` | object[] | `{ row, reason, raw }` for every row we could not publish. `reason` is one of `missing_ticker`, `invalid_ticker`, `summary_row`, `empty_row`, `misaligned_row`, `duplicate_row`. `raw` is a text preview of at most 200 characters. |
| `rows` | array | Sorted by ticker, then CUSIP, then name. Empty when `publishing_mode` is `link_only`. |
| `note` | string \| null | Present when a fund failed: the error we recorded. |

### `funds[].rows[]`

| Field | Type | Notes |
|---|---|---|
| `ticker` | string | As published, uppercased. Venue-suffixed names are kept intact (`601138 C1`, `YZJSGD SP`). |
| `name` | string \| null | Security name as published. |
| `cusip` | string \| null | 6 to 9 alphanumerics. The stable identity anchor for rename detection across days. |
| `shares` | number \| null | Rounded to 3dp. |
| `price` | number \| null | Rounded to 4dp. |
| `market_value` | number \| null | Rounded to 2dp. |
| `weight_pct` | number \| null | Percent, so `13.84` means 13.84%. Outside the 0 to 100 range it is null plus a flag. |
| `changed_ticker_from` | string | Present when this CUSIP had a different ticker in the previous snapshot, which is a rename rather than an exit plus an entry. |
| `flags` | string[] | Optional. `unparsed:<field>`, `out_of_range:weight_pct`, `missing_name`, `missing_date`. Present means "the issuer's cell was not usable"; the value is null, never guessed. |

---

## Latest pointer: `data/holdings/latest.json`

Small by design: a page can read it for the header (date, per-fund as-of, row counts) and then fetch
the dated file for rows.

| Field | Type | Notes |
|---|---|---|
| `dataset` | string | `"safastack.holdings.latest"`. |
| `date` | string | Date of the snapshot this points at. |
| `path` | string | Site-root-relative path: `data/holdings/YYYY-MM-DD.json` (never root-absolute, so it works on any host). |
| `content_sha256` | string | SHA-256 of that file's exact bytes. The validator re-reads the file and compares. |
| `revision` | integer | Mirrors the snapshot's revision. |
| `run` | object | Same shape as the snapshot's `run`. |
| `funds[]` | array | `fund`, `status`, `publishing_mode`, `as_of`, `rows`, `source_url`. |
| `disclaimer`, `attribution` | string | Same text as the snapshot. |

---

## Consumption notes for builders

- Dated files are immutable in practice: same URL, same bytes, forever. A same-day `revision` bump is
  the only exception, and it is visible in the file.
- `latest.json` is the only file whose URL is stable across days. Anything you cache should key on
  `content_sha256`.
- No CORS headers are set by a static host by default. If you need a browser-side fetch from another
  origin, host the file or ask us for the versioned feed (roadmap item).
- The terse rule for redistribution: **rows from `publishing_mode: "rows"` funds are attributed
  source data; rows from a `link_only` fund do not exist in our files at all.**

## Versioning

`schema_version` is `1.0.0`. It changes only for a breaking change (a renamed or removed field, or a
changed meaning of a value). Additive fields keep the same major and are noted here. Consumers should
ignore unknown fields and never key on field order, and neither should we, which is why the publisher
fixes its own key order.