# MacroVision: historical SIPP sample and precision audit (v84)

## Household and measurement follow-up (v85)

The [v85 household/food audit](HOUSEHOLD_FOOD_VALIDATION_V85.md) now defines a fixed index and any-member household exposure, and verifies answer-conditioned food coding. It retains 4,527 households with consistent baseline/outcome food codes, including 322 exposed households. No liquidity-hardship effect is fitted in v85. The person-candidate counts and precision scenarios below remain the v84 historical planning exercise and do not automatically describe the new household domain.

## Decision

The historical extension identifies **11,253 distinct candidate person IDs and 468 economic job separations** across five origin panels. There are **zero duplicate person-window keys** under the fixed assignment rule. This substantially expands the event inventory, but **the current design still does not meet the planned 5-percentage-point precision target** under the working assumptions examined.

Two validity issues must be resolved before fitting historical hardship effects: older food-security allocation and flag errors, and households containing both exposed people and comparison people. **No new food-security outcomes were inspected, no pooled empirical coefficient was fitted, and no model calibration was changed in this stage.** The previous v82/v83 effect estimates remain historical, inconclusive results.

## Fixed sample assignment

[The audit plan and code were committed before the new counts](https://github.com/idop1/macrovision/commit/45828daff04035a2b7b3c584659bfb74606dd997). Each origin panel P contributes one window only:

- Baseline reference year P-1, from release P.
- Separation reference year P, from release P+1.
- Potential outcome reference year P+1, represented by endpoint-release P+2 longitudinal weights. The outcome values and outcome-year residence are **not read** in this audit.

Origin panels 2018, 2020, 2021, 2022 and 2023 are used. The 2019-origin panel was terminated after its first wave and cannot support the required window. The 2024-origin panel's required three-year endpoint is outside the verified 2018-2025 public release inventory. Previously analyzed v82/v83 records are not appended as extra observations. All pool candidates are assembled afresh under one assignment per origin panel.

| Origin panel | Baseline / event / potential outcome | Candidate people | Economic separations | Baseline candidate households |
|---|---|---:|---:|---:|
| 2018 | 2017 / 2018 / 2019 | 5,235 | 184 | 3,870 |
| 2020 | 2019 / 2020 / 2021 | 1,692 | 156 | 1,253 |
| 2021 | 2020 / 2021 / 2022 | 1,168 | 37 | 855 |
| 2022 | 2021 / 2022 / 2023 | 2,035 | 65 | 1,487 |
| 2023 | 2022 / 2023 / 2024 | 1,123 | 26 | 824 |
| **Total** | One window per origin | **11,253** | **468** | **8,289** |

These are **candidate counts**, not a final hardship sample or national population estimates. Final outcome validity, outcome residence, net-wealth/debt and expanded-regression covariate completeness can reduce them. The number of distinct households with at least one candidate separation is **455**, so 468 person separations must not be described as 468 independent household shocks.

Person identity is checked using `SSUID+PNUM`; household grouping uses baseline `SSUID+ERESIDENCEID`. We preserve identifier strings and assert no reused person key across origin windows. This cannot detect an actual person re-enrolled under a different scrambled identifier. Disjoint sampled IDs also do not make overlapping calendar periods independent economic episodes.

## Eligibility retained from the earlier design

People have baseline `TAGE` 25-60 and December `RMESR=1 or 2`. Baseline residence must remain at the interview residence for all 12 observed months. Prior economic separations and ambiguous observed endings are excluded. All 12 baseline monthly household-income observations must be present, with a positive annual total. Six checking/savings ownership shares are aggregated once per household member at baseline year-end; unknown owned balances are not arbitrarily changed to zero. Household-level quantities copied to members are not summed repeatedly.

The event year must have 12 observations. Economic separation uses employer-job ending reasons 1, 2, 3 or 6, across all seven job slots; December endings require a zero continuation flag. Unknown reasons and ambiguous December continuation exclude the person. Comparison people have no observed employer separation of any reason in that year. Economic separation need not imply unemployment, permanent displacement or automation exposure.

The low-deposit threshold remains deposits below annual baseline household income divided by 12. It is **income coverage**, not months of expenditure or total safe liquid wealth. All extraction columns exclude `RFOODS`, `RFOODR`, food items and their response flags. Source archives contain the full survey, but those columns are not materialized or used by this audit.

## Replicate-weight-informed precision planning

For each origin window we use its endpoint `FINYR3` and **240 matching longitudinal replicate weights**. Reference periods are verified; `REPWGT0` must equal the point weight. Legacy `PANEL` is normalized to `SPANEL`; column case is normalized without losing identifier zeros. We do not substitute monthly weights or align replicate indices across release years as if joint cross-vintage replicates were supplied.

The planning contrast is:

> Change for event/low deposits minus event/high deposits, minus control/low deposits plus control/high deposits.

Within each of the four cells, point and replicate weights are normalized. Let a₀ᵢ and aᵣᵢ be the signed point and replicate influence coefficients for person i. Under a working null with independent baseline-household changes of common variance σ², the expected Fay variance is:

```math
E[\widehat{V}] = \frac{\sigma^2}{240(0.5)^2}
\sum_{r=1}^{240}\sum_h
\left[\sum_{i\in h}(a_{ri}-a_{0i})\right]^2.
```

We set σ² = 2p(1-p)(1-ρ), with assumed prevalence p=0.10 or 0.30 and pre/post correlation ρ=0 or 0.5. These are declared planning assumptions, not observed outcome prevalences. The formula sums within households before squaring, so copied household outcomes are not treated as independent person observations.

This is **conditional, working-model variance planning**, not an actual survey-power estimate or the variance of the expanded adjusted regression. It does not establish the outcome distribution, household stability after baseline, geographic dependence, confounding or macroeconomic transportability. Anticipated 80%-power detectable differences use the conventional normal approximation `(1.96+0.841621)×SE`.

The planned aggregate gives each origin-cohort contrast equal weight. With conditional sampling independence, its variance coefficient is ΣBₚ/5². A perfect positive error-correlation calculation uses `(Σ√Bₚ/5)²`. The latter is a dependence stress bound for this working model, not an estimated calendar correlation or a bound on all systematic bias.

| Assumed prevalence | Assumed pre/post correlation | Expected 95% half-width, independence benchmark | Approximate 80%-power detectable difference |
|---|---:|---:|---:|
| 10% | 0 | 11.46 pp | 16.38 pp |
| 10% | 0.5 | **8.10 pp** | **11.58 pp** |
| 30% | 0 | 17.51 pp | 25.02 pp |
| 30% | 0.5 | 12.38 pp | 17.69 pp |

All four miss both the **5-point half-width** and **5-point detectable-difference** targets. Under the perfect-positive-correlation stress case, the half-width range is **17.16-37.07 points**, emphasizing why calendar overlap must not be ignored.

As an additional design sensitivity calculated **after the blind counts**, optimally weighting cohort contrasts by inverse planning coefficients would give variance coefficient `1/Σ(1/Bₚ)=0.0119685681`, under the same conditional independence and common-variance assumptions. Even its most favorable scenario has half-width **6.43 points** and detectable difference **9.20 points**. These weights were not used to fit an effect, are not a new registered estimand, and do not repair measurement or identification problems. This is not a claim that every possible SIPP estimator is incapable of detecting a 5-point association: a revised household design, better measures or prespecified covariate adjustment require their own precision assessment.

## Why a larger N does not yet validate the test

### Older food-security measurement and allocation flags

Census documents a food-security hot-deck pathing/flag error in the **2018-2022 releases**, as well as the 2014 panel: some allocated cases were labeled outside the universe (flag 0) or logically allocated (flag 3), rather than hot-deck-imputed (flag 2). The note also describes errors in how valid donors were used. Later releases have additional allocation changes.

Therefore a flag of 0 on a conditional item cannot, by itself, certify an un-imputed structural skip in the historical releases. Before pooling hardship outcomes, reconstruct each food item's response universe from the preceding answers, check response values and flags against it, and verify scoring against official recodes. The v82 baseline uses the 2022 release, so its nominal flag-based sensitivity must not be presented as a certified imputation-free sample without this additional check. **v84 does not re-estimate or replace the earlier coefficients.** The v83 replication baseline is the 2023 release and is not within that older 2018-2022 pathing-error range; its own allocation and selection limitations remain as documented.

### Household exposure and comparison-group contamination

Among **455 candidate households with at least one separated candidate**, **177** also contain eligible comparison people who did not separate themselves. These are counts within the candidate population; events among other household members are not fully enumerated here. A person's own job status does not establish that their household had no job-loss shock. Household hardship and resources are shared, so an exposed person's partner may also appear in the person-level comparison group.

This does not establish the size or direction of bias, and replicate weights do not solve the exposure definition. Before a new effect test, choose a household-level shock definition and a baseline index/reference-person rule fixed without looking at outcomes. Define controls with no qualifying household-member shock, track membership changes, then reassess sample sizes and weighting. Do not count multiple family members as independent confirmations of a mechanism.

### Historical heterogeneity

The event years include pre-pandemic 2018, pandemic 2020, and later 2021-2023. Support programs, labor demand, interview conditions and selection differ. The 2021 release also has documented pandemic-era safety-net measurement concerns. An equal-cohort historical contrast is not a universal automation parameter. Record cohort heterogeneity and calendar dependence explicitly before considering a common coefficient.

## Next concrete research gate

1. **Household design:** freeze the baseline index-person/household definition, household-member separation rule, control rule and treatment of moves. Recount the four exposure/resource cells and household shocks before outcomes.
2. **Food crosswalk:** implement and test answer-conditioned universes and strict flag/value checks, then audit only classification, recode agreement and missingness by release, without comparing outcome effects by liquidity.
3. **Precision:** use the revised design and compatible weights to assess useful effect sizes. If a 5-point target remains implausible, report that limitation and select a genuinely larger or better-measured permitted source, rather than changing thresholds until significance appears.
4. Only then fix a new estimation plan. Contractual debt-service and actual consumption expenditure remain separate measurement requirements for H2 and the full consumption thesis. PSID remains excluded from this AI-assisted workflow.

## Reproduction and checks

Repository files:

- `data/metadata/sipp_pool_plan_v84.json`: fixed stage scope and cohort assignment.
- `data/metadata/sipp_pool_crosswalk_v84.json`: selected-field schema labels/types and source hashes for seven releases, with legacy naming notes; not a completed food harmonization.
- `scripts/sipp_pool_audit_v84.py`: extraction, candidate assignment, duplicate-key checks, four-cell influence coefficients and household-level expected Fay variance.
- `calibration/test_sipp_pool_audit_v84.py`: synthetic tests for outcome blindness, cell normalization, hand-calculated household variance, duplicate persons, legacy identifiers and cross-vintage variance combination.
- `data/processed/sipp_pool_audit_v84.json`: aggregate cohort counts, planning scenarios, code/plan/source hashes and closed research gates.

Place official public-use `puYYYY_csv.zip` files for releases **2018-2024** and endpoint `lgtwgtYYYYyr3_csv.zip` / `lgtrwYYYYyr3_csv.zip` files for **2020, 2022, 2023, 2024, 2025** outside the checkout. The tool reads no 2025 primary outcome file. Source URLs follow the Census release directories listed below. Install `requirements-research.txt`, then run:

```bash
python scripts/sipp_pool_audit_v84.py --data-dir /outside/sipp --output /outside/sipp_pool_audit_v84.json
python -m unittest discover -s calibration -p 'test_sipp_pool_audit_v84.py' -v
```

Selected-column caches are tied to archive hashes and column lists, outside the deployed repository. Only aggregates and metadata are committed or served. No microdata or person identifiers are published, and no model formulas or calibrated parameters are changed.

## Official sources

- [SIPP datasets and release inventory](https://www.census.gov/programs-surveys/sipp/data/datasets.html)
- [2018 release](https://www.census.gov/programs-surveys/sipp/data/datasets/2018-data/2018.html)
- [2020 release and three-year longitudinal files](https://www.census.gov/programs-surveys/sipp/data/datasets/2020-data/2020.html)
- [2021 release and pandemic measurement concerns](https://www.census.gov/programs-surveys/sipp/data/datasets/2021-data/2021.html)
- [Historical food hot-deck allocation error](https://www.census.gov/programs-surveys/sipp/tech-documentation/user-notes/2020-usernotes/2020-hot-deck-alloc-error.html)
- [2020 user's guide, Fay variance and replicate weights](https://www2.census.gov/programs-surveys/sipp/tech-documentation/methodology/2020_SIPP_Users_Guide_SEP23.pdf)
- [2022 user's guide](https://www2.census.gov/programs-surveys/sipp/tech-documentation/methodology/2022_SIPP_Users_Guide_SEP23.pdf)
- [2023 source and accuracy statement, including the discontinued 2019 panel](https://www2.census.gov/programs-surveys/sipp/tech-documentation/source-accuracy-statements/2023/SIPP-2023-SA-28-JUN24.pdf)
- [v83 results and preceding source notes](EMPIRICAL_REPLICATION_V83.md)