# Multi-Model Cross-Validation for Zi Wei Dou Shu AI Interpretation: A Reproducible Benchmark and Transparency Framework

**Authors:** Starnum Research Team
**Affiliation:** starnum.com.tw
**Date:** April 2026
**Version:** v1.0
**arXiv categories (intended):** cs.AI, cs.CY
**License:** CC BY 4.0
**Preprint URL:** https://starnum.com.tw/papers/
**Source:** https://starnum.com.tw

---

## Abstract

We present the first publicly documented AI benchmark and reproducible evaluation framework for Zi Wei Dou Shu (紫微斗數, Purple Star Astrology) interpretation. Traditional Chinese astrology poses unique challenges for large language models (LLMs): interpretive precision requires integrating hundreds of interdependent rules from a corpus of classical texts, while evaluation demands domain expertise. Our system employs a multi-model cross-validation architecture in which four frontier AI models (Claude Sonnet, GPT-4o, Gemini 2.5 Flash, xAI Grok-3) independently generate and validate astrological analyses. We introduce *starnum-bench*, a sanitized dataset of 37 fixed charts with six-dimensional scoring rubrics (D1–D6), and demonstrate that ensemble cross-validation achieves higher interpretive consistency and lower factual error rate than single-model baselines. We release our scoring rubric, evaluation script, and baseline results publicly. This work also establishes a transparency framework—machine-readable quarterly reports, commit-signed audit trails, and a Responsible AI disclosure—designed to meet emerging AI governance standards.

**Keywords:** astrology AI, multi-model validation, Chinese cultural AI, benchmark, transparency, LLM evaluation

---

## 1. Introduction

### 1.1 The Zi Wei Dou Shu Challenge

Zi Wei Dou Shu (ZWDS) is a classical Chinese divination system with an estimated 1,500-year history. A complete reading requires synthesizing:

- Fourteen major stars (主星) and over 100 minor stars
- Twelve palaces (宮位) representing life domains
- Four transformation rules (四化: 祿/權/科/忌) that vary by year stem
- Interaction patterns between palaces (三方四正)
- School-specific interpretive traditions (派別)

This combinatorial complexity—with 14! possible star arrangements, palace-specific weightings, and school-specific rules—creates a domain where LLM hallucination rates are unusually high. A model confident in incorrect four-transformation assignments can produce analyses that are internally consistent yet fundamentally incorrect.

### 1.2 Problem Statement

No public benchmark exists for evaluating AI performance on ZWDS interpretation. Without a reproducible evaluation framework, claims of "AI-powered astrology" are unverifiable and potentially misleading. This gap motivates our work.

### 1.3 Contributions

This paper makes four contributions:

1. **starnum-bench**: The first public, sanitized benchmark dataset for ZWDS AI evaluation (37 charts, six-dimensional rubric)
2. **Multi-model cross-validation architecture**: A practical ensemble method that leverages disagreement between frontier LLMs as a quality signal
3. **Transparency framework**: Machine-readable quarterly reports with commit-anchored tamper evidence
4. **Reproducible baseline**: Public scoring script and April 2026 baseline results

---

## 2. Related Work

### 2.1 LLM Evaluation Benchmarks

Benchmark development for specialized domains has established key principles: MMLU (Hendrycks et al., 2021) demonstrated multi-subject evaluation at scale; HumanEval (Chen et al., 2021) showed the importance of executable verification for code. BIG-Bench (Srivastava et al., 2022) highlighted the value of tasks requiring genuine reasoning over memorized facts.

For cultural and humanities domains, evaluation is harder: CMMLU (Li et al., 2023) addressed Chinese-language cultural knowledge, and C-Eval (Huang et al., 2023) extended this to Chinese professional domains. Our work addresses a gap in East Asian cultural AI evaluation—traditional divination systems that require both factual accuracy and interpretive coherence.

### 2.2 Multi-Model Ensemble Methods

Ensemble approaches to LLM reliability are well-established. Self-consistency (Wang et al., 2023) uses majority voting across sampled outputs. Mixture-of-agents (MoA) architectures (Wu et al., 2024) use multiple models as both generators and aggregators. Our cross-validation approach differs: we use disagreement (not agreement) as the primary quality signal, flagging outputs where models diverge for human review.

### 2.3 AI Transparency Standards

AI transparency requirements are emerging from multiple directions: the EU AI Act (2024) mandates documentation for high-risk AI systems; NIST AI RMF (2023) provides a risk management framework; Anthropic's Constitutional AI (Bai et al., 2022) and OpenAI's model cards (Mitchell et al., 2019) establish industry precedents for capability disclosure.

Traditional knowledge AI systems present additional concerns: cultural appropriation, deterministic framing of probabilistic traditions, and the tension between accessibility and accuracy. We address these in our ethics framework (Section 5).

---

## 3. Methodology

### 3.1 System Architecture

Our system comprises five layers:

```
User Input (birth data)
    ↓
[Layer A] Knowledge Base (KB)
    docs/kb/zwds/ — 88 files, classical sources
    docs/kb/numerology/ — 141 files
    ↓
[Layer B] Evidence Retrieval
    GraphRAG (Neo4j, 343 nodes / 681 edges)
    Vector Search (Qdrant, 12,378 chunks, BGE-M3 1024-dim)
    Rule Engine (kb-rules-engine.json, 127 rules)
    ↓
[Layer C] Multi-Model Generation
    Four frontier LLMs generate independent analyses
    ↓
[Layer D] Cross-Validation
    Disagreement detection → human review queue
    Logic validation (validate-astrology-logic.js)
    ↓
[Layer E] Output & Transparency
    Structured HTML output (22 blocks)
    Quarterly transparency reports
    Commit-anchored audit trail
```

### 3.2 Knowledge Base Construction

Our KB was constructed through systematic crawling of classical ZWDS sources, with strict quality controls:

- **Source selection**: Priority to 陸斌兆 (Lu Binzhao) school materials, annotated where other schools are cited
- **Quality threshold**: Minimum 300 words effective content per KB entry
- **School tagging**: Non-Lu-Binzhao content marked `⚠️ 非陸斌兆派`
- **Deduplication**: Semantic deduplication via Jaccard similarity (threshold: 70%)
- **Contradiction detection**: Automated detection of conflicting rules across entries

The resulting KB contains 88 ZWDS files and 141 numerology files, organized by category (rules / fundamentals / stars / palaces / sihua / patterns / advanced).

### 3.3 Evidence Retrieval Pipeline

For each reading request, the system executes three parallel retrieval paths:

1. **Graph traversal** (Neo4j): Starting from identified stars and palaces, expand via 681 edges to retrieve related patterns and rules
2. **Vector search** (Qdrant): Top-K semantic search across 12,378 chunks using BGE-M3 embeddings
3. **Rule engine lookup**: Direct lookup of applicable transformation rules from kb-rules-engine.json

Results are fused by evidence-fusion.js, which deduplicates, ranks by relevance, and checks for conflicts via detect-evidence-conflict.js.

### 3.4 Multi-Model Cross-Validation

Each analysis is independently generated by four frontier models:

| Model | Role | Input Limit |
|-------|------|-------------|
| Claude Sonnet (claude-sonnet-4-6) | Primary generator | 30K tokens |
| GPT-4o | Cross-validator | 30K tokens |
| Gemini 2.5 Flash | Cross-validator | 15K tokens |
| xAI Grok-3 | Cross-validator | 30K tokens |

**Disagreement protocol**: When models diverge on factual claims (four-transformation assignments, star positions, palace interpretations), the output is flagged for human review. Disagreement threshold: any factual claim where < 3/4 models agree.

**Voting format**: FIX / SKIP / DEFER — three-value voting reduces false positives from stylistic differences.

### 3.5 Output Specification

Each completed analysis contains 22 structured blocks in a standardized HTML format:

1. Traceable summary (可追溯摘要)
2. Core pattern identification (核心格局)
3. Ming palace analysis (命宮)
4. Career palace (事業宮)
5. Wealth palace (財帛宮)
6. Relationships palace (夫妻宮)
7. Health palace (疾厄宮)
8. [blocks 8-22: remaining palaces and synthesis sections]

Output is validated against format rules by validate-output-format.js (class whitelist, required bh-tags, ≤3 box elements).

---

## 4. Benchmark & Results

### 4.1 starnum-bench Dataset

**Dataset composition:**
- 37 fixed chart configurations (Golden Test Suite)
- Sanitized: no personally identifiable information (PII)
- Coverage: 14 main stars, 12 palace positions, multiple year stems
- Source: Supabase charts table, sampled to ensure diversity

**Dataset statistics:**

| Attribute | Count |
|-----------|-------|
| Total charts | 37 |
| Unique main stars | 9 |
| Charts with birth time | 28 |
| Charts without birth time | 9 |
| Year stems covered | 8 |

### 4.2 Scoring Rubric (D1–D6)

Each analysis is scored on six dimensions:

| Dimension | Description | Weight |
|-----------|-------------|--------|
| D1: Factual Accuracy | Correct star positions, four-transformations | 30% |
| D2: Rule Completeness | All applicable rules applied | 20% |
| D3: Interpretive Depth | Beyond surface-level description | 20% |
| D4: Internal Consistency | No contradictions within analysis | 15% |
| D5: Hook Quality | Engaging, specific, actionable | 10% |
| D6: Format Compliance | Correct HTML structure, all 22 blocks | 5% |

**Scoring scale:** 0–100 per dimension. Weighted sum = final score.

### 4.3 Baseline Results (April 2026)

Our current system achieves the following baseline scores on the 37-chart eval set:

| Dimension | Score | Notes |
|-----------|-------|-------|
| D1: Factual Accuracy | 87.3 | Four-transformation errors remain primary failure mode |
| D2: Rule Completeness | 82.1 | Minor stars coverage is the main gap |
| D3: Interpretive Depth | 79.8 | Improves with more KB evidence |
| D4: Internal Consistency | 91.2 | Cross-validation catches most contradictions |
| D5: Hook Quality | 84.6 | Highest variance; style-dependent |
| D6: Format Compliance | 96.4 | Near-perfect after validate-output-format.js |
| **Overall (weighted)** | **85.4** | Baseline established April 2026 |

**Cross-validation effect:** Single-model baseline (Claude only) achieves 79.2 overall. Multi-model cross-validation raises this to 85.4 (+6.2 points), primarily through D1 and D4 improvement.

### 4.4 Reproducibility

All results are reproducible using the public benchmark package:

```bash
git clone https://starnum.com.tw/benchmark/public/
cd public
node evaluate.js --input dataset/classical_cases.json --rubric evaluation/scoring_rubric.json
```

Expected output matches `results/baseline-2026-04.json`.

---

## 5. Ethics & Limitations

### 5.1 Responsible AI Principles

We operate under four core ethical principles for traditional knowledge AI:

1. **User autonomy (用戶自主)**: Analyses are advisory, not deterministic. All outputs include explicit disclaimers that ZWDS interpretations reflect classical traditions, not empirical predictions.

2. **Avoid determinism (避免決定論)**: We do not make predictions about death, severe illness, or fixed life outcomes. Our system includes hard-coded content filters for these categories.

3. **Data privacy (資料隱私)**: User birth data is stored encrypted. Free tier data is deleted after 1 month; paid tier after 6 months. VIP data is retained until explicitly deleted by user.

4. **Cultural respect (文化尊重)**: We distinguish between schools of ZWDS practice (派別) and clearly annotate when citing non-standard interpretations. We do not claim our AI system to be equivalent to expert human practitioners.

### 5.2 Known Limitations

**Technical limitations:**
- Four-transformation assignments vary by school; our system follows 陸斌兆 school which may conflict with users familiar with other traditions
- Charts without birth time (無時盤) have significantly lower confidence; this is disclosed in output
- LLM temperature and sampling variation means identical inputs may produce slightly different outputs

**Scope limitations:**
- Current KB focuses on classical (pre-modern) sources; contemporary interpretations may differ
- Evaluation covers 37 charts which, while diverse, cannot represent the full combinatorial space
- Numerology integration (生命靈數) uses a separate KB and is evaluated separately

**Ethical limitations:**
- We cannot fully eliminate risk of users treating AI analyses as authoritative predictions
- Cultural appropriation concerns exist; we address these through school attribution and disclaimer requirements
- Our evaluators are LLMs, not certified ZWDS practitioners; human expert validation is ongoing

### 5.3 Conflict of Interest

This research is conducted by the operators of starnum.com.tw, a commercial ZWDS interpretation service. We acknowledge this creates an inherent conflict of interest in evaluation. Mitigations: our eval-set and scoring rubric are public for independent verification; our cross-validation protocol uses models from competing providers.

---

## 6. Conclusion

We have presented the first public AI benchmark for Zi Wei Dou Shu interpretation, together with a multi-model cross-validation architecture that demonstrably improves output quality over single-model baselines. Our transparency framework—machine-readable quarterly reports, commit-anchored audit trails, and a Responsible AI page—provides a governance model for cultural AI systems that may be useful to other practitioners in this space.

Our key findings:
1. Multi-model cross-validation provides a +6.2 point quality improvement over single-model approaches on our six-dimensional rubric
2. Factual accuracy (D1) and internal consistency (D4) are the primary beneficiaries of ensemble validation
3. Machine-readable transparency infrastructure (JSON reports + Git-signed commits) is achievable at low cost and provides meaningful accountability

**Future work:** Expert human validation by certified ZWDS practitioners; expansion of the benchmark to 200+ charts; integration of formal uncertainty quantification; submission to cs.AI arXiv once academic co-author collaboration is established.

---

## References

Bai, Y., et al. (2022). Constitutional AI: Harmlessness from AI Feedback. *arXiv:2212.08073*.

Chen, M., et al. (2021). Evaluating Large Language Models Trained on Code. *arXiv:2107.03374*.

Hendrycks, D., et al. (2021). Measuring Massive Multitask Language Understanding. *arXiv:2009.03300*.

Huang, Y., et al. (2023). C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models. *arXiv:2305.08322*.

Li, H., et al. (2023). CMMLU: Measuring Massive Multitask Language Understanding in Chinese. *arXiv:2306.09212*.

Mitchell, M., et al. (2019). Model Cards for Model Reporting. *Proceedings of FAT 2019*.

NIST. (2023). AI Risk Management Framework (AI RMF 1.0). National Institute of Standards and Technology.

Srivastava, A., et al. (2022). Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models. *arXiv:2206.04615*.

Wang, X., et al. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. *arXiv:2203.11171*.

Wu, J., et al. (2024). Mixture-of-Agents Enhances Large Language Model Capabilities. *arXiv:2406.04692*.

---

## BibTeX Citation

```bibtex
@techreport{starnum2026preprint,
  title     = {Multi-Model Cross-Validation for Zi Wei Dou Shu AI Interpretation:
               A Reproducible Benchmark and Transparency Framework},
  author    = {{Starnum Research Team}},
  year      = {2026},
  month     = {April},
  institution = {starnum.com.tw},
  type      = {Preprint},
  url       = {https://starnum.com.tw/papers/},
  note      = {Version 1.0. Under review for arXiv cs.AI + cs.CY submission.}
}
```

---

## Appendix A: System Specifications

| Component | Specification |
|-----------|---------------|
| KB size (ZWDS) | 88 files |
| KB size (Numerology) | 141 files |
| Graph nodes | 343 |
| Graph edges | 681 |
| Vector chunks | 12,378 |
| Embedding model | BGE-M3 (1024-dim) |
| Reranker | bge-reranker-base |
| GPU | GTX 1080 Ti 11GB |
| LLM models used | Claude Sonnet, GPT-4o, Gemini 2.5 Flash, Grok-3 |
| Articles produced | See data/articles.json |
| Languages | 10 (zh-TW, zh-CN, en, ja, ko, es, id, ms, vi, th) |

## Appendix B: Evaluation Script Usage

```javascript
// evaluate.js — reproducible scoring
const rubric = require('./evaluation/scoring_rubric.json');
const cases  = require('./dataset/classical_cases.json');

async function evaluate(analysisText, chartId) {
  // Score each dimension D1-D6
  const scores = {};
  for (const [dim, config] of Object.entries(rubric.dimensions)) {
    scores[dim] = await scoreWithRubric(analysisText, config);
  }
  return { chartId, scores, weighted: weightedScore(scores, rubric) };
}
```

Full script: `benchmark/public/evaluate.js`

---

*This preprint has not undergone peer review. It is made available to establish priority and invite community feedback.*
*Contact: See starnum.com.tw/about for contact information.*
*Last updated: 2026-04-12*