Astrology Analysis Quality Benchmark

Test Overview

The Benchmark is an objective standard for measuring the quality of an astrology analysis system. We maintain a fixed set of 37 chart tests (the Golden Test Suite), automatically executed after each system update to confirm that analysis quality has not regressed.

The test set covers diverse chart combinations: different configurations of 14 main stars, 12 palace earthly branches, with and without birth time, different year stems — representing the full range of situations real users may encounter.

Test Suite Version: v1.0 | Test Set Size: 37 charts | Run Date: 2026-04-10

Six-Dimension Scoring (April 2026)

Multi-model cross-validation weighted total score: 85.4 / 100 (outperforms single-model baseline of 79.2, improvement +6.2)

Dataset Files (Public Download)

• 📊 dataset/classical_cases.json — 37 de-identified chart configurations, containing only astrology fields, no personal information
• 📋 evaluation/scoring_rubric.json — D1–D6 six-dimension scoring criteria, with rubric details and validation methods
• 📈 results/baseline-2026-04.json — April 2026 baseline results, multi-model weighted scores
• ⚙️ evaluate.js — Reproducible scoring and comparison script (Node.js)
• 📄 README.md — Full documentation and usage instructions

License: CC BY 4.0 · Citation: Starnum Research Team (2026). starnum-bench v1.0.

Test Methodology

Test Set Design

The Golden Test Suite consists of 37 fixed charts representing different difficulty levels and star combinations:

• Star diversity: Covers all 14 main stars — Zi Wei, Tian Ji, Tai Yang, Wu Qu, Tian Tong, Lian Zhen, Tian Fu, Tai Yin, Tan Lang, Ju Men, Tian Xiang, Tian Liang, Qi Sha, Po Jun
• Palace diversity: Life palaces distributed across 12 earthly branches, ensuring all palace combinations are covered
• With/without birth time: Includes both known birth time (precise charting) and unknown birth time (approximate charting)
• Year stem coverage: All 10 heavenly stems have representative cases, testing sihua calculation accuracy

Scoring Criteria

• Accuracy: Whether palace judgments, sihua flying entries, and pattern identification conform to the Lu Binzhao school standard — compared against all 127 hard rules
• Analysis Depth: Whether the plain-language interpretation covers the Life Palace main star characteristics, Life/Body palace relationship, major patterns, personality, and career tendencies
• Palace Coverage Rate: The proportion of the 22 standard analysis blocks actually completed

Validation Process

1. Fixed test set read from the Supabase chart database; same charts used for every test
2. Astrology logic validation tool automatically compares all 127 hard rules, flags potential errors
3. Multi-model cross-comparison: multiple independent tools analyze simultaneously, cross-confirming consistency
4. Professional editor human review of scoring, ensuring objective and reliable evaluation results
5. If any rule validation fails, the entire batch is paused and retested after correction

Comparison vs. General Astrology Websites

Comparison baseline is public content from major Chinese astrology sites in Taiwan and Southeast Asia. Evaluation date: April 2026.

Test Set Statistics

Data source: data/eval-set.json | Version: 1.0 | Generated: 2026-04-10

About This Report

This page is automatically generated by the Starnum Logic Engine v5.0 evaluation system, re-executed and updated with each system version update. The evaluation methodology references industry software quality assurance (QA) standards, combined with the specialized requirements of astrology analysis.

We chose to make this benchmark public because we believe transparency is the only way to build trust. If you have any questions about the evaluation methodology, you are welcome to contact @mychenan on Instagram.