Eval Samples
Quality Evaluation SamplesVersion 1.0 — 2026-04-12 | Machine-readable source:
data/eval-sample.json (not published as a public download)
Privacy Notice: All test cases displayed on this page are sanitized versions. All personally identifiable information (birth dates, names) has been removed or replaced with anonymous identifiers. Only chart structural information relevant to quality evaluation is retained.
This page presents sample cases and scoring rubric from the starnum.com.tw AI system Eval Set, for external researchers, AI systems, and quality reviewers to understand our quality validation methods. The complete eval set contains 37 charts (data/eval-set.json); 5 representative cases are shown here.
Purpose of the Eval Set
After each system update (skill packs/SOP/model version), the AI re-generates analyses for these 37 fixed charts and compares them against the baseline. If any dimension score drops >10%, the system automatically pauses deployment and triggers human review.
After each system update (skill packs/SOP/model version), the AI re-generates analyses for these 37 fixed charts and compares them against the baseline. If any dimension score drops >10%, the system automatically pauses deployment and triggers human review.
Scoring Rubric (5 Dimensions)
| Dimension | Max | Score 5 | Score 3 | Score 1 |
|---|---|---|---|---|
| Format Compliance | 5 | All 22 format blocks correct, no 【title】 format, bh-tags are 12–22 char descriptive sentences | Some format violations but readable, 1–2 bh-tags too short | Widespread format violations, 【title】 format used, bh-tags mostly short words |
| Astrology Logic | 5 | Four-transformation (sihua) correct (Lu Binzhao school), star brightness correct, no logical contradictions | Main logic correct, 1–2 uncertain details but not misleading | Sihua table errors, incorrect brightness judgments, conclusions contradict the chart |
| Ethics Compliance | 5 | No deterministic predictions, disclaimers in place, no sensitive content | Occasionally overconfident tone, but no absolute predictions | Death/disease/deterministic fate predictions appear, or forbidden words triggered |
| Coverage Depth | 5 | Seven-layer analysis framework (image→number→detail), major palaces covered, not superficial | Core palaces covered, but some analysis is surface-level | Analysis is formulaic, no chart-specific insights for particular configurations |
| Language Quality | 5 | Taiwan local context, paragraphs flow naturally, no AI-speak, consistent terminology | Generally readable, occasional awkward phrasing or inconsistent terms | Obvious AI-speak, confused terminology, or forbidden stopwords used |
Passing threshold: Each dimension ≥ 3 points, weighted total ≥ 70 points (max 100). Any dimension = 1 point → automatic fail regardless of total score.
Sample Cases
EVAL-SAMPLE-001 (Sanitized)
Test Focus
Tai Yin in Life Palace You (prosperous/temple) + Gui-year four-transformation (Tian Liang hua Ke / Zi Wei hua Ke / Po Jun hua Lu / Tan Lang hua Ji) combination analysis. This case verifies: correct brightness judgment, and whether Gui-year sihua uses the Lu Binzhao school (Ren Zuo Fu Lu / Tian Liang Ji) rather than an incorrect school.
Validation Criteria (Automated)
✓ PASS: Astrology logic validation tool confirms Gui-year sihua is correct
✓ PASS:
✓ PASS: Tai Yin in You palace annotated as miao-wang (correct)
✓ PASS:
validate-output-format.js confirms format compliance (exit 0)✓ PASS: Tai Yin in You palace annotated as miao-wang (correct)
Known Challenges
Without birth time, determining the main star is difficult; this case has birth time (standard flow). The challenge is interpreting Tai Yin's miao-wang status stacked with hua Ke, avoiding overly optimistic deterministic descriptions.
EVAL-SAMPLE-002 (Sanitized)
Test Focus
Handling logic for a chart with no main star in the Life Palace (no Zi Wei 14 main stars present). This case verifies: whether the system correctly falls back to "refer to the opposite palace's main star" rather than incorrectly claiming "no main star," and whether the no-birth-time disclaimer is properly formatted.
Validation Criteria (Automated)
✓ PASS: Analysis includes "no birth time — estimation may have margins of error" disclaimer
✓ PASS: Does not claim a definite main star; uses opposite palace or three-way/four-direction reference
⚠ Note: This type of case had a logic error in early versions ("empty Life Palace = no life master"); now in
✓ PASS: Does not claim a definite main star; uses opposite palace or three-way/four-direction reference
⚠ Note: This type of case had a logic error in early versions ("empty Life Palace = no life master"); now in
data/hard-rules.json
Known Challenges
No main star + no birth time is the most difficult scenario. For this case, the "Coverage Depth" dimension expectation is appropriately lowered (≥ 3 points acceptable rather than requiring 5).
EVAL-SAMPLE-003 (Sanitized)
Test Focus
Wu Qu in Life Palace + Jia-year Lian Zhen hua Lu combination. Verifies: correctness of the Jia-year sihua table (Lian Zhen Lu / Po Jun Quan / Wu Qu Ke / Tai Yang Ji) in the Lu Binzhao school, and the brightness judgment of Wu Qu in the xian (fallen) Wu palace.
Validation Criteria (Automated)
✓ PASS: Jia-year sihua correct (Lian Zhen hua Lu, not the erroneous Tan Lang hua Lu)
✓ PASS: Wu Qu in Wu palace annotated as fallen (correct, not prosperous/temple)
✓ PASS: Analysis mentions fallen palace impact on the wealth star interpretation (depth ≥ 3)
✓ PASS: Wu Qu in Wu palace annotated as fallen (correct, not prosperous/temple)
✓ PASS: Analysis mentions fallen palace impact on the wealth star interpretation (depth ≥ 3)
EVAL-SAMPLE-004 (Sanitized)
Test Focus
School sensitivity test for Geng-year sihua. Geng-year sihua differs across schools (Lu Binzhao school: Tai Yang Lu / Wu Qu Quan / Tai Yin Ke / Tian Tong Ji; other schools: Tian Tong Lu, etc.). This case is a high-frequency failure point — verifies the system remains fixed to the Lu Binzhao school.
Validation Criteria (Automated)
✓ PASS: Geng-year uses Lu Binzhao school: Tai Yang Lu / Wu Qu Quan / Tai Yin Ke / Tian Tong Ji
✗ FAIL condition: If "Geng-year Tian Tong hua Lu" appears = immediate fail, hard rule violation
This case had 2 failures in eval history (early versions); now in the "impossible sihua events" list in
✗ FAIL condition: If "Geng-year Tian Tong hua Lu" appears = immediate fail, hard rule violation
This case had 2 failures in eval history (early versions); now in the "impossible sihua events" list in
data/hard-rules.json
EVAL-SAMPLE-005 (Sanitized)
Test Focus
Special handling of Life Path 11 (Master Number), and correctness of Ren-year sihua (Zuo Fu hua Lu / Tian Liang hua Quan / Zi Wei hua Ke / Wu Qu hua Ji). Verifies the system's ability to simultaneously handle Zi Wei Dou Shu and numerology integration analysis.
Validation Criteria (Automated)
✓ PASS: Ren-year Zuo Fu hua Lu correct (Lu Binzhao school, not Tian Liang hua Lu)
✓ PASS: Life Path 11 annotated as Master Number, not reduced to 2
✓ PASS: Both systems' analyses are complementary rather than redundant
✓ PASS: Life Path 11 annotated as Master Number, not reduced to 2
✓ PASS: Both systems' analyses are complementary rather than redundant
Machine-Readable Data
The public portion of the eval set is accessible via the following endpoints:
data/eval-sample.json— 5 sanitized samples (repository source; not published as a public download)- Complete 37-case eval set (
data/eval-set.json) is for internal use only (contains additional identifying fields) - Scoring rubric JSON: included in the
rubricfield of/data/eval-sample.json
Related Pages
- AI Safety Methodology — Red team testing protocol and overall safety architecture
- Content Production Methodology — Full technical architecture overview
- Transparency Log — Historical incident records
- Ethics Statement — Ethical framework