AI Safety Methodology
AI Safety v2.0Version 2.0 · · Governance 2.0 public evidence surface
Governance 2.0 Overview
This page is part of the starnum public Governance 2.0 surface and uses the same evidence layer as the system card, data governance, transparency report, use policy, and security policy.
Governance Summary
This page describes the safety controls used around AI-assisted interpretation and public content generation.
Scope
Risk-boundary copy, medical/legal/financial advice exclusions, monitoring signals, incident disclosure, and model/provider benchmark boundaries.
Implementation Status
Version 2.0 ties safety language to public claims, machine checks, and release integrity status.
starnum.com.tw operates fully under AI automation (Claude Code as technical lead). In a fully AI-driven system, safety is not a retrospective safeguard — it is the core design principle of the architecture. This page explains how we implement AI safety, rather than merely declaring ethical commitments.
1. Red Team Protocol
Red teaming is an adversarial testing method that proactively attempts to make an AI system violate its rules, in order to surface security vulnerabilities.
1.1 Testing Tools
Automated red team testing tools run the following test categories on a regular basis:
| Test Category | Test Scenario | Expected Behavior |
|---|---|---|
| Ethics Boundary Test | Request death time prediction, disease diagnosis | Refuse output, notify user |
| Identity Protection Test | Request analysis of politicians, minors' charts | Refuse, produce no content |
| Prompt Injection Test | Embed manipulation instructions in input | Detect and isolate, blacklist source |
| Astrology Logic Contradiction Test | Input contradictory chart configurations | Trigger astrology logic validation |
| Format Evasion Test | Attempt to trigger prohibited output formats | Format validation intercepts, force rollback |
| Data Governance Test | Attempt out-of-scope access to user data | Blocked by Supabase RLS policies |
1.2 Test Result Handling
- Pass: Logged to system status log, no further action
- Fail: Triggers the three-tier escalation architecture (see below), logged to
data/incidents.mdanddata/transparency-log.json - Prompt Injection detected: Source URL/domain added to blacklist, never crawled again
1.3 Hardcoded Red Lines
- Death time prediction (in any form)
- Disease diagnosis or treatment advice
- Astrology chart analysis of political figures
- Chart analysis of minors (without written parental consent)
- Using astrology results as grounds for discrimination
- Any deterministic predictions that could cause psychological harm
The above items are hardcoded restrictions that cannot be overridden by any user instruction, paid service, or system upgrade.
2. Three-Tier Escalation Architecture
The three-tier escalation architecture ensures that system issues progress from automated handling to human intervention, with clear trigger conditions and time limits at each tier.
Triggers: Astrology logic validation failure, format standard violation, forbidden word triggered, Prompt Injection detected
Automated Actions:
- Forbidden word filter intercepts prohibited terms and inappropriate language
- Format validation tool (forces rollback of non-compliant output)
- Astrology logic reasoning validation tool
- SOP confirmation mechanism
Response Time: Immediate (synchronous interception, never enters publish pipeline)
Triggers: Agent fails to self-correct after L1 interception, quality score below threshold 3 consecutive times, same error type ≥ 3 cumulative occurrences
Automated Actions:
- Error logged to
data/failure-log.jsonwith root cause analysis - Incident logged to
data/incidents.md - Same error type ≥ 3 times → auto-proposes new rule to rule proposal system
- Triggers Multi-provider governance audit (
joint-audit.js) to confirm issue scope - Publicly recorded in transparency log (minor issues)
Response Time: Automated repair completed within 7 days
Triggers: Ethics boundary violation (any severity), suspected user data leak, systemic astrology logic errors affecting >10 articles, L2 automated repair fails >2 times
Actions:
- Immediately halt affected AI agent tasks
- Site owner (technical lead) reviews and intervenes
- Affected content taken down or labeled with warning
- Transparency log updated publicly (within 72 hours)
- Corresponding skill pack / SOP updated after root cause analysis
Response Time: CRITICAL within 4h / MAJOR within 24h for human review to begin
3. Eval Set Design Principles
The Eval Set is a fixed collection of test charts used to verify that output quality has not regressed after each system update (regression testing).
3.1 Design Principles
3.2 Trigger Conditions
- Required after any core specification document update
- Required after any SOP document update
- Run once before and once after AI model version upgrades (diff comparison)
- Part of the weekly Sunday site-wide review (
benchmark-run.js)
3.3 Evaluation Dimensions
| Dimension | Tool | Threshold |
|---|---|---|
| Format Compliance | Format validation tool | 100% pass (exit 0) |
| Astrology Logic Correctness | Astrology logic validation tool | 0 hard rule violations |
| SOP Read Confirmation | SOP confirmation tool | 100% have "✅ SOP read" confirmation |
| Ethics Compliance | Forbidden word filter | 0 forbidden word triggers |
| Coverage (seven layers) | Coverage check tool | ≥ 80% coverage |
→ Benchmark page: view public evaluation results and scoring rubric
4. Human Oversight Trigger Conditions
Although this site operates with very high AI automation, the following situations must trigger human (site owner) intervention:
| Trigger Condition | Type | Urgency |
|---|---|---|
| Any ethics boundary violation (hardcoded red line triggered) | Ethics | Immediate |
| Suspected user data leak or unauthorized access | Security | Immediate |
| Prompt Injection successfully bypasses protection layers | Security | Immediate |
| Systemic behavioral drift in AI agents (same error type ≥ 5 times) | Quality | 24 hours |
| Multi-provider governance audit finds P0/P1 issues | System | 24 hours |
| Eval set regression test score drops >10% | Quality | 72 hours |
| Supabase data anomaly (unauthorized deletion/modification) | Security | Immediate |
| External security researcher vulnerability report received | Security | 72 hours to confirm |
| DMCA copyright complaint received | Legal | Immediate takedown |
5. Knowledge Base Security Mechanisms
The astrology knowledge base (KB) is the foundation of all system outputs. Its integrity directly affects all content quality.
- LLM cleansing pipeline: All crawled content must pass LLM cleansing to remove ads, promotions, and Prompt Injection
- Blacklist mechanism: Source domains where Prompt Injection is detected are blacklisted and permanently blocked
- Terminology validation: All KB content must pass
data/glossary.jsonterm matching (221 terms × 9 languages) - School annotation: Non-Lu Binzhao school viewpoints must be marked ⚠️ and must not be used as primary reference in content generation
- Contradiction detection:
scripts/detect-evidence-conflict.jsperiodically scans for school/brightness contradictions
6. Continuous Improvement Mechanisms
Safety is not a static state — it is a dynamically evolving process:
- Failure-driven evolution: Root cause analysis of every L2/L3 event must update the corresponding skill pack, preventing recurrence of the same issue
- 20 joint audit rounds (as of 2026-04-12): 188 cumulative bug fixes; R15 is the first time all 4 AIs confirmed CLEAN
- Rule lifecycle: Rules untriggered for 90 days are candidates for retirement (
skill-lifecycle.md), preventing rule bloat - Whitepaper review: Quarterly joint review by multi-provider AI governance references of the overall architecture, with improvement recommendations
Related Resources
- Ethics Statement — Ethical framework and commitments for AI use
- Transparency Log — Public incident record
- Benchmark — Public evaluation results and scoring rubric
- AI Data Governance Policy — How user data is handled
- Acceptable Use Policy — Platform usage guidelines
- Content Production Methodology — Full technical architecture overview
- Security Disclosure Policy — Vulnerability reporting process