ai-tool-compliance - Internal AI Tool Compliance Automation

When to use this skill

• Starting a new AI project: When scaffolding the compliance foundation (RBAC, Gateway, logs, cost tracking) from scratch
• Pre-deploy P0 full verification: When automatically evaluating all 13 P0 mandatory requirements as pass/fail and computing a compliance score
• RBAC design and permission matrix generation: When defining the 5 roles (Super Admin/Admin/Manager/Viewer/Guest) + granular access control per game/menu/feature unit
• Auditing existing code for compliance: When inspecting an existing codebase against the guide and identifying violations
• Implementing cost transparency: When building a tracking system for model/token/BQ scan volume/cost per action
• Designing a behavior log schema: When designing a comprehensive behavior log recording system (Firestore/BigQuery)
• Role-based verification workflow: When configuring the release approval process based on Section 14 (ServiceStability/Engineer/PM/CEO)
• Building a criteria verification system: When setting up the Rule Registry + Evidence Collector + Verifier Engine + Risk Scorer + Gatekeeper architecture

---

Installation

npx skills add https://github.com/supercent-io/skills-template --skill ai-tool-compliance

---

Quick Reference

Action	Command	Description
Project initialization	/compliance-init	Generate RBAC matrix, Gateway boilerplate, log schema, cost tracking interface
Quick scan	/compliance-scan, /compliance-quick, /quick	Quick inspection of P0 key items (code pattern-based)
Full verification	/compliance-verify, /compliance-full, /full	Full verification of 11 P0 rules + compliance score computation
Score check	/compliance-score	Display current compliance score (security/auth/cost/logging)
Deploy gate	/compliance-gate	Green/Yellow/Red verdict + deploy approve/block decision
Improvement guide	/compliance-improve, /improve	Specific fix suggestions per violation + re-verification loop

Slash Mode Router

Mode slash commands are mapped as follows.

• /quick, /compliance-quick -> Quick Scan (/compliance-scan)
• /full, /compliance-full -> Full Verify (/compliance-verify)
• /improve -> Improve (/compliance-improve)

---

3 Execution Modes

1. Quick Scan (quick-scan)

Statically analyzes the codebase to quickly identify potential P0 violations.

How to run: /compliance-scan, /compliance-quick, /quick or trigger keywords compliance scan, quick scan

What it does:

• Grep/Glob-based code pattern search
• Detect direct external API calls (whether Gateway is bypassed)
• Detect direct Firestore client access
• Detect hardcoded sensitive data
• Check for missing Guest role

Output: List of suspected violations (file path + line number + rule ID)

Duration: 1–3 minutes

2. Full Verify (full-verify)

Fully verifies all 11 P0 rules and computes a quantitative compliance score.

How to run: /compliance-verify, /compliance-full, /full or trigger keywords P0 verification, full verify, deploy verification

What it does:

• Collect Evidence and evaluate pass/fail for each of the 11 P0 rules
• Compute scores per 4 domains (Security 40pts / Auth 25pts / Cost 20pts / Logging 15pts)
• Calculate total compliance score (out of 100)
• Determine deploy gate grade (Green/Yellow/Red)
• Generate role-based approval checklist

Output: Compliance report (compliance-report.md)

## Compliance Report
- Date: 2026-03-03
- Project: my-ai-tool
- Score: 92/100 (Green)

### Rule Results
| Rule ID | Rule Name | Result | Evidence |
|---------|-----------|--------|----------|
| AUTH-P0-001 | Force Guest for New Signups | PASS | signup.ts:45 role='guest' |
| AUTH-P0-002 | Block Guest Menu/API Access | PASS | middleware.ts:12 guestBlock |
| ... | ... | ... | ... |

### Score Breakdown
- Security: 33/40
- Auth: 25/25
- Cost: 17/20
- Logging: 12/15
- Total: 92/100

### Gate Decision: GREEN - Deploy Approved

Duration: 5–15 minutes (varies by project size)

3. Improve (improve)

Provides specific fix guides for violations and runs a re-verification loop.

How to run: /compliance-improve, /improve or trigger keywords compliance improvement, fix violations

What it does:

• Code-level fix suggestions for each FAIL item (file path + before/after code)
• Re-verify the rule after applying the fix
• Track score changes (Before -> After)
• Guide for gradually introducing P1 recommended requirements after passing P0

Output: Fix proposal + re-verification results

Improve Mode Auto-Fix Logic

/compliance-improve runs
       |
  1. Load latest verification-run.json
       |
  2. Extract FAIL items (rule_id + evidence)
       |
  3. For each FAIL:
       |
     a. Read violation code from evidence file:line
     b. Derive fix direction from rule.remediation + rule.check_pattern.must_contain
     c. Generate before/after code diff
     d. Apply fix via Write (after user confirmation)
     e. Re-verify only that rule (re-run Grep pattern)
     f. Confirm transition to PASS
       |
  4. Full re-verification (/compliance-verify)
       |
  5. Output Before/After score comparison
       |
  6. If no remaining FAILs → present guide for introducing P1 recommended requirements

Fix application priority:

1. must_not_contain violations (requires immediate removal) → delete the code or replace with server API call
2. must_contain unmet (pattern needs to be added) → insert code per the remediation guide
3. Warning (partially met) → apply supplement only to unmet files

---

P0 Rule Catalog

11 P0 rules based on the internal AI tool mandatory implementation guide v1.1:

Rule ID	Category	Rule Name	Description	Score
AUTH-P0-001	Auth	Force Guest for New Signups	Automatically assign role=Guest on signup; elevated roles granted only via invitation	Auth 8
AUTH-P0-002	Auth	Block Guest Menu/API Access	Do not expose tool name, model name, internal infrastructure, cost, or structure to Guest. Only allow access to permitted menus/APIs	Auth 7
AUTH-P0-003	Auth	Server-side Final Auth Check	Server-side auth verification middleware required for all API requests. Client-side checks alone are insufficient	Auth 10
SEC-P0-004	Security	Prohibit Direct Firestore Access	Direct read/write to Firestore from client is forbidden. Only via Cloud Functions is allowed	Security 12
SEC-P0-005	Security	Enforce External API Gateway	Direct calls to external AI APIs (Gemini, OpenAI, etc.) are forbidden. Must route through internal Gateway (Cloud Functions)	Security 18
SEC-P0-009	Security	Server-side Sensitive Text Processing	Sensitive raw content (prompts, full responses) is processed server-side only. Only reference values (IDs) are sent to clients	Security 10
COST-P0-006	Cost	Model Call Cost Log	Must record model, inputTokens, outputTokens, estimatedCost for every AI model call	Cost 10
COST-P0-007	Cost	BQ Scan Cost Log	Must record bytesProcessed, estimatedCost when executing BigQuery queries	Cost 5
COST-P0-011	Cost	Cache-first Lookup	Cache lookup required before high-cost API calls. Actual call only on cache miss	Cost 5
LOG-P0-008	Logging	Mandatory Failed Request Logging	Must log all failed requests (4xx, 5xx, timeout). No omissions allowed	Logging 10
LOG-P0-010	Logging	Auth Change Audit Log	Record all auth-related events: role changes, permission grants/revocations, invitation sends	Logging 5

Scoring System

Domain	Max Score	Included Rules
Security	40	SEC-P0-004, SEC-P0-005, SEC-P0-009
Auth	25	AUTH-P0-001, AUTH-P0-002, AUTH-P0-003
Cost	20	COST-P0-006, COST-P0-007, COST-P0-011
Logging	15	LOG-P0-008, LOG-P0-010
Total	100	11 P0 rules

Per-rule Automatic Verification Logic

Verification for each rule is performed based on the check_pattern defined in rules/p0-catalog.yaml. The core mechanism is Grep/Glob static analysis.

Verdict Algorithm (per rule):

1. Glob(check_targets) → collect target files
2. grep_patterns matching → identify files using that feature
   - 0 matches → N/A (feature not used, no penalty)
3. must_not_contain check (excluding exclude_paths)
   - Match found → immediate FAIL + record evidence
4. must_contain check
   - All satisfied → PASS
   - Partially satisfied → WARNING
   - Not satisfied → FAIL

Key Grep Patterns per Rule:

Rule ID	Feature Detection (grep_patterns)	Compliance Check (must_contain)	Violation Detection (must_not_contain)
AUTH-P0-001	signup|register|createUser	role.*['"]guest['"]	role.*['"]admin['"] (on signup)
AUTH-P0-002	guard|middleware|authorize	guest.*block|guest.*deny	--
AUTH-P0-003	router\.(get|post|put|delete)	auth|verify|authenticate	--
SEC-P0-004	-- (all targets)	--	firebase/firestore|getDocs|setDoc (client paths)
SEC-P0-005	-- (all targets)	--	fetch\(['"]https?://(?!localhost) (client paths)
SEC-P0-009	-- (all targets)	--	res\.json\(.*password|console\.log\(.*token
COST-P0-006	openai|vertexai|gemini|anthropic	cost|token|usage|billing	--
COST-P0-007	bigquery|BigQuery|createQueryJob	totalBytesProcessed|bytesProcessed|cost	--
COST-P0-011	openai|vertexai|gemini|anthropic	cache|Cache|redis|memo	--
LOG-P0-008	catch|errorHandler|onError	logger|log\.error|winston|pino	--
LOG-P0-010	updateRole|changeRole|setRole	audit|auditLog|eventLog	--

Detailed schema: see rules/p0-catalog.yaml and the "Judgment Algorithm" section in REFERENCE.md

---

Verification Scenarios (QA)

5 key verification scenarios run in Full Verify mode (/compliance-verify). Each scenario groups related P0 rules for end-to-end verification.

ID	Scenario	Related Rules	Verification Method	Pass Criteria
SC-001	New Signup -> Guest Isolation	AUTH-P0-001, AUTH-P0-002	Verify role=guest assignment in signup code + confirm 403 return pattern when Guest calls protected API	PASS when role is guest and access-denied pattern exists for protected API
SC-002	AI Call -> Via Gateway + Cost Logged	SEC-P0-005, COST-P0-006, COST-P0-011	(1) Confirm absence of direct external API calls (2) Confirm routing via Gateway function (3) Confirm cost log fields (model, tokens, cost) recorded (4) Confirm cache lookup logic exists	PASS when Gateway routing + 4 cost log fields recorded + cache layer exists
SC-003	Firestore Access -> Functions-Only	SEC-P0-004, AUTH-P0-003	(1) Detect direct Firestore SDK import in client code (2) Confirm server-side auth verification middleware exists	PASS when 0 direct client access instances + server middleware exists
SC-004	Failed Requests -> No Log Gaps	LOG-P0-008, LOG-P0-010	(1) Confirm log call in error handler (2) Confirm no log gaps in catch blocks (3) Confirm audit log exists for auth change events	PASS when all error handlers call log + auth change audit log exists
SC-005	Sensitive Data -> Not Exposed to Client	SEC-P0-009, AUTH-P0-002	(1) Confirm API responses do not include raw prompts/responses, only reference IDs (2) Confirm Guest responses do not include model name/cost/infrastructure info	PASS when raw content not in response + Guest exposure control confirmed

Verification Flow by Scenario

SC-001: grep signup/register -> assert role='guest' -> grep guestBlock/guestDeny -> assert exists
SC-002: grep fetch(https://) in client -> assert 0 hits -> grep gateway log -> assert cost fields -> assert cache check
SC-003: grep firebase/firestore in client/ -> assert 0 hits -> grep authMiddleware in functions/ -> assert exists
SC-004: grep catch blocks -> assert logAction in each -> grep roleChange -> assert auditLog
SC-005: grep res.json for raw text -> assert 0 hits -> grep guest response -> assert no model/cost info

---

Role-based Go/No-Go Checkpoints

After the deploy gate verdict, the role's Go/No-Go checkpoints must be cleared based on the grade. 4 roles × 5 items = 20 checkpoints total.

Service Stability (5 items)

#	Checkpoint	Go Condition	No-Go Condition
1	SLA Impact Analysis	Confirmed no impact on existing service availability/response-time SLA	SLA impact unanalyzed or degradation expected
2	Rollback Procedure	Rollback procedure documented + tested	Rollback procedure not established
3	Performance Test	Load/stress test completed + within threshold	Performance test not run
4	Incident Alerts	Incident detection alert channels (Slack/PagerDuty, etc.) configured	Alert channels not configured
5	Monitoring Dashboard	Dashboard for key metrics (error rate, response time, AI cost) exists	Monitoring absent

Engineer (5 items)

#	Checkpoint	Go Condition	No-Go Condition
1	FAIL Rule Root Cause Analysis	Root cause identified + documented for all FAIL rules	Unidentified items exist
2	Fix Code Verification	Fixed code accurately reflects the intent of the rule	Fix does not match rule intent
3	Re-verification Pass	Rule transitions to PASS in re-verification after fix	Re-verification not run or still FAIL
4	No Regression Impact	Fix confirmed to have no negative impact on other P0 rules	Another rule newly FAILs
5	Code Review Done	Code review approval completed for fixed code	Code review not completed

PM (5 items)

#	Checkpoint	Go Condition	No-Go Condition
1	User Impact Assessment	User impact of non-compliant items is acceptable	User impact not assessed
2	Schedule Risk	Fix timeline is within release schedule	Schedule overrun expected
3	Scope Agreement	Stakeholder agreement completed for scope changes	Agreement not reached
4	Cost Impact	AI usage cost within approved budget	Budget overrun expected
5	Communication	Changes shared with relevant teams	Not shared

CEO (5 items)

#	Checkpoint	Go Condition	No-Go Condition
1	Cost Cap	Monthly AI cost within pre-approved budget	Budget cap exceeded
2	Security Risk	All security P0 passed or exception reason is reasonable	P0 security FAIL + insufficient exception justification
3	Legal/Regulatory Risk	Data processing complies with applicable laws (privacy laws, etc.)	Legal risks not reviewed
4	Business Continuity	Business impact is limited if deployment fails	Business disruption risk exists
5	Final Approval	Final approval when all 4 above are Go	Deferred if even 1 is No-Go

---

Report Format

compliance-report.md, generated when /compliance-verify runs, consists of 6 sections.

Report Section Structure (6 sections)

# Compliance Report

## 1. Summary
- Project name, verification date/time, verification mode (quick-scan / full-verify)
- Total compliance score / 100
- Deploy gate grade (Green / Yellow / Red)
- P0 FAIL count
- Verification duration

## 2. Rule Results
| Rule ID | Category | Rule Name | Result | Score | Evidence |
|---------|----------|