ai-tool-compliance - Internal AI Tool Compliance Automation
When to use this skill
- Starting a new AI project: When scaffolding the compliance foundation (RBAC, Gateway, logs, cost tracking) from scratch
- Pre-deploy P0 full verification: When automatically evaluating all 13 P0 mandatory requirements as pass/fail and computing a compliance score
- RBAC design and permission matrix generation: When defining the 5 roles (Super Admin/Admin/Manager/Viewer/Guest) + granular access control per game/menu/feature unit
- Auditing existing code for compliance: When inspecting an existing codebase against the guide and identifying violations
- Implementing cost transparency: When building a tracking system for model/token/BQ scan volume/cost per action
- Designing a behavior log schema: When designing a comprehensive behavior log recording system (Firestore/BigQuery)
- Role-based verification workflow: When configuring the release approval process based on Section 14 (ServiceStability/Engineer/PM/CEO)
- Building a criteria verification system: When setting up the Rule Registry + Evidence Collector + Verifier Engine + Risk Scorer + Gatekeeper architecture
Installation
npx skills add https://github.com/supercent-io/skills-template --skill ai-tool-compliance
Quick Reference
| Action |
Command |
Description |
| Project initialization |
/compliance-init |
Generate RBAC matrix, Gateway boilerplate, log schema, cost tracking interface |
| Quick scan |
/compliance-scan, /compliance-quick, /quick |
Quick inspection of P0 key items (code pattern-based) |
| Full verification |
/compliance-verify, /compliance-full, /full |
Full verification of 11 P0 rules + compliance score computation |
| Score check |
/compliance-score |
Display current compliance score (security/auth/cost/logging) |
| Deploy gate |
/compliance-gate |
Green/Yellow/Red verdict + deploy approve/block decision |
| Improvement guide |
/compliance-improve, /improve |
Specific fix suggestions per violation + re-verification loop |
Slash Mode Router
Mode slash commands are mapped as follows.
/quick, /compliance-quick -> Quick Scan (/compliance-scan)/full, /compliance-full -> Full Verify (/compliance-verify)/improve -> Improve (/compliance-improve)
3 Execution Modes
1. Quick Scan (quick-scan)
Statically analyzes the codebase to quickly identify potential P0 violations.
How to run: /compliance-scan, /compliance-quick, /quick or trigger keywords compliance scan, quick scan
What it does:
- Grep/Glob-based code pattern search
- Detect direct external API calls (whether Gateway is bypassed)
- Detect direct Firestore client access
- Detect hardcoded sensitive data
- Check for missing Guest role
Output: List of suspected violations (file path + line number + rule ID)
Duration: 1–3 minutes
2. Full Verify (full-verify)
Fully verifies all 11 P0 rules and computes a quantitative compliance score.
How to run: /compliance-verify, /compliance-full, /full or trigger keywords P0 verification, full verify, deploy verification
What it does:
- Collect Evidence and evaluate pass/fail for each of the 11 P0 rules
- Compute scores per 4 domains (Security 40pts / Auth 25pts / Cost 20pts / Logging 15pts)
- Calculate total compliance score (out of 100)
- Determine deploy gate grade (Green/Yellow/Red)
- Generate role-based approval checklist
Output: Compliance report (compliance-report.md)
## Compliance Report
- Date: 2026-03-03
- Project: my-ai-tool
- Score: 92/100 (Green)
### Rule Results
| Rule ID | Rule Name | Result | Evidence |
|---------|-----------|--------|----------|
| AUTH-P0-001 | Force Guest for New Signups | PASS | signup.ts:45 role='guest' |
| AUTH-P0-002 | Block Guest Menu/API Access | PASS | middleware.ts:12 guestBlock |
| ... | ... | ... | ... |
### Score Breakdown
- Security: 33/40
- Auth: 25/25
- Cost: 17/20
- Logging: 12/15
- Total: 92/100
### Gate Decision: GREEN - Deploy Approved
Duration: 5–15 minutes (varies by project size)
3. Improve (improve)
Provides specific fix guides for violations and runs a re-verification loop.
How to run: /compliance-improve, /improve or trigger keywords compliance improvement, fix violations
What it does:
- Code-level fix suggestions for each FAIL item (file path + before/after code)
- Re-verify the rule after applying the fix
- Track score changes (Before -> After)
- Guide for gradually introducing P1 recommended requirements after passing P0
Output: Fix proposal + re-verification results
Improve Mode Auto-Fix Logic
/compliance-improve runs
|
1. Load latest verification-run.json
|
2. Extract FAIL items (rule_id + evidence)
|
3. For each FAIL:
|
a. Read violation code from evidence file:line
b. Derive fix direction from rule.remediation + rule.check_pattern.must_contain
c. Generate before/after code diff
d. Apply fix via Write (after user confirmation)
e. Re-verify only that rule (re-run Grep pattern)
f. Confirm transition to PASS
|
4. Full re-verification (/compliance-verify)
|
5. Output Before/After score comparison
|
6. If no remaining FAILs → present guide for introducing P1 recommended requirements
Fix application priority:
must_not_contain violations (requires immediate removal) → delete the code or replace with server API callmust_contain unmet (pattern needs to be added) → insert code per the remediation guide- Warning (partially met) → apply supplement only to unmet files
P0 Rule Catalog
11 P0 rules based on the internal AI tool mandatory implementation guide v1.1:
| Rule ID |
Category |
Rule Name |
Description |
Score |
| AUTH-P0-001 |
Auth |
Force Guest for New Signups |
Automatically assign role=Guest on signup; elevated roles granted only via invitation |
Auth 8 |
| AUTH-P0-002 |
Auth |
Block Guest Menu/API Access |
Do not expose tool name, model name, internal infrastructure, cost, or structure to Guest. Only allow access to permitted menus/APIs |
Auth 7 |
| AUTH-P0-003 |
Auth |
Server-side Final Auth Check |
Server-side auth verification middleware required for all API requests. Client-side checks alone are insufficient |
Auth 10 |
| SEC-P0-004 |
Security |
Prohibit Direct Firestore Access |
Direct read/write to Firestore from client is forbidden. Only via Cloud Functions is allowed |
Security 12 |
| SEC-P0-005 |
Security |
Enforce External API Gateway |
Direct calls to external AI APIs (Gemini, OpenAI, etc.) are forbidden. Must route through internal Gateway (Cloud Functions) |
Security 18 |
| SEC-P0-009 |
Security |
Server-side Sensitive Text Processing |
Sensitive raw content (prompts, full responses) is processed server-side only. Only reference values (IDs) are sent to clients |
Security 10 |
| COST-P0-006 |
Cost |
Model Call Cost Log |
Must record model, inputTokens, outputTokens, estimatedCost for every AI model call |
Cost 10 |
| COST-P0-007 |
Cost |
BQ Scan Cost Log |
Must record bytesProcessed, estimatedCost when executing BigQuery queries |
Cost 5 |
| COST-P0-011 |
Cost |
Cache-first Lookup |
Cache lookup required before high-cost API calls. Actual call only on cache miss |
Cost 5 |
| LOG-P0-008 |
Logging |
Mandatory Failed Request Logging |
Must log all failed requests (4xx, 5xx, timeout). No omissions allowed |
Logging 10 |
| LOG-P0-010 |
Logging |
Auth Change Audit Log |
Record all auth-related events: role changes, permission grants/revocations, invitation sends |
Logging 5 |
Scoring System
| Domain |
Max Score |
Included Rules |
| Security |
40 |
SEC-P0-004, SEC-P0-005, SEC-P0-009 |
| Auth |
25 |
AUTH-P0-001, AUTH-P0-002, AUTH-P0-003 |
| Cost |
20 |
COST-P0-006, COST-P0-007, COST-P0-011 |
| Logging |
15 |
LOG-P0-008, LOG-P0-010 |
| Total |
100 |
11 P0 rules |
Per-rule Automatic Verification Logic
Verification for each rule is performed based on the check_pattern defined in rules/p0-catalog.yaml. The core mechanism is Grep/Glob static analysis.
Verdict Algorithm (per rule):
1. Glob(check_targets) → collect target files
2. grep_patterns matching → identify files using that feature
- 0 matches → N/A (feature not used, no penalty)
3. must_not_contain check (excluding exclude_paths)
- Match found → immediate FAIL + record evidence
4. must_contain check
- All satisfied → PASS
- Partially satisfied → WARNING
- Not satisfied → FAIL
Key Grep Patterns per Rule:
| Rule ID |
Feature Detection (grep_patterns) |
Compliance Check (must_contain) |
Violation Detection (must_not_contain) |
| AUTH-P0-001 |
signup|register|createUser |
role.*['"]guest['"] |
role.*['"]admin['"] (on signup) |
| AUTH-P0-002 |
guard|middleware|authorize |
guest.*block|guest.*deny |
-- |
| AUTH-P0-003 |
router\.(get|post|put|delete) |
auth|verify|authenticate |
-- |
| SEC-P0-004 |
-- (all targets) |
-- |
firebase/firestore|getDocs|setDoc (client paths) |
| SEC-P0-005 |
-- (all targets) |
-- |
fetch\(['"]https?://(?!localhost) (client paths) |
| SEC-P0-009 |
-- (all targets) |
-- |
res\.json\(.*password|console\.log\(.*token |
| COST-P0-006 |
openai|vertexai|gemini|anthropic |
cost|token|usage|billing |
-- |
| COST-P0-007 |
bigquery|BigQuery|createQueryJob |
totalBytesProcessed|bytesProcessed|cost |
-- |
| COST-P0-011 |
openai|vertexai|gemini|anthropic |
cache|Cache|redis|memo |
-- |
| LOG-P0-008 |
catch|errorHandler|onError |
logger|log\.error|winston|pino |
-- |
| LOG-P0-010 |
updateRole|changeRole|setRole |
audit|auditLog|eventLog |
-- |
Detailed schema: see rules/p0-catalog.yaml and the "Judgment Algorithm" section in REFERENCE.md
Verification Scenarios (QA)
5 key verification scenarios run in Full Verify mode (/compliance-verify). Each scenario groups related P0 rules for end-to-end verification.
| ID |
Scenario |
Related Rules |
Verification Method |
Pass Criteria |
| SC-001 |
New Signup -> Guest Isolation |
AUTH-P0-001, AUTH-P0-002 |
Verify role=guest assignment in signup code + confirm 403 return pattern when Guest calls protected API |
PASS when role is guest and access-denied pattern exists for protected API |
| SC-002 |
AI Call -> Via Gateway + Cost Logged |
SEC-P0-005, COST-P0-006, COST-P0-011 |
(1) Confirm absence of direct external API calls (2) Confirm routing via Gateway function (3) Confirm cost log fields (model, tokens, cost) recorded (4) Confirm cache lookup logic exists |
PASS when Gateway routing + 4 cost log fields recorded + cache layer exists |
| SC-003 |
Firestore Access -> Functions-Only |
SEC-P0-004, AUTH-P0-003 |
(1) Detect direct Firestore SDK import in client code (2) Confirm server-side auth verification middleware exists |
PASS when 0 direct client access instances + server middleware exists |
| SC-004 |
Failed Requests -> No Log Gaps |
LOG-P0-008, LOG-P0-010 |
(1) Confirm log call in error handler (2) Confirm no log gaps in catch blocks (3) Confirm audit log exists for auth change events |
PASS when all error handlers call log + auth change audit log exists |
| SC-005 |
Sensitive Data -> Not Exposed to Client |
SEC-P0-009, AUTH-P0-002 |
(1) Confirm API responses do not include raw prompts/responses, only reference IDs (2) Confirm Guest responses do not include model name/cost/infrastructure info |
PASS when raw content not in response + Guest exposure control confirmed |
Verification Flow by Scenario
SC-001: grep signup/register -> assert role='guest' -> grep guestBlock/guestDeny -> assert exists
SC-002: grep fetch(https://) in client -> assert 0 hits -> grep gateway log -> assert cost fields -> assert cache check
SC-003: grep firebase/firestore in client/ -> assert 0 hits -> grep authMiddleware in functions/ -> assert exists
SC-004: grep catch blocks -> assert logAction in each -> grep roleChange -> assert auditLog
SC-005: grep res.json for raw text -> assert 0 hits -> grep guest response -> assert no model/cost info
Role-based Go/No-Go Checkpoints
After the deploy gate verdict, the role's Go/No-Go checkpoints must be cleared based on the grade. 4 roles × 5 items = 20 checkpoints total.
Service Stability (5 items)
| # |
Checkpoint |
Go Condition |
No-Go Condition |
| 1 |
SLA Impact Analysis |
Confirmed no impact on existing service availability/response-time SLA |
SLA impact unanalyzed or degradation expected |
| 2 |
Rollback Procedure |
Rollback procedure documented + tested |
Rollback procedure not established |
| 3 |
Performance Test |
Load/stress test completed + within threshold |
Performance test not run |
| 4 |
Incident Alerts |
Incident detection alert channels (Slack/PagerDuty, etc.) configured |
Alert channels not configured |
| 5 |
Monitoring Dashboard |
Dashboard for key metrics (error rate, response time, AI cost) exists |
Monitoring absent |
Engineer (5 items)
| # |
Checkpoint |
Go Condition |
No-Go Condition |
| 1 |
FAIL Rule Root Cause Analysis |
Root cause identified + documented for all FAIL rules |
Unidentified items exist |
| 2 |
Fix Code Verification |
Fixed code accurately reflects the intent of the rule |
Fix does not match rule intent |
| 3 |
Re-verification Pass |
Rule transitions to PASS in re-verification after fix |
Re-verification not run or still FAIL |
| 4 |
No Regression Impact |
Fix confirmed to have no negative impact on other P0 rules |
Another rule newly FAILs |
| 5 |
Code Review Done |
Code review approval completed for fixed code |
Code review not completed |
PM (5 items)
| # |
Checkpoint |
Go Condition |
No-Go Condition |
| 1 |
User Impact Assessment |
User impact of non-compliant items is acceptable |
User impact not assessed |
| 2 |
Schedule Risk |
Fix timeline is within release schedule |
Schedule overrun expected |
| 3 |
Scope Agreement |
Stakeholder agreement completed for scope changes |
Agreement not reached |
| 4 |
Cost Impact |
AI usage cost within approved budget |
Budget overrun expected |
| 5 |
Communication |
Changes shared with relevant teams |
Not shared |
CEO (5 items)
| # |
Checkpoint |
Go Condition |
No-Go Condition |
| 1 |
Cost Cap |
Monthly AI cost within pre-approved budget |
Budget cap exceeded |
| 2 |
Security Risk |
All security P0 passed or exception reason is reasonable |
P0 security FAIL + insufficient exception justification |
| 3 |
Legal/Regulatory Risk |
Data processing complies with applicable laws (privacy laws, etc.) |
Legal risks not reviewed |
| 4 |
Business Continuity |
Business impact is limited if deployment fails |
Business disruption risk exists |
| 5 |
Final Approval |
Final approval when all 4 above are Go |
Deferred if even 1 is No-Go |
Report Format
compliance-report.md, generated when /compliance-verify runs, consists of 6 sections.
Report Section Structure (6 sections)
# Compliance Report
## 1. Summary
- Project name, verification date/time, verification mode (quick-scan / full-verify)
- Total compliance score / 100
- Deploy gate grade (Green / Yellow / Red)
- P0 FAIL count
- Verification duration
## 2. Rule Results
|---------|----------|
