GrepAI Chunking Configuration

This skill covers how GrepAI splits code files into chunks for embedding, and how to optimize chunking for your codebase.

When to Use This Skill

• Optimizing search accuracy
• Adjusting for code style (verbose vs. concise)
• Troubleshooting search results
• Understanding how indexing works

What is Chunking?

Chunking is the process of splitting source files into smaller segments for embedding:

┌─────────────────────────────────────┐
│         Large Source File           │
│         (1000+ tokens)              │
└─────────────────────────────────────┘
                  ↓
┌─────────┐ ┌─────────┐ ┌─────────┐
│ Chunk 1 │ │ Chunk 2 │ │ Chunk 3 │
│ ~512    │ │ ~512    │ │ ~512    │
│ tokens  │ │ tokens  │ │ tokens  │
└─────────┘ └─────────┘ └─────────┘
                  ↓
          Each chunk gets
          its own embedding

Why Chunking Matters

Embedding models have optimal input sizes:

• Too large chunks: Less precise search results
• Too small chunks: Lost context, fragmented results
• Just right: Good balance of precision and context

Configuration

Basic Settings

# .grepai/config.yaml
chunking:
  size: 512      # Tokens per chunk
  overlap: 50    # Overlap between chunks

Understanding Parameters

Chunk Size

The target number of tokens per chunk.

Size	Effect
256	More precise, less context
512	Balanced (default)
1024	More context, less precise

Overlap

Tokens shared between adjacent chunks. Preserves context at boundaries.

Overlap	Effect
0	No overlap, may lose context at boundaries
50	Standard overlap (default)
100	More context, larger index

Visualization

With size=512 and overlap=50:

File: auth.go (1000 tokens)

Chunk 1: tokens 1-512
         ┌────────────────────────────────────┐
         │ func Login(user, pass)...          │
         └────────────────────────────────────┘
                                    ↘
                              50 token overlap
                                    ↙
Chunk 2: tokens 463-974
         ┌────────────────────────────────────┐
         │ ...validate credentials...         │
         └────────────────────────────────────┘
                                    ↘
                              50 token overlap
                                    ↙
Chunk 3: tokens 925-1000
         ┌──────────────┐
         │ ...return    │
         └──────────────┘

Recommended Settings by Language

Verbose Languages (Java, C#)

chunking:
  size: 768    # Larger to capture full methods
  overlap: 75

Concise Languages (Go, Python)

chunking:
  size: 512    # Standard size
  overlap: 50

Very Concise (Rust, Zig)

chunking:
  size: 384    # Smaller for precise results
  overlap: 40

Recommended Settings by Codebase

Small Functions (Microservices)

chunking:
  size: 384    # Capture individual functions
  overlap: 40

Large Classes (Monolith)

chunking:
  size: 768    # Capture more context
  overlap: 100

Mixed Codebase

chunking:
  size: 512    # Balanced default
  overlap: 50

How Tokens are Counted

GrepAI uses approximate token counting:

• ~4 characters = 1 token (for English text)
• Code varies based on identifiers and syntax

Example:

func calculateTotal(items []Item) float64 {
    total := 0.0
    for _, item := range items {
        total += item.Price * float64(item.Quantity)
    }
    return total
}

≈ 45 tokens

Impact on Index Size

Larger overlap = more chunks = larger index:

Size	Overlap	Chunks per 10K tokens	Index Impact
512	0	~20	Smallest
512	50	~22	Standard
512	100	~24	+10%
256	50	~44	+100%

Impact on Search Quality

Too Small Chunks (size: 128)

Query: "authentication middleware"

Result: "...c.AbortWithStatus(401)..."
        (Fragment, missing context)

Just Right (size: 512)

Query: "authentication middleware"

Result: "func AuthMiddleware() gin.HandlerFunc {
            return func(c *gin.Context) {
                token := c.GetHeader("Authorization")
                if token == "" {
                    c.AbortWithStatus(401)
                    return
                }
                // validate token...
            }
        }"
        (Complete function with context)

Too Large Chunks (size: 2048)

Query: "authentication middleware"

Result: "// Multiple unrelated functions...
        func AuthMiddleware()... (your match)
        func LoggingMiddleware()...
        func CORSMiddleware()..."
        (Too much noise)

Experimentation

Testing Different Settings

1. Try smaller chunks for more precise results:

chunking:
  size: 384
  overlap: 40

2. Re-index:

rm .grepai/index.gob
grepai watch

3. Test with searches:

grepai search "your query"

4. Adjust and repeat until satisfied.

Comparing Results

Before changing settings, save a search result:

grepai search "authentication" > before.txt

After changing settings and re-indexing:

grepai search "authentication" > after.txt
diff before.txt after.txt

Chunk Boundaries

GrepAI tries to split at logical boundaries:

1. Empty lines (function/class boundaries)
2. Closing braces
3. Statement ends

This means actual chunk sizes may vary slightly from the target.

Best Practices

1. Start with defaults: 512/50 works well for most codebases
2. Adjust based on code style: Verbose = larger, concise = smaller
3. Test with real queries: See what your searches return
4. Re-index after changes: Must regenerate embeddings
5. Consider overlap: Don't set to 0 unless index size is critical

Common Issues

❌ Problem: Search results are too fragmented ✅ Solution: Increase chunk size:

chunking:
  size: 768

❌ Problem: Search results have too much irrelevant context ✅ Solution: Decrease chunk size:

chunking:
  size: 384

❌ Problem: Results miss related code at function boundaries ✅ Solution: Increase overlap:

chunking:
  overlap: 100

❌ Problem: Index is too large ✅ Solutions:

• Decrease overlap
• Increase chunk size
• Add more ignore patterns

Output Format

Chunking status:

✅ Chunking Configuration

   Size: 512 tokens
   Overlap: 50 tokens

   Index Statistics:
   - Total files: 245
   - Total chunks: 1,234
   - Avg chunks/file: 5.0
   - Avg chunk size: 478 tokens

   Recommendations:
   - Current settings are balanced
   - Consider size: 384 for more precise results
   - Consider size: 768 for more context