GrepAI Storage with Qdrant
This skill covers using Qdrant as the storage backend for GrepAI, offering high-performance vector search.
When to Use This Skill
- Need fastest possible search performance
- Very large codebases (50K+ files)
- Already using Qdrant infrastructure
- Want advanced vector search features
What is Qdrant?
Qdrant is a purpose-built vector database offering:
- ⚡ Extremely fast vector similarity search
- 📏 Excellent scalability
- 🔧 Advanced filtering capabilities
- 🐳 Easy Docker deployment
Prerequisites
- Qdrant server running
- Network access to Qdrant
Advantages
| Benefit |
Description |
| ⚡ Performance |
Fastest vector search |
| 📏 Scalability |
Handles millions of vectors |
| 🔍 Advanced |
Filtering, payloads, sharding |
| 🐳 Easy deploy |
Docker-ready |
| ☁️ Cloud option |
Qdrant Cloud available |
Setting Up Qdrant
Option 1: Docker (Recommended)
docker run -d \
--name grepai-qdrant \
-p 6333:6333 \
-p 6334:6334 \
-v qdrant_storage:/qdrant/storage \
qdrant/qdrant
Ports:
6333: REST API6334: gRPC API (used by GrepAI)
Option 2: Docker Compose
version: '3.8'
services:
qdrant:
image: qdrant/qdrant
ports:
- "6333:6333"
- "6334:6334"
volumes:
- qdrant_storage:/qdrant/storage
environment:
- QDRANT__SERVICE__GRPC_PORT=6334
volumes:
qdrant_storage:
docker-compose up -d
Option 3: Qdrant Cloud
- Sign up at cloud.qdrant.io
- Create a cluster
- Get your endpoint and API key
Configuration
Basic Configuration (Local)
store:
backend: qdrant
qdrant:
endpoint: localhost
port: 6334
With TLS (Production)
store:
backend: qdrant
qdrant:
endpoint: qdrant.company.com
port: 6334
use_tls: true
With API Key (Qdrant Cloud)
store:
backend: qdrant
qdrant:
endpoint: your-cluster.aws.cloud.qdrant.io
port: 6334
use_tls: true
api_key: ${QDRANT_API_KEY}
Set the environment variable:
export QDRANT_API_KEY="your-api-key"
Configuration Options
| Option |
Default |
Description |
endpoint |
localhost |
Qdrant server hostname |
port |
6334 |
gRPC port |
use_tls |
false |
Enable TLS encryption |
api_key |
none |
Authentication key |
Verifying Setup
Check Qdrant is Running
curl http://localhost:6333/health
Check Collections (after indexing)
curl http://localhost:6333/collections
curl http://localhost:6333/collections/grepai
From GrepAI
grepai status
Qdrant Dashboard
Access the web dashboard at http://localhost:6333/dashboard:
- View collections
- Browse vectors
- Execute queries
- Monitor performance
Performance Characteristics
Search Latency
| Codebase Size |
Vectors |
Search Time |
| Small (1K files) |
5,000 |
<10ms |
| Medium (10K files) |
50,000 |
<20ms |
| Large (100K files) |
500,000 |
<50ms |
Memory Usage
Qdrant loads vectors into memory for fast search:
| Vectors |
Dimensions |
Memory |
| 10,000 |
768 |
~60 MB |
| 100,000 |
768 |
~600 MB |
| 1,000,000 |
768 |
~6 GB |
Advanced Configuration
Qdrant Server Configuration
Create config/production.yaml:
storage:
storage_path: /qdrant/storage
service:
grpc_port: 6334
http_port: 6333
max_request_size_mb: 32
optimizers:
memmap_threshold_kb: 200000
indexing_threshold_kb: 50000
Mount in Docker:
docker run -d \
-v ./config:/qdrant/config \
-v qdrant_storage:/qdrant/storage \
qdrant/qdrant
Collection Settings
GrepAI creates a collection named grepai with:
- Vector size: matches your embedding dimensions
- Distance: Cosine similarity
- On-disk storage for large datasets
Clustering (Advanced)
For very large deployments, Qdrant supports distributed mode:
cluster:
enabled: true
p2p:
port: 6335
Backup and Restore
Snapshot Creation
curl -X POST 'http://localhost:6333/collections/grepai/snapshots'
Restore Snapshot
curl -X PUT 'http://localhost:6333/collections/grepai/snapshots/recover' \
-H 'Content-Type: application/json' \
-d '{"location": "/path/to/snapshot"}'
Migrating from GOB
- Start Qdrant:
docker run -d --name qdrant -p 6333:6333 -p 6334:6334 qdrant/qdrant
- Update configuration:
store:
backend: qdrant
qdrant:
endpoint: localhost
port: 6334
- Delete old index:
rm .grepai/index.gob
- Re-index:
grepai watch
Migrating from PostgreSQL
- Start Qdrant
- Update configuration to use Qdrant
- Re-index (embeddings must be regenerated)
Common Issues
❌ Problem: Connection refused
✅ Solution: Ensure Qdrant is running:
docker ps | grep qdrant
docker start grepai-qdrant
❌ Problem: gRPC connection failed
✅ Solution: Check port 6334 is exposed:
docker run -p 6334:6334 ...
❌ Problem: Authentication failed
✅ Solution: Check API key:
echo $QDRANT_API_KEY
❌ Problem: Out of memory
✅ Solutions:
- Enable on-disk storage in Qdrant config
- Increase Docker memory limit
- Use Qdrant Cloud for managed scaling
❌ Problem: Slow initial indexing
✅ Solution: This is normal; Qdrant optimizes in background. Searches will be fast after indexing completes.
Qdrant vs PostgreSQL
| Feature |
Qdrant |
PostgreSQL |
| Search speed |
⚡⚡⚡ |
⚡⚡ |
| Setup complexity |
Easy (Docker) |
Medium |
| SQL queries |
❌ |
✅ |
| Scalability |
Excellent |
Good |
| Memory efficiency |
Excellent |
Good |
| Team familiarity |
Lower |
Higher |
Recommendation: Use Qdrant for large codebases or maximum performance. Use PostgreSQL if you need SQL integration or team is familiar with it.
Best Practices
- Use persistent volume: Mount
/qdrant/storage
- Enable TLS in production: Set
use_tls: true
- Secure API key: Use environment variables
- Monitor memory: Vector search is memory-intensive
- Regular snapshots: Backup before major changes
Output Format
Qdrant storage status:
✅ Qdrant Storage Configured
Backend: Qdrant
Endpoint: localhost:6334
TLS: disabled
Collection: grepai
Contents:
- Files: 5,000
- Vectors: 25,000
- Dimensions: 768
Performance:
- Connection: OK
- Indexed: Yes
- Search latency: ~15ms
