Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
Fetches spark-optimization from sickn33/antigravity-awesome-skills and configures it for Cursor.
3
Select Cursor when prompted
The CLI shows a list of agents. Use arrow keys and space to select Cursor:
◆ Which agents do you want to install to?
│
│ ── Universal (.agents/skills) ────────────────
│ · Cline · Codex · Goose · Windsurf
│ ●Cursor(selected)
│ · Cursor · Aider · Continue
4
Verify installation
Confirm successful installation by checking the skill directory location:
.cursor/skills/spark-optimization
Restart Cursor to activate spark-optimization. Access via /spark-optimization in your agent's command palette.
⚠
Security Notice
We perform automated surface-level scans (Gen AI Scanner, Socket, Snyk) during installation. These checks detect common vulnerabilities but do not guarantee complete security. Always review skill source code and verify the publisher's reputation before production use.
Skills execute code in your environment. Always review source, verify the publisher, and test in isolation before production.
Production patterns for optimizing Apache Spark jobs including partitioning strategies, memory management, shuffle optimization, and performance tuning.
Do not use this skill when
The task is unrelated to apache spark optimization
You need a different domain or tool outside this scope
Instructions
Clarify goals, constraints, and required inputs.
Apply relevant best practices and validate outcomes.
Provide actionable steps and verification.
If detailed examples are required, open resources/implementation-playbook.md.
Use this skill when
Optimizing slow Spark jobs
Tuning memory and executor configuration
Implementing efficient partitioning strategies
Debugging Spark performance issues
Scaling Spark pipelines for large datasets
Reducing shuffle and data skew
Core Concepts
1. Spark Execution Model
Driver Program
↓
Job (triggered by action)
↓
Stages (separated by shuffles)
↓
Tasks (one per partition)
2. Key Performance Factors
Factor
Impact
Solution
Shuffle
Network I/O, disk I/O
Minimize wide transformations
Data Skew
Uneven task duration
Salting, broadcast joins
Serialization
CPU overhead
Use Kryo, columnar formats
Memory
GC pressure, spills
Tune executor memory
Partitions
Parallelism
Right-size partitions
Quick Start
from pyspark.sql import SparkSession
from pyspark.sql import functions as F
# Create optimized Spark sessionspark =(SparkSession.builder
.appName("OptimizedJob").config("spark.sql.adaptive.enabled","true").config("spark.sql.adaptive.coalescePartitions.enabled","true").config("spark.sql.adaptive.skewJoin.enabled","true").config("spark.serializer","org.apache.spark.serializer.KryoSerializer").config("spark.sql.shuffle.partitions","200").getOrCreate())# Read with optimized settingsdf =(spark.read
.format("parquet").option("mergeSchema","false").load("s3://bucket/data/"))# Efficient transformationsresult =(df
.filter(F.col("date")>="2024-01-01").select("id","amount","category").groupBy("category").agg(F.sum("amount").alias("total")))result.write.mode("overwrite").parquet("s3://bucket/output/")
Patterns
Pattern 1: Optimal Partitioning
# Calculate optimal partition countdefcalculate_partitions(data_size_gb:float, partition_size_mb:int=128)->int:"""
Optimal partition size: 128MB - 256MB
Too few: Under-utilization, memory pressure
Too many: Task scheduling overhead
"""returnmax(int(data_size_gb *1024/ partition_size_mb),1)# Repartition for even distributiondf_repartitioned = df.repartition(200,"partition_key")# Coalesce to reduce partitions (no shuffle)df_coalesced = df.coalesce(100)# Partition pruning with predicate pushdowndf =(spark.read.parquet("s3://bucket/data/").filter(F.col("date")=="2024-01-01"))# Spark pushes this down# Write with partitioning for future queries(df.write
.partitionBy("year","month","day").mode("overwrite").parquet("s3://bucket/partitioned_output/"))
Pattern 2: Join Optimization
from pyspark.sql import functions as F
from pyspark.sql.types import*# 1. Broadcast Join - Small table joins# Best when: One side < 10MB (configurable)small_df = spark.read.parquet("s3://bucket/small_table/")# < 10MBlarge_df = spark.read.parquet("s3://bucket/large_table/")# TBs# Explicit broadcast hintresult = large_df.join( F.broadcast(small_df), on="key", how="left")# 2. Sort-Merge Join - Default for large tables# Requires shuffle, but handles any sizeresult = large_df1.join(large_df2, on="key", how="inner")# 3. Bucket Join - Pre-sorted, no shuffle at join time# Write bucketed tables(df.write
.bucketBy(200,"customer_id").sortBy("customer_id").mode("overwrite").saveAsTable("bucketed_orders"))# Join bucketed tables (no shuffle!)orders = spark.table("bucketed_orders")customers = spark.table("bucketed_customers")# Same bucket countresult = orders.join(customers, on="customer_id")# 4. Skew Join Handling# Enable AQE skew join optimizationspark.conf.set("spark.sql.adaptive.skewJoin.enabled","true")spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionFactor","5")spark.conf.set("spark.sql.adaptive.skewJoin.skewedPartitionThresholdInBytes","256MB")# Manual salting for severe skewdefsalt_join(df_skewed, df_other, key_col, num_salts=10):"""Add salt to distribute skewed keys"""# Add salt to skewed side df_salted = df_skewed.withColumn("salt",(F.rand()* num_salts).cast("int")).withColumn("salted_key", F.concat(F.col(key_col), F.lit("_"), F.col("salt")))# Explode other side with all salts df_exploded = df_other.crossJoin( spark.range(num_salts).withColumnRenamed("id","salt")).withColumn("salted_key"
✓
Make data-driven prioritization decisions faster
Stakeholder Communication
Draft PRDs, status updates, and stakeholder presentations
›Access to product documentation and roadmap tools (Jira, Notion, etc.)
›Understanding of product management frameworks (RICE, Jobs-to-be-Done, etc.)
›Stakeholder contact information and communication channels
Time Estimate
30-60 minutes to see productivity improvements
Steps
1Install product management skill
2Start with user story generation for known feature
3Progress to competitive analysis: research 2-3 competitors
4Use for roadmap prioritization: apply RICE/ICE scoring
5Draft stakeholder communications and refine based on feedback
6Build template library for recurring PM tasks
7Share effective prompts with product team
Common Pitfalls
⚠Not validating competitive research—verify facts before sharing
⚠Accepting user stories without involving engineering team
⚠Over-relying on frameworks without qualitative judgment
⚠Not customizing outputs to company culture and communication style
⚠Skipping stakeholder validation of generated requirements
Best Practices
✓ Do
+Validate research and competitive analysis with real data
+Collaborate with engineering when generating technical requirements
+Customize frameworks and templates to your company context
+Use skill for first drafts, refine with stakeholder input
+Document successful prompt patterns for PM tasks
+Combine AI efficiency with human judgment and intuition
✗ Don't
−Don't publish competitive analysis without fact-checking
−Don't finalize user stories without engineering review
−Don't make prioritization decisions solely on AI scoring
−Don't skip customer validation of generated requirements
−Don't ignore company-specific context and culture
💡 Pro Tips
★Provide context: company goals, constraints, customer feedback
★Ask for alternatives: 'Show 3 ways to prioritize this roadmap'
★Request stakeholder-specific formatting: 'Executive summary vs. engineering spec'
★Use skill for 70% generation + 30% customization to company needs
When to Use This
✓ Use when
Use for user story writing, competitive research, roadmap prioritization, stakeholder communication, and PRD drafting. Best for reducing repetitive documentation and research work.
✗ Avoid when
Avoid for strategic product vision (requires deep customer empathy), pricing decisions (needs market and financial expertise), or when face-to-face customer discovery is more valuable than speed.
Learning Path
1Basic: user stories, feature specs, status updates
