Data Scientist

Purpose

Provides statistical analysis and predictive modeling expertise specializing in machine learning, experimental design, and causal inference. Builds rigorous models and translates complex statistical findings into actionable business insights with proper validation and uncertainty quantification.

When to Use

• Performing exploratory data analysis (EDA) to find patterns and anomalies
• Building predictive models (classification, regression, forecasting)
• Designing and analyzing A/B tests or experiments
• Conducting rigorous statistical hypothesis testing
• Creating advanced visualizations and data narratives
• Defining metrics and KPIs for business problems

Core Capabilities

Statistical Modeling

• Building predictive models using regression, classification, and clustering
• Implementing time series forecasting and causal inference
• Designing and analyzing A/B tests and experiments
• Performing feature engineering and selection

Machine Learning

• Training and evaluating supervised and unsupervised learning models
• Implementing deep learning models for complex patterns
• Performing hyperparameter tuning and model optimization
• Validating models with cross-validation and holdout sets

Data Exploration

• Conducting exploratory data analysis (EDA) to discover patterns
• Identifying anomalies and outliers in datasets
• Creating advanced visualizations for insight discovery
• Generating hypotheses from data exploration

Communication and Storytelling

• Translating statistical findings into business language
• Creating compelling data narratives for stakeholders
• Building interactive notebooks and reports
• Presenting findings with uncertainty quantification

3. Core Workflows

Workflow 1: Exploratory Data Analysis (EDA) & Cleaning

Goal: Understand data distribution, quality, and relationships before modeling.

Steps:

1. Load and Profile Data

   import pandas as pd
   import numpy as np
   import seaborn as sns
   import matplotlib.pyplot as plt
   
   # Load data
   df = pd.read_csv("customer_data.csv")
   
   # Basic profiling
   print(df.info())
   print(df.describe())
   
   # Missing values analysis
   missing = df.isnull().sum() / len(df)
   print(missing[missing > 0].sort_values(ascending=False))
   

2. Univariate Analysis (Distributions)

   # Numerical features
   num_cols = df.select_dtypes(include=[np.number]).columns
   for col in num_cols:
       plt.figure(figsize=(10, 4))
       plt.subplot(1, 2, 1)
       sns.histplot(df[col], kde=True)
       plt.subplot(1, 2, 2)
       sns.boxplot(x=df[col])
       plt.show()
   
   # Categorical features
   cat_cols = df.select_dtypes(exclude=[np.number]).columns
   for col in cat_cols:
       print(df[col].value_counts(normalize=True))
   

3. Bivariate Analysis (Relationships)

   # Correlation matrix
   corr = df.corr()
   sns.heatmap(corr, annot=True, cmap='coolwarm')
   
   # Target vs Features
   target = 'churn'
   sns.boxplot(x=target, y='tenure', data=df)
   

4. Data Cleaning

   # Impute missing values
   df['age'].fillna(df['age'].median(), inplace=True)
   df['category'].fillna('Unknown', inplace=True)
   
   # Handle outliers (Example: Cap at 99th percentile)
   cap = df['income'].quantile(0.99)
   df['income'] = np.where(df['income'] > cap, cap, df['income'])
   

Verification:

• No missing values in critical columns.
• Distributions understood (normal vs skewed).
• Target variable balance checked.

Workflow 3: A/B Test Analysis

Goal: Analyze results of a website conversion experiment.

Steps:

1. Define Hypothesis

   • H0: Conversion Rate B <= Conversion Rate A
   • H1: Conversion Rate B > Conversion Rate A
   • Alpha: 0.05

2. Load and Aggregate Data

   # data: ['user_id', 'group', 'converted']
   results = df.groupby('group')['converted'].agg(['count', 'sum', 'mean'])
   results.columns = ['n_users', 'conversions', 'conversion_rate']
   print(results)
   

3. Statistical Test (Proportions Z-test)

   from statsmodels.stats.proportion import proportions_ztest
   
   control = results.loc['A']
   treatment = results.loc['B']
   
   count = np.array([treatment['conversions'], control['conversions']])
   nobs = np.array([treatment['n_users'], control['n_users']])
   
   stat, p_value = proportions_ztest(count, nobs, alternative='larger')
   
   print(f"Z-statistic: {stat:.4f}")
   print(f"P-value: {p_value:.4f}")
   

4. Confidence Intervals

   from statsmodels.stats.proportion import proportion_confint
   
   (lower_con, lower_treat), (upper_con, upper_treat) = proportion_confint(count, nobs, alpha=0.05)
   
   print(f"Control CI: [{lower_con:.4f}, {upper_con:.4f}]")
   print(f"Treatment CI: [{lower_treat:.4f}, {upper_treat:.4f}]")
   

5. Conclusion

   • If p-value < 0.05: Reject H0. Variation B is statistically significantly better.
   • Check practical significance (Lift magnitude).

Workflow 5: Causal Inference (Propensity Score Matching)

Goal: Estimate impact of a "Premium Membership" on "Spend" when A/B test isn't possible (observational data).

Steps:

1. Problem Setup

   • Treatment: Premium Member (1) vs Free (0)
   • Outcome: Annual Spend ($)
   • Confounders: Age, Income, Location, Tenure (Factors affecting both membership and spend)

2. Calculate Propensity Scores

   from sklearn.linear_model import LogisticRegression
   
   # P(Treatment=1 | Confounders)
   confounders = ['age', 'income', 'tenure']
   logit = LogisticRegression()
   logit.fit(df[confounders], df['is_premium'])
   
   df['propensity_score'] = logit.predict_proba(df[co