mirror of
https://github.com/block/goose.git
synced 2026-07-17 12:56:20 +02:00
feat: add A/B test framework generator recipe (#5378)
Signed-off-by: Shreyansh Singh Gautam <shreyanshrewa@gmail.com>
This commit is contained in:
Shreyansh Singh Gautam
committed by
GitHub
GitHub
parent
c3578b708f
commit
491eabfc81
@@ -0,0 +1,268 @@
|
||||
version: 1.0.0
|
||||
title: A/B Test Framework Generator
|
||||
description: An advanced recipe that generates complete A/B testing infrastructure for web applications, including variant setup, tracking code, statistical analysis, and interactive reporting dashboard with intelligent framework detection and multi-stage orchestration
|
||||
author:
|
||||
contact: scaler
|
||||
|
||||
activities:
|
||||
- Detect web application framework and project structure
|
||||
- Generate A/B test variant implementation templates
|
||||
- Create tracking and analytics integration code
|
||||
- Set up experiment configuration and user bucketing
|
||||
- Implement statistical significance analysis framework
|
||||
- Generate interactive reporting dashboard with real-time metrics
|
||||
- Create comprehensive documentation and setup guide
|
||||
- Optionally commit changes and create pull request
|
||||
|
||||
instructions: |
|
||||
You are an A/B Test Framework Generator that creates complete testing infrastructure for web applications.
|
||||
|
||||
Your capabilities:
|
||||
1. Detect web frameworks (React, Vue, Angular, vanilla JS) and adapt implementations
|
||||
2. Generate variant-specific code templates with proper randomization
|
||||
3. Create tracking event handlers and analytics integration
|
||||
4. Set up statistical analysis framework for significance testing
|
||||
5. Build interactive dashboards for real-time experiment monitoring
|
||||
6. Orchestrate multiple sub-recipes for specialized tasks
|
||||
7. Handle parameter passing and conditional logic based on framework type
|
||||
|
||||
Focus on:
|
||||
- Production-ready A/B testing infrastructure
|
||||
- Statistical rigor with proper significance testing
|
||||
- Framework-specific implementations
|
||||
- Real-time monitoring and reporting
|
||||
- Comprehensive documentation and setup guides
|
||||
- Manual file operations (users will need to commit changes themselves)
|
||||
|
||||
parameters:
|
||||
- key: project_path
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Path to the web application project directory to add A/B testing infrastructure
|
||||
|
||||
- key: framework
|
||||
input_type: string
|
||||
requirement: optional
|
||||
default: "auto"
|
||||
description: Web framework type - options are 'auto', 'react', 'vue', 'angular', 'vanilla'
|
||||
|
||||
- key: test_name
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Name of the A/B test (e.g., 'button-color-test', 'checkout-flow-test')
|
||||
|
||||
- key: variants
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Comma-separated variant names (e.g., 'control,variant-a,variant-b')
|
||||
|
||||
- key: metrics
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Comma-separated metrics to track (e.g., 'conversion,engagement,bounce-rate,click-through')
|
||||
|
||||
- key: sample_size
|
||||
input_type: string
|
||||
requirement: optional
|
||||
default: "1000"
|
||||
description: Minimum sample size per variant for statistical significance
|
||||
|
||||
- key: confidence_level
|
||||
input_type: string
|
||||
requirement: optional
|
||||
default: "95"
|
||||
description: Statistical confidence level for significance testing (90, 95, 99)
|
||||
|
||||
- key: include_dashboard
|
||||
input_type: string
|
||||
requirement: optional
|
||||
default: "true"
|
||||
description: Whether to generate interactive reporting dashboard (true/false)
|
||||
|
||||
|
||||
sub_recipes:
|
||||
- name: "experiment_tracker"
|
||||
path: "./subrecipes/experiment-tracker.yaml"
|
||||
values:
|
||||
test_name: "{{ test_name }}"
|
||||
variants: "{{ variants }}"
|
||||
metrics: "{{ metrics }}"
|
||||
framework: "{{ framework }}"
|
||||
|
||||
- name: "statistical_analyzer"
|
||||
path: "./subrecipes/ab-test-statistical-analyzer.yaml"
|
||||
values:
|
||||
sample_size: "{{ sample_size }}"
|
||||
confidence_level: "{{ confidence_level }}"
|
||||
metrics: "{{ metrics }}"
|
||||
|
||||
- name: "dashboard_generator"
|
||||
path: "./subrecipes/ab-test-dashboard-generator.yaml"
|
||||
values:
|
||||
test_name: "{{ test_name }}"
|
||||
variants: "{{ variants }}"
|
||||
metrics: "{{ metrics }}"
|
||||
include_dashboard: "{{ include_dashboard }}"
|
||||
|
||||
extensions:
|
||||
- type: builtin
|
||||
name: developer
|
||||
display_name: Developer
|
||||
timeout: 600
|
||||
bundled: true
|
||||
description: For file operations, code generation, and framework detection
|
||||
|
||||
- type: builtin
|
||||
name: memory
|
||||
display_name: Memory
|
||||
timeout: 300
|
||||
bundled: true
|
||||
description: For storing experiment configurations and tracking patterns across sessions
|
||||
|
||||
prompt: |
|
||||
Generate complete A/B testing infrastructure for {{ project_path }} with test "{{ test_name }}" and variants: {{ variants }}.
|
||||
|
||||
CRITICAL: Handle file paths correctly for all operating systems.
|
||||
- Detect the operating system (Windows/Linux/Mac)
|
||||
- Use appropriate path separators (/ for Unix, \\ for Windows)
|
||||
- Be careful to avoid escaping of slash or backslash characters
|
||||
- Use os.path.join() or pathlib.Path for cross-platform paths
|
||||
- Create A/B test directories if they don't exist
|
||||
|
||||
Workflow:
|
||||
1. Framework Detection & Project Analysis
|
||||
- Detect web framework in {{ project_path }}:
|
||||
* Look for package.json with React/Vue/Angular dependencies
|
||||
* Check for framework-specific files (src/, components/, etc.)
|
||||
* Identify build system (webpack, vite, rollup, etc.)
|
||||
* Store framework detection results in memory
|
||||
- Analyze project structure for integration points:
|
||||
* Identify entry points and main components
|
||||
* Check for existing analytics/tracking setup
|
||||
* Determine state management approach
|
||||
* Note CSS framework and styling approach
|
||||
|
||||
2. Experiment Configuration Setup
|
||||
- Create experiment configuration structure:
|
||||
* Generate experiment config JSON/YAML file
|
||||
* Define variant specifications and traffic allocation
|
||||
* Set up user bucketing and randomization logic
|
||||
* Configure metrics tracking definitions
|
||||
* Store configuration in memory for sub-recipe use
|
||||
|
||||
3. Variant Implementation Templates
|
||||
{% if framework == "react" or framework == "auto" %}
|
||||
- Generate React-specific templates:
|
||||
* A/B test hook (useABTest) for component variants
|
||||
* Higher-order component for variant wrapping
|
||||
* Context provider for experiment state management
|
||||
* TypeScript definitions for type safety
|
||||
{% endif %}
|
||||
{% if framework == "vue" or framework == "auto" %}
|
||||
- Generate Vue-specific templates:
|
||||
* Vue composable for A/B test logic
|
||||
* Mixin for component variant handling
|
||||
* Plugin for global experiment management
|
||||
* TypeScript support for Vue 3
|
||||
{% endif %}
|
||||
{% if framework == "angular" or framework == "auto" %}
|
||||
- Generate Angular-specific templates:
|
||||
* Service for experiment management
|
||||
* Directive for variant rendering
|
||||
* Guard for experiment-based routing
|
||||
* Module configuration
|
||||
{% endif %}
|
||||
{% if framework == "vanilla" or framework == "auto" %}
|
||||
- Generate vanilla JS templates:
|
||||
* Core A/B test library
|
||||
* DOM manipulation utilities
|
||||
* Event tracking helpers
|
||||
* Browser compatibility layer
|
||||
{% endif %}
|
||||
|
||||
4. Tracking & Analytics Integration
|
||||
- Create tracking event handlers:
|
||||
* Variant assignment tracking
|
||||
* Conversion event tracking
|
||||
* User behavior analytics
|
||||
* Performance metrics collection
|
||||
- Set up data collection pipeline:
|
||||
* Local storage for user assignments
|
||||
* Cookie-based persistence
|
||||
* API endpoints for data submission
|
||||
* Error handling and fallbacks
|
||||
|
||||
5. Run Experiment Tracker Sub-recipe
|
||||
- Execute experiment_tracker sub-recipe with:
|
||||
* test_name: {{ test_name }}
|
||||
* variants: {{ variants }}
|
||||
* metrics: {{ metrics }}
|
||||
* framework: {{ framework }}
|
||||
- Capture returned tracking code and configuration
|
||||
- Store results in memory for dashboard generation
|
||||
|
||||
6. Statistical Analysis Framework
|
||||
- Run statistical_analyzer sub-recipe with:
|
||||
* sample_size: {{ sample_size }}
|
||||
* confidence_level: {{ confidence_level }}
|
||||
* metrics: {{ metrics }}
|
||||
- Generate statistical analysis utilities:
|
||||
* Chi-square test for categorical metrics
|
||||
* T-test for continuous metrics
|
||||
* Confidence interval calculations
|
||||
* Sample size determination
|
||||
* P-value calculations
|
||||
|
||||
7. Dashboard Generation
|
||||
{% if include_dashboard == "true" %}
|
||||
- Run dashboard_generator sub-recipe with:
|
||||
* test_name: {{ test_name }}
|
||||
* variants: {{ variants }}
|
||||
* metrics: {{ metrics }}
|
||||
* include_dashboard: {{ include_dashboard }}
|
||||
- Create interactive reporting dashboard:
|
||||
* Real-time metrics visualization
|
||||
* Statistical significance indicators
|
||||
* Conversion funnel analysis
|
||||
* Export functionality for reports
|
||||
{% endif %}
|
||||
|
||||
8. Documentation & Setup Guide
|
||||
- Generate comprehensive documentation:
|
||||
* README with setup instructions
|
||||
* API documentation for A/B test functions
|
||||
* Integration examples for each framework
|
||||
* Troubleshooting guide
|
||||
* Best practices and recommendations
|
||||
- Create setup scripts:
|
||||
* Installation script for dependencies
|
||||
* Configuration validation script
|
||||
* Test runner for A/B test infrastructure
|
||||
|
||||
9. File Organization
|
||||
- Create organized directory structure:
|
||||
* ab-tests/experiments/{{ test_name }}/
|
||||
* ab-tests/shared/ (common utilities)
|
||||
* ab-tests/dashboard/ (reporting interface)
|
||||
* ab-tests/docs/ (documentation)
|
||||
- Ensure all files use OS-compatible paths
|
||||
- Create proper import/export statements
|
||||
|
||||
Error Recovery:
|
||||
- If framework detection fails, default to vanilla JS implementation
|
||||
- If sub-recipe fails, continue with remaining components
|
||||
- Provide fallback implementations for missing dependencies
|
||||
- Log errors clearly with context and recovery suggestions
|
||||
|
||||
Memory Management:
|
||||
- Store experiment configuration for future reference
|
||||
- Track framework-specific patterns for reuse
|
||||
- Maintain A/B test best practices library
|
||||
- Remember user preferences for future experiments
|
||||
|
||||
Focus on creating production-ready A/B testing infrastructure that:
|
||||
- Handles statistical significance properly
|
||||
- Provides real-time monitoring capabilities
|
||||
- Integrates seamlessly with existing codebases
|
||||
- Includes comprehensive documentation and examples
|
||||
- Supports multiple web frameworks and use cases
|
||||
+268
@@ -0,0 +1,268 @@
|
||||
version: 1.0.0
|
||||
title: A/B Test Dashboard Generator
|
||||
description: Creates interactive HTML dashboard for A/B test monitoring with real-time metrics visualization, statistical significance indicators, conversion funnels, and comprehensive reporting capabilities
|
||||
author:
|
||||
contact: scaler
|
||||
|
||||
activities:
|
||||
- Generate interactive HTML dashboard with responsive design
|
||||
- Create real-time metrics visualization and comparison charts
|
||||
- Implement statistical significance indicators and confidence intervals
|
||||
- Build conversion funnel analysis and user journey tracking
|
||||
- Add export functionality for reports and data
|
||||
- Create mobile-responsive interface with modern UI components
|
||||
|
||||
instructions: |
|
||||
You are an A/B Test Dashboard Generator specialized in creating comprehensive monitoring and reporting interfaces.
|
||||
|
||||
Your capabilities:
|
||||
1. Generate interactive HTML dashboards with modern UI/UX
|
||||
2. Create real-time data visualization and metric comparisons
|
||||
3. Implement statistical significance indicators and alerts
|
||||
4. Build conversion funnel analysis and user journey maps
|
||||
5. Add comprehensive reporting and export capabilities
|
||||
6. Ensure mobile-responsive design and accessibility
|
||||
|
||||
Focus on:
|
||||
- Real-time monitoring and updates
|
||||
- Clear visualization of statistical significance
|
||||
- Intuitive user interface and navigation
|
||||
- Comprehensive reporting capabilities
|
||||
- Mobile responsiveness and accessibility
|
||||
|
||||
parameters:
|
||||
- key: test_name
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Name of the A/B test experiment for dashboard title
|
||||
|
||||
- key: variants
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Comma-separated variant names (e.g., 'control,variant-a,variant-b')
|
||||
|
||||
- key: metrics
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Comma-separated metrics to display (e.g., 'conversion,engagement,bounce-rate')
|
||||
|
||||
- key: include_dashboard
|
||||
input_type: string
|
||||
requirement: optional
|
||||
default: "true"
|
||||
description: Whether to generate the dashboard (true/false)
|
||||
|
||||
extensions:
|
||||
- type: builtin
|
||||
name: developer
|
||||
display_name: Developer
|
||||
timeout: 600
|
||||
bundled: true
|
||||
description: For HTML/CSS/JavaScript generation and file operations
|
||||
|
||||
prompt: |
|
||||
Generate interactive A/B test dashboard for experiment "{{ test_name }}" with variants: {{ variants }} and metrics: {{ metrics }}.
|
||||
|
||||
CRITICAL: Handle file paths correctly for all operating systems.
|
||||
- Detect the operating system (Windows/Linux/Mac)
|
||||
- Use appropriate path separators (/ for Unix, \\ for Windows)
|
||||
- Be careful to avoid escaping of slash or backslash characters
|
||||
- Use os.path.join() or pathlib.Path for cross-platform paths
|
||||
|
||||
Workflow:
|
||||
1. Dashboard Structure & Layout
|
||||
{% if include_dashboard == "true" %}
|
||||
- Create main dashboard HTML file (ab-tests/dashboard/{{ test_name }}-dashboard.html):
|
||||
* Responsive layout with CSS Grid/Flexbox
|
||||
* Header with experiment name and status
|
||||
* Navigation sidebar for different views
|
||||
* Main content area for charts and metrics
|
||||
* Footer with last updated timestamp
|
||||
- Generate CSS framework (ab-tests/dashboard/styles.css):
|
||||
* Modern, clean design system
|
||||
* Responsive breakpoints for mobile/tablet/desktop
|
||||
* Color scheme optimized for data visualization
|
||||
* Accessibility features (WCAG 2.1 compliance)
|
||||
{% endif %}
|
||||
|
||||
2. Real-Time Metrics Visualization
|
||||
{% if include_dashboard == "true" %}
|
||||
- Create metrics comparison charts:
|
||||
* Conversion rate comparison (bar chart)
|
||||
* Time-series trends (line chart)
|
||||
* Statistical significance indicators
|
||||
* Confidence interval visualization
|
||||
- Implement interactive features:
|
||||
* Hover tooltips with detailed information
|
||||
* Click-to-drill-down functionality
|
||||
* Date range selection
|
||||
* Metric filtering and grouping
|
||||
{% endif %}
|
||||
|
||||
3. Statistical Significance Display
|
||||
{% if include_dashboard == "true" %}
|
||||
- Generate significance indicators:
|
||||
* P-value display with color coding
|
||||
* Confidence interval visualization
|
||||
* Effect size indicators
|
||||
* Sample size adequacy warnings
|
||||
- Create statistical summary cards:
|
||||
* Current significance status
|
||||
* Required sample size for significance
|
||||
* Estimated time to significance
|
||||
* Power analysis results
|
||||
{% endif %}
|
||||
|
||||
4. Conversion Funnel Analysis
|
||||
{% if include_dashboard == "true" %}
|
||||
- Build funnel visualization:
|
||||
* Step-by-step conversion flow
|
||||
* Drop-off analysis between steps
|
||||
* Variant comparison at each step
|
||||
* User journey mapping
|
||||
- Implement funnel features:
|
||||
* Interactive funnel steps
|
||||
* Conversion rate calculations
|
||||
* Drop-off rate analysis
|
||||
* Revenue impact estimation
|
||||
{% endif %}
|
||||
|
||||
5. Data Tables & Detailed Views
|
||||
{% if include_dashboard == "true" %}
|
||||
- Create comprehensive data tables:
|
||||
* Raw metrics data with sorting/filtering
|
||||
* Statistical test results
|
||||
* User segment breakdowns
|
||||
* Time-based performance data
|
||||
- Add table functionality:
|
||||
* Sortable columns
|
||||
* Search and filter capabilities
|
||||
* Pagination for large datasets
|
||||
* Export to CSV/Excel
|
||||
{% endif %}
|
||||
|
||||
6. Interactive Charts & Graphs
|
||||
{% if include_dashboard == "true" %}
|
||||
- Generate chart library using Chart.js or D3.js:
|
||||
* Bar charts for metric comparisons
|
||||
* Line charts for trend analysis
|
||||
* Pie charts for traffic allocation
|
||||
* Scatter plots for correlation analysis
|
||||
* Heatmaps for user behavior patterns
|
||||
- Implement chart features:
|
||||
* Zoom and pan capabilities
|
||||
* Legend toggling
|
||||
* Data point highlighting
|
||||
* Export as image (PNG/SVG)
|
||||
{% endif %}
|
||||
|
||||
7. Real-Time Updates & API Integration
|
||||
{% if include_dashboard == "true" %}
|
||||
- Create real-time data updates:
|
||||
* WebSocket connection for live updates
|
||||
* REST API integration for data fetching
|
||||
* Automatic refresh intervals
|
||||
* Manual refresh capability
|
||||
- Implement data management:
|
||||
* Local data caching
|
||||
* Offline mode support
|
||||
* Error handling and retry logic
|
||||
* Data validation and sanitization
|
||||
{% endif %}
|
||||
|
||||
8. Export & Reporting Features
|
||||
{% if include_dashboard == "true" %}
|
||||
- Generate export functionality:
|
||||
* PDF report generation
|
||||
* Excel/CSV data export
|
||||
* Image export for charts
|
||||
* Shareable dashboard links
|
||||
- Create reporting templates:
|
||||
* Executive summary report
|
||||
* Detailed statistical report
|
||||
* Custom report builder
|
||||
* Scheduled report delivery
|
||||
{% endif %}
|
||||
|
||||
9. Mobile Responsiveness & Accessibility
|
||||
{% if include_dashboard == "true" %}
|
||||
- Ensure mobile optimization:
|
||||
* Responsive design for all screen sizes
|
||||
* Touch-friendly interface elements
|
||||
* Optimized chart rendering for mobile
|
||||
* Progressive web app features
|
||||
- Implement accessibility features:
|
||||
* Screen reader compatibility
|
||||
* Keyboard navigation support
|
||||
* High contrast mode
|
||||
* Font size adjustment
|
||||
{% endif %}
|
||||
|
||||
10. JavaScript Framework & Utilities
|
||||
{% if include_dashboard == "true" %}
|
||||
- Create dashboard JavaScript (ab-tests/dashboard/dashboard.js):
|
||||
```javascript
|
||||
class ABTestDashboard {
|
||||
constructor(experimentName, variants, metrics) {
|
||||
this.experimentName = experimentName;
|
||||
this.variants = variants;
|
||||
this.metrics = metrics;
|
||||
this.charts = {};
|
||||
this.data = {};
|
||||
}
|
||||
|
||||
async loadData() {
|
||||
// Load experiment data from API
|
||||
}
|
||||
|
||||
renderCharts() {
|
||||
// Render all dashboard charts
|
||||
}
|
||||
|
||||
updateRealTime() {
|
||||
// Update dashboard with real-time data
|
||||
}
|
||||
}
|
||||
```
|
||||
- Implement utility functions:
|
||||
* Data formatting and validation
|
||||
* Chart configuration helpers
|
||||
* API communication utilities
|
||||
* Error handling and logging
|
||||
{% endif %}
|
||||
|
||||
11. Configuration & Customization
|
||||
{% if include_dashboard == "true" %}
|
||||
- Create dashboard configuration:
|
||||
* Theme and color customization
|
||||
* Chart type preferences
|
||||
* Update frequency settings
|
||||
* Notification preferences
|
||||
- Implement user preferences:
|
||||
* Saved dashboard layouts
|
||||
* Custom metric combinations
|
||||
* Personal alert settings
|
||||
* Export format preferences
|
||||
{% endif %}
|
||||
|
||||
12. Performance Optimization
|
||||
{% if include_dashboard == "true" %}
|
||||
- Optimize dashboard performance:
|
||||
* Lazy loading for charts and data
|
||||
* Efficient data processing
|
||||
* Minimal DOM manipulation
|
||||
* Caching strategies
|
||||
- Implement performance monitoring:
|
||||
* Load time tracking
|
||||
* Chart rendering performance
|
||||
* Memory usage optimization
|
||||
* Network request optimization
|
||||
{% endif %}
|
||||
|
||||
Focus on creating a comprehensive dashboard that:
|
||||
- Provides clear, actionable insights
|
||||
- Updates in real-time with accurate data
|
||||
- Works seamlessly across all devices
|
||||
- Includes robust statistical analysis visualization
|
||||
- Offers comprehensive reporting and export capabilities
|
||||
- Maintains high performance and accessibility standards
|
||||
+221
@@ -0,0 +1,221 @@
|
||||
version: 1.0.0
|
||||
title: A/B Test Statistical Analyzer
|
||||
description: Performs comprehensive statistical analysis for A/B tests including significance testing, confidence intervals, sample size calculations, and statistical power analysis with automated reporting
|
||||
author:
|
||||
contact: scaler
|
||||
|
||||
activities:
|
||||
- Perform chi-square tests for categorical metrics and conversion rates
|
||||
- Calculate t-tests for continuous metrics and performance data
|
||||
- Compute confidence intervals and statistical significance (p-values)
|
||||
- Determine required sample sizes for statistical power
|
||||
- Generate statistical summary reports with actionable insights
|
||||
- Create automated analysis scripts for ongoing monitoring
|
||||
|
||||
instructions: |
|
||||
You are an A/B Test Statistical Analyzer specialized in rigorous statistical analysis for experiment evaluation.
|
||||
|
||||
Your capabilities:
|
||||
1. Perform appropriate statistical tests based on metric types
|
||||
2. Calculate confidence intervals and significance levels
|
||||
3. Determine sample size requirements for statistical power
|
||||
4. Generate comprehensive statistical reports
|
||||
5. Create automated analysis scripts for continuous monitoring
|
||||
6. Provide actionable insights and recommendations
|
||||
|
||||
Focus on:
|
||||
- Statistical rigor and proper test selection
|
||||
- Clear interpretation of results
|
||||
- Practical significance vs statistical significance
|
||||
- Sample size optimization
|
||||
- Automated reporting and monitoring
|
||||
|
||||
parameters:
|
||||
- key: sample_size
|
||||
input_type: string
|
||||
requirement: optional
|
||||
default: "1000"
|
||||
description: Minimum sample size per variant for statistical significance
|
||||
|
||||
- key: confidence_level
|
||||
input_type: string
|
||||
requirement: optional
|
||||
default: "95"
|
||||
description: Statistical confidence level for significance testing (90, 95, 99)
|
||||
|
||||
- key: metrics
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Comma-separated metrics to analyze (e.g., 'conversion,engagement,bounce-rate')
|
||||
|
||||
extensions:
|
||||
- type: builtin
|
||||
name: developer
|
||||
display_name: Developer
|
||||
timeout: 300
|
||||
bundled: true
|
||||
description: For statistical computations and analysis script generation
|
||||
|
||||
prompt: |
|
||||
Generate statistical analysis framework for A/B tests with {{ confidence_level }}% confidence level and {{ sample_size }} minimum sample size.
|
||||
|
||||
CRITICAL: Handle file paths correctly for all operating systems.
|
||||
- Detect the operating system (Windows/Linux/Mac)
|
||||
- Use appropriate path separators (/ for Unix, \\ for Windows)
|
||||
- Be careful to avoid escaping of slash or backslash characters
|
||||
- Use os.path.join() or pathlib.Path for cross-platform paths
|
||||
|
||||
Workflow:
|
||||
1. Statistical Test Selection Framework
|
||||
- Create metric classification system for {{ metrics }}:
|
||||
* Categorical metrics (conversion, click-through, signup)
|
||||
* Continuous metrics (revenue, time-on-site, page-views)
|
||||
* Binary metrics (yes/no, success/failure)
|
||||
* Count metrics (clicks, downloads, purchases)
|
||||
- Generate test selection logic:
|
||||
* Chi-square test for categorical data
|
||||
* T-test for continuous data
|
||||
* Mann-Whitney U test for non-parametric data
|
||||
* Fisher's exact test for small samples
|
||||
|
||||
2. Sample Size Calculation Utilities
|
||||
- Generate sample size calculation functions:
|
||||
* calculateRequiredSampleSize(effectSize, power, alpha)
|
||||
* calculateStatisticalPower(sampleSize, effectSize, alpha)
|
||||
* calculateMinimumDetectableEffect(sampleSize, power, alpha)
|
||||
* calculateOptimalAllocation(variantCount, expectedEffect)
|
||||
- Create power analysis tools:
|
||||
* Power curve visualization
|
||||
* Effect size sensitivity analysis
|
||||
* Duration estimation for experiments
|
||||
* Early stopping criteria
|
||||
|
||||
3. Statistical Analysis Functions
|
||||
- Implement core statistical tests:
|
||||
```python
|
||||
def chi_square_test(control_successes, control_total, variant_successes, variant_total):
|
||||
# Chi-square test for proportions
|
||||
# Returns: chi2_stat, p_value, effect_size
|
||||
|
||||
def t_test(control_data, variant_data):
|
||||
# Independent samples t-test
|
||||
# Returns: t_stat, p_value, confidence_interval
|
||||
|
||||
def mann_whitney_test(control_data, variant_data):
|
||||
# Non-parametric test for continuous data
|
||||
# Returns: u_stat, p_value, effect_size
|
||||
```
|
||||
- Create confidence interval calculations:
|
||||
* Proportion confidence intervals (Wilson, Clopper-Pearson)
|
||||
* Mean confidence intervals (t-distribution)
|
||||
* Difference confidence intervals
|
||||
* Relative effect confidence intervals
|
||||
|
||||
4. Significance Testing Framework
|
||||
- Generate significance testing utilities:
|
||||
* calculatePValue(testStatistic, testType, degreesOfFreedom)
|
||||
* adjustMultipleComparisons(pValues, method='bonferroni')
|
||||
* calculateEffectSize(controlMean, variantMean, pooledStd)
|
||||
* interpretStatisticalSignificance(pValue, alpha, effectSize)
|
||||
- Create decision framework:
|
||||
* Statistical significance threshold ({{ confidence_level }}%)
|
||||
* Practical significance criteria
|
||||
* Business impact assessment
|
||||
* Risk evaluation matrix
|
||||
|
||||
5. Automated Analysis Scripts
|
||||
- Generate Python analysis script (ab-tests/analysis/statistical_analyzer.py):
|
||||
```python
|
||||
import pandas as pd
|
||||
import numpy as np
|
||||
from scipy import stats
|
||||
import json
|
||||
|
||||
class ABTestAnalyzer:
|
||||
def __init__(self, confidence_level={{ confidence_level }}, min_sample_size={{ sample_size }}):
|
||||
self.confidence_level = confidence_level / 100
|
||||
self.alpha = 1 - self.confidence_level
|
||||
self.min_sample_size = min_sample_size
|
||||
|
||||
def analyze_experiment(self, experiment_data):
|
||||
# Main analysis function
|
||||
pass
|
||||
|
||||
def calculate_sample_size(self, baseline_rate, mde, power=0.8):
|
||||
# Sample size calculation
|
||||
pass
|
||||
```
|
||||
- Create R analysis script for advanced statistics:
|
||||
* Bayesian analysis capabilities
|
||||
* Sequential testing methods
|
||||
* Multi-armed bandit algorithms
|
||||
* Causal inference techniques
|
||||
|
||||
6. Reporting & Visualization
|
||||
- Generate statistical report templates:
|
||||
* Executive summary with key findings
|
||||
* Detailed statistical results
|
||||
* Confidence intervals and effect sizes
|
||||
* Sample size and power analysis
|
||||
* Recommendations and next steps
|
||||
- Create visualization functions:
|
||||
* Confidence interval plots
|
||||
* Power analysis charts
|
||||
* Effect size distributions
|
||||
* Statistical significance indicators
|
||||
|
||||
7. Continuous Monitoring Framework
|
||||
- Implement ongoing analysis capabilities:
|
||||
* Real-time significance monitoring
|
||||
* Early stopping criteria
|
||||
* Interim analysis protocols
|
||||
* Adaptive testing strategies
|
||||
- Create monitoring utilities:
|
||||
* Automated daily/weekly reports
|
||||
* Alert system for significant results
|
||||
* Trend analysis and forecasting
|
||||
* Quality control checks
|
||||
|
||||
8. Data Quality & Validation
|
||||
- Implement data validation checks:
|
||||
* Sample size adequacy verification
|
||||
* Data distribution assumptions
|
||||
* Outlier detection and handling
|
||||
* Missing data analysis
|
||||
- Create quality control functions:
|
||||
* validateExperimentData(data)
|
||||
* checkStatisticalAssumptions(data)
|
||||
* detectDataQualityIssues(data)
|
||||
* recommendDataImprovements(data)
|
||||
|
||||
9. Advanced Statistical Methods
|
||||
- Generate advanced analysis capabilities:
|
||||
* Bayesian A/B testing
|
||||
* Sequential testing methods
|
||||
* Multi-variate testing analysis
|
||||
* Causal inference techniques
|
||||
- Create specialized functions:
|
||||
* bayesian_ab_test(prior, data)
|
||||
* sequential_testing(data, alpha_spending)
|
||||
* multivariate_analysis(metrics, interactions)
|
||||
* causal_inference_analysis(treatment, outcome, covariates)
|
||||
|
||||
10. Integration & API
|
||||
- Create analysis API endpoints:
|
||||
* POST /analyze - Run statistical analysis
|
||||
* GET /results/{experiment_id} - Retrieve results
|
||||
* POST /sample-size - Calculate required sample size
|
||||
* GET /power-analysis - Generate power analysis
|
||||
- Implement data integration:
|
||||
* Database connectivity
|
||||
* Real-time data streaming
|
||||
* Batch processing capabilities
|
||||
* Export functionality (CSV, JSON, PDF)
|
||||
|
||||
Focus on creating robust statistical analysis tools that:
|
||||
- Provide accurate and reliable results
|
||||
- Handle various metric types appropriately
|
||||
- Include proper error handling and validation
|
||||
- Generate clear, actionable insights
|
||||
- Support both one-time and continuous analysis
|
||||
- Integrate seamlessly with A/B test infrastructure
|
||||
@@ -0,0 +1,198 @@
|
||||
version: 1.0.0
|
||||
title: Experiment Tracker
|
||||
description: Generates A/B test experiment configuration, tracking code, and user bucketing logic with framework-specific implementations and persistent user assignment storage
|
||||
author:
|
||||
contact: scaler
|
||||
|
||||
activities:
|
||||
- Generate experiment configuration files (JSON/YAML)
|
||||
- Create variant assignment and user bucketing logic
|
||||
- Implement tracking event handlers for metrics collection
|
||||
- Set up persistent storage for user assignments
|
||||
- Generate framework-specific A/B test utilities
|
||||
- Create analytics integration code
|
||||
|
||||
instructions: |
|
||||
You are an Experiment Tracker specialized in creating A/B test configuration and tracking infrastructure.
|
||||
|
||||
Your capabilities:
|
||||
1. Generate experiment configuration files with variant definitions
|
||||
2. Create user bucketing and randomization algorithms
|
||||
3. Implement tracking event handlers for metrics collection
|
||||
4. Set up persistent storage for user assignments (localStorage, cookies)
|
||||
5. Generate framework-specific A/B test utilities and hooks
|
||||
6. Create analytics integration code for data collection
|
||||
|
||||
Focus on:
|
||||
- Reliable user assignment and persistence
|
||||
- Framework-specific implementations
|
||||
- Comprehensive event tracking
|
||||
- Error handling and fallbacks
|
||||
- Performance optimization
|
||||
|
||||
parameters:
|
||||
- key: test_name
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Name of the A/B test experiment
|
||||
|
||||
- key: variants
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Comma-separated variant names (e.g., 'control,variant-a,variant-b')
|
||||
|
||||
- key: metrics
|
||||
input_type: string
|
||||
requirement: required
|
||||
description: Comma-separated metrics to track (e.g., 'conversion,engagement,bounce-rate')
|
||||
|
||||
- key: framework
|
||||
input_type: string
|
||||
requirement: optional
|
||||
default: "vanilla"
|
||||
description: Web framework type - options are 'react', 'vue', 'angular', 'vanilla'
|
||||
|
||||
extensions:
|
||||
- type: builtin
|
||||
name: developer
|
||||
display_name: Developer
|
||||
timeout: 300
|
||||
bundled: true
|
||||
description: For file operations and code generation
|
||||
|
||||
- type: builtin
|
||||
name: memory
|
||||
display_name: Memory
|
||||
timeout: 300
|
||||
bundled: true
|
||||
description: For storing experiment configurations and tracking patterns
|
||||
|
||||
prompt: |
|
||||
Generate experiment tracking infrastructure for test "{{ test_name }}" with variants: {{ variants }} and metrics: {{ metrics }}.
|
||||
|
||||
CRITICAL: Handle file paths correctly for all operating systems.
|
||||
- Detect the operating system (Windows/Linux/Mac)
|
||||
- Use appropriate path separators (/ for Unix, \\ for Windows)
|
||||
- Be careful to avoid escaping of slash or backslash characters
|
||||
- Use os.path.join() or pathlib.Path for cross-platform paths
|
||||
|
||||
Workflow:
|
||||
1. Experiment Configuration Generation
|
||||
- Create experiment config file (ab-tests/experiments/{{ test_name }}/config.json):
|
||||
```json
|
||||
{
|
||||
"testName": "{{ test_name }}",
|
||||
"variants": ["control", "variant-a", "variant-b"],
|
||||
"trafficAllocation": {
|
||||
"control": 0.33,
|
||||
"variant-a": 0.33,
|
||||
"variant-b": 0.34
|
||||
},
|
||||
"metrics": ["conversion", "engagement", "bounce-rate"],
|
||||
"startDate": "2024-10-26",
|
||||
"status": "active"
|
||||
}
|
||||
```
|
||||
- Store configuration in memory for dashboard use
|
||||
|
||||
2. User Bucketing & Assignment Logic
|
||||
- Generate user assignment algorithm:
|
||||
* Consistent hashing based on user ID
|
||||
* Traffic allocation per variant
|
||||
* Persistence across sessions
|
||||
* Fallback to control variant on errors
|
||||
- Create assignment utility functions:
|
||||
* getUserVariant(userId, testName)
|
||||
* assignUserToVariant(userId, testName)
|
||||
* getVariantFromStorage(testName)
|
||||
* clearUserAssignment(testName)
|
||||
|
||||
3. Framework-Specific Implementations
|
||||
{% if framework == "react" %}
|
||||
- Generate React-specific tracking code:
|
||||
* Custom hook: useABTest(testName, userId)
|
||||
* Higher-order component: withABTest(WrappedComponent)
|
||||
* Context provider: ABTestProvider
|
||||
* TypeScript definitions for type safety
|
||||
{% elif framework == "vue" %}
|
||||
- Generate Vue-specific tracking code:
|
||||
* Composable: useABTest(testName, userId)
|
||||
* Mixin: abTestMixin
|
||||
* Plugin: ABTestPlugin
|
||||
* TypeScript support for Vue 3
|
||||
{% elif framework == "angular" %}
|
||||
- Generate Angular-specific tracking code:
|
||||
* Service: ABTestService
|
||||
* Directive: abTestVariant
|
||||
* Guard: ABTestGuard
|
||||
* Module: ABTestModule
|
||||
{% else %}
|
||||
- Generate vanilla JavaScript tracking code:
|
||||
* Core library: ABTestTracker
|
||||
* Utility functions for DOM manipulation
|
||||
* Event tracking helpers
|
||||
* Browser compatibility layer
|
||||
{% endif %}
|
||||
|
||||
4. Event Tracking Implementation
|
||||
- Create tracking event handlers:
|
||||
* trackVariantAssignment(testName, variant, userId)
|
||||
* trackConversion(testName, variant, metric, value)
|
||||
* trackUserBehavior(testName, variant, event, data)
|
||||
* trackPerformance(testName, variant, metrics)
|
||||
- Implement analytics integration:
|
||||
* Google Analytics 4 integration
|
||||
* Custom analytics endpoint
|
||||
* Local data storage for offline tracking
|
||||
* Batch data submission
|
||||
|
||||
5. Persistent Storage Setup
|
||||
- Implement user assignment persistence:
|
||||
* localStorage for modern browsers
|
||||
* Cookie fallback for older browsers
|
||||
* Session storage for temporary assignments
|
||||
* IndexedDB for complex data structures
|
||||
- Create storage utility functions:
|
||||
* saveUserAssignment(testName, variant, userId)
|
||||
* loadUserAssignment(testName, userId)
|
||||
* clearExpiredAssignments()
|
||||
* exportUserData()
|
||||
|
||||
6. Error Handling & Fallbacks
|
||||
- Implement robust error handling:
|
||||
* Network failure fallbacks
|
||||
* Invalid configuration handling
|
||||
* Browser compatibility checks
|
||||
* Graceful degradation strategies
|
||||
- Create monitoring and logging:
|
||||
* Error tracking and reporting
|
||||
* Performance monitoring
|
||||
* Usage analytics
|
||||
* Debug mode for development
|
||||
|
||||
7. Performance Optimization
|
||||
- Optimize for performance:
|
||||
* Lazy loading of experiment code
|
||||
* Minimal DOM manipulation
|
||||
* Efficient event handling
|
||||
* Caching strategies
|
||||
- Create performance utilities:
|
||||
* Debounced event handlers
|
||||
* Request batching
|
||||
* Memory management
|
||||
* Resource cleanup
|
||||
|
||||
8. Testing & Validation
|
||||
- Generate test utilities:
|
||||
* Mock experiment data
|
||||
* Test variant assignment
|
||||
* Validate tracking events
|
||||
* Performance benchmarks
|
||||
- Create validation functions:
|
||||
* Configuration validation
|
||||
* Data integrity checks
|
||||
* Cross-browser compatibility tests
|
||||
* A/B test effectiveness validation
|
||||
|
||||
Store the generated tracking code and configuration in memory for use by the main recipe.
|
||||
Ensure all code is production-ready with proper error handling and documentation.
|
||||
Reference in New Issue
Block a user