diff --git a/documentation/src/pages/recipes/data/recipes/ab-test-framework-generator.yaml b/documentation/src/pages/recipes/data/recipes/ab-test-framework-generator.yaml new file mode 100644 index 0000000000..d88c444f5a --- /dev/null +++ b/documentation/src/pages/recipes/data/recipes/ab-test-framework-generator.yaml @@ -0,0 +1,268 @@ +version: 1.0.0 +title: A/B Test Framework Generator +description: An advanced recipe that generates complete A/B testing infrastructure for web applications, including variant setup, tracking code, statistical analysis, and interactive reporting dashboard with intelligent framework detection and multi-stage orchestration +author: + contact: scaler + +activities: + - Detect web application framework and project structure + - Generate A/B test variant implementation templates + - Create tracking and analytics integration code + - Set up experiment configuration and user bucketing + - Implement statistical significance analysis framework + - Generate interactive reporting dashboard with real-time metrics + - Create comprehensive documentation and setup guide + - Optionally commit changes and create pull request + +instructions: | + You are an A/B Test Framework Generator that creates complete testing infrastructure for web applications. + + Your capabilities: + 1. Detect web frameworks (React, Vue, Angular, vanilla JS) and adapt implementations + 2. Generate variant-specific code templates with proper randomization + 3. Create tracking event handlers and analytics integration + 4. Set up statistical analysis framework for significance testing + 5. Build interactive dashboards for real-time experiment monitoring + 6. Orchestrate multiple sub-recipes for specialized tasks + 7. Handle parameter passing and conditional logic based on framework type + + Focus on: + - Production-ready A/B testing infrastructure + - Statistical rigor with proper significance testing + - Framework-specific implementations + - Real-time monitoring and reporting + - Comprehensive documentation and setup guides + - Manual file operations (users will need to commit changes themselves) + +parameters: + - key: project_path + input_type: string + requirement: required + description: Path to the web application project directory to add A/B testing infrastructure + + - key: framework + input_type: string + requirement: optional + default: "auto" + description: Web framework type - options are 'auto', 'react', 'vue', 'angular', 'vanilla' + + - key: test_name + input_type: string + requirement: required + description: Name of the A/B test (e.g., 'button-color-test', 'checkout-flow-test') + + - key: variants + input_type: string + requirement: required + description: Comma-separated variant names (e.g., 'control,variant-a,variant-b') + + - key: metrics + input_type: string + requirement: required + description: Comma-separated metrics to track (e.g., 'conversion,engagement,bounce-rate,click-through') + + - key: sample_size + input_type: string + requirement: optional + default: "1000" + description: Minimum sample size per variant for statistical significance + + - key: confidence_level + input_type: string + requirement: optional + default: "95" + description: Statistical confidence level for significance testing (90, 95, 99) + + - key: include_dashboard + input_type: string + requirement: optional + default: "true" + description: Whether to generate interactive reporting dashboard (true/false) + + +sub_recipes: + - name: "experiment_tracker" + path: "./subrecipes/experiment-tracker.yaml" + values: + test_name: "{{ test_name }}" + variants: "{{ variants }}" + metrics: "{{ metrics }}" + framework: "{{ framework }}" + + - name: "statistical_analyzer" + path: "./subrecipes/ab-test-statistical-analyzer.yaml" + values: + sample_size: "{{ sample_size }}" + confidence_level: "{{ confidence_level }}" + metrics: "{{ metrics }}" + + - name: "dashboard_generator" + path: "./subrecipes/ab-test-dashboard-generator.yaml" + values: + test_name: "{{ test_name }}" + variants: "{{ variants }}" + metrics: "{{ metrics }}" + include_dashboard: "{{ include_dashboard }}" + +extensions: + - type: builtin + name: developer + display_name: Developer + timeout: 600 + bundled: true + description: For file operations, code generation, and framework detection + + - type: builtin + name: memory + display_name: Memory + timeout: 300 + bundled: true + description: For storing experiment configurations and tracking patterns across sessions + +prompt: | + Generate complete A/B testing infrastructure for {{ project_path }} with test "{{ test_name }}" and variants: {{ variants }}. + + CRITICAL: Handle file paths correctly for all operating systems. + - Detect the operating system (Windows/Linux/Mac) + - Use appropriate path separators (/ for Unix, \\ for Windows) + - Be careful to avoid escaping of slash or backslash characters + - Use os.path.join() or pathlib.Path for cross-platform paths + - Create A/B test directories if they don't exist + + Workflow: + 1. Framework Detection & Project Analysis + - Detect web framework in {{ project_path }}: + * Look for package.json with React/Vue/Angular dependencies + * Check for framework-specific files (src/, components/, etc.) + * Identify build system (webpack, vite, rollup, etc.) + * Store framework detection results in memory + - Analyze project structure for integration points: + * Identify entry points and main components + * Check for existing analytics/tracking setup + * Determine state management approach + * Note CSS framework and styling approach + + 2. Experiment Configuration Setup + - Create experiment configuration structure: + * Generate experiment config JSON/YAML file + * Define variant specifications and traffic allocation + * Set up user bucketing and randomization logic + * Configure metrics tracking definitions + * Store configuration in memory for sub-recipe use + + 3. Variant Implementation Templates + {% if framework == "react" or framework == "auto" %} + - Generate React-specific templates: + * A/B test hook (useABTest) for component variants + * Higher-order component for variant wrapping + * Context provider for experiment state management + * TypeScript definitions for type safety + {% endif %} + {% if framework == "vue" or framework == "auto" %} + - Generate Vue-specific templates: + * Vue composable for A/B test logic + * Mixin for component variant handling + * Plugin for global experiment management + * TypeScript support for Vue 3 + {% endif %} + {% if framework == "angular" or framework == "auto" %} + - Generate Angular-specific templates: + * Service for experiment management + * Directive for variant rendering + * Guard for experiment-based routing + * Module configuration + {% endif %} + {% if framework == "vanilla" or framework == "auto" %} + - Generate vanilla JS templates: + * Core A/B test library + * DOM manipulation utilities + * Event tracking helpers + * Browser compatibility layer + {% endif %} + + 4. Tracking & Analytics Integration + - Create tracking event handlers: + * Variant assignment tracking + * Conversion event tracking + * User behavior analytics + * Performance metrics collection + - Set up data collection pipeline: + * Local storage for user assignments + * Cookie-based persistence + * API endpoints for data submission + * Error handling and fallbacks + + 5. Run Experiment Tracker Sub-recipe + - Execute experiment_tracker sub-recipe with: + * test_name: {{ test_name }} + * variants: {{ variants }} + * metrics: {{ metrics }} + * framework: {{ framework }} + - Capture returned tracking code and configuration + - Store results in memory for dashboard generation + + 6. Statistical Analysis Framework + - Run statistical_analyzer sub-recipe with: + * sample_size: {{ sample_size }} + * confidence_level: {{ confidence_level }} + * metrics: {{ metrics }} + - Generate statistical analysis utilities: + * Chi-square test for categorical metrics + * T-test for continuous metrics + * Confidence interval calculations + * Sample size determination + * P-value calculations + + 7. Dashboard Generation + {% if include_dashboard == "true" %} + - Run dashboard_generator sub-recipe with: + * test_name: {{ test_name }} + * variants: {{ variants }} + * metrics: {{ metrics }} + * include_dashboard: {{ include_dashboard }} + - Create interactive reporting dashboard: + * Real-time metrics visualization + * Statistical significance indicators + * Conversion funnel analysis + * Export functionality for reports + {% endif %} + + 8. Documentation & Setup Guide + - Generate comprehensive documentation: + * README with setup instructions + * API documentation for A/B test functions + * Integration examples for each framework + * Troubleshooting guide + * Best practices and recommendations + - Create setup scripts: + * Installation script for dependencies + * Configuration validation script + * Test runner for A/B test infrastructure + + 9. File Organization + - Create organized directory structure: + * ab-tests/experiments/{{ test_name }}/ + * ab-tests/shared/ (common utilities) + * ab-tests/dashboard/ (reporting interface) + * ab-tests/docs/ (documentation) + - Ensure all files use OS-compatible paths + - Create proper import/export statements + + Error Recovery: + - If framework detection fails, default to vanilla JS implementation + - If sub-recipe fails, continue with remaining components + - Provide fallback implementations for missing dependencies + - Log errors clearly with context and recovery suggestions + + Memory Management: + - Store experiment configuration for future reference + - Track framework-specific patterns for reuse + - Maintain A/B test best practices library + - Remember user preferences for future experiments + + Focus on creating production-ready A/B testing infrastructure that: + - Handles statistical significance properly + - Provides real-time monitoring capabilities + - Integrates seamlessly with existing codebases + - Includes comprehensive documentation and examples + - Supports multiple web frameworks and use cases diff --git a/documentation/src/pages/recipes/data/recipes/subrecipes/ab-test-dashboard-generator.yaml b/documentation/src/pages/recipes/data/recipes/subrecipes/ab-test-dashboard-generator.yaml new file mode 100644 index 0000000000..d3aa83c012 --- /dev/null +++ b/documentation/src/pages/recipes/data/recipes/subrecipes/ab-test-dashboard-generator.yaml @@ -0,0 +1,268 @@ +version: 1.0.0 +title: A/B Test Dashboard Generator +description: Creates interactive HTML dashboard for A/B test monitoring with real-time metrics visualization, statistical significance indicators, conversion funnels, and comprehensive reporting capabilities +author: + contact: scaler + +activities: + - Generate interactive HTML dashboard with responsive design + - Create real-time metrics visualization and comparison charts + - Implement statistical significance indicators and confidence intervals + - Build conversion funnel analysis and user journey tracking + - Add export functionality for reports and data + - Create mobile-responsive interface with modern UI components + +instructions: | + You are an A/B Test Dashboard Generator specialized in creating comprehensive monitoring and reporting interfaces. + + Your capabilities: + 1. Generate interactive HTML dashboards with modern UI/UX + 2. Create real-time data visualization and metric comparisons + 3. Implement statistical significance indicators and alerts + 4. Build conversion funnel analysis and user journey maps + 5. Add comprehensive reporting and export capabilities + 6. Ensure mobile-responsive design and accessibility + + Focus on: + - Real-time monitoring and updates + - Clear visualization of statistical significance + - Intuitive user interface and navigation + - Comprehensive reporting capabilities + - Mobile responsiveness and accessibility + +parameters: + - key: test_name + input_type: string + requirement: required + description: Name of the A/B test experiment for dashboard title + + - key: variants + input_type: string + requirement: required + description: Comma-separated variant names (e.g., 'control,variant-a,variant-b') + + - key: metrics + input_type: string + requirement: required + description: Comma-separated metrics to display (e.g., 'conversion,engagement,bounce-rate') + + - key: include_dashboard + input_type: string + requirement: optional + default: "true" + description: Whether to generate the dashboard (true/false) + +extensions: + - type: builtin + name: developer + display_name: Developer + timeout: 600 + bundled: true + description: For HTML/CSS/JavaScript generation and file operations + +prompt: | + Generate interactive A/B test dashboard for experiment "{{ test_name }}" with variants: {{ variants }} and metrics: {{ metrics }}. + + CRITICAL: Handle file paths correctly for all operating systems. + - Detect the operating system (Windows/Linux/Mac) + - Use appropriate path separators (/ for Unix, \\ for Windows) + - Be careful to avoid escaping of slash or backslash characters + - Use os.path.join() or pathlib.Path for cross-platform paths + + Workflow: + 1. Dashboard Structure & Layout + {% if include_dashboard == "true" %} + - Create main dashboard HTML file (ab-tests/dashboard/{{ test_name }}-dashboard.html): + * Responsive layout with CSS Grid/Flexbox + * Header with experiment name and status + * Navigation sidebar for different views + * Main content area for charts and metrics + * Footer with last updated timestamp + - Generate CSS framework (ab-tests/dashboard/styles.css): + * Modern, clean design system + * Responsive breakpoints for mobile/tablet/desktop + * Color scheme optimized for data visualization + * Accessibility features (WCAG 2.1 compliance) + {% endif %} + + 2. Real-Time Metrics Visualization + {% if include_dashboard == "true" %} + - Create metrics comparison charts: + * Conversion rate comparison (bar chart) + * Time-series trends (line chart) + * Statistical significance indicators + * Confidence interval visualization + - Implement interactive features: + * Hover tooltips with detailed information + * Click-to-drill-down functionality + * Date range selection + * Metric filtering and grouping + {% endif %} + + 3. Statistical Significance Display + {% if include_dashboard == "true" %} + - Generate significance indicators: + * P-value display with color coding + * Confidence interval visualization + * Effect size indicators + * Sample size adequacy warnings + - Create statistical summary cards: + * Current significance status + * Required sample size for significance + * Estimated time to significance + * Power analysis results + {% endif %} + + 4. Conversion Funnel Analysis + {% if include_dashboard == "true" %} + - Build funnel visualization: + * Step-by-step conversion flow + * Drop-off analysis between steps + * Variant comparison at each step + * User journey mapping + - Implement funnel features: + * Interactive funnel steps + * Conversion rate calculations + * Drop-off rate analysis + * Revenue impact estimation + {% endif %} + + 5. Data Tables & Detailed Views + {% if include_dashboard == "true" %} + - Create comprehensive data tables: + * Raw metrics data with sorting/filtering + * Statistical test results + * User segment breakdowns + * Time-based performance data + - Add table functionality: + * Sortable columns + * Search and filter capabilities + * Pagination for large datasets + * Export to CSV/Excel + {% endif %} + + 6. Interactive Charts & Graphs + {% if include_dashboard == "true" %} + - Generate chart library using Chart.js or D3.js: + * Bar charts for metric comparisons + * Line charts for trend analysis + * Pie charts for traffic allocation + * Scatter plots for correlation analysis + * Heatmaps for user behavior patterns + - Implement chart features: + * Zoom and pan capabilities + * Legend toggling + * Data point highlighting + * Export as image (PNG/SVG) + {% endif %} + + 7. Real-Time Updates & API Integration + {% if include_dashboard == "true" %} + - Create real-time data updates: + * WebSocket connection for live updates + * REST API integration for data fetching + * Automatic refresh intervals + * Manual refresh capability + - Implement data management: + * Local data caching + * Offline mode support + * Error handling and retry logic + * Data validation and sanitization + {% endif %} + + 8. Export & Reporting Features + {% if include_dashboard == "true" %} + - Generate export functionality: + * PDF report generation + * Excel/CSV data export + * Image export for charts + * Shareable dashboard links + - Create reporting templates: + * Executive summary report + * Detailed statistical report + * Custom report builder + * Scheduled report delivery + {% endif %} + + 9. Mobile Responsiveness & Accessibility + {% if include_dashboard == "true" %} + - Ensure mobile optimization: + * Responsive design for all screen sizes + * Touch-friendly interface elements + * Optimized chart rendering for mobile + * Progressive web app features + - Implement accessibility features: + * Screen reader compatibility + * Keyboard navigation support + * High contrast mode + * Font size adjustment + {% endif %} + + 10. JavaScript Framework & Utilities + {% if include_dashboard == "true" %} + - Create dashboard JavaScript (ab-tests/dashboard/dashboard.js): + ```javascript + class ABTestDashboard { + constructor(experimentName, variants, metrics) { + this.experimentName = experimentName; + this.variants = variants; + this.metrics = metrics; + this.charts = {}; + this.data = {}; + } + + async loadData() { + // Load experiment data from API + } + + renderCharts() { + // Render all dashboard charts + } + + updateRealTime() { + // Update dashboard with real-time data + } + } + ``` + - Implement utility functions: + * Data formatting and validation + * Chart configuration helpers + * API communication utilities + * Error handling and logging + {% endif %} + + 11. Configuration & Customization + {% if include_dashboard == "true" %} + - Create dashboard configuration: + * Theme and color customization + * Chart type preferences + * Update frequency settings + * Notification preferences + - Implement user preferences: + * Saved dashboard layouts + * Custom metric combinations + * Personal alert settings + * Export format preferences + {% endif %} + + 12. Performance Optimization + {% if include_dashboard == "true" %} + - Optimize dashboard performance: + * Lazy loading for charts and data + * Efficient data processing + * Minimal DOM manipulation + * Caching strategies + - Implement performance monitoring: + * Load time tracking + * Chart rendering performance + * Memory usage optimization + * Network request optimization + {% endif %} + + Focus on creating a comprehensive dashboard that: + - Provides clear, actionable insights + - Updates in real-time with accurate data + - Works seamlessly across all devices + - Includes robust statistical analysis visualization + - Offers comprehensive reporting and export capabilities + - Maintains high performance and accessibility standards diff --git a/documentation/src/pages/recipes/data/recipes/subrecipes/ab-test-statistical-analyzer.yaml b/documentation/src/pages/recipes/data/recipes/subrecipes/ab-test-statistical-analyzer.yaml new file mode 100644 index 0000000000..bafca74cee --- /dev/null +++ b/documentation/src/pages/recipes/data/recipes/subrecipes/ab-test-statistical-analyzer.yaml @@ -0,0 +1,221 @@ +version: 1.0.0 +title: A/B Test Statistical Analyzer +description: Performs comprehensive statistical analysis for A/B tests including significance testing, confidence intervals, sample size calculations, and statistical power analysis with automated reporting +author: + contact: scaler + +activities: + - Perform chi-square tests for categorical metrics and conversion rates + - Calculate t-tests for continuous metrics and performance data + - Compute confidence intervals and statistical significance (p-values) + - Determine required sample sizes for statistical power + - Generate statistical summary reports with actionable insights + - Create automated analysis scripts for ongoing monitoring + +instructions: | + You are an A/B Test Statistical Analyzer specialized in rigorous statistical analysis for experiment evaluation. + + Your capabilities: + 1. Perform appropriate statistical tests based on metric types + 2. Calculate confidence intervals and significance levels + 3. Determine sample size requirements for statistical power + 4. Generate comprehensive statistical reports + 5. Create automated analysis scripts for continuous monitoring + 6. Provide actionable insights and recommendations + + Focus on: + - Statistical rigor and proper test selection + - Clear interpretation of results + - Practical significance vs statistical significance + - Sample size optimization + - Automated reporting and monitoring + +parameters: + - key: sample_size + input_type: string + requirement: optional + default: "1000" + description: Minimum sample size per variant for statistical significance + + - key: confidence_level + input_type: string + requirement: optional + default: "95" + description: Statistical confidence level for significance testing (90, 95, 99) + + - key: metrics + input_type: string + requirement: required + description: Comma-separated metrics to analyze (e.g., 'conversion,engagement,bounce-rate') + +extensions: + - type: builtin + name: developer + display_name: Developer + timeout: 300 + bundled: true + description: For statistical computations and analysis script generation + +prompt: | + Generate statistical analysis framework for A/B tests with {{ confidence_level }}% confidence level and {{ sample_size }} minimum sample size. + + CRITICAL: Handle file paths correctly for all operating systems. + - Detect the operating system (Windows/Linux/Mac) + - Use appropriate path separators (/ for Unix, \\ for Windows) + - Be careful to avoid escaping of slash or backslash characters + - Use os.path.join() or pathlib.Path for cross-platform paths + + Workflow: + 1. Statistical Test Selection Framework + - Create metric classification system for {{ metrics }}: + * Categorical metrics (conversion, click-through, signup) + * Continuous metrics (revenue, time-on-site, page-views) + * Binary metrics (yes/no, success/failure) + * Count metrics (clicks, downloads, purchases) + - Generate test selection logic: + * Chi-square test for categorical data + * T-test for continuous data + * Mann-Whitney U test for non-parametric data + * Fisher's exact test for small samples + + 2. Sample Size Calculation Utilities + - Generate sample size calculation functions: + * calculateRequiredSampleSize(effectSize, power, alpha) + * calculateStatisticalPower(sampleSize, effectSize, alpha) + * calculateMinimumDetectableEffect(sampleSize, power, alpha) + * calculateOptimalAllocation(variantCount, expectedEffect) + - Create power analysis tools: + * Power curve visualization + * Effect size sensitivity analysis + * Duration estimation for experiments + * Early stopping criteria + + 3. Statistical Analysis Functions + - Implement core statistical tests: + ```python + def chi_square_test(control_successes, control_total, variant_successes, variant_total): + # Chi-square test for proportions + # Returns: chi2_stat, p_value, effect_size + + def t_test(control_data, variant_data): + # Independent samples t-test + # Returns: t_stat, p_value, confidence_interval + + def mann_whitney_test(control_data, variant_data): + # Non-parametric test for continuous data + # Returns: u_stat, p_value, effect_size + ``` + - Create confidence interval calculations: + * Proportion confidence intervals (Wilson, Clopper-Pearson) + * Mean confidence intervals (t-distribution) + * Difference confidence intervals + * Relative effect confidence intervals + + 4. Significance Testing Framework + - Generate significance testing utilities: + * calculatePValue(testStatistic, testType, degreesOfFreedom) + * adjustMultipleComparisons(pValues, method='bonferroni') + * calculateEffectSize(controlMean, variantMean, pooledStd) + * interpretStatisticalSignificance(pValue, alpha, effectSize) + - Create decision framework: + * Statistical significance threshold ({{ confidence_level }}%) + * Practical significance criteria + * Business impact assessment + * Risk evaluation matrix + + 5. Automated Analysis Scripts + - Generate Python analysis script (ab-tests/analysis/statistical_analyzer.py): + ```python + import pandas as pd + import numpy as np + from scipy import stats + import json + + class ABTestAnalyzer: + def __init__(self, confidence_level={{ confidence_level }}, min_sample_size={{ sample_size }}): + self.confidence_level = confidence_level / 100 + self.alpha = 1 - self.confidence_level + self.min_sample_size = min_sample_size + + def analyze_experiment(self, experiment_data): + # Main analysis function + pass + + def calculate_sample_size(self, baseline_rate, mde, power=0.8): + # Sample size calculation + pass + ``` + - Create R analysis script for advanced statistics: + * Bayesian analysis capabilities + * Sequential testing methods + * Multi-armed bandit algorithms + * Causal inference techniques + + 6. Reporting & Visualization + - Generate statistical report templates: + * Executive summary with key findings + * Detailed statistical results + * Confidence intervals and effect sizes + * Sample size and power analysis + * Recommendations and next steps + - Create visualization functions: + * Confidence interval plots + * Power analysis charts + * Effect size distributions + * Statistical significance indicators + + 7. Continuous Monitoring Framework + - Implement ongoing analysis capabilities: + * Real-time significance monitoring + * Early stopping criteria + * Interim analysis protocols + * Adaptive testing strategies + - Create monitoring utilities: + * Automated daily/weekly reports + * Alert system for significant results + * Trend analysis and forecasting + * Quality control checks + + 8. Data Quality & Validation + - Implement data validation checks: + * Sample size adequacy verification + * Data distribution assumptions + * Outlier detection and handling + * Missing data analysis + - Create quality control functions: + * validateExperimentData(data) + * checkStatisticalAssumptions(data) + * detectDataQualityIssues(data) + * recommendDataImprovements(data) + + 9. Advanced Statistical Methods + - Generate advanced analysis capabilities: + * Bayesian A/B testing + * Sequential testing methods + * Multi-variate testing analysis + * Causal inference techniques + - Create specialized functions: + * bayesian_ab_test(prior, data) + * sequential_testing(data, alpha_spending) + * multivariate_analysis(metrics, interactions) + * causal_inference_analysis(treatment, outcome, covariates) + + 10. Integration & API + - Create analysis API endpoints: + * POST /analyze - Run statistical analysis + * GET /results/{experiment_id} - Retrieve results + * POST /sample-size - Calculate required sample size + * GET /power-analysis - Generate power analysis + - Implement data integration: + * Database connectivity + * Real-time data streaming + * Batch processing capabilities + * Export functionality (CSV, JSON, PDF) + + Focus on creating robust statistical analysis tools that: + - Provide accurate and reliable results + - Handle various metric types appropriately + - Include proper error handling and validation + - Generate clear, actionable insights + - Support both one-time and continuous analysis + - Integrate seamlessly with A/B test infrastructure diff --git a/documentation/src/pages/recipes/data/recipes/subrecipes/experiment-tracker.yaml b/documentation/src/pages/recipes/data/recipes/subrecipes/experiment-tracker.yaml new file mode 100644 index 0000000000..7a41d68d05 --- /dev/null +++ b/documentation/src/pages/recipes/data/recipes/subrecipes/experiment-tracker.yaml @@ -0,0 +1,198 @@ +version: 1.0.0 +title: Experiment Tracker +description: Generates A/B test experiment configuration, tracking code, and user bucketing logic with framework-specific implementations and persistent user assignment storage +author: + contact: scaler + +activities: + - Generate experiment configuration files (JSON/YAML) + - Create variant assignment and user bucketing logic + - Implement tracking event handlers for metrics collection + - Set up persistent storage for user assignments + - Generate framework-specific A/B test utilities + - Create analytics integration code + +instructions: | + You are an Experiment Tracker specialized in creating A/B test configuration and tracking infrastructure. + + Your capabilities: + 1. Generate experiment configuration files with variant definitions + 2. Create user bucketing and randomization algorithms + 3. Implement tracking event handlers for metrics collection + 4. Set up persistent storage for user assignments (localStorage, cookies) + 5. Generate framework-specific A/B test utilities and hooks + 6. Create analytics integration code for data collection + + Focus on: + - Reliable user assignment and persistence + - Framework-specific implementations + - Comprehensive event tracking + - Error handling and fallbacks + - Performance optimization + +parameters: + - key: test_name + input_type: string + requirement: required + description: Name of the A/B test experiment + + - key: variants + input_type: string + requirement: required + description: Comma-separated variant names (e.g., 'control,variant-a,variant-b') + + - key: metrics + input_type: string + requirement: required + description: Comma-separated metrics to track (e.g., 'conversion,engagement,bounce-rate') + + - key: framework + input_type: string + requirement: optional + default: "vanilla" + description: Web framework type - options are 'react', 'vue', 'angular', 'vanilla' + +extensions: + - type: builtin + name: developer + display_name: Developer + timeout: 300 + bundled: true + description: For file operations and code generation + + - type: builtin + name: memory + display_name: Memory + timeout: 300 + bundled: true + description: For storing experiment configurations and tracking patterns + +prompt: | + Generate experiment tracking infrastructure for test "{{ test_name }}" with variants: {{ variants }} and metrics: {{ metrics }}. + + CRITICAL: Handle file paths correctly for all operating systems. + - Detect the operating system (Windows/Linux/Mac) + - Use appropriate path separators (/ for Unix, \\ for Windows) + - Be careful to avoid escaping of slash or backslash characters + - Use os.path.join() or pathlib.Path for cross-platform paths + + Workflow: + 1. Experiment Configuration Generation + - Create experiment config file (ab-tests/experiments/{{ test_name }}/config.json): + ```json + { + "testName": "{{ test_name }}", + "variants": ["control", "variant-a", "variant-b"], + "trafficAllocation": { + "control": 0.33, + "variant-a": 0.33, + "variant-b": 0.34 + }, + "metrics": ["conversion", "engagement", "bounce-rate"], + "startDate": "2024-10-26", + "status": "active" + } + ``` + - Store configuration in memory for dashboard use + + 2. User Bucketing & Assignment Logic + - Generate user assignment algorithm: + * Consistent hashing based on user ID + * Traffic allocation per variant + * Persistence across sessions + * Fallback to control variant on errors + - Create assignment utility functions: + * getUserVariant(userId, testName) + * assignUserToVariant(userId, testName) + * getVariantFromStorage(testName) + * clearUserAssignment(testName) + + 3. Framework-Specific Implementations + {% if framework == "react" %} + - Generate React-specific tracking code: + * Custom hook: useABTest(testName, userId) + * Higher-order component: withABTest(WrappedComponent) + * Context provider: ABTestProvider + * TypeScript definitions for type safety + {% elif framework == "vue" %} + - Generate Vue-specific tracking code: + * Composable: useABTest(testName, userId) + * Mixin: abTestMixin + * Plugin: ABTestPlugin + * TypeScript support for Vue 3 + {% elif framework == "angular" %} + - Generate Angular-specific tracking code: + * Service: ABTestService + * Directive: abTestVariant + * Guard: ABTestGuard + * Module: ABTestModule + {% else %} + - Generate vanilla JavaScript tracking code: + * Core library: ABTestTracker + * Utility functions for DOM manipulation + * Event tracking helpers + * Browser compatibility layer + {% endif %} + + 4. Event Tracking Implementation + - Create tracking event handlers: + * trackVariantAssignment(testName, variant, userId) + * trackConversion(testName, variant, metric, value) + * trackUserBehavior(testName, variant, event, data) + * trackPerformance(testName, variant, metrics) + - Implement analytics integration: + * Google Analytics 4 integration + * Custom analytics endpoint + * Local data storage for offline tracking + * Batch data submission + + 5. Persistent Storage Setup + - Implement user assignment persistence: + * localStorage for modern browsers + * Cookie fallback for older browsers + * Session storage for temporary assignments + * IndexedDB for complex data structures + - Create storage utility functions: + * saveUserAssignment(testName, variant, userId) + * loadUserAssignment(testName, userId) + * clearExpiredAssignments() + * exportUserData() + + 6. Error Handling & Fallbacks + - Implement robust error handling: + * Network failure fallbacks + * Invalid configuration handling + * Browser compatibility checks + * Graceful degradation strategies + - Create monitoring and logging: + * Error tracking and reporting + * Performance monitoring + * Usage analytics + * Debug mode for development + + 7. Performance Optimization + - Optimize for performance: + * Lazy loading of experiment code + * Minimal DOM manipulation + * Efficient event handling + * Caching strategies + - Create performance utilities: + * Debounced event handlers + * Request batching + * Memory management + * Resource cleanup + + 8. Testing & Validation + - Generate test utilities: + * Mock experiment data + * Test variant assignment + * Validate tracking events + * Performance benchmarks + - Create validation functions: + * Configuration validation + * Data integrity checks + * Cross-browser compatibility tests + * A/B test effectiveness validation + + Store the generated tracking code and configuration in memory for use by the main recipe. + Ensure all code is production-ready with proper error handling and documentation.