mirror of
https://github.com/block/goose.git
synced 2026-07-17 12:56:20 +02:00
Tania Chakraborty
9c9fdd9da8
This adds tooling to generate monthly Community Stars rankings for the goose repository.
Files added:
- scripts/community_stars.py: Python script that analyzes GitHub contributor stats
- scripts/community_stars_teams.txt: Team categorization lists (easy to update via PR)
Features:
- Analyzes contributions by time period (monthly or date range)
- Categorizes contributors into: Goose maintainers, Block non-goose, External, Bots
- Generates Top 5 Community All-Stars (external contributors)
- Generates Top 5 Team Stars (Block employees, non-goose)
- Outputs monthly leaderboard with all eligible contributors
- Scoring: commits + total lines changed (additions + deletions)
Usage:
curl -s -H 'Accept: application/vnd.github.v3+json' \
'https://api.github.com/repos/block/goose/stats/contributors' > /tmp/github_contributors.json
python3 scripts/community_stars.py 'November 2024'
The script can load team lists from local file or GitHub URL, making it usable
both locally and in automated workflows.
Goose Benchmark Scripts
This directory contains scripts for running and analyzing Goose benchmarks.
run-benchmarks.sh
This script runs Goose benchmarks across multiple provider:model pairs and analyzes the results.
Prerequisites
- Goose CLI must be built or installed
jqcommand-line tool for JSON processing (optional, but recommended for result analysis)
Usage
./scripts/run-benchmarks.sh [options]
Options
-p, --provider-models: Comma-separated list of provider:model pairs (e.g., 'openai:gpt-4o,anthropic:claude-sonnet-4')-s, --suites: Comma-separated list of benchmark suites to run (e.g., 'core,small_models')-o, --output-dir: Directory to store benchmark results (default: './benchmark-results')-d, --debug: Use debug build instead of release build-h, --help: Show help message
Examples
# Run with release build (default)
./scripts/run-benchmarks.sh --provider-models 'openai:gpt-4o,anthropic:claude-sonnet-4' --suites 'core,small_models'
# Run with debug build
./scripts/run-benchmarks.sh --provider-models 'openai:gpt-4o' --suites 'core' --debug
How It Works
The script:
- Parses the provider:model pairs and benchmark suites
- Determines whether to use the debug or release binary
- For each provider:model pair:
- Sets the
GOOSE_PROVIDERandGOOSE_MODELenvironment variables - Runs the benchmark with the specified suites
- Analyzes the results for failures
- Sets the
- Generates a summary of all benchmark runs
Output
The script creates the following files in the output directory:
summary.md: A summary of all benchmark results{provider}-{model}.json: Raw JSON output from each benchmark run{provider}-{model}-analysis.txt: Analysis of each benchmark run
Exit Codes
0: All benchmarks completed successfully1: One or more benchmarks failed
parse-benchmark-results.sh
This script analyzes a single benchmark JSON result file and identifies any failures.
Usage
./scripts/parse-benchmark-results.sh path/to/benchmark-results.json
Output
The script outputs an analysis of the benchmark results to stdout, including:
- Basic information about the benchmark run
- Results for each evaluation in each suite
- Summary of passed and failed metrics
Exit Codes
0: All metrics passed successfully1: One or more metrics failed