diff --git a/documentation/docs/guides/recipes/index.md b/documentation/docs/guides/recipes/index.md index f750cf5b18..3ca797e44f 100644 --- a/documentation/docs/guides/recipes/index.md +++ b/documentation/docs/guides/recipes/index.md @@ -52,6 +52,11 @@ import styles from '@site/src/components/Card/styles.module.css'; description="Learn how to save, organize, and find your Goose recipes for easy access and reuse." link="/docs/guides/recipes/storing-recipes" /> + diff --git a/documentation/docs/guides/recipes/recipe-reference.md b/documentation/docs/guides/recipes/recipe-reference.md index 8ce47d4d96..44337facd1 100644 --- a/documentation/docs/guides/recipes/recipe-reference.md +++ b/documentation/docs/guides/recipes/recipe-reference.md @@ -140,6 +140,7 @@ The `extensions` field allows you to specify which Model Context Protocol (MCP) | `name` | String | Unique name for the extension | | `cmd` | String | Command to run the extension | | `args` | Array | List of arguments for the command | +| `env_keys` | Array | (Optional) Names of environment variables required by the extension | | `timeout` | Number | Timeout in seconds | | `bundled` | Boolean | (Optional) Whether the extension is bundled with Goose | | `description` | String | Description of what the extension does | @@ -163,9 +164,35 @@ extensions: cmd: uvx args: - 'mcp_presidio@latest' - description: "For searching logs using Presidio" + + - type: stdio + name: github-mcp + cmd: github-mcp-server + args: [] + env_keys: + - GITHUB_PERSONAL_ACCESS_TOKEN + timeout: 60 + description: "GitHub MCP extension for repository operations" ``` +### Extension Secrets + +This feature is only available through the CLI. + +If a recipe uses an extension that requires a secret, Goose can prompt users to provide the secret when running the recipe: + +1. When a recipe is loaded, Goose scans all extensions (including those in sub-recipes) for `env_keys` fields +2. If any required environment variables are missing from the secure keyring, Goose prompts the user to enter them +3. Values are stored securely in the system keyring and reused for subsequent runs + +To update a stored secret, remove it from the system keyring and run the recipe again to be re-prompted. + +:::info +This feature is designed to prompt for and securely store secrets (such as API keys), but `env_keys` can include any environment variable needed by the extension (such as API endpoints, configuration values, etc.). + +Users can press `ESC` to skip entering a variable if it's optional for the extension. +::: + ## Settings The `settings` field allows you to configure the AI model and provider settings for the recipe. This overrides the default configuration when the recipe is executed. @@ -209,6 +236,7 @@ The `sub_recipes` field specifies the [sub-recipes](/docs/guides/recipes/sub-rec | `name` | String | Unique identifier for the sub-recipe | | `path` | String | Relative or absolute path to the sub-recipe file | | `values` | Object | (Optional) Pre-configured parameter values that are passed to the sub-recipe | +| `sequential_when_repeated` | Boolean | (Optional) Forces sequential execution of multiple sub-recipe instances. See [Running Sub-Recipes In Parallel](/docs/tutorials/sub-recipes-in-parallel) for details | ### Example Sub-Recipe Configuration @@ -312,7 +340,7 @@ The `response` field enables recipes to enforce a final structured JSON output f 1. **Validate the output**: Validates the output JSON against your JSON schema with basic JSON schema validations 2. **Final structured output**: Ensure the final output of the agent is a response matching your JSON structure -This **enables automation** by returning consistent, parseable results for scripts and workflows. Recipes can produce structured output when run from either the Goose CLI or Goose Desktop. +This **enables automation** by returning consistent, parseable results for scripts and workflows. Recipes can produce structured output when run from either the Goose CLI or Goose Desktop. See [use cases and ideas for automation workflows](/docs/guides/recipes/session-recipes#structured-output-for-automation). ### Basic Structure @@ -370,6 +398,20 @@ Advanced template features include: Default content {% endblock %} ``` +- `indent()` template filter + +### indent() Filter For Multi-Line Values + +Use the `indent()` filter to ensure multi-line parameter values are properly indented and can be resolved as valid JSON or YAML format. This example uses `{{ raw_data | indent(2) }}` to specify an indentation of two spaces when passing data to a sub-recipe: + +```yaml +sub_recipes: + - name: "analyze" + path: "./analyze.yaml" + values: + content: | + {{ raw_data | indent(2) }} +``` ## Built-in Parameters diff --git a/documentation/docs/guides/recipes/session-recipes.md b/documentation/docs/guides/recipes/session-recipes.md index 24f1735806..4b36b6032a 100644 --- a/documentation/docs/guides/recipes/session-recipes.md +++ b/documentation/docs/guides/recipes/session-recipes.md @@ -413,10 +413,11 @@ You can turn your current Goose session into a reusable recipe that includes the - :::info Privacy & Isolation + :::info Privacy, Isolation, & Secrets - Each person gets their own private session - No data is shared between users - Your session won't affect the original recipe creator's session + - The CLI can prompt users for required [extension secrets](/docs/guides/recipes/recipe-reference#extension-secrets) ::: @@ -568,6 +569,70 @@ retry: See the [Recipe Reference Guide](/docs/guides/recipes/recipe-reference#automated-retry-with-success-validation) for complete retry configuration options and examples. +### Structured Output for Automation + +Recipes can enforce [structured JSON output](/docs/guides/recipes/recipe-reference#structured-output-with-response), making them ideal for automation workflows that need to parse and process agent responses reliably. Key benefits include: + +- **Reliable parsing**: Consistent JSON format for scripts, automation, and CI/CD pipelines +- **Built-in validation**: Ensures output matches your requirements +- **Easy extraction**: Final output appears as a single line for simple parsing + +Structured output is particularly useful for: +- **Development workflows**: Code analysis reports, test results with pass/fail counts, and build status with deployment readiness +- **Data processing**: Results with counts and validation status, content analysis with structured findings +- **Documentation generation**: Consistent metadata and structured project reports for further processing + +**Example structured output configuration:** +```yaml +response: + json_schema: + type: object + properties: + build_status: + type: string + enum: ["success", "failed", "warning"] + description: "Overall build result" + tests_passed: + type: number + description: "Number of tests that passed" + tests_failed: + type: number + description: "Number of tests that failed" + artifacts: + type: array + items: + type: string + description: "Generated build artifacts" + deployment_ready: + type: boolean + description: "Whether the build is ready for deployment" + required: + - build_status + - tests_passed + - tests_failed + - deployment_ready +``` + +**How it works:** +1. Recipe runs normally with provided instructions +2. Goose calls a `final_output` tool with JSON matching your schema +3. Output is validated against the JSON schema +4. If validation fails, Goose receives error details and must correct the output +5. Final validated JSON appears as the last line of output for easy extraction + +**Example automation usage:** +```bash +# Run recipe and extract JSON output +goose run --recipe analysis.yaml --params project_path=./src > output.log +RESULT=$(tail -n 1 output.log) +echo "Analysis Status: $(echo $RESULT | jq -r '.build_status')" +echo "Issues Found: $(echo $RESULT | jq -r '.tests_failed')" +``` + +:::info +Structured output is supported in recipes run in both the Goose CLI and Goose Desktop. However, creating and editing the `json_schema` configuration must be done manually in the recipe file. +::: + ## What's Included A recipe captures: diff --git a/documentation/docs/guides/recipes/sub-recipes.md b/documentation/docs/guides/recipes/sub-recipes.md index 4a6a962534..a553f9140f 100644 --- a/documentation/docs/guides/recipes/sub-recipes.md +++ b/documentation/docs/guides/recipes/sub-recipes.md @@ -9,6 +9,10 @@ Sub-recipes are recipes that are used by another recipe to perform specific task - **Multi-step workflows** - Break complex tasks into distinct phases with specialized expertise - **Reusable components** - Create common tasks that can be used in various workflows +:::warning Experimental Feature +Running sub-recipes in parallel is an experimental feature in active development. Behavior and configuration may change in future releases. +::: + ## How Sub-Recipes Work The "main recipe" registers its sub-recipes in the `sub_recipes` field, which contains the following fields: @@ -26,15 +30,15 @@ Sub-recipe sessions run in isolation - they don't share conversation history, me ### Parameter Handling -Sub-recipes receive parameters in two ways: +Parameters received by sub-recipes can be used in prompts and instructions using `{{ parameter_name }}` syntax. Sub-recipes receive parameters in two ways: 1. **Pre-set values**: Fixed parameter values defined in the `values` field are automatically provided and cannot be overridden at runtime -2. **Automatic parameter inheritance**: Sub-recipes automatically have access to all parameters passed to the main recipe at runtime. +2. **Context-based parameters**: The AI agent can extract parameter values from the conversation context, including results from previous sub-recipes -Pre-set values take precedence over inherited parameters. If both the main recipe and `values` field provide the same parameter, the `values` version is used. +Pre-set values take precedence over context-based parameters. If both the conversation context and `values` field provide the same parameter, the `values` version is used. -:::info Template Variables -Parameters received by sub-recipes can be used in prompts and instructions using `{{ parameter_name }}` syntax. +:::tip +Use the `indent()` filter to maintain valid YAML format when passing multi-line parameter values to sub-recipes, for example: `{{ content | indent(2) }}`. See [Template Support](/docs/guides/recipes/recipe-reference#template-support) for more details. ::: ## Examples @@ -151,6 +155,10 @@ prompt: | ``` +:::tip +For faster execution when sub-recipes are independent, see [Running Sub-Recipes In Parallel](/docs/tutorials/sub-recipes-in-parallel) to execute multiple sub-recipes concurrently. +::: + ### Conditional Processing This Smart Project Analyzer example shows conditional logic that chooses between different sub-recipes based on analysis: @@ -284,6 +292,117 @@ prompt: | ``` +### Context-Based Parameter Passing + +This Travel Planner example shows how sub-recipes can receive parameters from conversation context, including results from previous sub-recipes: + +**Usage:** +```bash +goose run --recipe travel-planner.yaml +``` + +**Main Recipe:** + +```yaml +# travel-planner.yaml +version: "1.0.0" +title: "Travel Activity Planner" +description: "Get weather data and suggest appropriate activities" +instructions: | + Plan activities by first getting weather data, then suggesting activities based on conditions. + +prompt: | + Plan activities for Sydney by first getting weather data, then suggesting activities based on the weather conditions we receive. + +sub_recipes: + - name: weather_data + path: "./sub-recipes/weather-data.yaml" + # No values - location parameter comes from prompt context + + - name: activity_suggestions + path: "./sub-recipes/activity-suggestions.yaml" + # weather_conditions parameter comes from conversation context + +extensions: + - type: builtin + name: developer + timeout: 300 + bundled: true +``` + +**Sub-Recipes:** + +
+ weather_data + ```yaml + # sub-recipes/weather-data.yaml + version: "1.0.0" + title: "Weather Data Collector" + description: "Fetch current weather conditions for a location" + instructions: | + You are a weather data specialist. Gather current weather information + including temperature, conditions, and seasonal context. + + parameters: + - key: location + input_type: string + requirement: required + description: "City or location to get weather data for" + + extensions: + - type: stdio + name: weather + cmd: uvx + args: + - mcp_weather@latest + timeout: 300 + description: "Weather data for trip planning" + - type: builtin + name: developer + timeout: 300 + bundled: true + + prompt: | + Get the current weather conditions for {{ location }}. + Include temperature, weather conditions (sunny, rainy, etc.), + and any relevant seasonal information. + ``` +
+ +
+ activity_suggestions + ```yaml + # sub-recipes/activity-suggestions.yaml + version: "1.0.0" + title: "Activity Recommender" + description: "Suggest activities based on weather conditions" + instructions: | + You are a travel expert. Recommend appropriate activities and attractions + based on current weather conditions. + + parameters: + - key: weather_conditions + input_type: string + requirement: required + description: "Current weather conditions to base recommendations on" + + extensions: + - type: builtin + name: developer + timeout: 300 + bundled: true + + prompt: | + Based on these weather conditions: {{ weather_conditions }}, + suggest appropriate activities, attractions, and travel tips. + Include both indoor and outdoor options as relevant. + ``` +
+ +In this example: +- The `weather_data` sub-recipe gets the location from the prompt context (the AI extracts "Sydney" from the natural language prompt) +- The `activity_suggestions` sub-recipe gets weather conditions from the conversation context (the AI uses the weather results from the first sub-recipe) + ## Best Practices - **Single responsibility**: Each sub-recipe should have one clear purpose - **Clear parameters**: Use descriptive names and descriptions diff --git a/documentation/docs/tutorials/recipes-tutorial.md b/documentation/docs/tutorials/recipes-tutorial.md index 7100420095..f5aa563784 100644 --- a/documentation/docs/tutorials/recipes-tutorial.md +++ b/documentation/docs/tutorials/recipes-tutorial.md @@ -32,7 +32,7 @@ goose run --recipe trip.yaml ## Extensions -Goose recipes have a section where you can specify which extensions Goose can use during execution. Goose will only use the ones you specify. +Goose recipes have a section where you can specify which [extensions](/docs/guides/recipes/recipe-reference#extensions) Goose can use during execution. Goose will only use the ones you specify. Let's say we want to make sure we have good weather during our Europe trip. We can just add a weather extension (this example uses the [weather-mcp-server](https://github.com/TuanKiri/weather-mcp-server) by TuanKiri under the MIT License) to our recipe, modify the prompt a bit and now Goose will check the weather before adding a city to our trip. @@ -53,6 +53,8 @@ extensions: args: [] timeout: 300 description: "Weather data for trip planning" + env_keys: + - WEATHER_API_KEY ``` ## Parameters diff --git a/documentation/docs/tutorials/sub-recipes-in-parallel.md b/documentation/docs/tutorials/sub-recipes-in-parallel.md new file mode 100644 index 0000000000..e4d8d7c5fa --- /dev/null +++ b/documentation/docs/tutorials/sub-recipes-in-parallel.md @@ -0,0 +1,214 @@ +--- +title: Running Sub-Recipes In Parallel +sidebar_label: Sub-Recipes In Parallel +description: Run multiple sub-recipes instances concurrently with real-time progress tracking +--- + +Goose recipes can execute multiple [sub-recipe](/docs/guides/recipes/sub-recipes) instances concurrently using isolated worker processes. This feature enables efficient batch operations, parallel processing of different tasks, and faster completion of complex workflows. + +:::warning Experimental Feature +Running sub-recipes in parallel is an experimental feature in active development. Behavior and configuration may change in future releases. +::: + +Here are some common use cases: + +- **Monorepo build failures**: When 3 services fail in a monorepo build, use a "diagnose failure" sub-recipe with each build URL to diagnose all failures in parallel +- **Document summarization**: Process a CSV file with document links by running a "summarize document" sub-recipe for each link simultaneously +- **Code analysis across repositories**: Run security, quality, and performance analysis on multiple codebases simultaneously + +## How It Works + +Parallel sub-recipe execution uses an isolated worker system that automatically manages concurrent task execution. Goose creates individual tasks for each sub-recipe instance and distributes them across up to 10 concurrent workers. + +| Scenario | Default Behavior | Override Options | +|----------|------------------|------------------| +| **Different sub-recipes** | Sequential | Add "in parallel" to prompt | +| **Same sub-recipe** with different parameters | Parallel | • Set `sequential_when_repeated: true`
• Add "sequentially" to prompt | + +### Different Sub-Recipes + +When running different sub-recipes, Goose determines the execution mode based on: +1. **Explicit user request** in the prompt ("in parallel", "sequentially") +2. **Sequential execution by default**: Different sub-recipes run one after another unless explicitly requested to run in parallel + +In your prompt, you can simply mention "in parallel" in your prompt when calling different sub-recipes: + +```yaml +prompt: | + run the following sub-recipes in parallel: + - use weather sub-recipe to get the weather for Sydney + - use things-to-do sub-recipe to find activities in Sydney +``` + +### Same Sub-Recipe + +When running the same sub-recipe with different parameters, Goose determines the execution mode based on: +1. **[Recipe-level configuration](#choosing-between-sequential-and-parallel-execution)** (`sequential_when_repeated` flag) - when set to true, this forces sequential execution +2. **User request** in the prompt ("sequentially" to override default parallel behavior) +3. **Parallel execution by default**: Multiple instances of the same sub-recipe run concurrently + +If your prompt implies multiple executions of the same sub-recipe, Goose will automatically create parallel instances: + +```yaml +prompt: | + get the weather for three biggest cities in Australia +``` + +In this example, Goose recognizes that "three biggest cities" requires running the weather sub-recipe multiple times for different cities, so it executes them in parallel. + +If you wanted to run them sequentially, you can just tell Goose: + +```yaml +prompt: | + get the weather for three biggest cities in Australia one at a time +``` + +### Real-Time Progress Monitoring + +When running multiple tasks in parallel from the CLI, you can track progress through a real-time dashboard that automatically appears during execution. The dashboard provides: +- **Live progress tracking**: Monitor task completion in real-time with statistics for completed, running, failed, and pending counts +- **Task details**: View unique task IDs, parameter sets, execution timing, output previews, and error information as tasks progress from Pending → Running → Completed/Failed + +## Examples + +### Running Different Sub-Recipes in Parallel + +This example runs the `weather` and `things-to-do` sub-recipes in parallel: + +```yaml +# plan_trip.yaml +version: 1.0.0 +title: Plan Your Trip +description: Get weather forecast and find things to do for your destination +instructions: You are a travel planning assistant that helps users prepare for their trips. +prompt: | + run the following sub-recipes in parallel to plan my trip: + - use weather sub-recipe to get the weather forecast for Sydney + - use things-to-do sub-recipe to find activities and attractions in Sydney +sub_recipes: +- name: weather + path: "./sub-recipes/weather.yaml" + values: + city: Sydney +- name: things-to-do + path: "./sub-recipes/things-to-do.yaml" + values: + city: Sydney + duration: "3 days" +extensions: +- type: builtin + name: developer + timeout: 300 + bundled: true +``` + +### Running the Same Sub-Recipe in Parallel (with Different Parameters) + +This example runs three instances of the `weather` sub-recipe in parallel for different cities: + +```yaml +# multi_city_weather.yaml +version: 1.0.0 +title: Multi-City Weather Comparison +description: Compare weather across multiple cities for trip planning +instructions: You are a travel weather specialist helping users compare conditions across cities. +prompt: | + get the weather forecast for the three biggest cities in Australia + to help me decide where to visit +sub_recipes: +- name: weather + path: "./sub-recipes/weather.yaml" +extensions: +- type: builtin + name: developer + timeout: 300 + bundled: true +``` + +**Sub-Recipes:** + +
+ weather + ```yaml + # sub-recipes/weather.yaml + version: 1.0.0 + title: Find weather + description: Get weather data for a city + instructions: You are a weather expert. You will be given a city and you will need to return the weather data for that city. + prompt: | + Get the weather forecast for {{ city }} for today and the next few days. + parameters: + - key: city + input_type: string + requirement: required + description: city name + extensions: + - type: stdio + name: weather + cmd: uvx + args: + - mcp_weather@latest + timeout: 300 + ``` +
+ +
+ things-to-do + ```yaml + # sub-recipes/things-to-do.yaml + version: 1.0.0 + title: Things to do in a city + description: Find activities and attractions for travelers + instructions: You are a local travel expert who knows the best activities, attractions, and experiences in cities around the world. + prompt: | + Suggest the best things to do in {{ city }} for a {{ duration }} trip. + Include a mix of popular attractions, local experiences, and hidden gems. + {% if weather_context %} + Consider the weather conditions: {{ weather_context }} + {% endif %} + parameters: + - key: city + input_type: string + requirement: required + description: city name + - key: duration + input_type: string + requirement: required + description: trip duration (e.g., "2 days", "1 week") + - key: weather_context + input_type: string + requirement: optional + default: "" + description: weather conditions to consider for activity recommendations + ``` +
+ +## Choosing Between Execution Modes + +While parallel execution offers speed benefits, sequential execution is sometimes necessary or preferable. Here's how to decide: + +**Use Sequential When:** +- Tasks modify shared resources +- Order of execution matters +- Memory or CPU constraints exist +- Debugging complex failures in parallel mode + +**Use Parallel When:** +- Tasks are independent +- Faster completion is desired +- System resources can handle concurrent executions for up to 10 parallel workers +- Processing large datasets or multiple files + +**Recipe-Level Configuration:** + +For sub-recipes that should never run in parallel, set `sequential_when_repeated: true` to override user requests: + +```yaml +sub_recipes: + - name: database-migration + path: "./sub-recipes/migrate.yaml" + sequential_when_repeated: true # Always sequential +``` + +## Learn More +Check out the [Goose Recipes](/docs/guides/recipes) guide for more docs, tools, and resources to help you master Goose recipes. \ No newline at end of file