fix: improve smoke test prompt for reliable tool calling

The previous prompt 'please list files in the current directory' was
ambiguous and didn't explicitly require tool usage. Models like
qwen/qwen3-coder and z-ai/glm-4.6 would sometimes respond with text
describing what they would do instead of actually calling the tool.

The new prompt explicitly instructs the model to immediately call the
shell tool without asking for confirmation, which should improve
reliability for models with weaker tool-calling capabilities.
This commit is contained in:
Bradley Axen
2025-12-24 20:39:09 -08:00
parent a23e2cfcb2
commit ef0252bb3f
+1 -1
View File
@@ -85,7 +85,7 @@ for provider_config in "${PROVIDERS[@]}"; do
echo "Model: ${MODEL}"
echo ""
TMPFILE=$(mktemp)
(cd "$TESTDIR" && "$SCRIPT_DIR/target/release/goose" run --text "please list files in the current directory" --with-builtin "$BUILTINS" 2>&1) | tee "$TMPFILE"
(cd "$TESTDIR" && "$SCRIPT_DIR/target/release/goose" run --text "Immediately call the shell tool to run 'ls -la'. Do not ask for confirmation." --with-builtin "$BUILTINS" 2>&1) | tee "$TMPFILE"
echo ""
if grep -qE "$SUCCESS_PATTERN" "$TMPFILE"; then
echo "✓ SUCCESS: Test passed - $SUCCESS_MSG"