Skip to content

Compare Skill-Guided vs Freestyle

In this lab you will compare the output quality when your agent uses skills versus when it operates without them. This demonstrates the value that curated knowledge packages bring to agent-assisted AWS development.

Overview

Skills provide your agent with opinionated, production-ready patterns. Without skills, agents rely on their general training data, which may produce functional but less robust configurations. By comparing the two approaches, you will see concrete differences in IAM policies, error handling, resource configuration, and adherence to AWS best practices.

What You'll Learn

  • How skill availability affects agent output quality
  • Specific differences in IAM configuration, error handling, and best practices
  • When skills matter most (complex deployments, security-sensitive configurations)
  • How to evaluate agent output for production readiness

Instructions

Explore

Temporarily disconnect skills and ask the same deployment question, then reconnect and compare. Look for differences in IAM policy specificity, error handling patterns, and overall configuration quality.

Desired outcome: A side-by-side comparison showing how skill-guided deployments differ from freestyle in IAM, error handling, and best practice adherence.

Hints:

  1. First, capture what the agent produces without skills by asking it to just describe (not deploy) a serverless API architecture
  2. Then, with skills connected, ask the same question and compare the recommendations
  3. Focus on three areas: IAM policy scope, error handling completeness, and naming/tagging conventions
Step-by-Step Walkthrough

  1. Open your AI coding agent's chat interface

  2. Step A — Freestyle (without skills): Ask your agent to describe a deployment plan without actually deploying. Frame it so the agent relies on general knowledge:

Without using any AWS skills or MCP tools, describe how you would architect a serverless REST API with CRUD operations for a "products" table. Include the IAM policy, Lambda code structure, and API Gateway configuration. Just describe it — don't deploy anything.

  1. Note the response. Pay attention to:
  2. IAM policy: Is it scoped to specific resources or using *?
  3. Error handling: Does the Lambda code handle specific DynamoDB errors?
  4. Best practices: Are there tags, encryption settings, or capacity configurations mentioned?

  5. Naming: Does it follow any convention?

  6. Step B — Skill-guided: Now ask the same question but let the agent use its skills:

Using your AWS skills and MCP Server access, describe how you would architect a serverless REST API with CRUD operations for a "products" table. Include the IAM policy, Lambda code structure, and API Gateway configuration. Just describe it — don't deploy anything.

  1. Compare the two responses side by side. You should observe differences such as:
Aspect Freestyle Skill-Guided
IAM scope dynamodb:* on Resource: * Specific actions on specific table ARN
Error handling Generic try/catch DynamoDB-specific error codes (ConditionalCheckFailed, etc.)
API responses Basic 200/500 Proper 400, 404, 409, 500 with error bodies
Resource config Defaults Tags, encryption, point-in-time recovery
Naming Ad-hoc Consistent prefix convention
  1. Step C — Reflect: Consider these questions:
  2. Which output would you trust more in production?
  3. How much time would you spend reviewing and fixing the freestyle version?

  4. In what scenarios might the freestyle approach be acceptable?

  5. Optionally, ask your agent to summarize the differences:

Compare the two approaches above. What are the key differences in security, reliability, and operational readiness?

Validation

This lab is a comparison exercise — there are no additional resources to deploy. Your Module 2 completion is based on the resources created in the previous lab (Deploy a Serverless API).

Confirm your Module 2 resources are still in place:

  • ✅ Lambda function matching serverless-api-* exists
  • ✅ API Gateway REST API exists
  • ✅ DynamoDB table matching serverless-* exists

Agent-Specific Tips

To test the freestyle approach with Claude Code, you can prefix your prompt with explicit instructions to not use tools:

Without calling any MCP tools or using AWS skills, just based on your training knowledge, describe...

Then for the skill-guided version, simply ask normally and Claude Code will automatically discover and use relevant skills.

To compare approaches in Kiro, you can:

  1. For freestyle: Ask the agent to describe the architecture without invoking any powers or MCP tools
  2. For skill-guided: Ask the same question normally, allowing Kiro to use its full capabilities

Kiro's specs feature can help you document the comparison — create a quick spec capturing the differences you observed for future reference.

To isolate the freestyle response in Cursor:

  1. Ask the agent to describe the architecture "from general knowledge only, without using any MCP tools"
  2. Then in a new message, ask the same question normally to get the skill-guided version

Compare the two responses in the same conversation thread for easy side-by-side analysis.

:::alert{type="info"} Cursor does not have a toggle to disable MCP tools mid-conversation. Use prompt framing ("without using tools") to get the freestyle baseline.

:::

For the freestyle comparison with Codex:

  1. Ask the agent to describe the architecture "based only on your training knowledge, without calling any tools"
  2. Then ask the same question normally to get the skill-guided version

:::alert{type="info"} Codex does not have a UI toggle to disable MCP connections. Use prompt framing to request a tools-free response for comparison.

:::