Building Autonomous Agents: Plugins and Planners in Semantic Kernel

From Orchestration to Agency

In our last adventure, we dived into the basics of the Semantic Kernel—the magic glue letting you orchestrate AI and native functions smoothly. We learned how to manually chain functions together like a carefully crafted recipe. But what if instead of you telling AI every step to take, the AI could figure out which steps to take based on what the user actually wants? Welcome to the world of Agents or Planners.

Think of it this way: orchestration is you conducting an orchestra, pointing every instrument when to play their part. Agency, on the other hand, is giving the AI a sheet of music and letting it decide how best to perform the symphony. The goal here is simplicity and flexibility. Instead of manually chaining functions, you hand over the reins and say, “You decide which functions to call to fulfill this user’s request.” This empowers your AI to tackle complex, multi-step tasks dynamically, adapting as the conversation evolves.

Structuring Plugins for Agility

Now that we’re giving control to the AI agent, it’s critical to package your functions in a way that’s both logical and helpful for the AI’s decision-making. That’s where Plugins come into play. But no, it’s not about browser extensions or WordPress add-ons here — a Plugin in Semantic Kernel is a well-organized collection of semantic and native functions grouped by domain or purpose.

Imagine you’re building an assistant that needs to manage emails, calculate math problems, and fetch the weather. Instead of dumping all your functions into one giant bucket, you create an EmailPlugin for email-related functions, a MathPlugin for calculations, and so forth. This modular approach keeps things neat and makes it easier for the AI to pick the right tool for the right job.

One of the most underrated pieces in plugin design is the Description on each function. This is the AI’s north star that tells it exactly when and why to call this function. Getting the description just right means you can dramatically improve your AI’s accuracy in function selection. Think of it as writing a conversational annotation that explains to the AI, “Hey, use me when the task looks like this.”

Here’s how you might set up multiple plugins in your Kernel instance with C#:

using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Plugins.Web;

var kernel = Kernel.Builder.Build();

// Importing the built-in web search plugin for access to live info
var webSearchPlugin = new WebSearchPlugin();
kernel.ImportPlugin(webSearchPlugin, "web");

// Importing custom email and math plugins
var emailPlugin = new EmailPlugin(); // Assume this is your custom plugin
kernel.ImportPlugin(emailPlugin, "email");

var mathPlugin = new MathPlugin(); // Another custom plugin
kernel.ImportPlugin(mathPlugin, "math");

By giving each domain its own plugin and importing them with distinct names, the AI agent gets clarity on which function belongs where. It’s like giving the AI a neatly labeled toolbox instead of a messy junk drawer.

Understanding Planners

So how does the AI actually decide which functions to call and in what order? Enter the Planner. A planner is the agent’s strategist — it takes your lofty, natural language goal and figures out a multi-step plan that calls the right semantic or native functions to get the job done.

Semantic Kernel gives you a few flavors of planners. Two popular ones are the HandlebarsPlanner and the FunctionCallingStepwisePlanner. Think of the HandlebarsPlanner as a classic script-writer, mapping out plans with smart templates. The FunctionCallingStepwisePlanner is more of a step-by-step, real-time decision maker, perfect when you want your AI to adapt quickly at every step.

For example, say your user says, “Check the weather in New York and email it to Bob.” Instead of you painstakingly coding the sequence, the planner translates this request into a plan like:

• Call the weather plugin’s function to fetch New York weather
• Format the weather info for readability
• Call the email plugin’s function to send this info to Bob

This plan then guides the kernel’s execution engine, letting your AI autonomously complete the task.

Implementing the Agent Loop

Putting it all together, the execution workflow looks like this:

1. User Input: The user enters a high-level natural language command.
2. Planner: The planner takes this command and creates a multi-step plan.
3. Plan Execution: The kernel runs through the plan, invoking functions as decided.
4. Output: The AI returns the results, whether it’s an email sent or data displayed.

Let’s break down a basic example in code. We’ll instantiate a specific planner, create a plan from a user query, and then execute it step by step:

using Microsoft.SemanticKernel;
using Microsoft.SemanticKernel.Planners;

var kernel = Kernel.Builder.Build();
var planner = new FunctionCallingStepwisePlanner(kernel);

// A user query that requires multiple steps
string userQuery = "Check the weather in New York and email it to Bob.";

// Generate a plan based on the user query
var plan = await planner.CreatePlanAsync(userQuery);

// Execute the plan
var result = await plan.InvokeAsync(kernel);

// Display the step names chosen by the AI
foreach (var step in plan.Steps)
{
    Console.WriteLine($"Step: {step.Name}");
}

Console.WriteLine($"Final Result: {result}");

With this flow, your AI acts less like a simple function caller and more like an autonomous assistant, choosing and executing the right tools at the right time to fulfill the user’s intent.

Handling Context and Memory (Optional/Advanced)

Here’s where things get really interesting for the ambitious builder. As planners break down tasks into multiple steps, you need a way to pass variables or data gathered from one step to another. Semantic Kernel provides ContextVariables for this very purpose—think of it as the AI’s short-term memory.

For instance, after fetching the weather, the AI stores it in a context variable, which the email function then reads to craft the message. This seamless passing of data keeps your plan coherent and your agent on track without manual glue code.

Moreover, this architecture serves as a rock-solid foundation for Retrieval-Augmented Generation (RAG) applications, where your AI can retrieve knowledge chunks from external data sources, combine them with real-time functions, and generate responses dynamically.

In other words, your agent becomes not just task-driven but knowledge-aware—a real powerhouse for building advanced AI assistants.