Developer Experience in the Age of AI: Developing a Copilot Chat Extension for Data Streaming Engineers

Intro

Three in 4 programmers have tried artificial intelligence (AI). This factoid comes from a recent Wired survey on the habits of engineers with respect to AI tooling like GitHub Copilot. Though Wired used a pool of only around 700 engineers, Gartner’s prediction from a year ago was that 75% of enterprise software engineers would use AI by 2028. To many of us, it’s starting to feel like that’s already happened.

No matter the exact numbers, the seismic shift in professional coding processes is evident. As developer experience changes drastically, so does developer experience engineering. Here at Confluent, we’ve been hard at work providing tooling such as the Confluent VS Code extension. Our goal is simple: Meet data streaming engineers where they are. Within the context of a VS Code extension, this means working with the GitHub Copilot chat participant APIs to provide engineers with safe, fast access to data streaming–flavored AI. We’ve learned a few things along the way that we’d love to share with you. So what are we waiting for? Let’s “delve” into it.

Defining Developer Experience

What is developer experience? While the term first emerged somewhere around 2010-2011, user experience within software has existed since, well, users have used computers. We can argue that the first major leap forward was the invention of Assembly in 1949, which included human-readable syntax. Prior to that, developers would have to memorize numeric codes to program computers. This is the heart of developer experience: facilitating the efficient communication between humans and machines. It can be seen throughout modern best practices, as in these guidelines for creating a usable CLI, which recommends: “Catch errors and rewrite them for humans. If you’re expecting an error to happen, catch it and rewrite the error message to be useful.”

Symbiosis Between AI and Humans in the Developer Experience

The relationship between engineers and AI is symbiotic. AI models learn from the input and output of engineers, and engineers use AI output in their own work. Given this symbiosis, there are two aspects of improving AI tooling: The first is improving the models themselves, and the second is improving our use of them. Providing a good developer experience in the age of AI means being cognizant of the second concern. So what does it mean to provide a first-class developer experience when working on something like a GitHub Copilot chat participant?

Developing a GitHub Copilot Chat Participant

We had two primary concerns when developing the minimum viable product of our GitHub Copilot chat participant. First, safety was paramount. Next, we wanted to provide a solution for data streaming engineers that scaffolded to their needs. A recent Microsoft research paper stated:

“Knowledge workers face new challenges in critical thinking as they incorporate GenAI into their knowledge workflows. To that end, our work suggests that GenAI tools need to be designed to support knowledge workers’ critical thinking by addressing their awareness, motivation, and ability barriers.”

What are some of the challenges that engineers face when designing data streaming applications? How do we motivate them to build?

In the VS Code extension, we provide templates to help engineers kick-start projects. The reason we wrote them, in this age of AI, is that we wanted engineers to have access to complete data streaming application starters that included solid best practices as guardrails while helping those who were newer to data streaming application development.

The templates also provide a smooth developer experience in terms of reducing setup time. For example, to start building an Apache Kafka® client application, you may have to copy/paste several different resources from several different UIs: Kafka API key and secret, bootstrap server, schema registry URL, schema registry API key and secret … Add Apache Flink® capabilities, and the list continues! The VS Code extension eases this pain in a couple of ways. First, we provided a quick pick from the sidebar that enables you to generate a template with pre-filled options from a resource like a Kafka cluster:

And next, we hoped that when you typed “make me a Python project with a topic named ‘test’ and a bootstrap server ‘localhost:9092’” into our Copilot chat extension interface, it would prepare a pre-filled form for you.

We saw an opportunity to build on the symbiotic relationship between data streaming engineers and GitHub Copilot by extending our chat participant via tools. In case you’re not familiar with the concept of tools within chat participants, it means that you hand the AI chatbot the ability to call a function for the engineer. We wanted our tool to list the available projects, tell the engineer what values it needed for generating a project, and then generate the project.

Structuring Your VS Code LanguageModelTool API Integration

How does tool-calling from a Copilot chat extension work? VS Code outlines five major steps:

1. Copilot determines which tools are available.

2. Copilot sends the request to the large language model (LLM), along with appropriate context, and receives a response.

3. Copilot invokes tools with parameters from the LLM.

4. Copilot iterates in the case of errors.

5. Copilot returns the final response to the user.

Here’s the case of a tool that counts the open tabs in the editor. The user makes a request such as this: “I’d like to know how many python tabs I have open!” First, Copilot looks for applicable tools and finds the tabs_open tool. It then makes the request to the LLM, receiving a response with parameters, perhaps related to the language Python. Then Copilot invokes the tool and returns the response to the user.

Let’s take a step back and think about the relationship of a VS Code extension chat participant and the underlying LLMs, such as Claude, GPT, and Gemini. The VS Code’s chat participant APIs, such as LanguageModelTool, act as an interface to send inputs and receive outputs from each model.

There are two types of messages in the chat participant API to be aware of at this level: User messages, which are intended for the user in the conversation, and Assistant messages, which are sent to the model.

export class LanguageModelChatMessage {

       /**
        * Utility to create a new user message.
        *
        * @param content The content of the message.
        * @param name The optional name of a user for the message.
        */
       static User(content: string | Array<LanguageModelTextPart | LanguageModelToolResultPart>, name?: string): LanguageModelChatMessage;

       /**
        * Utility to create a new assistant message.
        *
        * @param content The content of the message.
        * @param name The optional name of a user for the message.
        */
       static Assistant(content: string | Array<LanguageModelTextPart | LanguageModelToolCallPart>, name?: string): LanguageModelChatMessage;

       /**
        * The role of this message.
        */
       role: LanguageModelChatMessageRole;

       /**
        * A string or heterogeneous array of things that a message can contain as content. Some parts may be message-type
        * specific for some models.
        */
       content: Array<LanguageModelTextPart | LanguageModelToolResultPart | LanguageModelToolCallPart>;

       /**
        * The optional name of a user for this message.
        */
       name: string | undefined;

       /**
        * Create a new user message.
        *
        * @param role The role of the message.
        * @param content The content of the message.
        * @param name The optional name of a user for the message.
        */
       constructor(role: LanguageModelChatMessageRole, content: string | Array<LanguageModelTextPart | LanguageModelToolResultPart | LanguageModelToolCallPart>, name?: string);
   }

The content of an Assistant message is an array of either LanguageModelTextParts or LanguageModelToolParts, and this determines the structure of our /chat folder.

Note: Formatting and wording matters. Sometimes it helps to make things more explicit in markdown when you’re sending a message for the AI model so that it “understands” it’s a user message:

      messages = `${messages}\n\nUSER: "${turn.prompt}"`;

The above is a far clearer approach than the below.

messages.push(userMessage(turn.prompt));

Here is the file structure for the whole participant:

Folder map of /vscode/src/chat
==================================================

├── summarizers/
│   ├── chatHistory.ts
│   ├── connections.test.ts
│   ├── connections.ts
│   ├── dockerContainers.ts
│   ├── environments.ts
│   ├── projectTemplate.ts
│   ├── README.md
│   └── topics.ts
├── tools/
│   ├── base.ts
│   ├── createProject.ts
│   ├── getConnections.ts
│   ├── getDockerContainers.ts
│   ├── getEnvironments.ts
│   ├── getTemplate.ts
│   ├── listTemplates.test.ts
│   ├── listTemplates.ts
│   ├── listTopics.ts
│   ├── README.md
│   ├── registration.ts
│   ├── toolMap.ts
│   └── types.ts
├── constants.ts
├── errors.ts
├── messageTypes.test.ts
├── messageTypes.ts
├── participant.ts
├── references.ts
├── telemetry.test.ts
├── telemetry.ts
└── types.ts

There are two subfolders: tools and summarizers. The logic for tool registration, the type definitions, and the implementation of the tools themselves lives here. The summarizers folder holds the functions that create LLM-friendly responses for each tool. For example, the summarizer for listing the topics holds this logic:

export function summarizeTopic(topic: KafkaTopic): string {
  const subjects: Subject[] = topic.children || [];
  if (subjects.length) {
    const subjectNames = subjects.map((subject) => subject.name).join(", ");
    const subjectLabel = subjects.length > 1 ? "Schema Subjects" : "Schema Subject";
    return (
      `- ## ${topic.name}\n` +
      `### Internal: ${topic.is_internal}\n` +
      `### Associated ${subjectLabel}\n${subjectNames}`
    );
  } else {
    return `- ## ${topic.name}\n` + `### Internal: ${topic.is_internal}\n`;
  }
}

Then, in tools/listTopics, the summary gets sent as a LanguageModelToolResult:

    const topicTextParts: LanguageModelTextPart[] = sampleTopics.map((topic) => {
      const topicSummary = summarizeTopic(topic);
      return new LanguageModelTextPart(topicSummary);
    });
    return new LanguageModelToolResult(topicTextParts);

The tool invocation that is intended to be sent back to the model is of type LanguageModelToolResultPart, where the content property is restricted to be an array of LanguageModelTextPart objects.

Since we aren't using LanguageModelPromptTsxPart, a tool's .invoke() method is only returning LanguageModelTextPart instances in its LanguageModelToolResult, which are then sent to the LanguageModelToolResultPart constructor. That LanguageModelToolResultPart will then be wrapped as a User message before being added to the message history.

Note on Safety

One of our top concerns was safety. This obviously meant that any tool written for a GitHub Copilot chat participant should not take in personally identifiable information (PII), but we also needed to modify our template creation functionality to not prompt for secrets. As stewards of good developer experience, we didn’t even provide opportunities for engineers to engage with unsafe practices in AI.

In order to be positive that a request for PII would never reach the developer, we modified the function that the tool calls itself instead of setting anything in the tool, like modifying the Assistant messages.

Here we modified getTemplatesList with a Boolean to determine whether or not the options were sanitized.

export async function getTemplatesList(
  collection?: string,
  sanitizeOptions: boolean = false,
): Promise<ScaffoldV1Template[]> {
  const client = (await getSidecar()).getTemplatesApi();
  const response = await client.listScaffoldV1Templates({
    template_collection_name: collection ?? "vscode",
  });

  const templates = Array.from(response.data) as ScaffoldV1Template[];
  return sanitizeOptions ? templates.map(sanitizeTemplateOptions) : templates;
}

The reason for this was twofold. First, we had a tool-calling chain where listTemplates could call getTemplate , which returned the options required. Then getTemplate could call createProject. It was easier and more maintainable to change the function called in listTemplates than to introduce safety gates at the level of each tool. Second, we just wanted to pick the safest choice, and this was the way to ensure that the prompt for PII never reached the tools at all.

One thing to take away: To get two tools to chain properly, you may need an intermediate tool. For example, we couldn’t just write a tool that listed the templates and then a tool that generated the form. We needed the tool to list the options for each template because those options were the input for the final tool.

If you’d like to use the tool yourself, it’s under an experimental flag. Open the Command Palette, navigate to Confluent: Support: Open Settings, and scroll down to Experimental: Enable Chat Participant in order to give it a try.

Zoom Back Out to the Wider Future of Code 

So what does developer experience look like in the age of AI? All in all, I think we’re just starting to figure that out. Making a chat participant for a VS Code extension is just one aspect of a burgeoning field.

Reflecting on the experience of working with this Copilot chat participant extension, one thing stands out: No matter how shiny the new tool is, old design principles still matter. The user (developer, that is) requires proper constraints, such as no prompting for PII. The feedback, like the user messages, needs to be clear and formatted well. The user should be aware that a tool is being called via the user messages as well, providing a mapping in their mind between the chat interface and the tools.

In all of this chat participant work, the end user experience remains the main concern. It’s good to start with the ideal conversation you’d like to have with the chatbot, then work backwards from there. You may end up writing more tools than you’d think to bridge chaining gap to get to the end user experience. Another word of advice is that it can take a little while to get used to developing chatbots. Prompting them can feel like an arcane art, but the only way to understand the shift in the programming approach is to try it yourself.

On that note, we’re open to contributions. :)