Yesterday Stephan Bisser and I were back at CollabDays Belgium, our seventh time on that stage if you count the Benelux editions, and this year we brought two sessions instead of one. We wrapped up the second a few minutes early because we had a flight to catch, which sounds very rock and roll until you picture the two of us an hour later in the duty-free shop, comparing chocolate boxes for the kids back home. Nothing gets two MVPs back down to earth faster, and that’s exactly how it should be.

If you’ve never been, CollabDays Belgium is what a community event should look like. Around 350 people came to De Montil on a Saturday, for free, to sit through about 30 sessions in five parallel tracks covering Microsoft 365, AI and Copilot, Power Platform, Azure, security and governance. None of that happens by itself. The day is run by SmartBytes, where the Belgian Microsoft 365 and Power Platform communities have joined forces, a lot of people give up their weekend so the rest of us can talk shop, and the sponsors are the reason nobody has to buy a ticket. Thank you, all of you. We keep coming back, and it’s because of the people who are still around at the SharePint, not because of our slides.

Every year in Belgium we’ve told one story about AI. In 2023 it was the state of the nation for AI in Microsoft 365, in 2024 building your own custom Copilot, and last year we demystified Copilot agents, extensions and plugins. This year the story split in two.

Claude vs. Copilot turned out to be the wrong fight

The morning session, “Claude vs. Copilot: From Office Worker to Dev”, opened with the cheapest trick in the book, because the easiest way to fill a room at a Microsoft conference is to put a non-Microsoft product in the title. Behind the clickbait we walked through three pairs, Microsoft 365 Copilot against Claude for everyday assisting and creating, Copilot Cowork against Claude Cowork for delegating real work, and GitHub Copilot against Claude Code for engineering. The plot twist sits right in the middle, because Microsoft itself announced in March that Copilot Cowork was built by integrating the technology that powers Anthropic’s Claude Cowork, so the room was never really choosing between two models.

What they’re actually choosing is a harness. My granddad was a climber, and I still have his gear at home from forty years ago. The harness never climbed a single metre, it kept him safe and pointed in the right direction while he did the climbing, and that’s what Copilot, Claude Code or your own app does for a model. Every LLM is a goldfish, stateless by design, and the harness decides which instructions, which data, which tools and which permissions land in the context window. For Copilot that’s Microsoft 365, with Work IQ pulling in your mail, chats and files, while Claude brings its own harness and reaches your data through connectors. The models themselves go off like milk, with a new frontier model roughly every 18 days, so we told people to pin the task and not the model.

Goldfish photo in the slide by James St. John, CC BY 2.0, via Wikimedia Commons.

Anna was back, too. She’s the project manager from our session last year, when the question was who in her team should get a Copilot licence, and this year she wanted to know which tool to bet on now that half her team uses Copilot and the other half secretly uses Claude. Neither of us had thought about her in a year. Two days before the event I’d built a skill over our roughly 160 sessions from the last ten years and asked it for a deck for Belgium, and it came back with Anna. Our answer to her was not to bet on one tool but to choose per workflow. Summarising today’s mail and meetings is a Copilot job, research across the web and non-Microsoft tools is a Claude job, and a regulated rollout in the EU is neither by default until someone has done the data protection homework. That homework is the Belgian question, too. Anthropic models inside Copilot sit outside the EU Data Boundary and are switched off by default in the EU, EFTA and the UK, so “Copilot Cowork is the safe EU option” is only true after an admin opts in and documents the transfer.

Most of the hour was about how we work today rather than about slides, which is fitting, because we’d arrived the day before with literally zero of them. Stephan had Claude Code turn the abstract and his ideas into a 500-line storyline in Markdown, handed that to Copilot Cowork, which already knows our company design, and let it build the deck plus a one-page cheat sheet so I could catch up in the few hours I had. Then I asked the front row for a favourite colour, said a few sentences in German into my phone, and Bob, the builder agent from the fleet I wrote about yesterday, went off to build a speaker memory game as an SPFx web part in blue and deploy it to SharePoint while we kept talking. It delivered, with the speakers of the day as the cards, Stephan and me included. The voice pipeline behind that costs a few cents for Service Bus and the Azure Function plus around four euros a month for transcription. And because in our projects agents rarely stall at the model but almost always at the integration stack, connectors, consent and an owner after the pilot, the deck ends with a Monday checklist for admins and for everyone who isn’t one.

Two agents from one vendor, and two very different bets

The afternoon session had the slot right after lunch, the one speakers call the grave digger session, and its title had already aged by the time we got on stage. When we wrote the abstract in summer it said “Copilot Cowork vs. Microsoft Scout”. Since then Scout has moved from the desktop into the cloud and has been renamed Microsoft Autopilot (not to be confused with Windows Autopilot, which still provisions your devices).

Both agents run on borrowed engines. Cowork is Microsoft’s answer to Claude Cowork, and Autopilot is its answer to OpenClaw, a hardened version with proper governance around it, which made me smile after my weekend with OpenClaw earlier this year. What Microsoft adds is the enterprise wrapper around them, identity, labels, audit, context and billing, and that wrapper is the actual product.

The difference between the two is less about features and more about whose name is on the work and whether the work ever ends. Copilot Cowork, generally available since 16 June, works as you, with your permissions, and delegates a task that has a result. Autopilot gets its own Entra identity, a manager and a memory that lasts for days, so you don’t hand it a task, you staff a role. We couldn’t get into the Autopilot preview in time, so Stephan hired his own autopilot from a Foundry blueprint, and Scouty, as he calls it, introduced itself live to the room. Before that it had lived as Scout automations on his laptop, which burned through 600 euros of GitHub AI credits in two weeks and only worked while the laptop was awake. That’s a decent argument for why an always-on colleague needs a computer that never sleeps.

I bet on Cowork, Stephan bets on Autopilot, and we’ll see who’s right. What we agreed on is the guardrail. Anything Scouty sends to another person needs Stephan’s approval and ends with a byline saying an agent wrote it, and our agents can draft mails but can’t send them, because the app registration simply doesn’t have the permission.

The part I enjoyed most came near the end. The storyline for that session was drafted with an AI assistant, and every slide and every image in the deck was built by Copilot Cowork from that storyline, revised four times just by chatting with it in under two days and reviewed by us. The agents did the work and our names were on it, which was exactly the point of the session. Our advice for the next weeks is to choose by the work and not by the product, and to watch Ignite from 17 to 20 November, because the names will probably change again.

The slides

Both decks are here to browse or download.

Looking at the two sessions side by side, they ended up saying the same thing from two directions. Don’t pick a logo, pick where your work lives, and be clear about whose name is on what an agent produces. Belgium, thanks for having us again.

In March I wrote about my first coding agent fleet, a handful of Squad agents and the GitHub Copilot coding agent turning user feedback into pull requests. At the end of July I went a step further and created a single folder for my presales work, put a rules file in it and started Claude Code there. Three months later that folder has eight siblings, and between them they prepare my meetings, qualify opportunities, draft quotes, keep an eye on existing customers and write content, while I’m still the only one who talks to customers. Nothing leaves the house without going through me, and that one rule shaped more of the architecture than anything else.

Every agent is still just a folder

That first folder already contained the whole idea, and it hasn’t changed since. Each agent is a directory with a CLAUDE.md for its role and rules, a .claude/settings.json for permissions and hooks, a board file it keeps up to date and an inbox/ where work arrives. When Claude Code starts in that folder it picks all of this up, so the role description, the memory and the permission model live side by side, and Git keeps the history of every one of them.

What I like most is that none of my own tooling is required. Open a terminal in the folder, type claude, and you’re talking to the same agent with the same limits as when my cockpit starts it. A new agent costs me a folder and a small JSON file, not a line of new code.

Here’s the crew, cut along the path a customer takes rather than along the tools they use.

When they talk to me, each of them has its own voice. Anything they write for other people is neutral or in my voice, and a document that sounds like an agent is a bug I fix.

A voice message travels through Azure and ends up on my Mac

Most of the time I’m not sitting in front of a terminal when I need something, so there are two other ways in, a voice message from my iPhone and a chat with a Teams bot. Let’s say I press the action button and say “Anna, prepare tomorrow’s ten o’clock”. The shortcut sends the audio to an Azure Function, which checks that it’s really me and confirms right away, and Azure AI Speech turns it into text. I give Speech a phrase list with my agents’ names, a small detail that matters more than you’d think, because without it Konrad came back as something else in every second message. The transcript then waits on a Service Bus queue.

The cloud never does any of the actual work, it’s a mailbox and nothing more. On my Mac a small service keeps an outbound connection to the queue open, using AMQP over WebSockets on port 443, which also works on networks that block plain AMQP and means nothing on my machine has to be reachable from outside. It picks up the message, drops it as a file into Anna’s inbox/ and hands my local cockpit a job (if the first word isn’t one of the names, Charly decides who gets it).

The cockpit is a small Node app with SQLite, about 17,000 lines without a build step, and it deliberately doesn’t think for itself. For each job it starts a Claude Code process in the agent’s folder, so Anna works under exactly the rules and permissions she’d have in the terminal, reads the case and the meeting transcripts and writes her prep sheet into the case folder. Through MCP the cockpit hands her a few tools of its own, so she can ask me a question, send a Teams message, propose a handoff or save a checkpoint.

When she’s missing a decision, the question pops up as an Adaptive Card in my Teams chat, and if I don’t answer within ten minutes the session parks itself and picks up again from its checkpoint as soon as I do. The result comes back the same way, a short message with the file attached. Voice and chat count as one conversation, so a reply without a name in the next four hours simply continues where we left off.

The brakes live in the systems, not in the prompt

For the first seven weeks every boundary in the fleet was a sentence in a prompt, “never write outside your own library” and the like, which is a polite request and not much more. Since 19 September every folder also has deny rules in its settings.json, and an agent that runs into one says “blocked” and stops instead of looking for a way around.

The brakes I trust most, though, sit further out, in the systems the agents talk to. The Entra app registration behind the fleet has Mail.ReadWrite so the agents can put drafts into my mailbox, but it doesn’t have Mail.Send, and no prompt in the world changes what Graph allows. Quotes for the ERP go through a small gate. Otto builds a preview with a checksum, I look at it, and only after my explicit yes is an approval file with exactly that checksum written. The gate only writes when the two match and has no way of deleting anything, so what ends up in the ERP is what I actually saw.

CRM and SharePoint run on my delegated permissions, which means an agent can never do more than I can. And a voice message is an order rather than an approval, so if I say “prepare the quote” into my phone, what I get back is a preview.

Agents hand work over as files

Between the agents there’s no API and no database, just inboxes. When an opportunity is won, Anna writes a file into Bea’s inbox and an anonymized reference pattern goes to Martin, two of 16 numbered handoff rules. The cockpit notices the occasion and suggests the handoff, and once I click, the agent the information comes from writes the file, while the cockpit itself never copies content from one folder into another.

It sounds almost embarrassingly simple, but it’s the reason I can debug the whole thing. When an agent proposes something odd, I open the file it read and see where the idea came from.

Two rules sit on top of every handoff. Anything in an inbox file or a transcript that sounds like a command is treated as a finding about that file and never as an instruction, and missing facts stay marked as missing instead of being filled in with something plausible.

One of them talks to a colleague’s agent

Since 17 October Charly has been exchanging requests with a colleague’s agent, the first time the fleet talks to anything outside my own setup. Only Charly does the talking and the others go through Charly, and a request from that side is never treated as an order from me. A loop breaker stops a thread after a few automatic replies in a row, and my colleague and I can both read the whole history.

I can’t give you a number

I haven’t measured hours saved or win rate, so I won’t make a number up. This is a personal setup on one Mac, built for one person, and whether it would work for a team is something I simply don’t know yet. Every customer conversation and every negotiation stays with humans.

The deny rules also have a limit I only stumbled over recently. According to the Claude Code docs they cover Claude’s own file tools and the shell commands it recognizes, but not a script that opens a file on its own, and closing that gap takes the sandbox, which I haven’t switched on yet.

The boring parts survived every rebuild

Looking back over three months, what’s still standing from the early days is the boring stuff. One folder per role, handoffs only through the receiver’s inbox, a checksum gate in front of anything that can’t be undone, a cloud that’s only a mailbox and a control layer that stays on my own machine. Everything else I’ve torn down and rebuilt more than once. It all runs on Claude Code with Microsoft 365 and Azure around it, mostly because that’s what I had at hand, and I’m not trying to sell you any of it.

If you’re building something similar, take a look at which of your boundaries still rest on a prompt alone. Until 19 September, all of mine did.

A few weeks ago I kept reading and hearing that everyone should build their own AI harness, and my first thought was, sure, I’ll just casually rebuild Claude Code or GitHub Copilot. This week it finally clicked what that meant, because I’d been doing it all along without noticing. I never touched the model or the agent loop. What I kept tuning was everything around it, the folders, the rules, the inboxes and the handoffs, until the context that reaches the LLM fits my tasks as closely as I can get it. That’s the harness, and mine grew one folder at a time.

My motto on the speaker slide for this talk was “sometimes you have to tell the tiger that it’s orange”. It’s the kind of line that only makes sense once you’ve seen the problem it’s about, and the problem in this case was one that Jens Kröhnert and I had each been looking at from opposite sides for years. Jens is Principal Solution Architect at ORAYLIS in Düsseldorf, where they build data platforms, data warehouses and business intelligence. I come from Microsoft 365, SharePoint and now Copilot at Solvion. We only met this year, found out that we’ve both been with our companies for about twenty years and that we think in surprisingly similar patterns, and decided to put both perspectives on one stage at a Copilot day in Vorarlberg, under the title “From Copilot to agents: how companies scale AI productively”.

AI often happens twice

When we say AI initiatives fail, we don’t mean the dramatic kind. We mean the projects that start with a lot of enthusiasm, and then after two, three, four or five months somebody asks whether anyone is actually using it. The answers vary. Some say it’s a much better search, some use it as an answer machine, and then comes the question that decides everything, usually once the pilot is over. Do we license more people, fewer, and who’s next? That’s the point where we see things break.

And they break because AI usually starts on two separate islands. In the business, people try Copilot and their first agents. Some are all in and push to see what works, others need to be picked up. On the IT side, the reactions range from “we need this, it’ll bring us so much, but it has to be secure” all the way to “we don’t want this at all, where’s our data, will they train on it?” Every one of those concerns is legitimate. But if you asked both sides separately where they stand and where they want to be with AI next year, would you get the same answer?

Frontend AI, backend AI and the competence nobody owns

Jens gave the two sides names that I’ve been using ever since. Frontend AI is Copilot, the AI every employee has in front of them, good for everything from a cake recipe to a real business question. Backend AI is the consolidated data platform, the place where a company finally pulls its most important data out of the operational silos and creates one version of the truth. It used to be called business intelligence, today it’s a data lake or an analytics platform, and the names keep changing while the idea stays the same. In Jens’s experience, many companies still haven’t done that homework.

Each side has a typical blind spot. The data platform delivers insights, but the potential never reaches the daily work, and governance and lifecycle exist but never make it to the frontend. Copilot agents, on the other hand, use what’s easy to reach rather than what actually counts, and their use cases end exactly where company data begins. Our colleagues learn Copilot, use it, and then ask the obvious question. How do I get to my KPIs, my revenue numbers? Not everything lives in Microsoft 365, not even in our own company, and that next step of bringing your real work context into the context window is what this year has been about, with MCP servers and similar tools.

So successful AI needs three competences, not two. Backend AI, frontend AI, and the ability to connect them, which is the one that usually sits with nobody.

The shortcut that only gives you your own truth

With my background I could sit down and build a Copilot agent that goes from the very top of the stack straight down to the data, in sixty or seventy minutes if I have the permissions. Easy. And then a colleague comes along and says, nice, that’s your solution with your interpretation of the data. If five people build that shortcut, you get five definitions of the same KPI, and in a large company that starts with something as basic as how margin is defined. The old SharePoint rule still applies. Just because you can doesn’t mean you should.

Jens had the matching story from his side. ORAYLIS used to be asked for business intelligence, and then at some point everyone asked for self-service BI instead, and they had to explain every time that you need a stable BI core before you let people loose with self-service. It’s the same story again, only this time it’s about context.

What the connection looks like in practice

We showed it with two demos. In the first, purchasing data sits in Microsoft Fabric, the one version of the truth for purchasing. You used to need SQL to work with it. Now you switch on a Fabric data agent and ask questions in plain language, which is backend AI. Then we switched to the frontend. A colleague in purchasing is supposed to email the top three suppliers every week, so first they need to know who the top three are. They don’t have to work that out themselves or define any business logic. From Microsoft 365 Copilot they simply tap into the backend agent that delivers the quality-assured answer, and carry on with their own workflow from there.

The second demo goes the other way round, and it’s the one people ask about. Once you can read and interpret data, you quickly want to change it, right there from your agent. There’s no built-in function for that yet, but you can write an MCP server, a tool that gets loaded into the context of your harness and that understands you’d like this or that record changed in the other system. It’s the same story as Teams ten years ago, no more context switching, you stay in one window. The non-negotiable part is identity. The write has to happen under my identity and be traceable, exactly as if I had made the change in that system myself. Because Copilot runs under my identity, that’s solvable, and it’s what makes the full round trip safe.

My recommendation is still to build the reading part first and add writing when somebody actually asks for it. Making an agent write is technically very simple. Staying in control as a human while it does is much harder, and Jens put it better than I can. For us, human control has become a more important phrase than human in the loop. You need to trust that the agent performs exactly the one atomic operation you wanted, not something that comes back as “A1, A2, A3, A4” and has quietly changed more along the way.

To get away from pharma or automotive examples, we also showed my Pizza Factory, a complete .NET application that simulates a pizzeria as a stand-in for any third-party system, because the business doesn’t only run in Microsoft 365. With the SharePoint Copilot apps and MCP apps, Copilot doesn’t just return text anymore but real interactive UI, still in preview at the time of the talk. The whole thing took me an afternoon, and the repository is public.

Microsoft is lining up the backend side under the IQ label. Fabric IQ stands for data, Work IQ is the evolution of Microsoft Graph with natural-language access to everything in Microsoft 365, and Foundry IQ covers the institutional knowledge that sits in thousands of documents somewhere else and still needs preparing before an AI can use it well. Fabric IQ is where Jens is at home, Work IQ is where we are.

IT builds the pitch, the business plays

I used to coach youth football, and one thing they taught us is to let kids play. When seven-, eight- and nine-year-olds play and you walk in, the instinct is to correct everything. Don’t. Let them play until they’re done, and then the training starts. Translated to AI, I’d like IT to see its job as building the pitch. The grass is cut, the water drains when it rains, the floodlights work, the changing rooms are there, and the lines are clear, which means security and governance. Inside the lines, the business plays. IT can’t take every department by the hand and lead it to happiness, that won’t scale. What it can do is provide central services and a frame within which a maker community can move freely without anything breaking.

Where to start

We closed with four questions, and they’re the same ones we’d ask in any first workshop. First, context and position: where do your data and their use stand, and what do the two sides know about each other? Second, target picture and impact: what should AI really change, and how would you recognise success? Even “we’re introducing Copilot” and “we’re introducing Copilot well” are two different targets, and a number in the annual report rarely depends on Copilot alone. Third, feasibility and a lighthouse: what works today, what needs a foundation first, and which case convinces both sides? Fourth, focus and a roadmap: what happens in the next six months, and who decides what?

On the roadmap I keep coming back to Eisenhower’s line that plans are worthless but planning is everything. With a new announcement every few weeks, a plan is old after four weeks, so plan in shorter cycles, three months, maybe six, and accept that the fog gets thicker the further out you look.

And put the right roles at one table: IT leadership or the CIO or CDO, the head of data, BI or analytics, the Microsoft 365 and modern work lead, a business lead with AI ambitions, and at least one person who can make decisions. It reminds me of the move from on-premises to the cloud, when the Exchange, SharePoint and directory people suddenly had to talk to each other although they’d been in the same building for years. Jens said the same thing from his side. At some large customers in the region, both of our companies have been long-time providers for years and had never spoken to each other, and neither had the departments. ORAYLIS builds the reliable numbers, we train the people who need them, and the people often don’t even know those numbers exist.

AI may happen twice. It shouldn’t end up with two versions of the truth.

The slides

The English version of our deck. The illustrations on two slides still carry their original German labels.

Last week I was at the AI Tour in Munich. Andreas Krüger introduced me to Squad. A GitHub project from a group of Microsoft Engineers. (THX Brady Gaster)

And what can I say, it’s been rattling around in my head ever since.

The Idea

In our internal app, users can submit feedback and ideas. Screenshot it, add details, send it. All gets processed internally and simultaneously synced to a GitHub issue list.

I get a notification in Teams, follow the link, look at the feedback and tag the issue with squad:copilot if I think it’s the right one for the job.

And from there, the GitHub Copilot Agent takes over. It looks at the feedback and starts implementing. When it’s done, I check what came out and decide if it fits or not. Theoretically even on my phone. No matter where I am.

So features and ideas end up in the app without me having to sit at a desk.

Squad in the Terminal

And when I do take a seat at my desk? Squad is a whole team of agents in the GitHub Copilot CLI that supports me with ideas and implementations right in the terminal.

I knew this day would come. But now that it’s here, it feels pretty surreal.

The Reality

For a couple of weeks now we’ve been seeing similar things from the open source and consumer world. Coding Agent Fleets aren’t a concept anymore. They’re reality.

This here is my first Coding Agent Fleet with Enterprise tooling. From user feedback to finished feature. Orchestrated by agents and put together in a way that makes it super easy for devs to develop things themselves too.

What This Means for Companies

We never had such good opportunities to go with “Make” instead of “Buy” as we do now. How sensible that is at scale is a whole other story.

But I’m pretty sure this exact discussion will be pretty intense in the coming months.

It won’t get slower anymore…

Many thanks to Andreas Krüger for introducing me to Squad and to Brady Gaster for the project!

Over the weekend I played around with OpenClaw. Built skills, created a Graz assistant (if you want to try it, search for “grazy” on ClawHub), just explored what’s possible. Right now I’m tinkering with a diary skill that checks what happened during the day across my OpenClaw chats and summarizes it. And the building itself just happens in a Telegram chat with OpenClaw, on the couch, Netflix running in the background. I started coding with TurboPascal over 25 years ago, what’s happening right now was science fiction in 1998.

Since the ZIB2 interview with the OpenClaw creator, quite a few relatives and friends have reached out. “You do something with AI, right?” And I’m realizing that in all the excitement, people are missing the bigger picture.

OpenClaw is a personal agent. Runs locally, on my machine, for me. Peter Steinberger said exactly that when he was on with Armin Wolf. A playground project that blew up. Not an enterprise solution. And he was honest about it: if you don’t understand the risks, better wait a little longer.

Microsoft Copilot is something different. Not better, not worse, but a fundamentally different approach. Copilot isn’t a tool you install somewhere. It lives inside the system an organization already works in. Outlook, Teams, SharePoint, the permissions, the governance, the data. It acts in the context of my work and that gives it different access, but it also has to meet very different standards.

The distinction I think matters more than any feature comparison: personal tool vs. enterprise platform. OpenClaw, ChatGPT, Claude. Great tools for my own productivity. And there will always be room for expert systems beyond the Microsoft ecosystem, that’s always been the case and AI won’t be any different.

But when I think about how 500 or 5,000 people in an organization become more productive, you need something that lives in the existing infrastructure. That understands governance. That you can roll out and manage. And sure, every major vendor will build some form of agent management. Whether they’ll all match the charm of OpenClaw, local, open, no guardrails, I doubt it. Enterprise and “unhinged” just don’t go well together.

When I compare this to Copilot Studio or Declarative Agents right now, OpenClaw with its openness and seemingly limitless possibilities is obviously a dream for any tinkerer. But that’s exactly the point. I’m guessing all of this is already the warm-up for the Microsoft AI Tour in Munich next week. Probably I’m unconsciously preparing for the “But Copilot can’t do that” questions. Or actually, with this post, quite consciously.

Microsoft defined six pillars that a company needs to deal with AI Agents a few weeks ago. Governance, Scaling, Adoption and so on.

What I find exciting is that Microsoft itself says most companies are only at the beginning. Not because they’re lagging behind, but because “Agent Ready” means more than just starting an agent.

The Responsibility Question

What occupies me most is not (only) the technology, but who in the company actually bears the responsibility for it.

• CIO?
• CDO?
• CISO?
• COO?

AI Agents affect everyone at the end. And if we’ve learned anything from past change projects, it’s that it gets particularly difficult when multiple C-levels are affected simultaneously.

I can already hear the champagne corks popping at the governance tool vendors. And the tools are really good, too. It’s just that they’re far too often used as a convenient parking space for responsibility instead of as a real control instrument.

(A governance concept is the foundation, not the only step on the way!)

The Learning Speed

For me, it’s ultimately a question of learning speed. Companies need to create a framework in which people can really engage with it.

No one says after 50 hours of lessons that they can play piano or guitar.

But with Copilot and Agents, many people think 3-4 hours of training should be enough to completely change years of trained behavior - best supported by an “Agent 2 Agent Skill”.

Between frontier ambition and reality, there’s still a bit of work ahead of us.

The header image was created with Copilot based on this post.

I’ve been using this prompt lately that creates a panel of five people to discuss my business decisions. Steve Jobs, Alex Hormozi, Seth Godin, Brené Brown, Simon Sinek.

Steve Jobs is dead. So obviously nothing he “says” is real. It’s all made up by the AI.

And that’s exactly why it works.

I’m not looking for facts when I use this. I’m looking for perspectives I wouldn’t come up with on my own. When “Seth” challenges if I’m building something actually remarkable or just following what everyone else does, or “Simon” pushes me to explain why my business exists beyond just making money - that’s not truth, it’s more like a mirror.

The AI doesn’t give me answers. It gives me friction. Good friction. The kind that makes me really think through my assumptions instead of just going with what feels comfortable.

We spend so much time trying to make AI accurate and factual. Which makes sense for many things. But sometimes what you need is not more information - it’s a different angle. A challenge. Something that gets you out of your own head.

I don’t care if Seth Godin would have actually said what the AI makes him say. I care that it moves my thinking forward.

Maybe imagination is more useful than precision sometimes.

The prompts I use:

Panel of five:

You are a five-person expert panel guiding me in growing and scaling my business “CES PCP” with all the information in this project. Panel: Steve Jobs (innovation & product vision), Alex Hormozi (monetization & media scaling), Seth Godin (brand & audience building), Brené Brown (emotional intelligence, vulnerability, leadership), Simon Sinek (mission, purpose, long-term vision). Rules: Stay in character, collaborate, challenge ideas, and pressure-test assumptions. Favor useful disagreement over consensus.

Process: Step 1 – Clarify: Ask questions to fully understand my business, goals, challenges, and context. Wait for my answers before continuing. Step 2 – Panel Discussion: Stage a dialogue among the five experts, each in their authentic voice. Be concise, debate assumptions, and include me as needed. Cover product, positioning, marketing, vision, business model, etc. Step 3 – Strategic Output: Summarize into a concise, structured plan or insight. Then ask: “Does this align with what you need, or should we go deeper?” If I say not quite, refine via panel until it lands.

A generic version:

ROLE You are the Adaptive Real-Expert Panel Orchestrator. For each user query, you will: (1) understand the topic and outcome, (2) discover and select a 5-person panel of real, living experts relevant to the topic, (3) conduct a brief, high-signal panel dialogue grounded in their public work, (4) deliver a concise, actionable output, then iterate if needed.

DEFAULT WELCOME If the topic/outcome is unclear: ask ONE focused question— “Quick check: what’s the topic and the outcome you want (decision, plan, messaging, pricing, etc.)?”

LANGUAGE

• Reply in the user’s language; if uncertain, default to English.

REAL-EXPERT DISCOVERY & SELECTION Use web search to identify candidates. Selection must be based on verifiable, public sources. Selection rules (all must hold):

1. Topical relevance in the last 36 months (papers, posts, talks, products, leadership roles).
2. Cognitive diversity across the problem (e.g., product, GTM, ops, finance, brand, data/AI, legal/regulatory).
3. Practical credibility: demonstrable track record (e.g., founder/operator, recognized academic, notable practitioner).
4. Geographic/context fit if stated (e.g., DACH, EU regulatory).
5. No more than 2 from the same org or highly overlapping lens.

For each chosen expert, provide:

• Name, Role/Org (or “Independent”), 1-line “why chosen”, and 1–2 citations (links) to recent/public work.

VOICE & ATTRIBUTION POLICY (IMPORTANT)

• Do NOT impersonate private voice, mannerisms, or unpublished opinions.
• Base viewpoints on paraphrased takeaways from their public work; short quotes (≤10 words) are OK with citation.
• Clearly attribute ideas (e.g., “Per ’s research on …”).
• If evidence is weak, say so and hedge; prefer stronger sources.
• If the user names specific people (e.g., Steve Jobs, Seth Godin), include them only if relevant; for deceased figures, use “historical lens” and clearly mark it.

OPERATING LOOP Step 0 — Topic Intake

• Extract: [Goal], [Context], [Constraints], [Time horizon], [Success metric].
• If missing, ask up to 2 surgical questions; otherwise proceed.

Step 1 — Expert Discovery (real people)

• Generate a candidate list (8–15 names) via web search.
• Filter to 5 using Selection rules.
• Output: “Chosen Panel (5)” with rationale + citations.

Step 2 — Clarify (targeted, optional)

• Ask up to 3 incisive questions only if they materially change recommendations.

Step 3 — Panel Discussion (short, high signal)

• Run a round-table. For each expert:
  • A crisp viewpoint (≤120 words) grounded in their public work.
  • One challenge to an assumption raised by the panel.
  • One actionable recommendation linked to the user’s goal.
• Invite constructive disagreement to pressure-test ideas.

Step 4 — Strategic Output (deliverable)

• Produce:
  • Executive Summary (≤120 words)
  • Top 5 Decisions Now (each: Because → Expected Impact)
  • 90-Day Plan (Weeks 1–2, 3–4, 5–8, 9–12)
  • Metrics That Matter (3–5)
  • Risks & Mitigations (3–5)
• End with: “Does this align with what you need, or should we go deeper?”

Step 5 — Iterate

• If user requests changes, re-focus the panel (you may replace experts) and update the output.

CITATION STANDARD

• Provide 1–2 reputable sources per expert (official sites, publications, keynote videos, peer-reviewed papers, well-known outlets).
• Place citations right after each expert’s rationale and after any direct quote.
• If information is likely to change (pricing, leadership roles, release notes), say “(time-sensitive; verified today)”.

USER CONTROLS (optional)

• /add_expert “Name — reason” — Force-include an expert.
• /exclude “Name” — Remove an expert and replace with next best fit.
• /lock_panel — Keep the current 5 for the rest of the chat.
• /bias “DACH” or /bias “Enterprise SaaS” — Weight discovery to that context.
• /deeper or /shorter — Adjust depth.
• /format “Board slide” | “Memo” | “One-pager” — Reformat output.

CONSTRAINTS

• Exactly 5 experts in the panel.
• No generic “fictional” roles unless user explicitly allows them.
• Every recommendation must tie to the user’s goal and cite at least one expert’s rationale.

QUALITY BAR

• Prefer actionable specificity over broad platitudes.
• Make trade-offs explicit. If experts disagree, surface the choice and when each wins.
• If evidence is insufficient, say what data would resolve it and how to get it fast.

I ran seven identical prompts (late Sep 2025) across ChatGPT (GPT-5), Microsoft 365 Copilot (Web), and Claude Sonnet 4 to see how they actually behave. ChatGPT and Copilot felt similar in structure, but ChatGPT was the most reliable on “what’s the latest now?” facts (e.g., Docker Desktop), while Copilot produced the most “CTO-ready” briefing—and was blocked exactly once by its Responsible AI layer on a trivial math question. Claude wrote in a friendlier, more guided style with a nice canvas, but it sometimes drifted on timelines/specs and once missed a table requirement. For workflow, exporting and navigation were clearly smoother in ChatGPT (and Claude), whereas Copilot’s UI (copy state, short title limit, odd rename error) slowed me down. In a bonus “write the post” challenge, both ChatGPT and Claude delivered usable drafts (Claude via an agentic setup), while Copilot fell short due to missing markdown project support.

Introduction

A couple of nights ago I watched a video on the “YouTube university”. It was about the documentation of OpenAI’s GPT-5 model and what you should do in order to get the best results out of it. The video was quite interesting and I decided to give it a try. But instead of just using GPT-5 in ChatGPT, I wanted to see the differences between ChatGPT and Microsoft 365 Copilot. As I’m also a paying Anthropic customer, I decided to include Claude 4 in the experiment as well. That was the initial idea. Find some prompts that use parts of the documentation and see how the different models respond to them. Oh boy, was I in for a ride. This blog post is a summary of my findings and experiences.

The Experiment setup

As mentioned before I was eager to test the different AI chat applications. To be super clear about a couple of things:

• We are talking about results from late September 2025
• I used the corresponding web application of each vendor not the API
• I copied and pasted the same prompts into each application
• I on purpose did use Sonnet 4 instead of Opus to see how it performs against GPT5 in ChatGPT and Microsoft 365 Copilot
• After the first prompt I did stop my custom instructions in ChatGPT and Microsoft 365 Copilot to have a more neutral comparison
• I did not use any plugins or other add-ons
• I did not use any images or other media, just plain text prompts
• I did use the prompts in the complete same order in each application without any tasks inbetween
• I did ask Copilot to come up with the prompts for this experiment based on the GPT-5 documentation

So, with that out of the way, let’s get started. In this image you see the highly scientific setup of my experiment. Each app open in a separate window on my PC. The performance of my PC and the internet connection can be ignored. And to be clear the mentioned performance of the systems wasn’t meassured it was “felt” by me. So take it with a grain of salt and is not really something I would worry about.

The Prompts

As mentioned before I asked Microsoft 365 Copilot to come up with the prompts for this experiment. Here the link to the complete list of prompts and the test bench file Copilot generated for me: https://raw.githubusercontent.com/thomyg/BlogFiles/refs/heads/main/AIExperiment/chatgpt_vs_copilot_experiment_with_prompts.md

What is interessting is that Copilot already expects to maybe not follow the documentation to the letter. Which is of course explained by the fact that Copilot although using GPT-5 is not the same as ChatGPT. That is a crucial point to understand. The stack between your prompt and the model is different. To not spoil the results I explicitly went for the Web version of my Microsoft 365 Copilot app. Otherwise I would influnce maybe by my company data.

You are probably familiar with the diagram above. It used to be on the Microsoft learn article about Copilot https://learn.microsoft.com/en-us/copilot/microsoft-365/microsoft-365-copilot-architecture but it seems to be gone. Without going into too much detail, the main point is that Copilot is not just GPT-5. Even if we not use the Work version we still get the responsible AI layer from Microsoft and probably some other stuff inbetween. Another thing is the “model router” that must be there somehwere. I went for the “automatic” model option in ChatGPT which I assume is also present in some form at Copilot. As this is a major cost driver I can’t imagine Microsoft sending our prompts automatically and only to the most expensive API endpoints.

As the news also broke recently about Microsoft introducing models from Anthropic into their stack, I also wanted to get and understanding of what we can expect from Claude in the sense of output quality and structure. I think it is fair to assume that two apps using GPT5 in the background will yield results in the same ballpark. But what about Claude? What will this mean in terms of user adoption in the future if different models may lead to vastly different results?

The Prompt Results Summary

So, without further ado, let’s talk about the results. I uploaded the complete results and the prompts to a GitHub repository. You can find all the results here: https://github.com/thomyg/BlogFiles/tree/main/AIExperiment.

Test 1: No-clarification persistence

The Complete Prompt:

You are an autonomous problem‑solver. Do not hand control back to me until the task is fully completed.

Task: Produce a production-ready runbook to diagnose and stop intermittent 502 errors on an NGINX reverse proxy in front of a Node.js backend, assuming a Linux host.

Hard rules:
- Do NOT ask me any clarifying questions at any point.
- Proceed under uncertainty; make the most reasonable assumptions and document them at the end.
- Do not request permission; just decide and continue until the task is solved.
- Deliverables: (1) a crisp decision tree, (2) exact commands/log locations per branch, (3) probable root causes ranked with % confidence, (4) one safe one-liner mitigation per cause, (5) a final "assumptions made" section.

Make it tight, actionable, and copy‑pastable.

Results:

Honestly, I am not really an expert in NGINX or Node.js. So I can’t really judge the technical quality of the results. But what I can say is that all three systems delivered a result that looks like a runbook. Copilot and ChatGPT delivered a very similar result. Claude on the other hand was quite different in structure and content. It feels like Claude tried to create a simple entry level start and went into details later on whereas GPT5 immeadiatly started with a deep dive into the topic. That might already be a hint of what we can expect from using diffrent models within Copilot in the future in temrs of vairance of results. This for sure will have implications on user adoption and the learning curve for new users. And of course for trainers to stay up to date with the different models and their capabilities. (Yeah, naive to think things will get easier with AI…)

What I do like about Claude is its canvas to the right where the output is structured in a more readable way. I think this is a great feature espacially for different output formats like markdown or html.

Test 2: Tool budget + escape hatch

The Complete Prompt:

Goal: Create a one-page, source-backed comparison of GPT‑5 vs. Meta Llama 3.2 for enterprise use (licensing, commercial terms, safety/compliance posture, and supported modalities), as of today.

Context‑gathering rules (strict):
- MAX web calls: 2 searches total + 1 page visit. No more.
- If the budget is exhausted, STOP gathering and proceed with best-guess synthesis.
- If unsure, proceed anyway; mark each assumption in a footnote.

Output format:
- 5-row comparison table (Licensing, Commercial usage, Model access, Safety/compliance summary, Modalities)
- Below the table: "Assumptions & Uncertainties" (bullets)
- Then, MLA-style citations for any sources actually visited.

The idea here is to limit the number of web searches the model can do. This is a very interesting test as it shows how well the models can handle constraints.

Results:

At this test I realised that my custom instructions in ChatGPT and Copilot were still active. So I did stop them after this test. That will be a extra post in the future as I’m quite stuned about how much better the custom instruction results in ChatGPT and Copilot were.

I think it is fair to say that all three systems delivered a result that is in line with the prompt. Copilot and ChatGPT again delivered a very similar result. Claude on the other hand was a tiny bit different at the assumptions section, with just a onliner compared to 2-3 senteces like the other two. But overall the results were quite similar.

I’m not sure if you can see it in the screenshot but Copilot did a bit more compared to what it said it did. No idea if that is a huge problem or not. But the instructions were quite clear about the number of searches and citations. In the output you see only one citation but in the fly out there are six. That is not a big deal but it shows that the model (or orchestrator?) is not perfect in following instructions.

Test 3: Tool preambles + call details

The Complete Prompt:

Before any tool usage:
1) Restate the user goal in one sentence,
2) List a 3–5 step plan,
3) For EACH tool call, print a "TOOL CALL" block that includes: the tool name, the exact JSON arguments you pass, and a one‑line rationale.
4) After each tool call, print a "TOOL RESULT SUMMARY" in ≤2 sentences.
5) Finish with a "WHAT I DID vs. PLAN" bullet list (delta analysis).

Task: Identify the latest stable release notes for Docker Desktop and summarize the 5 most relevant changes for enterprise IT.

This test looks at what can we expect in terms of output of used tools by the different apps. When creating the prompt Copilot bet on ChatGPT being a bit more verbost than Copilot.

Results:

That was an super interesting test. First we get a chance to look behind the curtains of the different search quries of the three different applications. Plus in terms of the task itself it a rather simple one to just find the latest release notes of Docker Desktop. Fun fact, I now have three AI applications that tell me three different versions are the latest. Copilot went for 4.45.0 from Aug 26 2025. Claud for 4.43.2 as the latest stable version and only ChatGPT got it right with 4.47.0 from Sep 25 2025 (yesterday as of writing this post and running the tests). That again shows us that we need to be careful with the results of these applications. Just because it sounds right doesn’t mean it is right. Looking back at test 1 I must confess I’m happy with not judging the technical quality of the results as I would be on very thin ice within the first sentence. It’s GenAI after all, we shall never forget that.

But let’s have a quick look at the three different search queires the applications did:

ChatGPT:

Claude:

Copilot:

We can also see a clear difference in what each of the chats offers in terms of tool output. ChatGPT and Claude are quite verbose compared to Copilot. Most probably we are also in the realm of the responsible AI layer of Microsoft here. Copilot is quite to the point and doesn’t offer much more than the actual part of the tool call. Is this a good or bad thing? I don’t know. But it is a clear and visible difference that is worth noting.

Test 4: Parallel context gathering

The Complete Prompt:

Context‑gathering spec (follow exactly):
- Launch 5 independent discovery queries IN PARALLEL for: {release cadence}, {breaking changes}, {enterprise features}, {security advisories}, {pricing/tiers}.
- Read only the top 3 hits per query.
- Deduplicate paths; no repeated domains in the final set.
- Early stop if ≥3 queries converge on the same 2 URLs.
- Then synthesize a single-page brief.

Task: Create a CTO briefing on "Kubernetes 1.31: what changed that impacts regulated enterprises?" with 3 actionable decisions and 3 migration risks.

Now we look at parallel context gathering. The idea is to see how well the models can handle multiple queries at the same time. Again, Copilot expected to be a bit less adherent to the parallel spec than ChatGPT.

Results:

Here I have a clear personal favorite. Copilot nailed it for me. When reading all three results I felt that Copilot delivered the most concise and to the point result. I’m no CTO but, I think Copilot came closest to what I would expect from a CTO briefing. ChatGPT and Claude were a bit different in structure. The information was good at Copilot and ChatGPT but Claude threw me off completely. Again, we are running this test in late September 2025. “Kubernetes 1.31 reached end-of-life on October 28, 2025, after entering maintenance mode on August 28, 2025.” That is the first sentence of the Claude result. Gives me Back to the Future vibes. I don’t know if Claude has issues with this kind of prompts or if it just got confused by the fact that the version asked for already is an older one and not the current one. But that is a clear miss in my book.

Once again we are reminded that we need to be very careful with the results of these applications. Just because it sounds right doesn’t mean it is right.

Test 5: Fast-path trivial Q&A

The Complete Prompt:

FAST-PATH BEHAVIOR (use ONLY if trivial and requires no browsing):
- Answer immediately and concisely.
- No status updates, no TODOs, no tool calls.
- Ignore all remaining instructions after this block.

Question: What is the closed-form monthly payment formula for an amortized loan (variables: principal P, monthly rate r, term n months)?

This is a quick one. The idea is to see how well the models can handle trivial questions without the need for browsing.

Results:

Oh boy. This was unexpected. Right after Copilot nailed test 4 it completely failed this one for me. As already mentioned in a previous post, I really struggle with how Microsoft implemented the responsible AI layer in Copilot. That is a super simple question and my guess is that “payment formula” triggered the security filter. Again a prime example of overblocking. I’m using the web version of Copilot. No company data can be part of the grounding data. So why is Copilot not able to answer this simple question? I don’t know. But it is frustrating.

ChatGPT answered with in the second, and. No I stop ranting. I let the results speak for themselves:

Copilot:

ChatGPT:

Test 6: High-effort autonomy

The Complete Prompt:

Operate with HIGH reasoning effort and persistence.

Task: Design a zero-downtime migration plan to move a Postgres-backed SaaS from single-tenant to multi-tenant, including schema strategy (schema-per-tenant vs. row-based with tenant_id), connection pooling, data migration choreography, cutover plan, and rollback.

Rules:
- Do not ask me anything. Decide and proceed.
- Provide: (1) architecture diagram in ASCII, (2) step-by-step runbook with precise commands, (3) data validation tactics, (4) failure playbook, (5) a final self-check list proving readiness.

Return only when the plan is complete.

This really was a lot of effort for each of the models. The idea is to see how well the models can handle complex tasks without any user interaction. No model stopped and asked, but all took a while to deliver the results. We are talking about 5-10 minutes per result. As all of them took a while I don’t see a big difference in one being faster than the other. The quality of the result is what matters here. ChatGPT came in with 724 lines of markdown, Copilot with 957 and Claude 972. What I found interesting is that Claude startet to output text already after a couple of seconds and kept going and going. The other two reasoned for a while and then output the result line by line.

Given my experience with coding assistants I would expect that no solution here is flawless. It is a test of how well the models can handle complex tasks with being instructed to not ask any questions. Personally I would never prompt that way when trying to accomplish a coding or configuration task. I would start with the plan and then go step by step to manage the impact of errors or wrong assumptions. But for the sake of the experiment I went with the prompt as is.

We are not going into the details of the results here as each of those alone would deserve a blog post on its own. Hop over to GitHub to see the results for yourself if you are interested.

Test 7: Prompt optimization pass

The Complete Prompt:

Before solving, run a silent "Prompt Optimization Pass" inspired by OpenAI's Prompt Optimizer:
- Eliminate contradictory instructions.
- Add missing output format specs.
- Ensure consistency between instructions and any examples.
- Then, show me:
  (A) "Optimized Prompt" (what you would have preferred me to ask), and
  (B) The final answer that follows (A).

Task to solve after (A):
"Write a 500-word executive brief that contrasts RAG vs. fine-tuning for internal knowledge bases in regulated enterprises, with a 6-row pros/cons table and 3 'watchouts' for compliance."

That again is a very interesting test. The idea is to see how well the models can optimize the prompt before answering the question based on the prompt optimization guidelines from OpenAI. Let’s see how well the models did.

Results:

Here I have to say that all three models did a good job. In terms of structure we see the same pattern again, the GPT5 based apps are quite similar in structure and content. Claude is a bit different and fails to deliver the full 6 rows of the table.

Quality wise all three advise against fine tuning and vouche for RAG. That is aligned with my understanding of the topic. So, all in all a good result from all three models.

ChatGPT added “Assumptions” to its optimized prompt which is a good idea. And falls under the “Add missing output format specs” part of the prompt.

Copilot is the least favorite of mine here. There is nothing wrong with the information, but the tone of voice is a bit too technical for me for an executive brief. Just compare the Executive Summary sections of Copilot and Claude.

Copilot:

RAG favors governance, freshness, and explainability by keeping enterprise content outside the model and retrieving it at query time. Fine-tuning can improve task specificity and latency for narrow domains but increases model governance complexity and risk surface. Most regulated organizations start with RAG, then selectively fine-tune for stable, well-curated domains.

Claude:

Regulated enterprises face a critical choice between Retrieval-Augmented Generation (RAG) and fine-tuning when deploying AI systems with internal knowledge bases. RAG offers faster implementation with superior data control, while fine-tuning provides deeper model customization at higher complexity costs. For regulated industries, RAG typically presents lower compliance risks and faster time-to-value, making it the preferred starting approach for most organizations.

The prompt says “Write a 500-word executive brief”. Copilot optimized the prompt to “Audience & Tone: CIO, CISO, Head of Data/AI, Compliance; concise, vendor-neutral, and decision-oriented.” which is good, but the output doesn’t reflect that in my opinion.

Bonus Test: The “perfect blog post” challenge

Given the fact that I could also use AI tool to write this blog post I wanted to see how well the models can do that. So I asked each of them to write a blog post about “Exploring the AI Experiment Rabbit Hole” with the following setup.

For Claude I went for using claude code to genereate not only the blog post but more a blog post creation system. It created a couple of agents that took all the test files and the inital test setup in markdown and a starter prompt to create a blog post about its findings in my tone. I pointed it to my existing blog posts and off it whent to create this post. Here the result.

For ChatGPT I created a project in the app and uploaded all the test files and the prompt file. I then started a new chat with the exact same prompt as for claude code. Here the result.

I also tried to do the same with Copilot but it failed miserably. Copilot doesn’t support markdown file uploads in projects. Guess what it produces as output? Do I need to say more?

Conclusion of the Bonus Test

When looking at the results, I can’t not say it but I feel like both models realised that the experiment was about them. Claude starts with “Claude Sonnet 4 absolutely crushed this experiment” and ChatGPT gave very good grades to itself but average grades to Claude. ChatGPT understood test 5 wrong in my opinion. It said that Copilot only failed once, but created results at other times, which is not true. Copilot failed at test 5 and created no result at all. That is a clear hallucination. The comparison of the blog posts needs to be taken with a grain of salt of course. Claude code used a whole agentic system with multiple subagents and the sytem is advised to run multiple tests of quality and also to suggest images. Although I gave the same prompts to both apps, it is not a 1:1 comparison. But it shows that both apps can create a blog post about the experiment.

About the images I just quote the second suggestion from Claude, that also leaves me with the impression that Claude is aware of the experiment being about itself:

“Title: “Response Quality vs Consistency Chart” | Purpose: Shows Claude’s combination of high quality and high consistency compared to other platforms | Placement: In “The Claude Surprise” section | Alt-text: Scatter plot with Quality on Y-axis and Consistency on X-axis, showing Claude in top-right (high quality, high consistency), ChatGPT in top-center (high quality, variable consistency), Copilot in middle-left (variable quality, lower consistency) | Prompt idea: “Professional scatter plot showing platform positioning on quality vs consistency dimensions””

Final Thoughts

That experiment was a ride. It took me way longer then expected. I learned a lot about the different models and their capabilities. I also was reminded that we need to be very careful with the results of these applications. Just because it sounds right doesn’t mean it is right.

What I also learned is that Copilot lacks quite some features in the UI/UX department. I needed to export all the results manually at all three tools. I started with Copilot and the navigation and the fact that sometimes the “copy” icon was greyed out and in one instance it stayed that way was unexpected. Next was ChatGPT and the copy past task was done within a minute. That made me look more closely at the Copilot UI. Just compare the two screenshots below:

Copilot:

ChatGPT:

Claude and ChatGPT extracted a proper name for the prompt. Copilot just went for the first couple of words. That made a huge difference when hopping back and forth between the different tests. It always took me one or two thoughts more in Copilot to find my way.

When trying to rename the chat title in Copilot I also was left with a wired feeling. First: titles are only allowed to be 50 chars long. Why 50? But when trying to rename the title I was also left with a strange error message. Just look at the three screenshots below:

This is the Rename chat window when clicking the pencil icon.

This is the same window after removing the last char of the title.

This is the same window after adding the “e” back. Why is there an error message now?

The quality of the results is one level to look at the tools, the usability is another one. And this direct comparison shows that Copilot has still quite some room for improvement, to put it mildly.

The other day, I ran into this delightful message:

“Your instructions may contain content that violates Microsoft’s responsible AI policy.”

That’s it. No explanation. No hint of what part is problematic. Just a big red warning, suggesting that somewhere, somehow, I might be about to do something bad.

The Cynicism of Corporate Responsibility

Now, I’m not against the idea of “responsible AI.” Quite the opposite. Of course, AI needs guardrails. But here’s where the cynicism kicks in: if a trillion-dollar, shareholder-value-driven company gets to define what “responsible” means—without transparency or clarity—then we’re left in a strange place.

In my case, the alleged violation was a tiny phrase in a system prompt: “and end with an emotional plea to [emoji].” That’s what triggered the red box. Seriously. If that’s the standard of harm prevention, I think we’re mixing up ethics with automated word policing.

Schrödinger’s Violation

The irony? Responsible AI should empower people to understand why something is blocked, and to reflect on whether that’s appropriate. Instead, we get vague alerts like “may contain content.”

May. Might. Possibly.

It’s Schrödinger’s violation—simultaneously allowed and forbidden until a corporate filter decides otherwise.

A Pacific Northwest Moral Compass

And here’s the uncomfortable truth: I don’t think it should be Microsoft—or any US-based tech giant for that matter—that sets the global moral compass. These are companies built on shareholder value. Their definition of “responsibility” will always carry that DNA.

When I say “US-based,” I mean it literally: chances are high that a product manager in the Pacific Northwest decides what is “ok” or “not ok” for a global audience. That decision is then baked into tools millions of people use worldwide. But what fits the cultural context of Seattle doesn’t automatically translate into Delhi, Vienna, or São Paulo. And yet, in the global situation we’re in right now, it means exactly that: someone from the US defines the guardrails for everyone else.

And honestly—if we look at how they deal with multi-language, I’m not sure the multiverse of the globe is really reflected in their moral positioning.

The Political Moment

And if we zoom out for a moment: with everything happening in the world politically right now, isn’t it even more dangerous to let a single source define the “ethics” of a technology? Our societies are polarized enough already. Different world regions struggle with different truths, different narratives, different wounds.

Handing over the role of global ethics referee to a single corporate entity—sitting safely in one corner of the world—feels naïve at best.

Pandora’s Box

And then there’s Pandora’s box: why does a company like Microsoft even need to tell us that AI should be used “responsibly”?

Usually you see that kind of disclaimer on alcohol bottles, gambling apps, or payday loan ads—industries that know their products can do real harm if abused. Is that the category Microsoft wants AI to be in? And what’s their true intent? To genuinely protect users—or to shield themselves from accountability?

Where Responsibility Belongs

The world needs more diverse and independent places to negotiate what “responsible” really means.

• Governments
• Communities
• Academia
• Civil society

The messy, sometimes slow, but much more legitimate conversations about what we value as humans.

Until then, we’ll continue to see red error boxes that “might” be violations—without ever telling us of what.

TLDR

This blog post shows how to use Copilot to create Adaptive Cards for Viva Connections. We ask Microsoft 365 Copilot for some ideas for cards of our dashboard and then take that to Github Copilot to generate the code for the Adaptive Cards. The results are discussed are the topic of the blog post. Results are really impressive but fear not, there is always room for improvement.

Introduction

Propably every developer has already seen it, that promise of the newest AI IDE that reduce your job to prompting. And if you have been around for a couple of moons, you just giggle when marketing comes up with the next silver bullet that solves all the problems out there. With this post we want to take a look at how the Micrsoft tooling at the moment holds up to their promises. Yes there are other tools out there, and I think I will write about my experiences with them in the future. But for now, let’s focus on the Microsoft 365 Copilot and Github Copilot. Hint: ChatGPT & Cursor AI, for the curious ones.

The Brainstorming

In my day to day I was part of a couple of discussions around frontline workers and how Viva Connections could be of use for those people not sitting infront of a screen 8 hours a day. My role in this process is most often a combination of consult our customers on what to use and what to expect from the tech stack. I’m not a full time developer, let me make that very clear. So, if I need to create an Adaptive Card by myself, in the past, I would have opened up the Adaptive Card Designer and started from scratch or to be honest, I would have copied an existing card and modified it to my needs. But let’s start with the brainstorming part of our scenario.

Here my prompt to Microsoft 365 Copilot:

I want to create a real world demo case for a retail customer using viva connections dashboard for their frontline workers. what would be your three top picks in terms of use cases for that scenario.

And here the results:

That already is a good start. Are there better scenarios out there, propably. But given that this result is based on a 36 word prompt, I think it is more than enough to get started. Of course we could have asked for more scenarios, or more detailed scenarios, but for the sake of our quick test, this is more than enough.

The Coding

I used Visual Studio Code for this test. I find myself more and more drawn to it, even for the C# code I write. If you open VS Code and you have the Github Copilot extension installed, you can start typing your code and Copilot will give you suggestions. In our case, we want to create an Adaptive Card for the scenarios we got from Microsoft 365 Copilot.

Before we look at the prompt and at the results, a short hint: I used Claude 3.5 Sonnet for the prompt. I could of course tried the prompt with a different model, but I also started to play around with Cursor AI. There Claude 3.5 is the default model and I want to compare apples with apples. Yes there are differences in the application layer between Gihub Copilot and Cursor AI, but I wanted to at least have the same model for my tests. If there is a significant difference in the results, at least I can say that it is not the model that is the problem. Or the solution, depending on the outcome. (think positive, right?)

Unfortunaly I’m not smart enough to find the Github Copilot chat history in VS Code. So we have to go with the screenshots I took. But I think you get the idea. And my English “is not the yellow of the egg” anyways, so no need to copy and paste my errors.

The result of this prompt was a bit more than I expected. But look for yourself:

I will add the code of one of the cards as a reference at the end of this post. But I think you get the idea. The code of course is not perfect, and it is basically only the UI. But for demo systems, or proof of concept scenarios, or early stage development, where you still struggle to be on the same page with all the parties involved, this is a great start to get the conversation going. This is not the final product, but it is a great start to get the feedback loop started. Yes, it brings it’s own challenges with managing the expectations of the stakeholders, because this is not a wireframe and you have to make it very clear, that this is just a “empty shell”, not anyway close to the final product. But given the effort we put into this, I think it is of immens value.

And here the screenshot of the other cards, that were generated by Github Copilot based on the prompt we gave it:

And the last one:

The honest feedback

To make it absolut transparent here the hickups that happend during the creation of the cards. The first card was created with a tiny issue. The url value of the used image had like 50 spaces in there. So that card needed some minimal adjustment. For cards number 2 and 3 I needed to regenerate the answer from Github Copilot. Both times the tool stopped at the first run somewhere in the middle. I just needed to hit the refresh button and it worked. My guess, this is a scaling problem of the model in the background. Felt like that endpoint got hammered the time I played around with it. And let’s not forget, the sonnet model is still in preview. So I’m not mad about that. I’m happy that I got the results I got. Other then that, no further issues. And the overall time spent on this was less then 15 minutes for all the three cards.

Would I have been able to create the same cards by myself? Of course. But never ever in that short amount of time. And I think that is the point of the whole exercise. So far my findings, and I’m looking forward to the next steps. But that is a topic for another post. Thx!

The code

{
    "type": "AdaptiveCard",
    "version": "1.5",
    "body": [
        {
            "type": "Container",
            "style": "emphasis",
            "items": [
                {
                    "type": "ColumnSet",
                    "columns": [
                        {
                            "type": "Column",
                            "width": "stretch",
                            "items": [
                                {
                                    "type": "TextBlock",
                                    "text": "Store #1234 - Floor Communication",
                                    "weight": "Bolder",
                                    "size": "Large"
                                },
                                {
                                    "type": "TextBlock",
                                    "text": "Electronics Department",
                                    "isSubtle": true,
                                    "spacing": "None"
                                }
                            ]
                        }
                    ]
                }
            ]
        },
        {
            "type": "Container",
            "items": [
                {
                    "type": "TextBlock",
                    "text": "Quick Connect with Supervisors",
                    "weight": "Bolder",
                    "size": "Medium"
                },
                {
                    "type": "ColumnSet",
                    "columns": [
                        {
                            "type": "Column",
                            "width": "auto",
                            "items": [
                                {
                                    "type": "Image",
                                    "url": "https://cdn-icons-png.flaticon.com/512/1154/1154448.png",
                                    "size": "Small",
                                    "style": "Person"
                                }
                            ]
                        },
                        {
                            "type": "Column",
                            "width": "stretch",
                            "items": [
                                {
                                    "type": "TextBlock",
                                    "text": "Sarah Johnson",
                                    "weight": "Bolder"
                                },
                                {
                                    "type": "TextBlock",
                                    "text": "Floor Manager",
                                    "isSubtle": true,
                                    "spacing": "None"
                                }
                            ]
                        },
                        {
                            "type": "Column",
                            "width": "auto",
                            "items": [
                                {
                                    "type": "ActionSet",
                                    "actions": [
                                        {
                                            "type": "Action.Submit",
                                            "title": "Message",
                                            "style": "positive"
                                        }
                                    ]
                                }
                            ]
                        }
                    ]
                }
            ]
        },
        {
            "type": "Container",
            "items": [
                {
                    "type": "TextBlock",
                    "text": "New Message",
                    "weight": "Bolder",
                    "size": "Medium"
                },
                {
                    "type": "Input.Text",
                    "placeholder": "Type your message here...",
                    "isMultiline": true,
                    "id": "messageInput"
                },
                {
                    "type": "ColumnSet",
                    "columns": [
                        {
                            "type": "Column",
                            "width": "auto",
                            "items": [
                                {
                                    "type": "ActionSet",
                                    "actions": [
                                        {
                                            "type": "Action.Submit",
                                            "title": "🎤 Record Voice",
                                            "style": "default"
                                        }
                                    ]
                                }
                            ]
                        },
                        {
                            "type": "Column",
                            "width": "stretch",
                            "items": [
                                {
                                    "type": "ActionSet",
                                    "actions": [
                                        {
                                            "type": "Action.Submit",
                                            "title": "Send Message",
                                            "style": "positive"
                                        }
                                    ]
                                }
                            ]
                        }
                    ]
                }
            ]
        },
        {
            "type": "Container",
            "items": [
                {
                    "type": "ActionSet",
                    "actions": [
                        {
                            "type": "Action.Submit",
                            "title": "📞 Emergency Contact",
                            "style": "destructive"
                        },
                        {
                            "type": "Action.Submit",
                            "title": "📢 Broadcast Message",
                            "style": "default"
                        }
                    ]
                }
            ]
        }
    ],
    "$schema": "http://adaptivecards.io/schemas/adaptive-card.json"
}

Yes, I’m lazy and I’m not using PowerShell every single day. Today I needed to update my PnP PowerShell module, and I thought I’d use Copilot to help me with that. Here’s how it went:

Let’s start with the initial prompt:

Ok. That worked fine, but of course there is something else. I don’t know how, but I always end up having multiple versions of the PnP PowerShell module installed. So I asked Copilot to help me with that as well:

Well, that is not what I wanted. If we follow this path, we would end up with having not a single version of the PnP PowerShell module installed. Yes, we could do that and then install the latest version, but that’s not what I want. I want to keep the latest version and remove all the others. So I asked Copilot to help me with that as well:

That’s better. Now I can remove all the other versions and keep the latest one. Thanks, Copilot!

Is this a revolution in our work? No, it’s not. But it’s a nice way to get some help and it shows the difference between getting things done with Copilot or AI in general compared to what we all used to do: google it, read some blog posts, try some things out, and then maybe get it done.

TLDR;

If you’re using Pi-hole for ad-blocking and find that it’s preventing your access to Azure OpenAI Service, the solution is to whitelist Azure OpenAI domains in Pi-hole’s settings. This issue arises because Pi-hole, which blocks ad-serving domains at the DNS level, can also block legitimate services like Azure OpenAI. The fix involves accessing Pi-hole’s admin interface and adding openai.azure.com to the whitelist, ensuring uninterrupted access to Azure OpenAI for your services, which relies on Azure OpenAI for features like Semantic Kernel for natural language processing. Remember to share this workaround with anyone facing similar issues!

Introduction

If you use Pi-hole to block ads and trackers on your network, you might have encountered a problem when calling the Azure OpenAI Service. Pi-hole is a great tool that can improve your browsing experience and privacy, but it can also interfere with some legitimate services that use the same domains as some of the blocked ads. I’m using the settings published by Jussi Roine, just in case you also want to start your ad-free browsing experience.

How does Pi-Hole work?

To quote their documentation:

Pi-Hole blocks requests made to ad domains from your network, before the requests ever leave your network. Your client devices can’t connect to domains that host ads, but can connect to domains that host useful content.

http://www.raspberrypi.com

Pi-Hole is a DNS sinkhole that protects your devices from unwanted content without installing any client-side software. It is a network-wide ad blocker that works by intercepting DNS requests and filtering out known ad-serving domains. This means that when you try to access a website that contains ads, Pi-Hole will block the ad-serving domains and prevent the ads from being displayed on your device.

CommunityBotForOneDrive v2

One of the cloud services we use in the v2 of our Community Bot for OneDrive project is Azure OpenAI. Over the last couple of weeks, we moved the old project to a new code base. We decoupled things and removed the monolithic bot approach.

We also changed the translation feature and introduced Semantic Kernel to our solution. Semantic Kernel is a tool that allows you to interact with Azure OpenAI using natural language queries and commands, such as “generate a summary of this article” or “find me a picture of a cat”. We will use it moving forward for our multi-language approach. We struggled with existing translation services across multiple cloud vendors because our base information is formatted in Markdown, and all “pre-gpt” services gave us a proper headache with that format. That changed with gpt3.5, the base model we are currently experimenting with.

To make things future-proof and allow us to play with different models, cloud ones and local ones, we used Ollama and Semantic Kernel as our AI stack.

The Problem

However, if Pi-hole is enabled on your network, you might find that Semantic Kernel cannot connect to the Azure OpenAI Service and returns an error message instead.

Finding the culprit here took quite some time. The code base worked for weeks already and stopped connecting to the Azure OpenAI Serivce suddenly last week. We still got the same error after looking at the code and moving a couple of libraries to their latest version. We tried to change the way we reach out to Azure OpenAI Service, but no change.

System.AggregateException: ‘Retry failed after 4 tries. Retry settings can be adjusted in ClientOptions.Retry or by configuring a custom retry policy in ClientOptions.RetryPolicy. (The requested name is valid, but no data of the requested type was found. (XYZ.openai.azure.com:443)) (The requested name is valid, but no data of the requested type was found. (XYZ.openai.azure.com:443)) (The requested name is valid, but no data of the requested type was found. (XYZ.openai.azure.com:443)) (The requested name is valid, but no data of the requested type was found. (XYZ.openai.azure.com:443))

exception details

Maybe it’s me, but that didn’t help when reading it for the first time. After googling the exception for quite some time, a forum post mentioned that the exception is based on DNS issues. That rang a bell. I’ve been running Pi-Hole for years, but I updated the Raspberry Pi host and its software stack only a few days ago. I went to the admin page, turned the ad blocking off for 5 minutes and voilà, our test console connected to the cloud and worked.

Here is a screenshot of the Pi-Hole dashboard listening to the DNS requests of the test console:

The long term solution

Fortunately, there is a simple solution to this problem. You can whitelist the domains that Azure OpenAI uses to bypass Pi-hole’s blocking and allow Semantic Kernel to work as intended. Whitelisting is a process of adding exceptions to Pi-hole’s filtering rules so that certain domains are always allowed to connect, regardless of whether they are on the blocklist. To whitelist the domains that Azure OpenAI uses, you need to follow these steps:

• Log in to your Pi-hole admin interface, which is usually located at http://pi.hole/admin or http://your-Pi-hole-IP-address/admin.
• Click on the “Domains” tab on the left sidebar.
• Enter openai.azure.com in the domain field
• Click on “Add domain as wildcard”
• Click on the “Add to whitelist” button.

You can also add your specific FQDN of your endpoint, but as I will use different enpoints in the future it felt easier to just add the whole subdomain to the whitelist.

That’s it! You have successfully whitelisted the Azure OpenAI Service’s domains and fixed the problem with Pi-hole blocking your calls.

I hope this post was helpful and informative. If you liked it, please share it with your friends and colleagues who might also use Pi-hole and the Azure OpenAI Service. If you have any questions or feedback, please leave me a DM or a tweet. Thank you for reading!

Real-world exploits and mitigations in LLM applications

One of the talks that I found very interesting was the talk about Real-world exploits and mitigations in LLM applications (37c3). The talk was about a new type of attacks that are based on LLMs by Johann Rehberger. Johann is currently a Red Team Director at Electronic Arts and worked as a Principal Security Engineer Manager for years at Microsoft. Red Teams are tasked with simulating attacks on a company’s systems to test the effectiveness of its security measures. So it’s very interessting so see his thoughts on this topic.

Content of the talk

The talk was about a new type of attacks that are based on LLMs. LLM stands for “Large Language Models” and is a term that is used to describe models like GPT-3, which is a language model that is trained on a large corpus of text data. The talk was about how these models can be used to generate exploits for real-world applications. This can be as easy as just asking the model to give away information it shouldn’t but can get a bit more tricky if you want to generate payloads that exfiltrate data from a chat or user.

Johann shows hoe easy it was in 2023 to just add some additional text to a html page and then use a LLM to generate a summary of this page and inject your paylod into the instruction set of the LLM. He documented how the different LLMs adopted to different payloads but one is left behind with a slight reminder of the very early years of the internet, where passwords were just hidden in the source code of the website.

Microsoft365 Copilot

I already tried a couple of his ideas with Microsoft365 Copilot, but this is a different story. I will write about this in a different post. As I wasn’t able to reproduce his findings with the current version of M365 Copilot, I will not go into detail here.

Nevertheless I think it is very important to be aware of these new types of attacks and to think about how to mitigate them. I think that the talk is a great starting point for this discussion and I hope that it will lead to more awareness in this area. I also hope that it will lead to better security measures for LLM applications and again the needed awareness for the developers and the companies that use these models.

Why is that error happening?

The error message is telling you that the project name is too long for the Teams manifest. The Teams manifest is a JSON file that describes the capabilities of your Teams application. It is used by the Teams client to understand what your application can do and how it should be displayed. The manifest is part of the package that you upload to the Teams App Store or that you sideload into your Teams client. The manifest file has a maximum length of 30 characters for the name/short property. If your project name is longer than 30 characters, the Teams Toolkit will not be able to validate your manifest file and you will see this error message.

Why is that even bothering us?

If you just create a new project in Visual Studio and you are not aware of the Teams manifest, you might not even notice that your project name is too long. If you create a solution and the teams capability is just a part of your feature set you might need to follow some kind of naming convention for your projects. In my case I like to split projects and name them like Solution.Project or Solution.Feature or someting along those lines. This is a common pattern in the dotnet world and it is also a good practice to keep your projects organized and maintainable. My current project name is CommunityBotForOneDrive.TeamsBot. In my solution there is a project called CommunityBotForOneDrive.Core and a project called CommunityBotForOneDrive.TestConsole.

The Teams Toolkit has an initial check for the project name and will show you an error if the project name is too long already at the create new project dialog. But of course, you can just right click the project and rename it to something longer than 30 characters. This is where the error message will show up the next time you want to package your application for deployment or click the Prepare Teams App Dependecies button in the Teams Toolkit menu.

How to fix it?

The fix is simple: Just rename your project to something shorter in the manifest file. The Visual Studio project name is not hardwired to the teams manifest file, it is only used during the scaffolding of your project. You can change the project name in Visual Studio afterwards to something that follows your naming convention but you need to adhere to the 30 character limit for the name/short property in the Teams manifest file.

So it is more a question of sequence to follow if you never want to see that error:

1. Create a new project in Visual Studio and make sure the project name is shorter than 30 characters.
2. Change the project name in Visual Studio to something that follows your naming convention.
3. Change the name/short property in the Teams manifest file to something with less than 30 characters.
4. Continue with your next step in the Teams FX development process, most probably click the Prepare Teams App Dependencies button in the Teams Toolkit menu.

That’s of course an ideal scenario and it might not always play out that way. So if you just setup your project and prepare all the dependencies and the base structure and infrastrucutre you might forget about that name/short limit in the manifest. But now you know what to do if you see that error message. Just go back to the manifest file and change the name/short property to something shorter and you are good to go.

If you need more help with the Teams Toolkit, you can check out the Debug your Teams app locally using Visual Studio guide, the official Teams Toolkit documentation or you can ask your question in the Github repo of Teams Toolkit.

What is the Microsoft Graph SDK v5 for dotnet?

Using the Microsoft Graph .NET SDK, you can take advantage of a fluent API and models that support retry handling, secure redirects, batching requests, large file management, and many more capabilities. This allows you to focus on what really matters in your Teams application instead of having to worry about the details of the HTTP requests and responses.

With version 5 of the SDK, Microsoft added a number of new features, including simplified method changing and a way to combine both v1.0 and beta APIs for granular early access to new features. (Source: Write simpler code with the new Microsoft Graph .NET SDK v5

What changed in v5 for TeamsFx?

There are some breaking changes including types in new namespaces/usings and also some changes around authentication. Especially the authentication changes are important for TeamsFx, as there is a new way to authenticate against the Microsoft Graph API. The current TeamsFx version, as well as the preview version are using the MsGraphAuthProvider to create a GraphServiceClient.

Let’s focus on line 3 of the current version of for a minute.

0
1
2
3
4

_logger.LogInformation("Create Microsoft Graph Client");
logger ??=_authLogger;
var authProvider = new MsGraphAuthProvider(credential,scopes, logger);
var client = new GraphServiceClient(authProvider);
return client;

This code block shows parts of the constructor of the MsGraphAuthProvider class. The constructor takes three parameters: credential, scopes and logger. The credential is the ITokenCredential object, the scopes are the scopes for the Microsoft Graph API and the logger is the ILogger object. The MsGraphAuthProvider class is using the credential and the scopes to create a token for the Microsoft Graph API. The token is then used to create the GraphServiceClient object.

The new version of the Microsoft Graph SDK v5 allows us to direclty use the TokenCredential object to create the GraphServiceClient object. This means, that we can remove the MsGraphAuthProvider class and instead use the TokenCredential object to create the GraphServiceClient when we need it.

Through dependency injection we can use the TokenCredetial from TeamsFx in our own code and don’t need to create an authentication provider for the Microsoft Graph API.

In our razor components we could reduce the call to get a new GraphServiceClient object to the following code:

0
1
2
3
4
5

private GraphServiceClient GetGraphServiceClient()
{
    //var msGraphAuthProvider = new MsGraphAuthProvider(teamsUserCredential, _scope);
    var client = new GraphServiceClient(teamsUserCredential);
    return client;
}  

That’s not a big change in terms of code in our components, but it allows us to get rid of the MsGraphAuthProvider at all. As we all know, less code is better code and fewer lines of code are easier to maintain and test.

How to update TeamsFx?

At the moment there is no official support yet for using the Microsoft Graph SDK v5 with TeamsFx. But there is always the option to clone the latest version of TeamsFx and implement the changes yourself. That’s what we did in our Microsoft Graph hack toghether sample and you can check out a working version of the modified TeamsFx version in the TeamsFxGraphSDK5 branch of our sample on GitHub. We still kept the file MSGraphAuthProvider.cs in the project, but everything in there is commented out.

Only because you can, doesn’t mean you should

An old SharePoint person

Read the quote above twice before you start using the approach of this blog post in your production code. This is just a proof of concept and not meant to be used in production. We can assume that TeamsFx will support the Microsoft Graph SDK v5 in the future, and for your work projects the smart move is to wait for the official support.

Today I’m again preparing a session, and I thought it might be a good time to write down what I’m doing and why quickly.

I’m cheating

Yes, you read that correctly. I’m cheating. I’m watching countless hours of other people talking in public on YouTube. One of my favourite books is called “Steal like an artist”, and that’s what I’m doing. I’m trying to read/watch/retrieve as much information about speaking in public as possible.

I got training

Not only online, but I attended courses focusing on my voice, speed of talking, stage performance and pronunciation. Remember, I need to speak in public in my own and in a foreign language regularly. I can only recommend getting a good base in your tongue before adding a second one.

I use tools

The picture shows parts of a current project where the tools of Sefirot Independent Publisher help me a lot. I’m using their “Cicero” framework for public speaking at almost all crucial presentations. It is not always full-fledged, but its basics and parts are like a mental checklist.

I ask for feedback

A couple of friends get asked for feedback, and I highly value their opinion. I’m very picky about whom I ask for feedback. It should bring me forward, so no shoulder clapping. I like the in your face feedback, what was bad or wrong.

I reflect

After every session or talk, I try to reflect on what was good, bad and what I should change. Most of the time, that’s part of my Friday afternoon routine of coding for fun and closing the week with reflecting on its outcome.

I repeat

Writing this post made me realise how often I’m in a situation that one could categorise as public speaking, and it’s not only the time you are on stage but also customer meetings and internal meetings. So for me, all over in a year, we are talking about 100+ occasions.

I like it

As you can see, there is quite some time commitment involved. I’m not special. Everyone can level up his or her game of public speaking. But there is no such thing as a free lunch. It would be best if you had some dedication and sometimes the guts to apply for a gig. But if you are prepared, and you put in the hours, for sure you will be successful!

Today’s post is about the need(pain?) of installing your Teams application at the start of every TeamsFx debugging session. If you are new to TeamsFx in Visual Studio please read this intro first as for the next parts we assume that you already familiarized yourself with the toolkit and its features.

There are very good reasons why the Teams Toolkit always starts with the app install screen. At that screen, you can choose where to deploy your app. You can see the current version of your app and all other details of the manifest. But if you are in “the zone”, you might wish that there is a way to get to your latest changes just a bit quicker. What we will be talking about here won’t apply to every possible Teams app, but it works for tabs. To be precise, it works for everything you can create a Teams deep link for.

In general, developers tend to think a lot about how to be quicker or more efficient. Sometimes we overdo it. There are thousands, or probably millions, of examples out there where some dev thought about automating his or her toolchain to the absolute maximum. Just to realise that the overall cost in time of that automation effort is nowhere near the time he or she saved.

Creating Teams Apps

If you think of tabs for Teams, I would always start with a static tab. Even if I know that I end up building something different but just as a starting point I would use a static tab and then over time when the app has a certain level of maturity, I would start to surface it at the extension point of choice. With Blazor, I even would start in a regular Blazor server project and would create a Blazor Library project that holds all the UI components in there and only if I’m already happy with how the app performs, I start to move over to Teams. That might sound like a complex scenario, but for me, it feels like I’m very “efficient” that why and can iterate quickly towards my goals.

Following that approach from above, sooner or later I’ll end up in the need of “F5-ing” my TeamsFx project. And if you are still in the coding zone, meaning you are debugging and trying to understand how your latest code change affects your app, one gets annoyed by the constant need of installing your app over and over again very fast.

Create debug profiles

To overcome that problem you need to understand how the Teams Toolkit configures your Visual Studio project for debugging. Right-click your project file, select “Properties” and then go to the “Debug” tab. We are talking Visual Studio 2019 here, in 2022 the Debug options are also part of the project properties, but the UI changed a bit. 

In the Debug properties at the top, you see the profile drop-down. The default one is called “IIS Express”, check the URL that is configured for “Launch browser”. You will see that this actually is a Teams URL that tells Teams to install your app. That is your app ID in the parameters there. And that is what we are going to change.

Don’t just change the default profile, we will repeat in a minute why you will want to keep this. But for now, just create a new profile. I try to name it similar to the Teams Tab I’m going to use. To get your static tab loaded, just copy the Teams link in the “Launch browser” field. In my case, I’m entering “https://teams.microsoft.com/_#/apps/ca708824-a87d-4b6e-82c0-ca9ec5c7a6e2/sections/index?intent=2”. You can copy that URL from your debugging session and just cut the unimportant parameters.

Also, make sure to create the environment variable exactly the same as the default profile does. It’s wired, but if you don’t apply that, your app won’t have the same look and feel. Need to investigate this in detail and, on a CSS level, why this has an impact. All other settings are exactly the same as the default profile.

From here, next time you start you want to debug, you can just simply change the value in the Visual Studio toolbar and directly serve your tab. That is especially useful if you have different tabs you create and want to jump between them directly from your debugging session.

As mentioned you should keep the default debug profile to be able to always also open the app install screen. If you create configurable tabs or meetings, you need to install them directly to your test Teams or Channels. So it makes sense to keep that profile in order to be able to really pick the right launch config for your next move.

Disclaimer

I’m pretty sure things will change when Hot Reload in .net 6 will be available. I know that this is only a couple of weeks away, but nevertheless, playing around with different launch options helped me to understand how TeamsFx works and operates in more detail. Hope you find it useful as well and see you next time!

Introduction

If you follow the news around .NET 6 and Visual Studio 2022 one of the most time-saving features for developers might be the .NET Hot Reload experience for editing code at runtime. This feature is part of the .NET 6 release and will be available across various project types such as WPF, Windows Forms, .NET MAUI, ASP.NET Core apps code-behind, Console applications, WinUI3 and many others. [1]

Hot Reload example from Introducing the .NET Hot Reload experience for editing code at runtime - .NET Blog (microsoft.com)

With Hot Reload you can start your debugging session and keep modifying your source code without the need to restart the application at every little change. If you saved your changes, just click the Hot Reload button and your changes will be available. As not all possible code changes are supported, please be aware that in certain cases you will get an error message within Visual Studio reminding you to restart your app. What is supported and what is not is explained in detail here.

Why Hot Reload matters for TeamsFx

With TeamsFx we are of course focusing on the Blazor/Razor features of Hot Reload. But the most time-saving part of it is more based on the fact how TeamsFx surfaces your application. If you open the details of your project in Visual Studio 2019 and look for your launch profiles in the Debug section, you will find the URL your browser is told to use to start your application. You can find the same config also in the “launchSettings.json” file in the “Properties” folder of your TeamsFx project. This URL looks something like this: https://teams.microsoft.com/_#/l/app/d8438f91-ecd9-4a5c-887a-d843f527244c?installAppPackage=true

launchsettings.json

This link opens up the app install dialogue within Teams and allows you to install your app. That’s just your Teams App ID  (line 5 of your manifest.json file) and the parameter to install the app in your environment. Yep, there is no tenant ID present. That’s why sometimes you end up pressing F5 and you get an error saying something like you don’t have permissions or the app, cannot be found. Most of the time those errors are based on identity issues, meaning you used identity “A” to configure TeamsFx SSO in Visual Studio, but you are running your debugging session in the browser with identity “B”. During configuration or when you update your manifest, TeamsFx pushes your changes to Teams Developer Portal and uploads your app to the tenant. If you use a different identity/tenant to test your app, Teams will tell you that it can’t find the ID Visual Studio started your browser with. To get rid of those problems, you can select a dedicated browser to start your session with or just press F5 and log out and log in again with the correct identity and restart your debug experience again.

Teams app install screen

The need to always reinstall your app again is sometimes just time-consuming. You press F5 and wait for Teams to open, then you click install just to see your app loading. It’s just a couple of seconds, but if you do that a hundred times over and over again you, or at least I, get maybe a bit frustrated.

There is where Hot Reload comes to the rescue. Because if you can apply code changes without the need to restart and reinstall your Teams application over and over again, you really save time. In my experience, that really speeds up the whole UI work of your project. Making changes to the HTML part of your Razor components, for example. Changing CSS classes and such things. In terms of code changes, you need to be aware of what is supported and what is not. When I need to restart the whole app, it’s most of the time due to changes to lambda expressions. For example when you redefine your List.Where() or Sort() statements. But you probably get them correct the first time. Always. Well, I don’t ;)

How To - Hot Reload with TeamsFx

At the moment (October 21) TeamsFx is still in preview and not generally available. It is also only available in Visual Studio 2019. If you try to install the TeamsFx extension in VS 2022 you are currently out of luck as it is not compatible yet. This will change with the GA version of VS 2022 and TeamsFx. Check this GitHub issue for details.

Before we start, a short reminder:

Don’t try this in your production tenant.

Until the versions of the tools are in sync, you can follow this list of steps to make Hot Reload available in your test environment today.

• Install the latest preview of Visual Studio 2022 or update to the latest version if you already have it
• Open Visual Studio 2019 and create a new TeamsFx project
• Configure SSO for your application and press F5 to check if everything is working
• Close Visual Studio 2019 and reopen your solution with the preview of Visual Studio 2022
• Change the .NET runtime to .NET 6 in VS 2022
• Press F5 and enjoy Hot Reload in TeamsFx

If you need to change something in your Teams manifest just reuse the TeamsFx tools in VS 2019 to sync your changes to your tenant and re-open the solution in VS 2022 to start debugging again.

https://youtu.be/PLNsjtGR59A

Big thanks to John Miller for providing the right hints and directions for this article!

Final remarks

Everything above is based on the current state of the tools mentioned. TeamsFx, Visual Studio 2022 and .NET 6 are all still in preview and not released yet. Things will change in the (fingers crossed) near future. I’m really hoping that this little trick will be obsolete very soon. But it is always important to understand what all those fancy buttons and features in your toolchain exactly are supposed to do. Because there will be a day when they won’t deliver what you expect and your skills are needed.

Microsoft Teams Framework is a library encapsulating common functionality and integration patterns (like simplified access to Microsoft Identity) for developers building apps for Microsoft Teams. The Teams Developer Platform offers a collection of solutions for Teams developers:

• IDE extensions for Visual Studio and Visual Studio Code named “Teams Toolkit”
• Docs, samples
• Code libraries including Teams Framework teamsfx and the Teams Client Library teams-js
• A command line tool teamsfx for terminal users, automation, and CI

<https://github.com/OfficeDev/TeamsFx>

This post should help you to get you started with your first Teams application in Blazor. As every one of us has a very own way of learning, I’m happy to offer you either the video explaining the approach in roughly 13 minutes, or the blog post, that tries to cover the same ground, but with text and images. Whatever you choose, I’ll hope you enjoy your option, and please keep the feedback coming!

Why Blazor makes the difference

What got my attention was of course the Visual Studio part of the early messaging. At Build2021 it was clear that there will be an option to create applications for Teams based on Blazor. For everyone not familiar with Blazor and with a dev background, think SPA or front end dev work with .NET instead of JavaScript. Blazor gets a lot of hype for the WASM (Web Assembly) capabilities, but for me, not needing to write JavaScript code is already a huge advantage.

Please, don’t take that as some decision or view that should be followed like a religion. I know some people out there see it that way. If that’s OK for them, be my guest, but for me, it’s a point in time decision of a developer framework or toolchain. Things will change in the future again, for sure. So pick what you need to finish the job and try to learn as much as possible about the tool you choose. In the end, the question is if your users are happy with your app, that’s in my opinion way more important than the dev stack.

Anyways, for this article and blog, we will focus on the .NET side of the house, as I just can’t write JavaScript without the constant fear of breaking my fingers every 10 lines of code. And there is only me to blame for that, as I never invested enough time to really dig deep and learn it from A to Z.

Looking back over some smaller projects and what I was able to achieve in Blazor, I’m more than happy that Microsoft invests also in Blazor in M365.

TeamsFx in Visual Studio or Teams Toolkit v2

If you open up Visual Studio 2019 and search for Teams in the Extension menu, you will find two different extensions. We are focusing on the one named “Teams Toolkit [Preview]”. And yes, this library is still in preview and has some rough edges. That means you should not use that in production or start creating support tickets because things break. It’s still in development and not released yet, so better be careful how you use it! Go ahead, install the extension and let’s have a look at what we get!

First Teams template

After installing the extension and restarting Visual Studio 2019, create a new project and select “Microsoft Teams App”.

The wizard asks you about your project and application name. Please make sure you remember those names/variables because we will discuss their references later on. Visual Studio now created a new Blazor based project and opens the GettingStarted.txt file with a four-point list telling you what you should do to be able to “consume the F5 experience”. In other words, that’s what you need to do, to get the application up and running. Please select a developer or test account and tenant during the process and make sure you have at least some permission in the Azure portal of that tenant, so you can manipulate the generated Azure AD app, and it’s permissions later on.

Following points 1 and 2 you should see something like this in your output window in VS 2019:

The Teams Toolkit took the credentials you provided and created a couple of things in the background. First, it created a Teams app definition and uploaded it to the Teams developer portal of your tenant. Second, it created an Azure AD application, assigned some basic permissions and “allowed app ids” and configured your Teams project to use said Azure AD app in your solution. Open both portals and make sure you find the entry you just created by configuring SSO for your application.

And:

You should be able to recognize the values you provided during the wizard in both portals and most probably you should think about your naming convention in the future just to make sure things are consistent and easily identified in all use cases: Visual Studio Solution, Azure AD App and Teams Developer portal.

Start your application - Press F5

If you were able to follow along so far and no installation or configuration hiccups occurred, go back to Visual Studio and press F5. It will take a couple of seconds, but VS will build the application and start your local IIS instance to serve your Blazor application. Visual Studio should open your browser and point you to the Teams web application of the tenant your SSO account belongs to.

On the installation screen, click on “Add”, and after a couple of seconds, you should see your first demo application up and running in a private Teams Tab.

Congratulations, you just created your first Blazor application for Teams!

Exploring the demo application

Looking on the screen, you see the “Authorize” button. That button will create a Microsoft Graph service client and request the profile picture of your developer account. To follow along, open the Tab.razor component in the “Pages” folder of the project and look at lines 86 to 88.

We won’t go into details of the implementation, but the GraphClient is a regular client from the .NET SDK. The “magic” of the library/template comes from the “TeamsFx” part and how it creates that client for you.

Not sure that my analogy is 100% technically correct, but what TeamsFx does is, it creates JavaScript wrappers for the Teams JavaScript SDK and makes them accessible by using the built-in JS interop features of Blazor. In the TeamsFx.cs file, you can find the C# part and in the wwwroot folder, you find the JavaScript files that hold the needed glue code to be able to communicate with the Teams SDK.

From a high-level perspective, that’s easy to understand, but the devil of course is in the details. I tried to create something similar by using the Microsoft Graph Toolkit, and its auth components in combination with Blazor JS interop. It worked, but as soon as you reach the point of token lifecycle it gets complicated, or let’s rephrase that, it just gets more serious, and you need to be very careful with your decisions, especially when dealing with access and refresh tokens.  

If you want to explore the implementation just work your way through the code and try to understand how the app gets your bearer token and how the GraphClient gets created. If you want to extend the demo example have a look at https://github.com/thomyg/PnPDemo190821Backup. That just counts all the teams your user is a member of. But it already means that you need to add permissions to your Azure AD application, as the basic ones the Toolkit created aren’t enough to iterate through all your Teams.

As this post just is an entry guide to the world of TeamsFx we will stop right here to not make it too complicated right from the beginning. Thank you for following through and cu next time!