Copilot: What you do &
don't know:
How Copilot actually thinks

Hi everyone. Welcome to episode three of Copilot. The inforcer way I guess is the term or the title that we've been using. This episode is really about Copilot, how Copilot actually thinks. And before I do anything further, I have been rightly pointed out that I've not even introduced myself other than I'm just sat on YouTube with an introduction to say Copilot. But just really quickly, my name is Tim. I'm one of our 365 solutions architects here at inforcer.

I've been working here now for probably about four or five months from the date this was published and spend my life kind of working around AI in integrations Microsoft 365 stack focusing on kind of improvements changes enhancements features baseline management alignments you name it I'm involved in most things hopefully that gives you a good understanding I prior to this kind of job I was an MSP working MSP based in central London that focused around financial services for around 5 years and I've been in the MSP space now for around 12 years which has been a long time for me feels like forever.

Love every minute of it as always and I look forward to kind of going through this series moving forward. So without further ado, let's focus on what we're essentially here to talk about and that's how Copilot thinks when we talk about the Microsoft 365 environment. Understanding the mind behind the AI assistant. So let's start with how Copilot works or when does it get confused. I think it's a really good start. There's always going to be moments with AI particularly as it adjusts and it starts to get better or grow its understanding through the LLM.

There are elements as it starts this process where it's going to get confused. Call it the Copilot brain fart if you will. What you start with, what you're asking based on what it delivers. And sometimes that comes down to the descriptions we give.

We're very much used to as humans when we're doing a search in the Google search engine, we're clear cut and direct with what we're looking to deliver and what we're looking to. So off the back of that, we're very much used to kind of giving us simple answers, i.e. Summarize last quarter sales pipeline. It's going to come back and it's going to go through an information and go, I'm not too sure exactly what you're after, but let me have a look and I'll come back to you.

And it's going to come back. As you can see with the example, summarize last quarter sales pipeline. Your team sold three llamas and a time machine. It's confused. It doesn't understand. And that's really down to the fact that perhaps the description or the request is a bit too vague. And it's common. We're used to it. We're very much in a position where we like to give clear-cut instructions. But with AI to adapt and for it to grow as an LLM, it needs to gain more knowledge or it needs more precise information.

A conversation is probably the way I would start to use this. Asking it a question with please can you look at the following? This is where I would be looking. So let's talk through how it works.

So how does Copilot think the human analogy? So Copilot isn't magical. It's just a really fast pattern of matching words. It looks through and sifts through the information. It creates predictions, pattern predictions that identify what it's looking at and what it's looking for. Think of it as, and I'm actually reading out from the screen, but think of it like a colleague who reads everything in the Microsoft 365 tenant, but has the memory of a goldfish. It's a great example.

It can always go back and retrieve the information that's readily available. But it's absolutely right. It doesn't have the brain functionality that we have where we might think of an idea and that provide an outcome. It's learning. It's trying to get to that point with Gen AI, but it's not to that point at this moment in time. And that's where we have to kind of think of how we request or we write down the words we're looking at.

Really good example at the moment fits around picture requirements. When we go to AI image platforms and we give a description of what we're looking for, it's using pattern recognition to a degree. It's matching up your words with previous image conversations or images and it's reanalyzing those and it's coming out with an outcome. The more descriptive you are when you're looking for an image, sometimes more complex it makes it. And some sometimes we're literally just after provide me a Copilot image that has a couple of descriptions of what the Copilot for example the image you see on the screen in front of you is AI generated.

The human analogy is essentially what it's looking for. It's taken the information it's funneled it through and it's created and generated this image. The quality formula I think adds some huge value. So the quality of your prompt and the quality of your data equals the quality of the result. So the better your prompt injection, prompt understanding, prompt responses and how well you've governed and protected your data or managing your data will determine the result you're going to get.

And fundamentally that's how Copilot really works within the Microsoft 365 environment. So the real workflow and I again I have haven't updated this completely but just to kind of put this into prospect Work IQ is available in Microsoft 365. It's very much the same as graph but it's got a bit more fluidity and it sits really there for Copilot AI. Want to highlight that. Please have a look in a bit more detail on it. So let's get more technical whilst keeping it human.

Let's have a look at behind the scenes. We create the prompt.

That prompt needs to be an instruction and it needs to be clear, precise but fluid and it needs to be a bit longer than providing the analysis of Q1 sales. Where does it need to provide that analysis? The way it works is graph will retrieve those context. So or Work IQ will sift through the relevant documents, messages and permissions that you have. I might emphasize it's the permissions and purposes or connections that you have specifically.

It'll then use the secure LLM in the background. The mo model that sits behind or in front or managed by our tenant. So it's that secure boundary. It will generate a response utilizing that information. So text based on patterns in your data returned from it from the apps that have been requested. Privacy at this point is guaranteed and I think this is really useful. Nothing is stored or reused for the model that's being trained. It's all about utilizing your response request, removing that sensitive information that you've maybe put in as a request.

So the model is still learning and it will then at the end produce you that response. So the last bit is once it's collected that information, made sure that it meets those privacy guarantees and requirements, it then pushes that response out to you. And I want to touch on and it's not I don't think included in this libraries or this slide deck, but the way that it works is it's all about ensuring over permissioning is protected.

We need to make sure that the data that we are utilizing and sifting through has the right permissions for the user that's using it. There's a real big hot topic on how do we manage and provide access to Copilot and I've said it in the series or the episodes before. We can leverage data loss prevention now with sensitivity classification around AI information and sensitive information being provided or passed through I should say Copilot and we need to leverage that.

We should be utilizing data loss prevention governance within purview to protect the data and we should be doing that from day one because ma remanaging overshared SharePoint permissions is a pain in the backside. It takes forever to do so. So what measures can we do in advance or at the forefront of this delivery if people are asking for Copilot? Straight up answer let's leverage Purview data loss prevention. Leveraging inforcer's policy baseline management to deliver these.

You can standardize what you want to implement for Purview protection. So data loss prevention and push that out to your customers. We can leverage that information, leverage that alignment and baseline delivery utilizing inforcer. So let's go through that kind of hallucination in a bit more detail.

Why Copilot hallucinates? What's the cause of the hallucination? It starts with poor grounding, data missing, not accessible to the system. I.e. You haven't got the right permissions to access. That's the common cause for the things going wrong. And it will always leverage if it can't find the information on your local system and you've got Bing enabled or the external access enabled. It's going to start looking through information that's on the web.

It's not stored in the LLM. It's not in misinformation, but it's leveraging the tools it has available, but it starts with our data. So sometimes we have data that's missing or the pit as said it's not accessible in the system. The other area where we get hallucinations is vague prompts. We just say, "Tell me about our projects. What's it going to look for?" It's going to provide you perhaps information that's irrelevant because the LLM isn't strong enough to understand exactly what you're after.

And this is why prompts need to be precise, clear, and agile. Cross domain confusion, mixing Teams chats with random documents. It has at the moment a lot of integration with different areas. Sometimes it's sifting through information in the wrong location. And that can happen. It's an AI agent. It's learning. It's understanding how things work. There's going to be misinformation. And that normally comes down to poor prompting poor prompt injection. And then the other one is permission issue.

It doesn't actually have access to the information or the real data that we're looking for. And that's because your permissions don't have the right access. So, where do we go to resolve most of these hallucinations? It starts with a well-grounded prompt.

Summarize Apollo project updates from last week's Teams channel. You're saying look for the specific project and look in the specific location. You're providing a clearcut, accurate contextual summary with the right references. So, the AI knows where it needs to go. If you were to say, "Tell me about projects." We're going to be or you should be expecting mix mixed information and unrelated information. Information that's irrelevant. Copilot isn't providing you the wrong information. It's going based on the information you've requested.

So, the more that your prompt is accurate or precise or concise, we can focus on making sure that we're going to get the right information. We will get the right results if we're pointing in the right lo in the right location.

So, let's talk about prompting like a pro. My colleague Freddy does this really well and he's got a lot of information around this. I'm sure we can get some more context around how do we prompt correctly, but it starts with four areas. Your role, who are in the context, what's the context of the role or the request we're making. It's going to look at our job title, our description, understand who we are as a business or as a model.

So, it starts with what do we look for? How are we asking for that request? Set the task. Understand what it needs to do. So what specific action do you want AI to perform? Is it a summary? Does it need to create something? Is it analyzing? Adding those three words or a word like this allows it to understand what prompt or what response it's going to project. If it's an analysis, it's going to consider a pi diagram or it's going to provide you with bullet points for information.

If it's a summary, if you're asking it to create something, it's going to be utilizing an application in order to produce that report for you. The context is probably the most important part. So, where should Copilot look? If you know that the request you're after is in a document on a SharePoint site, you should be including that. If it's a date range, it's going to look at modified dates version history because it has the ability to do so if you have the right permissions.

The more concise and your context is, the more likely you're going to receive the right result. And then finally, the output. What's the format of the response you want? Do you want it as just a set of results? Do you want it as bullet points, a table, a graph, a pie diagram, an email draft is put there on point 4. How do you want that output to look? These four areas determine how well that prompt and that response is going to give back to you, how it's going to look as it provides you the right response.

So again, another example from the previous slide and this one summarize my project updates is too broad. There's no context and it's unpredictable results. You need to be prepared to expect misinformation or information or hallucinations that are inaccurate. Whereas the structure responses as a project manager has my job title summarize last week's updates for the project Apollo using teams messages and files in the Apollo channel. Present this as bullet points for me. You provided that clear role you've created that specific task.

You've defined the context and then you've requested it in the right format or the format that you were after. The more concise you are, the clearer the results are going to be and it focuses around what we've made as a request.

So thinking depends on clean inputs and this starts with making sure that we follow as part of the same series I've got as defend govern. It's following a clear structure and this is where inforcer really leverages and provides a good context response and encompassing that profile delivery for Copilot readiness. So inforcer makes sure that you can provide clean inputs and clean outputs because we measure controls around defending i.e. The configuration of your environment the governance putting in the right controls over permission management security management SharePoint delivery more importantly conditional access policies and data loss prevention.

So accessing that's that governance piece detection and more importantly the readiness the proof that we are in a position to be able to do the delivery and that's where the other series and I'm highlighting it now because it's coming out very shortly the defend improve webinar and YouTube series we've got coming out really brings to light what you see on this screen now we've got to make sure that our security baselines are at the forefront of how we deliver a secure complex but secure Microsoft 365 environment.

We need to make sure our customers are aligned to those processes. They are meeting our best practices and MSP. Then we need to make sure the access controls around it are in place. That starts with conditional access. Maybe it's a license uplift from business standard to business premium. So we've got that better security control. Delivering data loss prevention to secure that AI model. Sensitive information should not be going through AI unless you have the right permissions to do so.

And we're locking down HR files, information that shouldn't be relevant to AI. Should it be, should it not be, that's controls we need to be measuring. Data loss prevention policies can be deployed using inforcer. We can predefine them as an MSP and offer that as a service, whether that's commercial gain, commercial delivery, project delivery as professional services. All those functions are available for us to leverage. Sensitivity labeling coverage. It's something we should be doing already. It's recommended by Microsoft when we talk about Copilot.

We all know it's difficult to do from day one. It's a journey we have to take our customers on. That's governance as a service there. Finally, detecting errors or drifts from the configuration. That's our governance piece. Making sure that we are stopping and we're enforcing things and we're keeping on top of it. Drift detection is so useful. We need to know someone's been excluded from a set of policies. And then finally, that readiness assessment.

We have those three controls. The next step on our journey is that readiness piece. Making sure that we can demonstrate to a customer we've made these changes after speaking to you previously. This is the result of your Copilot readiness. This is where we see you as individuals within a business. We see the benefit or ROI for the following staff and we think this is a value you're going to get. You know, Copilot in 365 particularly as a security piece is not cheap.

We need to make sure that the right processes are in place and we're as secure as we can be and it starts with those right models. Finally, that productivity revolution. So, how do we go from that configuration and that build? How do we make sure that what it's thinking is accurate and contextual?

And it's not creating it's not creative chaos, it's contextual clarity. We want to make sure that as we go down this route that we focus on the right things, the clear context that we're after for the clear results that Copilot's going to give us. Starts with that baseline management that we're offering at the moment. Making sure that we are keeping things secure and safe within the in the environment. And as a result, providing we follow those four key points, that role, the context, the wording, and the permission management or the context or results that we're looking for, clearcut points, we're going to get the right results, and we want to make sure it's secure when we do this.

When we when we make sure how Copilot is actually thinking, that information it's provided is secure and safe. So, what's next? To wrap up this thankfully it's less than 20 minutes. Hallelujah. We want to focus on where do we go? What's the productivity revolution? What Copilot changes for everyone? How do we adapt to what it's now offering? A AI is identifying over permissioning messy environments at the moment. So what's that productivity revolution? And that's going to be episode four coming up.

We'll see how this magic changes the way we actually work. Meetings, emails, documents that ROI delivery the effectiveness of moving forward utilizing and using Genai to make things more efficient for ourselves and that comes with a productivity revolution. So stay tuned for episode 4. Any questions in the meantime, reach out to inforcer. Let's get some demos booked. Let's get these conversations rolling. Thank you very much for your time, guys. All the best.