Continuing my series of posts on Power BI Copilot and the type of things you should be including in AI Instructions, today I want to talk about helping the end user understand the scope of the data: what data your model does contain, what data it could contain but doesn’t, and what data it could never contain.
Let me show you some examples to explain what I mean, using the semantic model I introduced in the first post in this series containing real estate transaction data from the Land Registry in the UK. On a model with no AI Instructions added, if I give Power BI Copilot the prompt:
show average price paid for the city manchesterI get the correct answer back:
That’s good and as you would expect. Now consider the following two prompts which both return no data:
show average price paid for the city edinburgh
show average price paid for the locality Boode
A user asking a question and not getting an answer is unavoidable in many cases but it still represents a failure: the user wanted information and the model couldn’t help. As a semantic model developer you need to minimise the impact of this and help the user understand why they didn’t get an answer. In both cases Copilot does the right thing and tells the user that it can’t find data for these places. However, there’s an important difference between these two places.
Boode is a very small village (somewhere I found at random) in England. There’s no data for it in the semantic model because there happen to be no real estate sales for it. Edinburgh on the other hand is a large city, so why is there no data? It’s because this semantic model only contains data for England and Wales, not Scotland, Northern Ireland or anywhere else. An end user might expect that a dataset from a UK government agency would contain data for the whole of the UK but this one doesn’t. If the user was looking at a report displaying this data then this information would probably be displayed somewhere in a textbox on an information page, or be obvious from the values visible in slicers or filters, but with Copilot we have to tell the user this in other ways.
Similarly, consider the following two prompts:
show average price paid for January 2025
show average price paid for January 2024
Why is a blank displayed for January 2024? Were there no sales in January 2024? The answer is that the semantic model only contains data between January 1st 2025 and April 30th 2025, so there’s no point asking for data before or after that – and we need to tell the user so.
Here is some text that I added to the AI Instructions for this semantic model to tell Copilot what data is and isn’t present:
##What data this semantic model contains
This semantic model only contains data for England and Wales. If the user asks for data outside England and Wales, tell them that this model does not contain the data they are looking for.
This model only contains data for the date range 1st January 2025 to 30th April 2025. Tell the user this if they ask for data outside this date range.With these instructions in place let’s rerun some of the prompts above. First, let’s ask about Edinburgh again:
This is a much more helpful response for the user I think. Copilot knows that Edinburgh is in Scotland and it tells the user that it only has data for England and Wales. Copilot doesn’t know every place name in Scotland but it does pretty well; for example (to take another random place name) it knows that Arrina is in Scotland:
Similarly, here’s what Copilot now says for January 2024:
Again, a much better response than before: it explicitly tells the user that the model only contains data for January 1st 2025 to April 30th 2025.
Giving Power BI Copilot clear instructions on what data the semantic model does and doesn’t contain means that it can set end users’ expectations correctly. This then means that users are more likely to ask questions that give them useful information and therefore builds trust. Conversely, not explaining to end users why their questions are returning data means they are less likely to want to use Copilot in the future.
Thanks for the details with examples. I am currently struggling with Copilot to first look at the data and answer questions about data rather than answer questions based on visuals. I feel I get more consistent result when copilot looks into data and answers.Any thoughts
There are a number of issues that we’re fixing that could cause this problem. But if you have specific examples I’d love to hear about them. You can contact me via the Contact page on this blog.
This is spot on—and it mirrors a challenge I’ve been facing recently. In my case, the issue is on a much larger scale. Stakeholders interacting with Copilot are frequently not getting the expected results, largely because the underlying reporting data is derived from call transcripts.
The types of questions being asked are quite broad and nuanced, such as:
“What are common reasons we’re losing clients?”
“How many clients have expressed disinterest or requested to be removed from our contact list after multiple calls?”
“Can you analyze the transcripts to determine if pricing is a key reason clients are not interested?”
Given the wide range of questions, I’m trying to figure out the best approach to ensure Copilot delivers meaningful responses. Should I be manually interpreting these questions and preparing the data accordingly in the ‘prep data’ phase? If so, doesn’t that undermine the time-saving value Copilot is supposed to offer?
I’d love to hear your thoughts—especially if you’ve found a scalable way to handle this kind of open-ended Q&A.
This is a question we’re all wrestling with! I guess you have to understand the different types of questions that people ask:
1) Questions that Copilot can answer today
2) Questions that Copilot could answer if you prepared your existing model appropriately (either with AI Instructions, new measures or different modelling)
3) Questions that Copilot could answer if you added more data to the existing model
4) Questions that Copilot will never be able to answer, or at least answer accurately
…and then train end users appropriately and also add AI Instructions to help Copilot provide helpful responses for scenarios 2-4