Power BI Preview Review Part 2: Q&A

Today the natural language query functionality in Power BI, Q&A, was enabled in my Power BI Preview tenant. This is the last major piece of functionality to be added to the Preview, and of course as soon as I heard that it was there I had to go and play with it, despite being on a slow mobile WIFI connection and having tons of emails to catch up with. So apologies to anyone who is waiting for an email from me!

First thing to point out: you can only use it on some sample workbooks provided by Microsoft, and these get added to your Power BI site when you enable Q&A. The ability to use it on your own data is coming soon but I don’t think this stopped me from getting a feel for how well it works.

With Q&A switched on my Power BI site now looks like this:

Notice that the top half of the page now has a ‘Popular Questions’ area which takes you to some pre-defined questions on the sample data, while in the bottom half you see the sample workbooks along with any other workbooks you have uploaded. The two sample workbooks contain data on medals won at the Olympics and sales of drinks in a bar.

Here’s what one of the sample questions looks like when you click on it:

At the top of the screen you have a search box where you can type your question; immediately underneath that box is how Q&A has interpreted the question. On the left-hand side you see the workbook used as the data source, and you can see alternative visualisations for the data that you can choose. On the right-hand side there are some suggested questions for this data and other hints and tips. It’s all pretty straightforward.

But enough of the descriptions – there are two big questions that need to be answered here:

1. Does it work? That’s to say, will a real user be able to ask questions and find they data they want?
2. Will anyone use it, or is it just demo-ware?

I’m aware these are loaded questions and I run the risk of upsetting my friends at Microsoft if I say rude things about this product. I’m also aware that I have only had a very limited amount of time to play with it; that it is still a Preview and will undoubtedly improve a lot over the next few months (however good it is now); and most importantly that I am not the target audience for this product therefore it will be impossible for me to judge how well it works – I know too much, and I am likely to be able to be able to write questions that work much better than a normal user can. I had hoped to try this out on my wife to get some real world feedback but unfortunately I’m travelling at the moment and she’s at home with the kids…

With that in mind, let me give you some idea of how well it works by showing questions I tried and the output I got:

The first question I tried to find an answer for was how many medals Australia had won for Swimming in 2012. I typed “how many medals did Australia win for swimming in 2012” and this was the result:

I had wanted a single value and instead I got a count of medals for swimming broken down by name; they were Australians (I assume) but it’s not exactly what I was after. I then tried “total number of medals won for swimming by Australia in 2012” and hit the jackpot:

My next question “list the cities where the games have been held” got the right answer first time:

A similar question, “total number of times the games have been held in each city”, also worked first time (but maybe because I had learned that Q&A can interpret the phrase “total number” properly):

I don’t know why it thought a map was appropriate here though.

One final test: I wanted to know which athletes had won medals at more than three different Olympics. I tried “who has won medals at more than three games” but got a list of countries that had won medals at more than three games, which of course was quite a long one; “athletes that have won medals at more than three Olympics” got a list athletes and the games they had won medals at. Eventually after several more tries I gave up.

Of course this last question was deliberately difficult question and entering the same text into Google didn’t give any useful results either. I would have been very surprised if I had got the correct answer. I think, though, this question gets to the heart of the reason why a lot of us were sceptical about Q&A when they first heard about it: end users won’t know which questions a product like this can reasonably answer and which ones are just too difficult, and they won’t know how to frame questions appropriately, and therefore a product that seems to suggest it can answer any question you type will always, ultimately, disappoint them. It will be very difficult to set users’ expectations appropriately.

That said, I was pleasantly surprised at its success rate and I’m looking forward to trying it on my own data. To answer my questions above:

1. Does it work? Yes, I think it does a good job of answering any question I would have expected it to be able to answer. Furthermore, as I said, I know it will improve over time, not only because we’re still in Preview but also because this is a cloud-based solution and MS can keep tweaking after it has been released as well. I wonder if MS are capturing usage metrics and questions and using it to tune the system? I bet they are.
2. Will anyone use it? This is a much trickier problem. I’ve never come across an end user who has asked for a product like this, but just because no-one has ever asked for something doesn’t mean they won’t love it if they get it. It’s clearly aimed at people who find every other BI tool available today too difficult (and there are plenty of them), but are these people interested in using any BI tool however easy it is to use? Will they just go and ask a colleague or an analyst or a minion to go and find the answer for them instead? Frankly, I don’t know at this point. We shall see though!