I was looking at the list of M functions supported in custom connectors and not in Power BI Desktop (using the technique I blogged about here) in the latest version of the Power Query SDK when I came across an intriguing new function: List.ParallelInvoke(). It doesn’t seem to be documented anywhere, but I think I’ve worked out what it does and it’s very exciting!
Consider the following M function, declared in a custom connector:
[sourcecode language=”text”]
SlowFunction = () as number =>
Function.InvokeAfter(()=>1, #duration(0,0,0,5));
[/sourcecode]
When you call it, it waits 5 seconds and returns the value 1. If you call it three times and sum up the results, as follows:
[sourcecode language=”text”]
List.Sum({SlowFunction(), SlowFunction(), SlowFunction()})
[/sourcecode]
…then after 15 seconds you get the value 3 back.
Now, consider the following expression:
[sourcecode language=”text”]
List.Sum(
List.ParallelInvoke(
{SlowFunction, SlowFunction, SlowFunction}
)
)
[/sourcecode]
When this is evaluated in a custom connector, you get the value 3 back after 5 seconds – so it looks like List.ParallelInvoke() allows you to invoke a list of functions in parallel. There’s also an optional second parameter called concurrency, which seems to control the amount of parallelism. So, for example:
[sourcecode language=”text”]
List.Sum(
List.ParallelInvoke(
{SlowFunction, SlowFunction, SlowFunction},
2
)
)
[/sourcecode]
…returns after 10 seconds, suggesting that only two function calls at a time are invoked in parallel.
I can imagine all kinds of uses for this, for example making multiple parallel calls to data sources or doing expensive calculations in parallel. I wonder if it will ever be allowed to be used outside custom connectors?
UPDATE: see Curt Hagenlocher’s comment below for some important information about this function.
This currently works in only some extremely-limited ways, as the engine code on the whole isn’t thread-safe.
OK, so the message is not to use it yet, I guess.
Some parallelism, or at least concurrency would surely be nice.
Also, I wonder if we have a Python script which does multi-threading or parallelism, how that works when published.
Dear Chris, I need a little help in relation to show a list from a table before my connector call the service to display list of tables in the navigation pane, my full query is below:
https://community.powerbi.com/t5/Developer/Custom-data-connector-SDK-help-required/td-p/547696
Any possibility You can help me with? Really struggling for quite sometime.
This function has been removed in the latest PowerBi
Hi Chris,
Thanks for sharing this function, your posts are incredible!
I saw this function was deprecated and I find no PowerQuery parallelism discussion on forums… Do we have any news about this? I know we have “Table.Partition()” and it’s sibblings currently, but honestly it doesn’t seem to work concurrently and I’m looking for more content.
Thanks !
There’s no news unfortunately