I was looking at the list of M functions supported in custom connectors and not in Power BI Desktop (using the technique I blogged about here) in the latest version of the Power Query SDK when I came across an intriguing new function: List.ParallelInvoke(). It doesn’t seem to be documented anywhere, but I think I’ve worked out what it does and it’s very exciting!
Consider the following M function, declared in a custom connector:
SlowFunction = () as number => Function.InvokeAfter(()=>1, #duration(0,0,0,5));
When you call it, it waits 5 seconds and returns the value 1. If you call it three times and sum up the results, as follows:
List.Sum({SlowFunction(), SlowFunction(), SlowFunction()})
…then after 15 seconds you get the value 3 back.
Now, consider the following expression:
List.Sum( List.ParallelInvoke( {SlowFunction, SlowFunction, SlowFunction} ) )
When this is evaluated in a custom connector, you get the value 3 back after 5 seconds – so it looks like List.ParallelInvoke() allows you to invoke a list of functions in parallel. There’s also an optional second parameter called concurrency, which seems to control the amount of parallelism. So, for example:
List.Sum( List.ParallelInvoke( {SlowFunction, SlowFunction, SlowFunction}, 2 ) )
…returns after 10 seconds, suggesting that only two function calls at a time are invoked in parallel.
I can imagine all kinds of uses for this, for example making multiple parallel calls to data sources or doing expensive calculations in parallel. I wonder if it will ever be allowed to be used outside custom connectors?
UPDATE: see Curt Hagenlocher’s comment below for some important information about this function.
This currently works in only some extremely-limited ways, as the engine code on the whole isn’t thread-safe.
OK, so the message is not to use it yet, I guess.
Some parallelism, or at least concurrency would surely be nice.
Also, I wonder if we have a Python script which does multi-threading or parallelism, how that works when published.
Pingback: Invoking M Functions In Parallel Using List.ParallelInvoke() | Pardaan.com
Dear Chris, I need a little help in relation to show a list from a table before my connector call the service to display list of tables in the navigation pane, my full query is below:
https://community.powerbi.com/t5/Developer/Custom-data-connector-SDK-help-required/td-p/547696
Any possibility You can help me with? Really struggling for quite sometime.
This function has been removed in the latest PowerBi