Custom Data Connectors · M · Power BI

Invoking M Functions In Parallel Using List.ParallelInvoke()

I was looking at the list of M functions supported in custom connectors and not in Power BI Desktop (using the technique I blogged about here) in the latest version of the Power Query SDK when I came across an intriguing new function: List.ParallelInvoke(). It doesn’t seem to be documented anywhere, but I think I’ve worked out what it does and it’s very exciting!

Consider the following M function, declared in a custom connector:

[sourcecode language=”text”]
SlowFunction = () as number =>
Function.InvokeAfter(()=>1, #duration(0,0,0,5));
[/sourcecode]

When you call it, it waits 5 seconds and returns the value 1. If you call it three times and sum up the results, as follows:

[sourcecode language=”text”]
List.Sum({SlowFunction(), SlowFunction(), SlowFunction()})
[/sourcecode]

…then after 15 seconds you get the value 3 back.

Now, consider the following expression:

[sourcecode language=”text”]
List.Sum(
List.ParallelInvoke(
{SlowFunction, SlowFunction, SlowFunction}
)
)
[/sourcecode]

When this is evaluated in a custom connector, you get the value 3 back after 5 seconds – so it looks like List.ParallelInvoke() allows you to invoke a list of functions in parallel. There’s also an optional second parameter called concurrency, which seems to control the amount of parallelism. So, for example:

[sourcecode language=”text”]
List.Sum(
List.ParallelInvoke(
{SlowFunction, SlowFunction, SlowFunction},
2
)
)
[/sourcecode]

…returns after 10 seconds, suggesting that only two function calls at a time are invoked in parallel.

I can imagine all kinds of uses for this, for example making multiple parallel calls to data sources or doing expensive calculations in parallel. I wonder if it will ever be allowed to be used outside custom connectors?

UPDATE: see Curt Hagenlocher’s comment below for some important information about this function.

9 thoughts on “Invoking M Functions In Parallel Using List.ParallelInvoke()

    1. Chris Webb – My name is Chris Webb, and I work on the Fabric CAT team at Microsoft. I blog about Power BI, Power Query, SQL Server Analysis Services, Azure Analysis Services and Excel.
      Chris Webb says:

      OK, so the message is not to use it yet, I guess.

  1. Hi Chris,
    Thanks for sharing this function, your posts are incredible!

    I saw this function was deprecated and I find no PowerQuery parallelism discussion on forums… Do we have any news about this? I know we have “Table.Partition()” and it’s sibblings currently, but honestly it doesn’t seem to work concurrently and I’m looking for more content.

    Thanks !

    1. Chris Webb – My name is Chris Webb, and I work on the Fabric CAT team at Microsoft. I blog about Power BI, Power Query, SQL Server Analysis Services, Azure Analysis Services and Excel.
      Chris Webb says:

      There’s no news unfortunately

Leave a ReplyCancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.