I’m sure you already know how to configure scheduled refresh for your semantic models in Power BI. While the options you have for controlling when refresh takes place are generally good enough – you can configure daily or weekly refreshes and set up to eight times a day for refreshes to take place – there are some scenarios it doesn’t work for, such as monthly refreshes. Up to now the workaround has been to use Power Automate to trigger refreshes (see here for an example) or to call the refresh API from another application. Now, with Fabric, you have a much better option for scheduling refreshes: Data Pipelines.
The semantic model refresh activity in Fabric Data Pipelines was released last year and at the time all the focus was on the extra control it gives you over what happens in a refresh: it allows you to refresh individual tables or partitions and control the amount of parallelism, for example; it also allows you to schedule your refresh after other ETL operations, which in Fabric will probably also be scheduled using Pipelines, have completed. What I want to draw your attention to is the fact that Fabric Data Pipelines use the new Fabric scheduler which offers more flexibility for controlling when they run.
There’s some documentation here on how to schedule a Data Pipeline run but it’s very straightforward to use. I created a Fabric Pipeline consisting of a single semantic model refresh activity like so:
..then hit the Schedule button on the toolbar, selected “Monthly” on the Repeat dropdown and configured it to run on the first Sunday of the month:
Apart from the option to run on the Nth instance of a given day of the week you can also run the Data Pipeline on a given day number of the month; you can also run every N months, add up to 10 times to run per day, and set a start and end date.
There are two other options for scheduling that aren’t available in the scheduler for semantic models: the ability to run the Data Pipeline every N hours or minutes.
Be warned though: refreshing your semantic model every few minutes is dangerous because it can result in excessive CU consumption on your capacity and maybe even throttling if you’re not careful.
The same options to run a Data Pipeline daily and weekly that exist in the scheduler for semantic models with one notable limitation: the semantic model scheduler allows you to specify up to 48 times to refresh every day for models stored on a Premium/Fabric capacity, whereas the Fabric scheduler used by Pipelines only allows you to specify 10 times per day.
Of course you need a capacity to be able to use Fabric Data Pipelines but orchestration activities only cost 0.0056CU hours per activity run, so using Pipelines to refresh a semantic model in this way will only use a tiny fraction of even the smallest capacity’s resources.
Even if you don’t think you’re interested in anything that Fabric offers beyond Power BI, it’s features like this that, in my opinion, still make it worthwhile to flip the switch to enable Fabric and make your life as a Power BI developer easier.
@Chris – It’s great. However, organizations nowadays looking event driven dataset refresh for Power BI rather than scheduled one. Do we have any functionaity in MS Fabric where once the DBX Data Load gets complete, it can automatically trigger the Semantic Model refresh without using Power BI APIs?
Yes – take a look at https://learn.microsoft.com/en-us/fabric/real-time-hub/explore-fabric-job-events for example
Happy to see that the refreshes go down to table and partition level. If only there were anything comparable in the Dataflow area as well – ability to refresh individual buckets etc.
Can pipelines monitor refresh status?
It depends on what you mean by “monitor refresh status” – you can make the refresh activity wait until a refresh has completed.
Hey Chris, is there any word on when you will be able to refresh specific HIDDEN tables using the pipeline action? I tried to manually code it into the definition, but it always refuses to accept a hidden table.
Are these refreshable via the REST API call?
Do you mean can you run the Pipeline via the API? If so, then I think yes: https://learn.microsoft.com/en-us/fabric/data-factory/pipeline-rest-api#run-on-demand-item-job
yes, you can call any hidden table using the API – so you can use a notebook to refresh a hidden table, but my point is that limiting data factory to only visible tables is not ideal.
I’ve been told it’s possible to refresh hidden tables using Pipeline expressions – have you tried that?
i did try this, but every time i entered the expression value, it would just disappear after hitting save. i tried as a string, json object, the array of tables as objects, and the json as a string, and couldn’t get it ðŸ˜
Thank you, Chris. I’m having trouble setting up a refresh for a specific table. The pipeline gets stuck in the loading dataset table list step, and nothing happens afterward. Is it just me, or is there something I might be missing?
I think there’s a bug here. I’ll ask someone to look into it.
Just checking – what storage mode does your model use?
Import
I’ve been told this is a known issue. Hopefully it will be fixed soon.
Chris, I refreshed connection based on organizational account, seems it work now. I need to refresh hidden tables (related to RLS and user access), but they are not in drop down list for table refresh. I wonder if you could provide feedback to product group.
Thanks so much to Chris for this informative post! Im a PM on the Power BI team supporting the Private Preview for this feature with Penny. As an update, we are interested in hearing feedback from MVPs, Partner / Enterprise Voice, and the Data Community on how we can improve this experience of using Fabric Data Pipelines with Power BI for semantic model refresh. If you are interested in sharing your scenarios or use cases, feel free to fill in this form and we will followup with next steps while we are making improvements to the feature. https://forms.office.com/r/yzmG9hrKLS
Hey Chris,is there a way to send two seperate refresh request simultaneously for two partition A and B of same time.but they are separate requests via api.I have a scenario where I want user to give ability to refresh specific portion of the table.Right now i am queuing them ,but I would be great if we can have concurrent refresh request
You can’t send two separate refresh requests. You need to send a single request that refreshes the two partitions in a single batch.