Microsoft Graph JSON batching using PowerShell

Microsoft Graph JSON batching using PowerShell

 , , , , , , ,

I’m working on a new big(ger) project that is related to working with groups and Intune. However, I quickly realized that depending on the customer environment I would have to chew through a lot of data. It also has to be fast. As I had worked little with Graph and usually with the precision of an axe I asked the community. Thanks to a mention by Alec 1 I came across JSON batching 2 which allows for 20 requests to Graph at once and (using dependsOn3) with interdependence. While there is already some content 4 5 6 around this (and a video 7 and an module 8) I wanted to share my approach on this, especially showing the vast speed difference this provides.

JSON batch processing requires some learning

Yes, it really does, but lets lead with an example first. JSON itself already has what feels like a finicky syntax. If I wanted to be able to work through thousands of items, I had to think in terms of speed while staying within the limits of Graph9. You may have already noticed what filters can do for you if you’re used to working with things like Get-ADUser. So here’s a stern reminder: Before you dive into JSON batching, try using $filter and $select first. The latter in particular will actually reduce your ‘resource unit cost’, making a “429 Too Many Requests” error less likely 10 . Here’s an example of the data going through all stages of the request..

# Create a [PSCustomObject] and [System.Collections.ArrayList] - the arraylist should never contain more than 20 items.
$PSObject = [PSCustomObject]@{
        id      = $ID
        method  = $Method
        URL     = $URL
        headers = $Headers #Not mandatory
        body    = $Body #Not mandatory
    }
$Array = [System.Collections.ArrayList]
$Array.add($PSObject)
# However, all requests need to be surrounded by a 'requests' item.
$BatchRequestBody = [PSCustomObject]@{requests = $Array }
# Now all we need to do is convert this into our request body - don't forget that be default ConvertTo-Json doesn't go deeper than two items. 
$JSONRequests = $BatchRequestBody | ConvertTo-Json -Depth 4 
# Send the requests and make sure to catch the results
$Results = Invoke-MgGraphRequest -Method POST -Uri 'https://graph.microsoft.com/v1.0/$batch' -Body $JSONRequests -ContentType 'application/json' -ErrorAction Stop

The resulting JSON looks like this. In the example below, I have omitted PUT and PATCH as methods.

{
    "requests": [{
            "id": 0,
            "method": "POST",
            "URL": "/groups/",
            "headers": {
                "Content-Type": "application/json"
            },
            "body": {
                "MailEnabled": false,
                "MailNickName": "ROLwFDBuZWyVkCz",
                "description": "Group-Documentation Batch",
                "securityEnabled": true,
                "DisplayName": "GD | ROLwFDBuZWyVkCz"
            }
        }, {
            "id": 1,
            "method": "DELETE",
            "URL": "/groups/9a0ce429-daad-4308-9b4d-478f38ef1da7",
        }, {
            "id": 2,
            "method": "GET",
            "URL": "/groups/?$select=displayName,id&$filter=startswith(displayName,'GD | ')",
        }
    ]
}

Note that not every request needs to have a headers or body attribute. Keep in mind that some body attributes can be really complex, so using 4 as a depth may not be enough. To keep the post shorter, I’ll just show a snippet of a response for the new group created by the JSON.

"id": "10",
    "status": 201,
    "headers": {
      "Location": "https://graph.microsoft.com/v2/d36eed9c-6ef6-45dc-9813-0061b0b8030d/directoryObjects/38e5796b-376a-4034-8ad1-c90628363345/Microsoft.DirectoryServices.Group",
      "OData-Version": "4.0",
      "Cache-Control": "no-cache",
      "Content-Type": "application/json;odata.metadata=minimal;odata.streaming=true;IEEE754Compatible=false;charset=utf-8",
      "x-ms-resource-unit": "1"
    }
  },
  {
    "body": {
      "onPremisesSamAccountName": null,
      "groupTypes": "",
      "classification": null,
      "createdDateTime": "2023-09-11T20:40:55Z",
      "proxyAddresses": "",
      "preferredLanguage": null,
      "onPremisesNetBiosName": null,
      "onPremisesLastSyncDateTime": null,
      "resourceProvisioningOptions": "",
      "membershipRule": null,
      "isAssignableToRole": null,
      "mail": null,
      "id": "74279ebc-5e8d-4391-80d1-f701c170f787",
      "securityIdentifier": "S-1-12-1-1948753596-1133600397-33018240-2281140417",
      "deletedDateTime": null,
      "creationOptions": "",
      "mailEnabled": false,
      "renewedDateTime": "2023-09-11T20:40:55Z",
      "displayName": "GD | BoCsvqQDSXriRcZ",
      "preferredDataLocation": null,
      "theme": null,
      "onPremisesProvisioningErrors": "",
      "expirationDateTime": null,
      "serviceProvisioningErrors": "",
      "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#groups/$entity",
      "mailNickname": "BoCsvqQDSXriRcZ",
      "resourceBehaviorOptions": "",
      "onPremisesDomainName": null,
      "description": "Group-Documentation Batch",
      "onPremisesSecurityIdentifier": null,
      "securityEnabled": true,
      "onPremisesSyncEnabled": null,
      "membershipRuleProcessingState": null,
      "visibility": null
    }

Gotchas

Here are some of the gotchas that I had when working with JSON batching for the first time.

  • Each request in your requests will be processed randomly unless the dependsOn attribute is used.
  • Responses are most likely out of order and need to be matched using ID from the original request.
  • PowerShell makes it incredibly easy to export objects that use the key = value format to JSON – nesting included. All you need is ConvertTo-JSON
  • If using a larger GET request, paging 11 will occur. The nextLink required to request the next ‘page’ will be an attribute of the response with the matching ID (and keep the optimizations in mind!).
  • If you want to use advanced queries you need to add an additional header called ConsistencyLevel depending on the query type.12
  • headers don’t require a body attribute – but vice versa, because you need to specify the Content-Type.

Optimization

Here’s a couple of things that I found that make batching work best.

  • Try grouping similar requests. If you need to delete 2000 groups, you should do just that. Don’t queue multiple different items into your next batch as shown above. This is especially true for the runtime of said requests, because…
  • The slowest request in your requests (couldn’t resist) determines the response time of your batch. A response is not sent until the slowest item has been processed.
  • If you are working with GET, be sure to also use $select in your requests to keep the ‘request unit cost’ low.

How fast can we get?

Speed comparison between JSON batching and single request processing

For the sake of keeping the post short: Pretty fast! Especially if you keep the optimizations in mind. If you would like to try for yourself wrap your code in a stopwatch as shown in the following code. I really like learning new things regarding PowerShell and this exercise has helped my code come along plenty! The example on the left is the difference between using JSON batching and the Microsoft Graph PowerShell SDK build in New-MgGroup. So yes, you’re looking at a tenfold increase in speed (example size used: POST 250 Groups)!

$Stopwatch = [System.Diagnostics.Stopwatch]::StartNew()
Get-GroupMembers #aka your function here
$Stopwatch.Stop()
$Stopwatch.Elapsed

Conclusion

If you’re like me and want to use PowerShell as much as possible for compatibility reason because you’re as stubborn as I am you should look into JSON batching. It takes a bit of getting used to, the fact you need to convert to JSON every time is annoying and on top of that, there’s no built-in function in the SDK that provides this functionality out of the box. Luckily there’re some good examples out there to draw inspiration from.

Footnotes

  1. awebie (Alec Weber) (github.com) Again, Thanks Alec! ↩︎
  2. Combine multiple requests in one HTTP call using JSON batching – Microsoft Graph | Microsoft Learn ↩︎
  3. Combine multiple requests in one HTTP call using JSON batching – Microsoft Graph | Microsoft Learn Explains dependsOn ↩︎
  4. Batching w/ Microsoft Graph API – Part II | console.log(‘Charles’); (charleslakes.com) ↩︎
  5. Making batch calls to the Microsoft Graph API using the PowerShell SDK · A blog @ nonodename.com ↩︎
  6. Batching Microsoft Graph API Requests with JSON Batching and PowerShell – Kloud Blog ↩︎
  7. Microsoft Graph Batching – YouTube ↩︎
  8. PowerShell Gallery | GraphFast 2.0.8 – This module is out of date in terms of authentication ↩︎
  9. Microsoft Graph service-specific throttling limits – Microsoft Graph | Microsoft Learn General Information ↩︎
  10. Microsoft Graph service-specific throttling limits – Microsoft Graph | Microsoft Learn Information about resource unit cost in identity requests ↩︎
  11. Paging Microsoft Graph data in your app – Microsoft Graph | Microsoft Learn ↩︎
  12. Microsoft Graph advanced queries for directory objects are now generally available – Microsoft 365 Developer Blog ↩︎