Setting up Microsoft 365 to use the Remove-DuplicateItems script

The Remove-DuplicateItems PowerShell script by Michel de Rooij is a powerful script used to clean up duplicate messages within Microsoft Exchange mailboxs, including Microsoft Office 365.

The script has been around for many years and Michel has done a great job at keeping it up to date and optimizing it. While they do provide some instructions that are kept up to date, I found they were lacking when it came to using the script with Microsoft 365 Online.

The script is easy enough to use if you can use Basic auth (just a username and password) but for the most part, you must use Modern auth (with multi-factor authentication) these days, which makes it harder to run from a PowerShell script that doesn't support that natively.

These instructions are here as my notes on one way (there are several valid ways) to get it set up and working with Office 365.

Download the script and its supporting files

Download the PowerShell script and the supporting files from GitHub and save them all together in a folder.

You may be able to get away with only the PowerShell script and not the .dll files if you run the following commands:

Register-PackageSource -Provider NuGet -Name nugetRepository -Location
Install-Package Exchange.WebServices.Managed.Api

This downloads the files from and installs them in C:\Program Files\PackageManagement\NuGet\Packages\

While I did do this, I also downloaded the copies from GitHub because I had more success getting those to work. Possibily because I'm missing a command to get the Microsoft.Identity.Client files?

For what it's worth the files from NuGet were identical to the ones in Michel's GitHub. (If I made sure the line endings of the .xml file matched -- CR LF vs just LF)

Register an "app" in Entra Identity (Azure AD)

Load up what used to be Azure AD, but is now the Microsoft Entra admin center

Register an application for the script to use

Under Identity > Applications > App registrations

Tap New registration at the top

Name it whatever you want, I called mine Remove Duplicate Items Script

Supported account types: Accounts in this organizational directory only (Single tenant)

You can leave the Redirect URI area blank since we aren't actually making a web app. Tap Register

Take note of the Application (client) ID and the Directory (tenant) ID as we will need both of these later.

Assign API permissions

We'll have to give the script permissions to access our mailboxes in order to run.

While looking at your newly created app registration, tap API Permissions. You'll see by default it already has permissions to Microsoft Graph's User.Read.

Tap Add a permission

Choose the APIs my organization uses tab and search for Office 365 Exchange Online

We are going to use Application permissions and give it access to full_access_as_app so it can access the full mailbox. Tap Add permissions

You'll need to tap the Grant admin consent for organization in order to actually grant that permission.

Create a secret

Still looking at the App registration created above, tap Certificates & secrets.

Under Client secrets, tap New client secret

Enter any description and set an expiration date for this secret. Remember, this app has full permissions to all your mailboxes on Exchange Online, so realize how important this secret is to keep safe! Copy the secret value that it creates. After you leave this screen, it will not show it to you again (but you can create another one later).

You can use certificates as well with the script, but these instructions don't cover that scenario.

Running the script

To run the script on Exchange Online, include the TenantId, ClientId, and Secret from above, and set Server to

Secret must be a PowerShell SecureString, so you can use the command $secret = ConvertTo-SecureString -String "the secret value that was given earlier" -AsPlainText -Force

Then, for example: .\Remove-DuplicateItems.ps1 -Identity -Server -TenantId 00000000-0000-0000-0000-000000000000 -ClientId 00000000-0000-0000-0000-000000000000 -Secret $secret -Report -Verbose -WhatIf

Obviously replace the secret and ids with your own. The -WhatIf flag lets you see what the script would do without it actually removing any messages until you are ready.

When you are done with the script, feel free to delete the app's Client Secret or even the entire App registration if you never plan on running the script again, or don't mind setting it back up when you do. Because remember that anyone with your tenant id, app's client id, and secret has full access to your organization's emails.

Last updated: 2023-07-24