Reducing Label Fatigue: Auto-Applying Sensitivity Labels with Trainable Classifiers

The Problem: The "Manual Tagging" Bottleneck

Data protection policies often fail not because the rules are wrong, but because users forget to apply them. When a company relies on employees to manually select a sensitivity label (e.g., "Highly Confidential") from a dropdown menu, the result is usually a mix of under-labeled sensitive data and over-labeled mundane documents. This creates a gap in security and a burden on the end user.

The goal is to move from user-driven labeling to system-driven labeling without creating a flood of false positives that frustrate employees.

Moving Beyond Simple Keywords

Most administrators start with Sensitive Information Types (SITs), which look for patterns like credit card numbers or Social Security numbers. However, SITs struggle with context. A document containing a list of numbers might be a financial report or just a grocery list.

Trainable Classifiers solve this by using machine learning to recognize the "shape" and language of a document. Instead of looking for a specific string, a classifier learns what a "Contract," a "Financial Statement," or a "Medical Record" looks like based on sample data you provide. Once trained, these classifiers can trigger the auto-application of sensitivity labels across Exchange, SharePoint, and OneDrive.

How Auto-Application Works Across Workloads

Auto-labeling via classifiers does not behave the same way in every app. Understanding this distinction is critical for setting user expectations:

• Client-Side (Outlook, Word): Labeling often happens in real-time. As a user writes or saves a document, the classifier evaluates the content and suggests or applies the label.
• Server-Side (SharePoint, OneDrive): Processing is asynchronous. When a file is uploaded or modified, the Purview service scans the document in the background and applies the label. There is often a delay between the upload and the label appearing.

The Hierarchy of Labeling

By default, Microsoft 365 respects the user's intent. If a user has already manually applied a label, the auto-labeling engine typically will not override it. This prevents the system from "fighting" the user, though administrators can configure specific policies to force a higher-security label if the classifier's confidence is high enough.

Example: Automating "Project X" Confidentiality

Imagine you need to protect all documents related to a secret merger called "Project X." Keywords are too risky because "Project" is a common word.

1. Training: In the Microsoft Purview compliance portal, create a new Trainable Classifier. Upload 50–100 examples of actual Project X documents (seeds) and a similar number of non-related business documents to provide contrast.
2. Publishing: Once the model is trained and the confidence score is acceptable, publish the classifier.
3. Label Configuration: Edit your "Confidential" sensitivity label. Under the Auto-labeling settings, select the "Project X" classifier as the trigger.
4. Threshold Setting: Set the confidence threshold to "High." This ensures that only documents the AI is very certain about get labeled, reducing the risk of labeling a random memo as "Confidential."

Trade-offs and Limitations

Verifying the Result

To confirm the system is working without guessing, use these three verification steps:

• The Upload Test: Upload a document that matches your training set to a SharePoint library. Wait 15–30 minutes and check the File Properties to see if the label was applied.
• The Audit Log: Navigate to the Purview Audit logs. Filter for the Label applied event. This will tell you exactly which classifier triggered the action and when.
• Content Explorer: Use the Content Explorer in Purview to see a distribution of items labeled by your classifier. Compare this against a manual sample to calculate the precision (how many were correct) and recall (how many were missed).