Photoshop Convert to Smart Object loses vector mask in 2022+

The Resolution

In Photoshop 2022 and newer, converting a raster layer with a vector mask into a Smart Object results in the mask being rasterized. To preserve the vector editability of your mask, you must convert the layer to a Shape Layer first or ensure the content is encapsulated as a Vector Smart Object from an external source (like Adobe Illustrator) before applying the mask.

Explanation of Behavior

The "Convert to Smart Object" command is designed to encapsulate the current visual state of a layer into a nested PSB file. When a vector mask is applied to a raster layer, Photoshop treats the mask as a transparency modifier for those pixels. During the encapsulation process in recent versions, the software prioritizes the visual output (the resulting alpha channel) over the path data, effectively flattening the vector mask into a raster mask within the Smart Object.

Confirmed Facts vs. Assumptions

• Confirmed: Converting a raster layer with a vector mask to a Smart Object results in a rasterized mask in the resulting nested document.
• Confirmed: Shape Layers (which are natively vector) behave differently and maintain their vector properties when converted to Smart Objects.
• Assumption: This change in behavior is likely an optimization of the PSB encapsulation engine to ensure visual consistency across different hardware accelerations, though it is not explicitly detailed in the public release notes.

Recommended Workflow

To maintain vector control while gaining the benefits of a Smart Object, use this sequence:

1. Convert to Shape: If your mask is a simple path, convert the layer to a Shape Layer before converting to a Smart Object.
2. External Vectoring: Create the vector mask in Illustrator and place it as a Linked Smart Object. This ensures the path data remains external and fully editable.
3. Nested Editing: If you have already converted the layer, double-click the Smart Object thumbnail to enter the PSB. Check if the path still exists as a Work Path in the Paths panel; if it has been rasterized, you must recreate the path inside the nested document.

Verification Step

To verify if your specific layer type is being rasterized, run this test:

Create Raster Layer → Add Vector Mask (Pen Tool) → Convert to Smart Object → Double-click Thumbnail

If the mask in the nested PSB is a grayscale thumbnail rather than a vector path icon, the rasterization has occurred.

Diagnostic Detail Needed: Are you applying these masks to standard raster layers, or are you using "Fill" layers with vector masks? The behavior differs significantly between the two.