I've found that the following process causes the subsystem not to be shown as a sublibrary in the Library Browser.
- Select the subsystem, right-click and select Mask > Edit Mask (Ctrl + M)
- Open the Parameters & Dialogs tab
- Click one of the controls in the left column to add a control to the dialog.
- Click "Apply"
- Delete the added control.
- Click "OK" to close.
- Save the library.
- Open the Library Browser and refresh.
This doesn't seem like "correct" behavior, but it solves my problem.