You can either start from the settings menu, or you can make a label while in an email. Click the gear icon on the top right next to the search box and open "See all settings." Click the "Labels" tab.

One such issue at launch was the lack of an option to not combine taskbar icons (also called ungrouping) and show labels for each window. Thankfully, starting with Windows 11 version 23H2 ...