The Pen Tool (G) gives you precise control for creating custom mask shapes with straight lines and smooth curves exactly where you need them.
Steps
Select your target layer and press G to activate the Pen Tool. Click to add straight points.

Click and drag to create bezier handles for smooth curves.

Close the path by clicking the first point you made. The mask appears under the layer with a Mask Path you can edit or adjust anytime.

One more thing
If you accidentally create a shape layer instead of a mask, deselect everything with F2, then re-select the layer you want masked and draw again with the Pen Tool.
Related how-tos
How to use trim paths on shapes in After Effects
How to fill shapes with color in After Effects
How to create a circle mask in After Effects
Need more help? Explore all our How-tos in one place.