Feta is a PDO (Protected Designation of Origin) product from Greece. It tastes quite different from many the other cheese.
Primarily due to the fact that feta is either made of 100% sheep milk or mixes with (up to 30% of) goat milk. And these animal can only be raised in Greece, which the milk produced will contain the region-specific bacteria.
With that, it gives feta a unique taste unlike those made from cow milk, such as mozzarella and parmesan.
Sheep milk does not have the strong taste like goat milk, but it is richer and buttery. See comparison of Gorgonzola vs goat cheese here.
If you ate aged cheese before, you probably knew the umami taste. The umami taste is often described as savory — salty, spicy, and tangy.
The longer it aged, the stronger the taste.
When making the cheese, the lamb rennet will be added to the milk to aggregate into curd. The rennet contains protein-cutting enzymes which later cut the peptides compound off from the casein, the milk protein. The peptides are what give the bitterness/ umami taste for the cheese (see pictures in this post).
Other than that, the bitterness can also be contributed by glutamic acids. As the maturation of cheese involves breaking down of casein, which later leads to an increase of glutamic acid in the cheese — thus bitterness (1).
You may ask, why does it taste salty then?
The salty comes during feta making, where the cheese will be brined for days or months in wooden barrel.
Did you know why feta cheese smell like feet? Read the answer here.