Some people have a perception of age as quantized. Either you're 22 or you're 23. No in-between. I think they like to quantize age in order to rationalize the concept of age in their head. SO, on your birthday, you are all of a sudden one year older.
I like to think of age as a continuous concept, not a discrete one. I do fall into the trap of saying that someone is 54 years old, etc, but that really does make it easier to rationalize. For my personal life, though, I don't think that I turned a year older on my birthday. I turned a day older. Or even less than that. You get older with every breath. Time only moves in one direction (around this part of the universe).
When people try to make you "older" aka more mature/with more life experience as a result of having a birthday, I think they are just using an expectations operator...they are saying that at a certain age, you should have had this many of that sort of experiences (on average). If there weren't such a high information cost, they could much more accurately pin down what they're looking for by knowing all of your accomplishments, failures, and other life experiences (travels, deaths of those close to you, exposure to violence or culturally enriching experiences, etc).
So, maybe you do get older with every breath...but more importantly, you get older when you do something or when something happens to you. When you experience something. You grow up a bit after the first time you ride on a plane by yourself. Or after your first kiss. Or after your 100th test this semester is over.