I think there are variables at play that need to be considered.
First, do you genuinely enjoy hanging out with this person? If so, and the money is the problem, just go do things that don't cost much, or just pay things separately.
If it's an issue of them not making the effort to reciprocate your efforts, when they are capable of doing so, then I would have a problem with that.
I have a friend who I get along with, but it's immensely hard to hang out with him. He never wants to go out with other people I might know, it's pretty much mutual high school friends only, or the two of us. He rarely drives out to my neighborhood, on the pretense that gas is expensive, his car is in poor condition, and parking is a bitch to find when he goes back home.
Gas is expensive for me too, and parking is most definitely a bitch when I visit him. In fact, for a while I would only ride my bike, due to the parking situation, but then we can't go anywhere, and he doesn't like people to be in his apartment because he shares it with his dad. Whatever his reasons are, it's really hard to just hang out with him. It always feels like a struggle and there are barriers to everything.
Go get food? Restaurants aren't cheap enough. Okay, fine, fast food? Food is too shitty. Let's go somewhere, I have to drive every time. Okay, let's hang out at home, can't, dad's home. Okay, then come over and chill at my place, too far away, don't want to drive.
I'm sure you guys see the picture by now.
I am not cutting him out of my life, he is still my friend. But for the time being, I will not be going out of my way to socialize with him.
OP, I suggest you do the same, and assess how much you really enjoy hanging out with your friend. If there is a genuine friendship there, maybe it's worth telling them how you feel. If the friendship is true, they will at the least accept your concerns and not hold them against you. If they are really your friend, they will make adjustments.
If they are really offended, you didn't lose a friend, and you can move on without a second thought.