The four costs everyone names
Ask people to name the biggest expense in retirement and you get the same four answers: taxes, healthcare, advisor fees, and market risk. I used to think the same thing. Run the numbers and one cost is doing more damage than those four combined, and it's the one nobody lists.
Taxes are real. If you're not careful about how you structure retirement, you can end up paying more in taxes than you did while working, because RMDs push your income up and cause your Social Security to get taxed. Sloppy withdrawals can trigger IRMAA penalties, which raise your Medicare premiums. Taxes are a big problem in retirement. They're still not the biggest cost you need to cut.
Healthcare comes with a scary number. Fidelity's study put average spending after age 65 at $172,500 per person, or $345,000 for a couple, and that doesn't include long-term care. It's a huge figure, and it isn't really a cost you can cut. You can plan for it, budget for it, and try to stay healthy. That's about all you can do.
Advisor fees deserve attention. A 1% fee on a $500,000 portfolio runs $5,000 a year, and over 20 years that's $100,000, more once you count the growth you missed on that money. You should absolutely know what you're being charged. It still isn't the thing that makes or breaks a retirement.
Market risk is the one people lose sleep over. If the market drops 30% the year you retire and you have to sell shares to pay the bills, you're doing permanent damage to your portfolio. Sequence of returns risk can do real damage. There's still one cost that does more.
Debt followed a generation into retirement
It's debt, and it costs a lot more than you think.
60% of Americans 65 and older are carrying debt right now. Back in the 1990s that number was under 35%. It almost doubled in one generation. The share of homeowners aged 64 to 79 who still have a mortgage went from 24% to 41% over the past 30 years. For those 80 and older, it went from 3% to 31%.
Your parents' generation paid off the house before they retired. That was the default back then. Somewhere along the way we stopped doing it. Did it get harder to do? Yes, absolutely. It is what it is.
Mortgages aren't the whole story. Nearly half of baby boomers carry a credit card balance, and Gen X averages $28,000 in auto loans. Over half the people 65 and older who have car loans are paying between $300 and $500 a month on that alone. People are walking into retirement with more debt than any generation before them, and many haven't done the math on what that debt costs once the paycheck stops.
The $450,000 sitting there to pay your lenders
Say you have $1,500 a month going to debt payments, a mortgage, a car payment, credit cards, or some combination of those. That's $18,000 a year in debt service. If your retirement income comes from your portfolio and you use the 4% withdrawal rule, you need $450,000 in savings to cover that debt service alone.
Sit with that for a second. You might owe $200,000 across all of it. To make the payments you need $450,000 set aside, more than double what you actually owe. That $450,000 isn't working for you. It isn't funding your travel and it isn't there for emergencies. It sits there so you can pay your lenders every month.
Debt payments are also fixed. Same amount, same interest rate, every single month. In a bad market year, when you should be pulling less out of your portfolio, you pull the same dollars, which means selling a larger number of shares. Groceries, travel, entertainment, you can cut those if you have to. Your payments are your payments.
Taxes make the number worse
That $1,500 a month doesn't cost you $1,500. If the money comes out of a traditional IRA or 401k, you pay taxes on it before it can pay a bill. Depending on your bracket and your state taxes, you might pull $1,800 or $1,900 to net $1,500.
And it doesn't stop there. The more you withdraw, the more your other income gets taxed, the more of your Social Security gets taxed, and you can trigger IRMAA surcharges that push your Medicare premiums up. It's a chain reaction. You pull money out to cover debt, your income goes higher, higher brackets kick in, and you end up owing tax on money you weren't planning to touch.
To actually put $1,500 in your lender's pocket each month, you might have to pull $2,200 to $2,500 out of your retirement accounts. Over a full year that's $25,000 to $30,000 instead of the $18,000 you budgeted for. Apply the 4% rule to that and you need somewhere between $608,000 and $750,000 in your portfolio to service the debt. All of it for something that could have been eliminated before you retired.
When keeping the debt makes sense
Paying off debt isn't the right move for every person in every circumstance. If you snagged a mortgage around COVID at those historically low rates near 3% or 4%, that's an amazing loan, and you can almost certainly get a better return on your money elsewhere. My rough tipping point is 5%. At 5% or lower it can make sense to keep the debt and buy income producing assets that cover the payment. Above 5%, just pay the mortgage off. You're getting a great risk free return at that point.
Be clear about what income producing means. It rules out the stock market, mutual funds, and anything that gets repriced daily. Those prices move up and down, and you don't want to bet your mortgage on that. You want a fixed rate that pays you the same amount every month, every quarter, or every year, so the money is there when the payment is due. There's a big difference between putting money into speculative markets and buying an asset that hands you a fixed income payment.
Here's an example. You owe $300,000 on your mortgage, and instead of taking $300,000 from your portfolio you leave it in the stock market. The market drops 30%, which has happened before. That $300,000 is now $210,000, and you still owe $300,000. That's what can happen when you rely on market appreciation to beat your interest rate. It's speculation at that point. Put that $300,000 into income producing assets instead, things like bonds, rental properties, or secured mortgage notes, and when the stock market drops you still have an asset paying you every month so you can make the mortgage payment.
A lot of people stop here, because pulling money out of a retirement account means taxes, plus penalties if you're under 59 and a half. That's true if you're paying the mortgage off. If you're buying income producing assets, the money can stay inside the retirement account. Individual bonds and bond ladders can generally be bought inside a 401k or IRA. Rental property and secured mortgage notes go inside a self-directed IRA, held with an SDIRA custodian. There are dozens of them. You roll your funds over from the traditional account and tell the custodian where you want those funds invested. It's a little detailed, and a custodian will walk you through it.
The critical point is this. If you do carry debt into retirement, make sure the rate is low and make sure you've bought assets producing income that covers the payment every month.
Which debt to pay off first
Credit cards come first, always. Their rates can run above 20%. If you've got multiple cards with different balances and different rates, there are two schools of thought. The waterfall method has you pay off the lowest balance card first, then roll that entire payment onto the next lowest balance. The other approach lines the cards up by interest rate and starts with the highest. That's the one costing you the most every month, so that's the strategy I recommend.
Car loans come next. Over half of people 65 and older with car payments are paying $300 to $500 a month, so clearing one frees up $3,600 to $6,000 a year for something else. HELOCs follow, because they're variable rate and they tend to spike at the worst possible time. The mortgage is last. It's usually the largest balance and usually the lowest rate as well.
Every payment you eliminate lowers your monthly expenses. That's less money coming out of your portfolio, less money you're taxed on, and more money that stays yours. It also makes the next debt easier to knock out, because you've freed up cash flow.
All of it comes down to needing less every month, whether that money comes from Social Security, a pension, income producing assets, or portfolio withdrawals. The less you need, the less you have to touch the portfolio, and the more of that $450,000 goes to travel, seeing the grandkids, and hobbies instead of debt service.