How To Actually Tax The Rich

SUMMARY

Scott Galloway and Ed Elson interview Ray Madoff, a Boston College Law School professor and author, to analyze how the wealthiest Americans legally avoid taxes and the systemic flaws enabling this. The discussion explores the ineffectiveness of current estate and income tax structures, the mechanics of wealth preservation, and potential reforms to address inequality.

MAIN POINTS

• The estate and gift tax raises only a fraction of what is expected from the wealthiest Americans due to widespread avoidance. [0:00]
• Ray Madoff explains that while the top 1% pay a large share of income taxes, the wealthiest often avoid income taxes entirely by not reporting taxable income. [7:01]
• Ultra-wealthy individuals like Buffett and Bezos take minimal salaries, instead relying on appreciating assets and borrowing against them to fund their lifestyles. [12:02]
• Discussion of possible reforms includes taxing appreciation at the point of transfer and skepticism about the effectiveness of an alternative minimum tax. [18:02]
• Inheritance, gifts, and life insurance proceeds are largely untaxed, relying on an estate tax system that has become ineffective due to loopholes. [25:00]
• A campaign in the 1990s led to the estate tax being undermined, with Congress ceasing to close loopholes and the tax losing its effectiveness. [29:01]
• Madoff argues for eliminating the estate tax and instead taxing inheritances as ordinary income, with reasonable exemptions for recipients. [35:01]
• Capital gains are taxed at lower rates than labor income, and the panel agrees on the need to equalize these rates while adjusting for inflation. [39:01]
• Madoff outlines three key reforms: taxing unrealized gains at transfer, repealing the estate tax in favor of inheritance taxes, and equalizing capital gains and income tax rates. [44:00]
• A federal wealth tax is deemed impractical due to constitutional concerns and valuation difficulties, and international cooperation is needed for effective corporate taxation. [48:02]
• The primary fiscal problem is insufficient tax revenue, not excessive government spending, as the richest Americans avoid taxes on investment gains and inheritances. [53:02]
• The rise of AI and automation threatens tax revenue from labor, highlighting the need to tax capital more effectively in the future economy. [57:01]
• Public education is essential to build support for tax reforms that address the disparity between taxes on labor and those on investments and inheritances. [59:01]

DETAILED ANALYSIS

The episode opens with a stark statistic: despite the top 1% of Americans holding $55 trillion in wealth, the estate and gift tax system—ostensibly designed to tax large transfers at a 40% rate—collects only $28 billion, representing a negligible 0.06% of the total. This discrepancy is not due to high exemptions or low rates, but rather the ability of the wealthy to circumvent the tax entirely. The conversation quickly pivots to the broader issue of wealth inequality, noting that the top 1% now control about a third of the nation's wealth, while the bottom half controls just 3%.

Audit rates for the ultra-wealthy have also plummeted, further exacerbating the problem.

Ray Madoff, a leading expert on tax law and author of 'The Second Estate,' is introduced to dissect these issues. She addresses a common rebuttal: that the top 1% pay 40% of federal income taxes. Madoff clarifies that this statistic refers to high-income earners, such as lawyers and bankers, not necessarily the wealthiest individuals.

The ultra-wealthy often avoid income taxes by minimizing taxable income, placing them among the 40% of Americans who pay no income tax at all. For example, the average effective tax rate for the 400 wealthiest Americans is 24%, compared to 30% for the average American.

The mechanics of wealth accumulation and preservation among the ultra-wealthy are then explored. Unlike most Americans, who earn income through labor and are subject to both income and payroll taxes, the wealthiest individuals derive their fortunes from the appreciation of assets. Notably, figures like Warren Buffett and Jeff Bezos take minimal salaries—Buffett has never earned more than $100,000 in salary and bonus, while Bezos has kept his salary at $82,000, even qualifying for the child tax credit.

Instead, they rely on the increasing value of their stock holdings. To fund their lifestyles, they borrow against these appreciating assets, securing favorable loan terms due to their immense collateral. These loans can be continually rolled over, and the interest payments are modest relative to their wealth.

This strategy allows them to avoid realizing taxable income, thereby sidestepping both income and payroll taxes.

The discussion moves to potential policy solutions. Madoff is skeptical of the alternative minimum tax (AMT) as a fix, arguing that it primarily addresses deductions and does not tackle the core issues of untaxed appreciation and inheritances. She advocates for taxing appreciation at the point of transfer—whether by gift, sale, or inheritance—similar to the Canadian model.

This approach would ensure that gains are taxed when ownership changes hands, rather than allowing assets to pass untaxed across generations. Madoff also criticizes the current estate tax structure, noting that its effectiveness has been eroded by decades of loopholes and avoidance strategies. A pivotal moment came in the 1990s, when a campaign funded by wealthy families rebranded the estate tax as the 'death tax,' making it politically toxic and halting Congressional efforts to close loopholes.

As a result, the estate tax now raises a trivial amount of revenue and serves more as a shield for the wealthy than a meaningful source of taxation.

Madoff proposes eliminating the estate tax entirely and instead integrating inheritances, gifts, and life insurance proceeds into the income tax system. This would tax recipients based on their income bracket, with reasonable exemptions (such as allowing each individual to inherit $1–2 million tax-free). Such a reform would align the treatment of inherited wealth with other forms of income, closing a significant gap in the current system.

Attention then turns to the disparity between capital gains and labor income tax rates. Historically, capital gains were taxed at higher rates than labor income, but this changed in recent decades. The panel agrees on the need to equalize these rates, with Madoff suggesting an adjustment for inflation to prevent taxing illusory gains.

She points out that arguments against raising capital gains taxes—such as claims that it would reduce investment—are only persuasive because current loopholes allow gains to go untaxed if assets are not sold. Closing these loopholes would neutralize such concerns.

Three primary reforms are outlined: taxing unrealized gains at the point of transfer, repealing the estate tax in favor of taxing inheritances as income, and equalizing capital gains and ordinary income tax rates. Madoff dismisses the feasibility of a federal wealth tax, citing constitutional challenges and the practical difficulty of valuing complex assets. She also notes that international cooperation is essential for effective corporate taxation, as companies can exploit differences between national tax systems.

The conversation addresses the broader fiscal context, emphasizing that the main issue is insufficient tax revenue rather than excessive government spending. In 2024, the federal government collected just under $5 trillion in revenue but spent $6.8 trillion, resulting in a $1.8 trillion deficit. Meanwhile, the richest 1% held $55 trillion in wealth, much of it untaxed. Madoff argues that bringing investment gains and inheritances into the tax system could easily cover such deficits.

Looking ahead, the rise of artificial intelligence and automation poses new challenges for tax policy. As labor's share of the economy declines and capital's share grows, reliance on income and payroll taxes will become increasingly untenable. Madoff stresses the importance of adapting the tax system to capture revenue from capital, whether through corporate taxes, business taxes, or new mechanisms targeting emerging technologies.

Finally, the discussion underscores the need for public education to build support for these reforms. Many Americans are unaware that the wealthy can avoid taxes on investments and inheritances, while labor income is heavily taxed. Educating the public about these disparities is seen as a crucial first step toward enacting meaningful change.

LINKS

• Ray Madoff's book, The Second Estate: How the Tax Code Made an American Aristocracy
• Prof G Markets Tour tickets and dates
• Prof G Markets newsletter subscription
• Order Notes On Being A Man
• Scott Galloway's Instagram
• Ed Elson's Instagram
• Ed Elson's X (Twitter) profile
• Ed Elson's Substack
• Prof G Markets on Spotify
• Prof G Markets on TikTok
• Prof G Media homepage
• Monarch personal finance app
• Navan business travel and expense management
• SoFi Small Business Loan Marketplace

KEYWORDS