In The Problem of Political Authority, Michael Huemer tells us that “libertarian political philosophy rests on three broad ideas”:
i) A non-aggression principle in interpersonal ethics. Roughly, this is the idea that individuals should not attack, kill, steal from, or defraud one another, and in general, that individuals should not coerce one another, aside from a few special circumstances.
ii) A recognition of the coercive nature of government. When the state promulgates a law, the law is generally backed up by a threat of punishment, which is supported by credible threats of physical force directed against those who would disobey the state.
iii) A skepticism of political authority. The upshot of this skepticism is, roughly, that the state may not do what it would be wrong for any nongovernmental person or organization to do.
He then asserts that (i) is not really controversial:
The main positive ethical assumption of libertarianism, the nonaggression principle, is the most difficult to precisely articulate. In truth, it is a complex collection of principles, including prohibitions on theft, assault, murder, and so on. I cannot completely articulate this principle or set of principles. Fortunately, it is not the locus of disagreement between libertarians and partisans of other political ideologies, for the “non-aggression principle,” as I use the term, is simply the collection of prohibitions on mistreating others that are accepted in common sense morality. Almost no one, regardless of political ideology, considers theft, assault, murder, and so on morally acceptable.
The crux of the disagreement is actually (iii):
Libertarians are skeptical about authority, whereas most accept the state’s authority in more or less the terms in which the state claims it. This is what enables most to endorse governmental behavior that would otherwise appear to violate individual rights: nonlibertarians assume that most of the moral constraints that apply to other agents do not apply to the state.
If Huemer wants to persuade nonlibertarians, the phrase “most of” is key. Nonlibertarians usually assume that some moral constraints that apply to other agents do apply to the state. What are those moral constraints? What line do states—even our own—have cross to forfeit moral legitimacy? There is only one clear-cut candidate: killing their own civilian citizens. States can kill civilians as long as they are citizens of other countries—and they can kill their own citizens as long as they’re combatants (e.g., rebel soldiers, terrorists, etc.). But when state X deliberately kills noncombatants who are officially citizens of X, most nonlibertarians talk like libertarians: “That’s cold-blooded murder—even if you’re wearing a uniform and following the leader’s orders.”
Question: Why precisely is this the place that so many nonlibertarians draw the line? The moral distinction between combatants and noncombatants is plain enough. When you kill a combatant, you can argue self-defense. But the moral distinction between citizens and noncitizens is not plain. After all, suppose a government (a) strips a hated civilian subgroup of its citizenship, and then (b) kills them? Is that substantially less wrong than doing (b) without (a)?
A cosmopolitan nonlibertarian might grant the wrongness of killing any noncombatant. But this is still an odd place to draw the line of moral legitimacy. Suppose government A randomly murders .001% of its people every year, government B strictly enforces a 99.99% income tax, and government C strictly requires everyone to perform 50 years of forced labor. If we condemn A, it is hard to see how we could fail to similarly condemn B and C. After all, who wouldn’t prefer to take his chances in A rather than endure the certain misery of B or C?
My point is that there’s more than one way to skin a cat—or a fellow human being. Many policies are as bad or worse than murder, if they’re sufficiently broadly or strictly enforced. Libertarians may sound like cranks when they group “life” with “property” or “liberty.” But anyone willing to equate government killing of civilians with murder should at least be open to equating taxation with robbery and conscription with slavery.
 Killing combatant citizens, strangely, inspires far more controversy than killing noncombatant non-citizens.
If forced labor seems irrelevant to modern public policy, note that it continues to inspire nostalgia in many Americans, as long as the victims are healthy young males and their “employer” is the military.