When people hear I’m a libertarian, they often assume that I am heavily into rights. There is a sense in which I am—I care about rights--but I do not consider myself a “rights guy." I consider myself someone very concerned with what rights are for (or what they do for us). The primary reason I don't consider myself a "rights guy" has always been that I think there are too many intractable debates surrounding rights talk.
Consider the following:
A. After every mass shooting there are demands for gun control. Presumably, people think limiting rights to guns would prevent further gun violence.
B. There are often calls to limit speech on college campuses when controversial figures are invited. Presumably, people think limiting rights to speech would prevent some harm or offense by controversial speech.
With regard to (A), I assume it's clear that those killed or wounded in mass shootings have their rights violated. I'm also sure that whether or not gun control would limits anyone's rights is less clear. Some will loudly proclaim yes (responsible gun owners shouldn't be penalized because of some bad actors); others will loudly proclaim no (you have no right to it in the first place). (It will also be debated whether such limits would prevent the violence.)
With regard to (B), I assume all will agree that speech--ignoring cases of fraud, libel, slander, or incitement to violence--is valuable. I'm also sure that whether or not limiting such speech infringes rights will be less clear. Some will loudly proclaim yes (say what you want; as long as you don't limit me, you shouldn't be limited); others will loudly proclaim no (you have no right to say X, Y, or Z).
The disagreements about rights limitations in A and B can be intense and are not likely to result in any good resolution. But disagreement about rights is not confined to questions of what we have rights to (still, see Wellman). Political philosophers and philosophers of law may argue more about what puts the meat on the bones of rights than about what we have rights to. There are two dominant theories (FWIW, I suspect neither is right): some believe that rights serve to protect our ability to choose (see Hart) while others believe that rights protect our interests (see Raz). And, lest we forget, some of our unfortunate utilitarian brethren may well think that rights are nothing but "nonsense upon stilts" (see this about Bentham).
We have disagreement about the nature of rights and we have disagreement about what we have rights to. There are also disagreements about whether rights can be sacrificed (or are inalienable), whether they can be overridden (and if so, by what), whether they are natural or social constructs, and more. Despite all these controversies surrounding rights, rights are supposed to be somehow “sacred,” "basic," or “fundamental.” They are sometimes thought to be the ground upon which our social and political order stand. They are sometimes thought to be that which the entire social and political order should protect. All of this despite the fact that what rights are, what we have rights to, etc are questions subject to intense and significant disagreement.
As I hope is well understood, disagreement doesn't mean much (see this old BHL post of mine). It can be true that X and there be disagreement about X. It might be that Y is morally good and there be disagreement about whether Y is morally good. Facts are facts, but not all facts are known by everyone--indeed, some facts are known by no one. The presence of disagreement about a right to gun ownership or about a right to speech does not mean there is no such right (nor that there is such a right). Still, I have to wonder if we can’t improve the discourse by doing away with rights talk.
My own view is that rights talk is really just shorthand for something of genuine importance. On one view, rights are just a way of indicating that persons are of moral significance such that there are normative constraints surrounding them (such constraints mean they are not to be interfered with in a variety of ways). (See Mack; from a different angle, see Rainbolt). So why don't we just talk about that moral significance or those normative constraints?
I would suggest that one way to improve our discourse about guns and speech both is to talk of a simple normative constraint: the constraint against doing harm (this is a principle of non-maleficence; we can also talk of the harm principle, which indicates interference is permitted with an actor that has caused, is causing, or will cause harm). This will push us to discuss what is meant by harm and whether what occurs in any specific case is in fact a harm. That seems to me exactly the way we should go.