ack Phillips, the embattled Colorado baker, finally got his day in Court. Crowds on both side staged rallies starting early Tuesday, when the Supreme Court heard oral arguments in Masterpiece Cakeshop v. Colorado Civil Rights Commission. When the thrusts and ripostes were finished, Phillips’s friends left with the buoying sense that he’d had quite a good day; that the case was likely to be decided his way. But the seasoned observers, savants, and kibitzers, gathered outside the Court, were having trouble guessing just how the Court would frame its decision: Phillips had struck a brave stance against the forces set upon him by law in Colorado but upon what grounds would the Court vindicate that bravery?

Phillips ran afoul of a Colorado statute that barred discriminations in private businesses open to transactions with the public. Those kinds of laws have become quite familiar with cases of racial discrimination, but the Colorado Anti-Discrimination Act (CADA) broadened the coverage of these laws to include discriminations based on “sexual orientation.” Phillips readily served gays and lesbians in his business, but he declined when asked to make a cake to celebrate the wedding of a gay couple. Phillips had moral reservations about same-sex marriage anchored in his religious convictions. The request for that cake, bearing that meaning, was just a step too far for Phillips; it would have marked his acceptance of the cake’s purpose. That elementary point seemed to elude Justices Ginsburg, Kagan, and Sotomayor: with that simple involvement, Phillips would become complicit in something he regarded as wrongful.

And yet, by holding back, he was marked a serious wrongdoer under Colorado law. He faced fines and compulsory counseling, the object of which was to purge him and his employees of “bigoted” reflexes. Phillips was not compelled to speak words of acceptance and praise for same-sex marriage. But his “performative” act, in making the cake, could have been taken as a moral endorsement. A witness appearing years ago before the Judiciary Committee professed that he had no moral judgment on the matter of abortion, only to have Orrin Hatch point out the inconvenient fact that the witness, Dr. Berkowitz, had performed abortions. Weighing in, Daniel Robinson noted: “Whether you say you favor abortion or not, if in fact you actively performed these during your professional career, this is a sufficient statement; it is a non-verbal statement.” This is the sort of non-verbal statement Phillips made, and it formed the principal argument of his lawyers: namely, that being forced to bake a cake would be coerced speech, the kind of coercion that has been barred under the First Amendment. This was a matter of requiring the speaking of words, the affirming of convictions, that the speaker regarded as deeply objectionable. For Phillips and his supporters, this was closer to the law that required Jehovah’s Witnesses’ children to pledge and salute the American flag despite their religious objections.

But that leading argument on speech encountered serious reservations, even among conservatives. The hazards lay precisely in the claim that certain acts—such as baking a cake—could be translated into “expressions” of political speech. Justice Scalia himself had warned that virtually any act could be converted into a claim of “expression”: driving through a red light, for example, could be an expression of opposition to laws restraining the “liberty to travel.” And if acts of “expression” have a trumping quality, a crowd of thugs that breaks up a demonstration is itself engaged in an act of expression. And being barred from violating the rights of expression of the thugs, the government would not be able to protect the rights of the original demonstrators under the First Amendment.

It was Justice Breyer who saw the connection right away to the interests at stake in the Civil Rights Acts. He mentioned Ollie’s Barbecue, the restaurant in Birmingham, Alabama, that in Katzenbach v. McClung tested the Civil Rights Act of 1964: might it be argued now that by forcing the owners of the establishment to treat black customers in the main dining room, the law was forcing the owners to engage in performative acts of expression that ran counter to their own moral convictions?

On the other hand, if the concern in this case was not for coerced speech but religious conviction, that argument too suffered embarrassment when put up against the case of Ollie’s Barbecue. For it never occurred to anyone that the Civil Rights Act of 1964 would offer a “religious” exemption to those who professed religious grounds for their attachment to the private right of owners to arrange the terms of their private business in accord with their own, private standard. Or that they could not continue to respect their own views on rightful and wrongful treatment of black people. The legislators who framed the law in Colorado thought their law was as deeply just as the laws that barred discriminations based on race. And anyone who agreed with that judgment could see no more ground for a religious exemption in Masterpiece Cakeshop than in the cases on racial discrimination.

It is certainly worth reminding ourselves, in that vein, that even when the Court struck down the laws that barred marriage across racial lines in Loving v. Virginia (1967), there were earnest ministers, like the Reverend Bob Jones, who found a scriptural basis for rejecting the mingling of races in marriage. Does anyone seriously think that the law now would give a pass to bakers or florists who professed a religious ground for refusing to engage their arts in support of interracial marriage? Clearly, the difference cannot be “religious belief.” The difference between the two cases can turn—can only turn—on the fact that we now regard racial discrimination as deeply unjust, but that many of us have not been persuaded that there is something truly unjust in confining marriage to the coupling of one man and one woman.

In the midst of another moral divide in the country, Lincoln said of the defenders of slavery, “Their thinking it right, and our thinking it wrong, is the precise fact upon which depends the whole controversy.” And here was the elusive key to the puzzle of this case: whether the argument is cast against coerced speech, or on behalf of religious freedom, the argument in either case cannot be resolved without making its way to the issue that stands at the core of the disagreement—the moral acceptability of same-sex marriage.

And so, in the oral argument, Chief Justice Roberts noted that “Catholic Legal Services…provide pro bono legal representation to people who are too poor to [afford] it and they provide it to people of all different faiths.” But if they were asked to represent the gay couple in Masterpiece Cakeshop, wouldn’t they be free to refuse precisely because they could not take up a cause in serious conflict with Catholic moral teaching? And if they held back, asked Roberts, would they not be in violation of this law in Colorado?

From another angle, Justice Gorsuch offered the hypothetical of a baker who made a cake topped with a red cross. One customer buys it as a celebration of the American Red Cross while another wants to celebrate the Ku Klux Klan. Would the baker not be well within his rights by refusing the supporter of the KKK? And in a comparable case, Justice Alito offered the example of the customer who wanted to mark his wedding anniversary with a cake saying “November 9, the best day in history.” But then another customer asks for a cake with precisely the same wording, but this time for the purpose of celebrating Kristallnacht, with the attack on Jews in 1938. May the baker not refuse, even though it is the same product, with the same inscription, but marked now for a purpose so strikingly different?

Once again, though, the matter hinges on the fact that we think people quite warranted in opposing the KKK or the Nazis in their assaults on Jews, but the people who brought forth the law in Colorado see nothing comparable. For they cannot see any plausible ground on which any one may find fault any longer with same-sex marriage, or cast a critical moral judgment on the homosexual life. “If slavery is right,” said Lincoln at Cooper Union, “all words, acts, laws, and constitutions against it, are themselves wrong, and should be silenced, and swept away.” If slavery were right, he could accede to the demand to censor the federal mails to screen out the incendiary, abolitionist broadsides. And there could be no holding back:

This, and this only [will appease the partisans of slavery]: cease to call slavery wrong, and join them in calling it right. And this must be done thoroughly—done in acts as well as in words. Silence will not be tolerated—we must place ourselves avowedly with them.

The opponents of slavery had to be made to confess the rightness of slavery and their own wrongness in opposing it. And that is what is precisely engaged now on the matter of same-sex marriage. My libertarian friends rightly point out that many other bakers and florists readily cater to same-sex weddings. But the libertarians, ever quick to deride the flaring of “moral” issues, can’t quite seem to grasp the point that the proponents of same-sex marriage take their positions as profoundly serious as a moral matter. It doesn’t matter that other bakers are available. Jack Phillips, in denying the rightness of same-sex marriage, was a wrongdoer. And as a wrongdoer he had to be publicly reproached and punished by the law.

But what exactly had the law sought to forbid and punish when this case had first arisen? Justice Alito recalled a couple of salient facts that were strangely screened from the record: in 2012 the laws of Colorado offered no acceptance of same-sex marriage, or even of civil partnerships. Phillips himself was never averse to serving gays or lesbians. He rejected only the notion that marriage may be regarded legitimately as anything other than a legal union between one man and one woman. That was the decisive point on which his judgment turned, and nothing in that judgment was stamped as wrongful in the laws of Colorado at the time. Had it simply become wrongful in the sub-legal mores of Colorado to express moral reservations about same-sex marriage and the homosexual life? In other words, when the case was viewed from this angle, one could challenge the very ground of the law, without needing to raise any argument about religious belief or coerced speech.

Still, the case drew on some plausible connections to other cases on speech. Justice Alito noted that there are bakers who have refused to bake cakes carrying a message of opposition to same-sex marriage. The authorities did not target those bakers for punishment. As Alito pointed out, this would clearly be a case of “viewpoint discrimination.” True enough, but that charge dissolves if indeed the people who deny same-sex marriage are rightfully turned away as customers. There would be no moral parity then between the viewpoints. Once again, the difference between the cases turns solely on the question of whether it is legitimate any longer to express opposition to same-sex marriage.

Justice Gorsuch, though, did point up the part of the case that did indeed offer an instance of coerced speech: that Phillips and his employees were compelled to undergo those rounds of compulsory counseling, to purge them of their retrograde views. Justice Ginsburg quickly intervened to object and set matters straight: Phillips and his entourage were simply being tutored in the requirements of the law. But in her typical way, she glided past the moral point so plainly in sight: she glided past the point that Phillips and his workers were being compelled to acknowledge—and to absorb the understanding—that the moral and religious convictions that animated their lives were not only contrary to the law, but deeply and morally wrong.

What was at work at every turn among the drafters and enforcers of the law was that sense of those people so depraved, so backward in their understanding of sexuality, that they would harbor reservations about same-sex marriage. Whether those sentiments had been codified in the statute or not, the record revealed rather clearly that they were animating the members of the commission who had targeted Jack Phillips for prosecution. It was the brutally explicit recording of those sentiments that finally triggered the recoil of Anthony Kennedy. Kennedy cited the remark made by one of the commissioners enforcing the law, Commissioner Hess, who remarked that the claim of religious belief brought forth to justify discrimination in these cases is “a despicable piece of rhetoric.” He then put the question to Frederick Yarger, the Solicitor General of Colorado, defending the law: “Did the Commission ever disavow or disapprove of that statement?” It had not. Kennedy put the question then more sharply: “Do you disavow or disapprove of that statement?” Yarger wavered for a while until he finally conceded that he would indeed disavow that sentiment. His punch landing, Justice Kennedy was moved to fill out his declamation:

Counselor, tolerance is essential in a free society. And tolerance is most meaningful when it’s mutual. It seems to me that the state in its position here has been neither tolerant nor respectful of Mr. Phillips’ religious beliefs.… And because accommodation is quite possible, we assume there were other shops that—other good bakery shops that were available.

In Obergefell Kennedy was liberal-minded enough to acknowledge that many people, holding to the understanding of marriage as we have known it, were decent, well-meaning people. Apparently it was not entirely their fault that they were willing now to fall into an attitude that demeaned and denied the dignity of gay and lesbian people.

But where had the commissioners in Colorado been schooled in these sentiments? Had it not occurred to Kennedy that he himself had been the prime tutor here as he had lectured the country from on high on the deep wrongness of casting moral judgments on the way people led their sexual lives? Was he not the justice who had been arguing since Romer v. Evans (1996) and Lawrence v. Texas (2003) that the laws barring same-sex marriage could claim no reasoned ground of support? Was he not the one who had said that those laws had to be attributed then to an irrational “animus”? In Obergefell, he argued that the opposition to same-sex marriage had to be attributed mainly to a “baredesire to harm a politically unpopular group.” And did it pass his recognition that his complaint now was nothing less than an echo of Justice Scalia’s dissent in the Windsor case (2013) when Scalia pointed out the subtext, or the deep rhetoric, of Kennedy’s opinion: that the people who were defending the laws on marriage, and resisting same-sex marriage, were painted now as “hostes humani generis, enemies of the human race.”

It is hardly likely that Justice Kennedy had experienced an epiphany, comparable to the light that struck Saul of Tarsus on the road to Damascus. What has likely been triggered here is Justice Kennedy’s libertarian side, his aversion to people getting stigmatized and beaten up. But if he is truly willing to carve out a space for the people who bear reservations about same-sex marriage, he could be taking the step that begins a critical scaling back in the reach of the Obergefell case. It would be a notable move away from the insistence that same-sex marriage is so deeply rightful that anyone who calls it into question may be rightly condemned and punished. And if that move takes place, it could break the engine of repression that has been set in motion now and expressing itself in these ways in different parts of the country: Christian schools being put under serious pressure to recognize LGBT groups, and accept same-sex couples in their medical plans; parents being threatened with the removal of their children if they bring in counselors to counsel against the gay-lesbian life; and priests threatened with prosecution for hate speech if they expound, in homilies, the teaching of the Catholic Church on marriage and sexuality. It is hard to imagine that Justice Kennedy would look with sympathy on a rolling back of his teaching on sexuality and marriage. But Kennedy could indeed take this as a moment to rein in the demons he has launched in the land. With a simple turning in this case on Jack Phillips, he could gently launch the move to scale back the wave of repression that has sprung from the moral premises he has planted in the law. And if that comes to pass, Jack Phillips, the baker, will have wrought even better than he knew.

Hadley Arkes is the Ney Professor of Jurisprudence Emeritus at Amherst College and the Founder/Director of the James Wilson Institute on Natural Rights & the American Founding.