On Monday, Mark Zuckerberg published a 6,500-word manifesto about personal AI, largely centered on the possibilities of the “personal superintelligence” systems Meta AI is building. The ideas in the post are not entirely new. A version of the essay ran in The Wall Street Journal two weeks ago, and Zuckerberg has discussed similar themes on Meta earnings calls before. But this is probably the most detailed version he has shared.

Some observers described the essay as an anti-doomer piece, but that characterization is not quite right. It is more of an explanation of why Zuckerberg is personally excited about the ways AI will change society. Yet, even as he tries to paint a picture of a wonderful future shaped by abundant superintelligence, he keeps reminding readers of all the ways it is likely to go wrong.

Key facts from the manifesto

• Zuckerberg published a lengthy essay arguing that personal superintelligence, delivered broadly, will benefit society.
• He claims that giving everyone access to AI tools will create a future of personalized tutors, superintelligent lawyers, and fairer access to compute.
• The essay frames Meta’s freemium model as a way to distribute intelligence widely, including a dynamic auction mechanism for those who want extra compute.
• Critics say the manifesto ignores real-world harms such as AI-generated homework, legal complexity, and the trust deficit caused by social media.
• Public skepticism is compounded by a recent survey showing 64% of Americans believe social media has been harmful to democracy, as well as a $567 million fine against the company for harm to children.

The trust problem is impossible to ignore

AI is exciting to many journalists, researchers, and product developers, but a large share of the public still finds it creepy and unpleasant. That reaction is not random. It is shaped by a decade of social media scandals, data privacy violations, election interference debates, and mounting evidence that large platforms can cause real psychological harm. Zuckerberg is central to that history. Facebook as a product and Zuckerberg as a person are both unpopular with the U.S. public.

The recent numbers are stark. A survey found that 64% of Americans believe social media has been harmful to democracy, and a similar percentage believe it should be more heavily regulated. Those numbers cut evenly across partisan lines. Just this past weekend, a court fined the company $567 million for being harmful to children. The vibes are bad.

None of this means Zuckerberg should withdraw to the wilderness in shame. But the fallout from social media is one of the central reasons we now see so much anxiety about AI. Whether it is fair or not, the public does not trust tech executives to make sure new technologies have a positive impact on society. Instead of acknowledging that and trying to win back trust, the manifesto demonstrates over and over again how that trust was lost in the first place.

Vague promises about intelligence

A large part of the essay is devoted to hazy generalities about intelligence, which sound very similar to the hazy generalities Zuckerberg and other tech leaders used to give about free speech in the early social media era. For example, he writes: “As everyone gains more powerful tools, each person will become more capable of shaping the future, not less. People and institutions with competing interests naturally check and balance each other to lead towards positive outcomes. The best and most realistic path to building a positive AI future is by delivering superintelligence to everyone.”

That argument is hard to accept without serious caveats. Competing interests do not always lead to positive outcomes; they can just as easily lead to gridlock, exploitation, or mutually assured destruction. But the weirder issue is that this argument is presented as an earnest philosophical conclusion, rather than as a pitch for a specific product called “personal superintelligence” that Meta is bringing to market. If someone already distrusts Zuckerberg, they may worry that he is writing all of this to convince himself that nothing bad can happen.

AI in education is not just a dream

That feeling intensifies when Zuckerberg gets to specific examples. Many of them are alarmingly out of touch with how AI tools are actually being used. For instance, here is his take on AI in education: “Everyone will have a personalized tutor and coach with a PhD in every subject and unlimited patience to help you learn anything you want. Students will have extra help in areas they need it that is currently only available to those whose parents can pay. Adults will have a superintelligent learning assistant that knows exactly how to teach you new job skills, new languages, new hobbies, or anything else you’re interested in.”

Of course, the product Zuckerberg is describing already exists. This is a consumer chatbot like ChatGPT, Claude, and Gemini. These tools really can be helpful if you want to learn about a certain topic, but the main way they are used in education is to avoid learning. A personalized tutor can do your homework and write your assigned essay. Because there is no robust watermarking system in place, teachers cannot be sure which essays were generated with AI.

This is a relatively low-stakes example of AI harm, but it is happening right now. Sometimes you design a technology to do good things and it has unintended consequences that make things worse. When one of the most powerful people in the world refuses to acknowledge this, it makes people understandably nervous.

The superintelligent lawyer thought experiment

Zuckerberg also offers an example of how AI will change the legal system for the better. He writes: “As a thought experiment, imagine only one person had a superintelligent lawyer. They would have an unfair advantage in court — even if they were wrong on the merits. That would lead to a worse society. But now imagine everyone has a superintelligent lawyer. In this case, justice would be carried out much more fairly and efficiently than it is today when there is often an imbalance in skills and resources in litigation.”

That is a loaded example. Access to legal AI might lead to a more just society, but it might also add more complexity to an already bureaucratic hellscape. It could unleash a new wave of vexatious litigants who clog the courts with the legal equivalent of spam. And if everyone has a superintelligent lawyer, the real bottleneck might shift to judges, court staff, and enforcement — none of which are being superintelligent. It is hard to feel calm about any of this, and the fact that Zuckerberg is not worried makes others more worried.

Freemium AI and the surge pricing problem

In other passages, even descriptions of Meta’s existing business practices take on an oddly abstract character. For instance, Zuckerberg describes Meta’s commitment to a freemium model this way: “Everyone will have free or affordable access to these tools. We will offer free versions that will be accessible to billions of people. For those who want to pay to use more compute, there will be a dynamic auction mechanism that will guarantee that everyone gets the lowest price possible for the intelligence and compute they’re using while also ensuring the capacity is used for whatever people collectively find most valuable. This will ensure the benefits of superintelligence are distributed widely.”

In broad strokes, the underlying argument has merit. Zuckerberg is right that access is an issue, and a freemium model allows for more access than requiring everyone to pay for compute up front. Dynamic markets for spot compute already exist, so the idea is not completely outside the norm. But there is a reason every consumer AI product insulates end users from the spot price of the compute they are using. Surge pricing is a terrible user experience, especially when it is deployed on a tool you rely on for work.

Again, it is fair to assume Zuckerberg understands this and is not actually planning to set token prices through dynamic auctions in the consumer product. But after reading the essay, that assumption feels less certain. By describing the idea in abstract terms, without acknowledging the obvious user experience problems, he makes readers wonder whether he has thought through the practical consequences.

What better AI communication looks like

There are better ways to talk about this stuff. For all their faults, other AI leaders like Sam Altman and Dario Amodei do a reasonably good job when they are directly communicating with the public. They do exactly what Zuckerberg resists in this essay: they acknowledge the dangers of AI, emphasize their own precautions, and try to convince listeners that they can be trusted.

That kind of communication can only do so much. Sometimes, particularly recently, safeguards fail and the inherent danger of the enterprise is made clear. But without some kind of trust-building, the industry may not always be able to survive those failures. The public has seen too many broken promises from social media to accept pleasant abstractions about superintelligence. The next wave of AI leadership will need to do more than promise everyone a PhD-level tutor or a superintelligent lawyer. It will need to show that the people building these systems genuinely understand how they can go wrong, and that they are willing to design for the messy, imperfect reality of human behavior.

Source:TechCrunch News