The iPhone has long been a symbol of the importance of design in technology. Faster and smaller chips, touch screens and wireless internet are hard technological developments that made the smartphone era possible, but the story that is told, is that Apple brought the right design, that made people adopt the technology because the design gave the right form to the technology.
The new technology is multimodal LLMs, and there is likewise here a problem of design. One might think that designing it right might be making the right device, making the right choices in whether it is to be chat, voice conversations etc., in short that the question is just about finding the right mode of interaction in a way that still remains within the existing framework of interaction design.
What we are seeing is that the most fundamental part of the design is now actually the design of personality. Since ChatGPT, the primary form factor of general AI has been the chat interface, in which the AI system has a persona. Especially Anthropic, with their Claude model, has taken the design of personality extremely seriously, with the (former) philosopher Amanda Askell playing an important part in designing the personality of the Claude chatbot.
For the iPhone, the story is that brilliant minds like Johhny Ive and Steve Jobs saw certain design possibilities, actualized them, and presented them to the common man who could immediately appreciate the design. The idea is that even if the common man can appreciate the final product and confirm that he prefers it to a Blackberry phone, he might not have been much use in letting the designers take all the steps that led them to the final form. It required a high level design expertise.
The idea is that an applied philosopher, has a certain expertise in thinking that allows her to make certain choices in terms of the design of the personality of the chatbot, that the user will appreciate but they might not have been able to formulate it themselves. More importantly, it is not merely that they would not have been able to formulate it, the personality could not have arisen, have been found, by the user telling the chatbot what answers they did not like and what they liked.
One can visualize design as a search: There is a large space of possible designs, and the designer has to find the points in this space that corresponds to good designs. The many choices give a large space, but more importantly there is a path from the current design to the better design. We think that talented people can think in a deeper way, have a better understanding of the space, that allows them to better explore the space and think about how to navigate it and think about what characterizes a good design.
The alternative to intentional design of good systems, is algoritmic optimization: We have the reinforcement learning algorithms that allow for the optimization of personality for the cases where we have a signal to be optimized. This means that providers of AI products can measure certain signals that they care about and optimize against them. It is well accepted that this is what is happening on social media, that algorithms are directly applied to optimize retention, spent time, etc. It is also well accepted that optimizing for these things is problematic, and arises out of misalignment between the incentives of the companies offering the services and the users of these services. At the consumer level, human-AI interaction stand to have similar optimization of crude signals and misaligned incentives.
There is a psychological aspect to the problem. It is well known, that one cannot view a human being as an agent that simply acts according to its preferences. Everyone will tell you that they do things they do not wish to do. That they eat things that they would prefer not to, that there are certain things they wish they did, but do not always have the willpower to. Often this is explained as conflict between cortical higher level executive function and lower level subcortical impulses. When people install app blockers on their phone, it is the higher level system that makes a decision in an attempt to aid the entire system in making better choices on the whole, because the higher level system is limited in its reach.
The picture is one in which the higher level system organizes the overall framework within which the human lives their life, building good habits and structure within which the lower-level system can act, thrive and limit damage. Technologies such as app blockers, or features within social media which allow the user to make higher level choices about e.g. the content of their social media feeds, are empowering users. The action of installing an app blocker or configuring settings for a feed are not themselves intrinsically rewarding or enjoyable activities, they provide a space wherein the human’s higher cognition can exert its influence, make good choices with its limited influence.
If a core aspect of the problem we have seen with social media is that misaligned incentives reduce user empowerment, then it might be argued that to make progress on the problem what is needed for good human-AI interaction will be the follwing:
- Design of organization: We need organizations that produce AI systems whose incentives are aligned with the user’s incentives, that is the higher level non-impulsive interests of users, in short organizations aligned with human empowerment.
- Design of empowerment: And we need that the ability for the user’s high level executive function to efficiently exert its influence, that is the AI systems should improve empowerment.
A dialogue
Skeptic: I understand that you say that TikTok has incentives to get user’s addicted to their app, which is not what the user actually wants, and this is also a risk for human-AI interaction. I have two objections: As Sam Altman from OpenAI has noted, because running the AI is much more costly in terms of computer resources than running TikTok, OpenAI actually is not incentivised to make the user use the AI as long as possible, rather they are incentivied to make a product that the user is willing to pay for.
Non-skeptic: Right now, in 2025, it might be the case that what you quote Sam Altman as saying might hold, but this is not true in general, it is only because of the specific monetization strategy of a specfici product at the current moment. Efficiency of the systems is continually improving, and the amount of data that can be collected from the user from interacting with AI systems might be much more valuable than what could previously be connected, so even if we do not have the ability to predict what exact monetization strategies will be employed, this argument is not that strong.
Skeptic: Okay, even if we grant you that, companies will always have profit motive, and it is only companies that can make services that are sufficiently good that user’s will use them. Even if there were “charities” that made AI systems, they would never be able to compete on the quality of the systems, people would use the capitalistic AI systems more.
Non-skeptic: I grant you that this should be our base expectation, that it is always going to be difficult to bet against the power of capital. I will also grant you that it is very likely that what you say will be true, that the space will be dominated by such companies. However, the dominance of certain internet services by Big Tech has not hindered the spread of alternatives. I think at a minimum it is still worth thinking about what alternatives means in the context of human-AI interaction: For an alternative to Gmail, it is not hard to see how you can get some idealistic talented software developers make an alternative. For human-AI interaction, one could imagine the resources required both in terms of talent, algorithm, data and compute would be so big, that it is not easy how to see how a idealistic talented minority could make a worthy alternative. I actually think that one could make arguments for why it is possible to make services that are not only alternatives on the fringe, but real competitors, but the arguments are not strong enough, so I will instead argue for the this weaker point about the possibility of worthy alternatives. But first notice one thing, the core of what I am arguing is not capitalistic products vs charities. I only identified the profit motive as a core of the misaligned incentive. The core of what I believe is necessary, is alignment with user empowerment.
Skeptic: Hmm, I feel like you are retreating a bit. But let me raise another point: For social media, it is easy to argue that spending a lot of time scrolling a feed is against their best interest. Is it the same with human-AI interaction though, could one not imagine that there is instead a convergence where the most engaging human-AI interaction will be one where the user is interacting with a way in which they flourish and solve their problems and become better, learn new things and so on?
Non-skeptic: Yes, that is a good point. If that is the case, then I merely care that we do what is needed to ensure that is the outcome. I would guess the main problem, that this outcome is harder to reach than one might think. Just as I started by talking about the design of personality versus optimization, I think there are aspect of human personality that makes it so that without being careful, we will hit a whole different optimum and one that will be more easily monetizable.