To Control is To Align

1 The central question

How can people convey their true intention to computational models as completely and faithfully as possible?

This essay is a companion to my essay Computation Shaped by Intention [1], which I call the main essay. It expands the main essay with more detailed guiding ideas, starting from its first argument, that to control is to align with human intents and human priors. The main essay asks what computation mechanism is supposed to align with the true intention of humans, and answers that the alignment should happen in the shape of the mechanism, not only in its parameters. The question here is asked from the other side: how a person can convey that intention to a computational model. By the true intention I mean, as in the main essay, what a person wants, as opposed to what they settle for when the mechanism cannot express it.

The question has three sub-questions. How do we know that what the model gives us is what we really want? If the intention arises before we enter the system, how can the model receive what is in our minds? If the intention is established while we interact with the system, how can we be sure that the model can provide the soil and the potential for that interaction, and can keep to what we have already settled so as to give us the exploration we need?

Whether a system conveys an intention is falsifiable, a property inherited from the main essay, and workarounds can refute it. A workaround, such as translating the intention into a form the person would not otherwise use or repairing the result outside the system, shows where an intention was not conveyed. It can also be proved directly, in the positive direction. For example, given two systems, people choose the one that better expresses what they intend, so the choices of the people who hold an intention can prove which system conveys it better. The proof is relative: it ranks two systems, and it does not say how much of the intention either one conveys. These checks bear on the first sub-question: a workaround shows that what a model gave us is not what we wanted, and a choice shows which of two systems comes closer. They also test my answer: if two systems are judged by the two alignments before people choose between them, the answer predicts that people will choose the one that is better on both.

2 My answer

My answer is that to control is to align with human intents and human priors.

True intention is hard to express, because what is in a person’s mind cannot connect with a computational model directly, fully and without loss. So people must map, or project, the intention onto some medium, such as text, images, video, sound, code, or even abstract mathematical rules. This projection is an essential human need. People need it even when they interact with one another, and they need it all the more when they interact with machines. And the projection always loses something: some of the intention is always lost.

To convey the true intention to a computational model, a person must control the model, and the medium onto which the person projects the intention is then the control medium. I hold that conveying the intention as completely and faithfully as possible is equivalent to communicating it to the model as losslessly as possible, so its necessary and sufficient condition is that two alignments hold, both at once: the alignment between the control medium and human intent, and the alignment between the control medium and the computational model. More specifically, the intention travels from intent to control medium to computational model, and one alignment reduces the loss from intent to medium, the other the loss from medium to model. When the subject is intention, lossless can also be called seamless.

The two must hold at once because the loss splits into the two steps. Measure the loss by how far the results are from what a person intends, on average over their requests, or in the worst case for a part that must hold on every request (main essay, Section 3). Take the loss of the person served worst, so that no rare intention is averaged away. One part of it is the least loss that any way of computing results from the medium could reach: no reading can restore what the medium does not carry, or what its form cannot hold, as an opaque RGB image cannot hold a transparent layer. This part is the loss from intent to medium. The other part is how much more the model’s results lose than that best reading, which is the loss from medium to model. Neither part can be negative, because the model’s way of computing results is one of all the possible ways. So the whole loss is small exactly when both parts are small, and a small loss in one step cannot make up for a large loss in the other. A medium that people write well is wasted on a model that cannot read it, and a medium that a model reads well is wasted if people cannot write their intention in it. Both failures occur. In ControlNet’s user study, people ranked the results of five methods that all received the same hand-drawn sketches, and the average rank for fidelity to the sketch ranged from 1.02 to 4.28, where 5 is best [2]. The medium was the same, but the methods read it differently. In the other direction, for a given prompt and settings, a diffusion model’s result is fixed by its random seed, yet a person cannot write an intention in a seed and can only try seeds and keep one.

Sections 3 and 4 take the two alignments in turn, and with them the second sub-question, how the model can receive what is in our minds. Section 5 adds what an intention that forms in interaction also needs, which is the third sub-question, and turns to divergence. The answer is meant for every medium named above, and the evidence in this essay comes mostly from image and video generation, as in the main essay.

3 The first alignment: the control medium and human intent

For the step from intent to medium to lose less and to be more seamless, four things are key.

First, the medium itself should be determined by the intention, including the form in which the result comes back. If people want transparent images, a medium that gives only opaque RGB channels is the wrong medium with any model. The main essay argues this case: in a control mechanism the medium is part of the shape, and no parameters give a transparent layer to a model without an alpha channel (Section 4.1). People also show this when they can choose. In the user study of LayerDiffuse, 14 participants preferred transparent layers generated with an alpha channel in 97.1% of cases to layers made by the usual workaround, generating an RGB image and then estimating its alpha [3]. This is close to the choice described in Section 1, made for prompts the participants were given.

Second, people must be able to understand the medium. Language and pictures are easy to understand. A map of the Fourier domain, a panel of hundreds of sliders, or a classifier-free guidance (CFG) scale is hard to understand. The Fourier transform of an image loses nothing of it, yet a person cannot see in it where an object is. A CFG scale is one number for the whole image, so nothing in it tells a person which parts of the image will change. People can control only what they understand.

Third, people should be able to write the medium: to edit it within their capability, and not in something as inhuman as binary. Better still, the medium can be improved step by step, and has enough room and potential to carry the intention. A medium a person can read and change is a surface for iteration: the person looks at the medium and at the result, compares them with what they mean, adjusts the medium, and so puts the intention onto it one piece at a time. Media differ greatly in room: a choice among k options carries at most log₂ k bits of an intention (main essay, Section 4.2), while a sketch has room for any shape the person can draw. Capability differs from person to person. In the study of CineVision, a storyboard system for film directors and cinematographers, the pairs who drew their storyboards by hand communicated more effectively when they had stronger drawing skills [4].

Fourth, where several people are involved, the medium should even let them exchange views. Projection is needed between people too, and a medium serves a group when each person can see what the others meant and answer it, so that the group forms among themselves the intention it gives the model. In the CineVision study, directors and cinematographers worked in pairs on the same script, and each rated, from 1 to 7, how well the partner had understood their shooting intentions: 6.5 on average with CineVision, 5.12 with an image generator driven by prompts, and 4.13 with hand-drawn storyboards. The pairs with CineVision tried the changes their partners suggested and discussed why one effect was better than another [4]. The study is small, with eight people in each condition, and it compares whole systems, but its rating measures what this key is for: whether each partner understood what the other meant.

A medium should meet these keys, and the better it meets them, the better it is aligned with the intention; otherwise it is not. Language is easy to understand and to write, yet it leaves much of an image open, and for the pose of a figure or the outline of an object it is not the medium the intention determines. A prompt such as “on the left” leaves the exact place open, and “cleaner” means slightly cleaner to one person and spotless to another, so people revise the prompt again and again, guessing at the model each time. When a person draws the contour instead, the shape in the result is defined at once, and the person sees at a glance whether it is where they meant it to be. Sketches, edge maps and poses, the media ControlNet added, are determined by such an intention, and people can read them and make them, by drawing or from a reference image [2].

The second and third keys are close to what Hutchins, Hollan and Norman called the gulfs of evaluation and execution [5]. In the interfaces they studied, the machine read a command exactly as it was defined, so the step from medium to machine lost nothing by construction. A learned model reads a medium through what it has learned, and that step is the subject of the second alignment.

4 The second alignment: the control medium and the computational model

When a control medium meets a computational model, a control mechanism appears: the arrangement through which the model reads the medium and computes a result. The second alignment is then equivalent to the control mechanism being lossless and seamless, and what matters most is that human priors take part in establishing the control mechanism. This is the main essay’s answer seen from the medium: evidence and priors from human activity take part in establishing the topology of control, the structure through which a control input reaches the output (Section 2).

By human priors I mean the priors in human data and in everything human activity produces, from the images people make and keep to the laws of physics they have found. A model learns part of them from human data in pretraining, and these are the priors the main essay speaks of. A mechanism can bring in another part, such as a physical law, by imposing it on the data in training. Even the priors in pretraining data are human. Consider every combination of pixels in an image of 256 by 256 pixels. Almost all of them are noise. The images a person can read, such as spectra or brain scans, form a small region of this space, and natural images, such as cats, dogs or houses, form a smaller region inside it. The billions of images a model is pretrained on sound like many, but against the possibilities of one such image they are a tiny and biased slice, and people decided which slice it is: what they photograph, draw and keep. So the prior a model learns in pretraining is shaped by human selection.

The core of the second alignment is this. The control medium is closely bound to human behavior, because the first alignment fits it to what people intend, understand and write. For the model to make good use of such a medium, it must be able to digest what people produce in that medium, and that ability comes from building the control mechanism itself on human data priors. The examples of a new medium are also few: the largest datasets for single control tasks, such as human poses, hold about 100,000 pairs, 50,000 times fewer than the LAION-5B images Stable Diffusion was trained on [2]. So most of the ability to read the medium should come from priors far more abundant than the medium’s own examples. There are several ways to build on such priors, including these, and in each, human data or human knowledge already holds what the mechanism needs in order to read the medium. A model may have learned many priors from human data: ControlNet reads a sketch with a trainable copy of the pretrained model’s encoder, so the sketch is read with what the base model learned about composition [2]. Human data may hold an inherent consistency: in videos, content close in time or in features is related, and FramePack packs a video’s history into a context of fixed length, giving more of it to the frames most related to the section being generated [6]. Human data may hold an inherent contrast: Stable Diffusion, from its first versions to SDXL, reads a prompt with text encoders trained to tell the captions that match an image from captions that do not, on image and text pairs that people posted online [7, 8]. And even a physical law from human knowledge can serve as a constraint on the data: IC-Light imposes the linearity of light transport in training, so that what it learns from varied images is relighting that keeps the color of surfaces fixed [9].

Taking away the route through the priors suggests how much the mechanism stands on them. TinyHistory reads a video’s history through two branches. One passes a low-resolution copy of the history through the pretrained model’s own input path, its encoder and first layers, and the other passes the full-resolution history through a new, lightweight module. With the same number of context tokens for each second of history, removing the new branch lowered a measure of character consistency from 80.19 to 74.29, while removing the branch through the pretrained path lowered it to 67.73, although that branch sees less detail [10]. The same holds outside images. A shift cipher loses nothing of a message, yet GPT-4 decoded at least half of the messages shifted by one letter and fewer than 3% of those shifted by two letters. Shifts of one, three and thirteen letters are common in text on the internet, and other shifts are rare [11]. A medium can carry everything and still go unread if its form is not among the priors. For people, Larkin and Simon made a similar distinction, between representations that carry the same information and those that also make the same inferences easy [12]; for a model, what is easy is set by its priors.

A diffusion model shows exactly what is lost in reading a medium. It is trained to predict, at each noise level, the clean image, or equivalently the noise, from a noisy copy of the image and the medium made from it, under squared error. For any model, this error is that of the best prediction, the mean of the images that could have produced the noisy copy and the medium, weighted by their probability, plus the model’s squared distance from that mean. This is the split of Section 2 in exact form: the first term is fixed by the data and the medium, and only the second depends on the model, so of two mechanisms trained on the same medium and pairs, the one with the lower held-out error is closer to the best reading, by exactly the difference. The mean blurs where the noisy copy and the medium leave the image open (main essay, Section 4.2), but the blur does not reach the results: the probability flow ODE carries noise to an image by following these predictions down the noise levels, and when each equals the mean, its images are distributed exactly as the data are, given the medium [13]. A model that reads only part of the medium can at best predict the mean given that part, which blurs more, and the ODE then fills from the data what the unread part should have settled. Training can lower only the second term, so the rise in reconstruction quality seen as a diffusion model trains is this term falling. A pretrained model has already lowered it, on billions of the images people make and keep, for everything except what the medium adds to the best prediction, so a mechanism that stands on that model needs the medium’s few examples mainly for that part.

The medium need not reach the model in the form in which people write it. A designer reads and writes a transparent layer as color and alpha, while LayerDiffuse gives the alpha to the model as a small offset in the latent space the model already reads [3]. The first alignment sets the form people write, and the form that reaches the model belongs to the control mechanism, which is what the main essay calls giving the new medium a form the priors can read (Section 4.1). This is one reason the two alignments can both be met. The other is that both are alignments with people. A medium that meets the first alignment usually corresponds to regularities that fill human data, even when examples of the medium itself are rare. Before LayerDiffuse, the largest open datasets of transparent images held fewer than 50,000 images, but images, designs and illustrations everywhere show one thing in front of another, and that understanding of layers and occlusion is the prior LayerDiffuse builds on. The mechanism must also correspond to the medium people understand, and read it as they do. People read a prompt with its word order and know which attribute belongs to which object, while models trained by contrast on image and text pairs, such as CLIP, match images and captions much like bags of words, probably because their training data rarely require more [14]. If the control mechanism stands on these human priors and corresponds to the medium people understand, then the control medium and the model are aligned as well.

5 Following, exploring and divergence

Some intentions are established while a person interacts with the system, which is the third sub-question. For the model to provide the soil and the potential for that interaction, the medium must have room for it: an intention can form only in what the medium can carry, and a medium that people can revise step by step, with room to carry more, lets it form there. For a part the person is still unsure of, there is no settled intention yet to convey, and the medium should leave that part open. The mechanism must then keep to what the person has already settled so as to give the exploration they need. This is following versus exploring, and both alignments carry its boundary: the medium marks which parts are settled, as the form of an input often does, and the mechanism keeps to them. The main essay’s third argument places this boundary in the structure of the mechanism (Section 4.3). Choosing among options is also a way to convey an intention. A person who cannot yet say what they want can recognize it, and the option they accept becomes a settled part that they can read and edit, unlike a kept seed, which settles the whole result in a form they cannot read.

The human priors and the control mechanism must also be able to represent divergence. Human data hold divergence as well as what people share, because different people and practices leave different regularities in them. An intention that is still forming needs this, because the person often finds it among what the mechanism offers, and a mechanism that pools these regularities offers mostly the common pattern. Pooling also fills what the medium leaves out, including what a person holds but never states, with the common pattern. That fill is right where a person wants what others want, but wrong where the person departs from it, and the main essay holds that the departure is more often the valuable part (Section 4.2). So a mechanism should stand on priors selectively and keep such a regularity apart rather than average it: through the medium, which then carries the departure, or through a relation imposed in learning, as IC-Light does with the law of light transport. This is not a third condition beside the two alignments, which are necessary and sufficient. In the split of Section 2, a departure that the medium does not carry belongs to the loss of the first alignment, a loss that, as the main essay shows, no setting of the parameters can remove, and a departure that the medium carries but the mechanism averages away belongs to the loss of the second. This part returns to the main essay: its second argument takes up divergence, and its third the boundary of following and exploring.