Thiago Cardoso :: Optimizing anamorphic sculptures

Thiago Cardoso

About

Blog

Publications

GitHub

Optimizing anamorphic sculptures

2026-09-07
aiart

A pig made of cars, pigs and pigeons.

At my workplace there’s an anamorphic sculpture that reveals a recognizable image only when viewed from a specific vantage point. The work of Jonty Hurwitz
explores that technique with mirrors (catoptric anamorphosis) and with perspective (oblique anamorphosis). In Blue Singularity, Jonty’s own head sculpture is displayed in slices. Those slices compose the head when looking from a specific angle.

Blue Singularity from Jonty Hurwitz

The sculpture as seen by the vantage point. Blue Singularity from Jonty Hurwitz

My office’s sculpture always gets me thinking about how I would optimize for that kind of result. To experiment with that, I wrote an application that, given a set of 3d meshes, arrange them in the 3d space so that a target silhouette is rendered from a specific camera angle.

There should have been a video here but your browser does not seem
to support it.

A pig made of cars, pigs and pigeons.

Random meshes in, a silhouette out
Sculptures are generated by optimizing the position, size, and rotation of different meshes with gradient descent. Meshes are placed randomly in the scene, the scene silhouette is rendered. The render is then compared to a target image, a loss is computed, and errors are backpropagated to the input parameters (meshes positions, rotations, and scale). At each step meshes are moved and adjusted so the rendered silhouette comes closer to the target image.

Overview of the optimization process. Meshes are placed at random, the silhouette is rendered and compared to the target image.

Although the process has few steps, the implementation is not simple. There are multiple implementation details necessary to generate visually interesting images. First and most important, to be able to backpropagate errors from the loss to the input, the rendering step should be differentiable. The loss function should also be tuned to value details. In other words, just getting most of the image right is not enough to get a similar silhouette.
The rasterizer has no gradients
A rasterizer is a function that maps vertices to pixels. It is responsible for converting a list of triangles (3d mesh) into an image. For every pixel it asks whether that pixel is covered by the triangle. This is a binary operation: covered or not. Nudging a triangle slightly changes nothing until its edge crosses a pixel center, and then the pixel flips. This means that the derivative is zero almost everywhere and undefined on the boundary. To backpropagate errors in this setting, some constraints need to be relaxed. Luckily, Liu et al.
had this problem before and published a paper on soft rasterizers.
The main idea in SoftRas is to replace the boolean (pixel inside or outside triangle) with a sigmoid of the signed squared distance to the triangle’s boundary. For pixel \(i\) and triangle \(j\), let \(d_{ij}\) be the distance from the pixel center to the triangle’s outline, \(\delta_{ij}\) = +1 if inside, -1 if outside triangle, and \(\sigma\) a constant to control the transition. Then the “belonging” of a pixel in a triangle can be measured by \(p_{ij}\):
$$
p_{ij} = \operatorname{sigmoid}\!\left(\frac{\delta_{ij}\, d_{ij}^2}{\sigma}\right)
$$

Experiment with different \(\sigma\) values
Move the pixel center across a triangle edge. A smaller \(\sigma\) makes the transition sharper; a larger \(\sigma\) lets a missed shape send a gradient farther. A hard rasterizer is the \(\sigma\) → 0 limit.

Interactive figure — requires JavaScript.

As you can see, different \(\sigma\) values have different trade-offs. With a small \(\sigma\), errors are only propagated close to the triangle edge. Meshes placed far away won’t be moved. Having a large \(\sigma\) will make the silhouette more diffuse. The adopted solution was to anneal \(\sigma\): start large and shrink it as the optimization progresses. A large \(\sigma\) is the renderer squinting. The silhouette is a blur, but a mesh far from where it should be still feels a pull. Shrinking \(\sigma\) opens the renderer’s eyes and sharpens the edges.
Another optimization was splitting the meshes in bins. Not every pixel should be checked against every triangle. Also recomputing gradients during backprop helps keeping the memory in check.
Edges are what make a silhouette recognizable
The naive approach is to compare the rendered silhouette with the target image and adjust the objects. The problem with that is that although the silhouette can have a lot of “mass” in the center, the edges are what make silhouettes distinguishable. On top of that, I wanted the generated sculptures to be manufacturable in the real world, which meant avoiding collisions between placed components.
The loss function computes a single number that represents how good or bad the current arrangement is. The optimizer’s objective is to make this number smaller by nudging where each object sits, how it is turned, and how big it is. The loss is composed by four different components:

The intersection between the computed silhouette and the target (comparison).
A penalty for painting outside the target silhouette (spill).
The quality of the silhouette edges (outline).
A penalty for colliding objects (collisions).

Comparing silhouettes
To compare silhouettes the Intersection over Union is used. This function is widely used for comparing masks and bounding boxes in different computer vision problems. Given the generated image \(\alpha\), and the target image \(t\) the IoU is given by:

$$IoU = \dfrac{|\alpha \cap t |}{|\alpha \cup t|}$$
In other words, the pixels shared between the two silhouettes divided by the union of “painted” pixels. To be compatible with loss, \(1 - IoU\) is used as the metric. In some experiments, I also added a band IoU loss in which only the surroundings of the silhouette are considered.
Reducing spill
Painting outside of the target silhouette hinders IoU but its impact can be small. Since most of the mass is around the core of the image, small variations in the edges, or around the image, don’t impact the loss sufficiently to generate close enough silhouettes. Adding an additional component to penalize out of place meshes increases the final quality of the image. Given a weight \(c\), the total number of pixels in the image \(|F|\), the spill is given by the sum of painted pixels outside of the target image:

$$\text{spill} = c \cdot \frac{1}{|F|}\sum_i \alpha_i (1 - t_i)$$Improving the outline
Even with those components, the generated silhouette can have a jagged edge. For example, the little pigeon feet can leak under the pig silhouette generating spikes in the edge. The outline component adds a penalty for perimeters larger than the target image requires.
To capture that information we use a Total variation (TV) metric. It is the sum, over all pixels, of how sharply the image changes at that pixel.

$$\operatorname{TV}(\alpha) = \sum_{i,j} \sqrt{(\partial_x \alpha)^2 + (\partial_y \alpha)^2 + \epsilon}$$
The outline component of the loss is given by the ratio between the rendered image and the target TV.

$$\text{outline} = \dfrac{TV(\alpha)}{TV(t)}$$Penalizing collisions
The final component of the loss formula aims to reduce overlap between meshes. This way the sculptures could be really built in the real world with the composition of different objects.
The collision penalty is computed with axis-aligned bounding boxes (AABB) for every pair of objects (\(P\) is the set of every pair of objects). With \(\text{pen}_{ab}\) being the penetration between objects \(a\) and \(b\), and \(\text{width}_{ab}\) the average frame width in the image.
$$\text{collisions} = \frac{1}{|P|}\sum_{(a,b) \in P} \frac{\text{pen}_{ab}}{\text{width}_{ab}}$$
The implementation optimizes this collision to only compute the metric for pairs that can actually collide, which cuts the cost of each optimization step.
Loss formula
All components are multiplied by a weight and summed together. Given the generated image \(\alpha\), and the target image \(t\), the hand-tuned weights \(c, s, w\) the loss function is given by:
$$
L = \underbrace{\bigl(1 - IoU(\alpha, t)\bigr)}_{comparison}
\;+\; c \cdot \underbrace{\frac{1}{|F|}\sum_i \alpha_i (1 - t_i)}_{\text{spill}}
\;+\; s \cdot \underbrace{\frac{\operatorname{TV}(\alpha)}{\operatorname{TV}(t)}}_{\text{outline}}
\;+\; w \cdot \underbrace{\frac{1}{|P|}\sum_{(a,b) \in P} \frac{\text{pen}_{ab}}{\text{width}_{ab}}}_{\text{collisions}}
$$Optimizing for multiple views
There is an interesting byproduct of this optimization process. It is possible to assemble different images for different camera angles with the same set of objects.

There should have been a video here but your browser does not seem
to support it.

One sculpture with a friendly dog viewpoint and a rabbit assembled from the same components.

In this case, there is an additional restriction. Objects inside the intersection of the two camera frustums are visible from both vantage points, so both target images have to agree on what is painted there. Where they disagree, one of the views ends up with components it never asked for.
To maximize the image freedom, I am using two cameras with a 90 degree angle between them. Cameras’ losses are combined with the harmonic mean, which is pulled down by the worse of the two. The optimizer cannot sharpen the dog by letting the rabbit fall apart.
What I want to render next

There should have been a video here but your browser does not seem
to support it.

Another pig, this time made only of pigeons. From any other angle it is a flock going nowhere.

There are still things to explore: rendering with more components, making the result hold up at angles near the vantage point, and using complementary views better. This is not the first time I used gradient descent for art. I did the same for abstract machine learning art
, generating images that fool a classifier while staying recognizable to us. What would you like me to render next?

See also

2026-07-09
Pare de corrigir o resultado. Corrija o loop.

2026-07-09
Stop fixing the output. Fix the loop.

2026-03-27
The rules of code quality have changed

2026-02-21
OpenClaw brings social engineering to digital systems

2024-07-07
Abstract Machine Learning Art

← All posts

© 2026 Thiago Cardoso

GitHub

X

Google Scholar

LinkedIn

RSS

Thiago Cardoso explores the optimization of anamorphic sculptures, working to generate 3D arrangements of meshes that yield a desired silhouette when viewed from a specific vantage point. This work builds upon techniques used in art, such as catoptric and oblique anamorphosis, by applying computational methods to ensure the resulting sculptures are optimized and manufacturable.

The core of the optimization process involves arranging a set of 3D meshes in space to match a target silhouette from a specified camera angle. This is achieved using gradient descent, where the positions, rotations, and scales of the meshes are iteratively adjusted to minimize a computed loss function. Initially, meshes are placed randomly, a silhouette is rendered, and the resulting image is compared to the target image to determine the error. This error is then backpropagated through the rendering process to adjust the input parameters.

A significant technical challenge in this process is making the rendering step differentiable, as traditional rasterizers operate on a binary logic (pixel inside or outside the triangle) and lack gradients. To enable error backpropagation, the approach employs soft rasterizers, inspired by work on soft rasterizers, which replace the binary result with a sigmoid function of the signed squared distance from a pixel center to the triangle's boundary. The parameter sigma controls the transition sharpness; a large sigma results in a diffuse silhouette, while a smaller sigma sharpens the edges. Cardoso found that annealing sigma, starting large and reducing it during optimization, effectively sharpened the rendered edges. Further efficiency optimizations included splitting meshes into bins and recomputing gradients during backpropagation.

The overall loss function is designed to address several criteria simultaneously: silhouette comparison, spill, outline quality, and object collisions. The comparison component utilizes the Intersection over Union (IoU) metric, where minimizing (1 minus IoU) ensures the rendered silhouette aligns with the target. A secondary component penalizes "spill," which measures the area of painted pixels outside the target silhouette, weighted by a factor $c$. To address jagged edges, the outline component incorporates the Total Variation (TV) metric, which quantifies the sharpness of the image changes, penalizing perimeters that are less smooth than the target, calculated as the ratio of the rendered image's TV to the target's TV, weighted by $s$. The final component addresses physical constraints by penalizing collisions between objects, where overlap is assessed using axis-aligned bounding boxes (AABB) and penetration distances relative to the frame width, weighted by $w$.

The complete loss formula combines these elements: L equals the comparison loss, the spill penalty, the outline penalty, and the collision penalty. This comprehensive loss function is multiplied by specific weights to fine-tune the trade-offs between silhouette accuracy, edge fidelity, and geometric feasibility.

Furthermore, the optimization process allows for exploring multiple views by generating different images for various camera angles using the same set of objects. To maximize image freedom when combining views, Cardoso utilized two cameras separated by a ninety-degree angle and combined their respective losses using the harmonic mean, which pulls the resulting optimization toward the more restrictive constraint. The research suggests that while optimizing for a single view, the generated structure possesses the potential to be assembled into various configurations, highlighting the inherent flexibility of the optimization process.