Programmable assembly of particles on a Chladni plate

Nature-inspired algorithm iteratively assembles particles into complex user-specified shapes on a vibrating plate.

Abstract

In nature, simple building units can be assembled into complex shapes through long-term time-varying external stimuli that are often spatially nonlinear. In contrast, most artificial methods of externally directed assembly rely on field- or template-based energy minimization. However, methods directing the assembly process by controlling time-varying external stimuli instead of attaining the lowest-energy state remain largely unexplored. In this study, we introduce a method that applies time-varying and spatially nonlinear vibration fields to assemble particles into a desired two-dimensional shape. Our assembly method predicts, controls, and monitors the vibration-induced particle motion to iteratively minimize the difference between the desired shape and the actual particle distribution. We applied our method to a centrally actuated vibrating plate, also known as a Chladni plate, and assembled up to a hundred submillimeter particles into complex recognizable shapes. The method allows programmable formation of shapes beyond the intrinsic limits of periodic patterning of the plate.

INTRODUCTION

In nature, variations of wind and water flow can gradually form ordered structures and complex shapes, some of which are recognizable. For example, multidirectional wind regimes create pyramid-shaped dunes (1), and actions of water produce arches and bridges of rock (2). These naturally formed shapes are often the results of long-term external stimuli, which are time-varying and spatially nonlinear (1, 2). Although mimicking the natural processes of shape formation has attracted great attention, most state-of-the-art methods either emulate the collective shape formation of living beings with programmable robot swarms (3) or direct the assembly process through field- or template-based energy minimization and force balance. For example, magnetic fields can trigger formations, such as chains (4), ribbons (5), patterns (6), and vortices (7), determined by local magnetic dipole-dipole interactions. In addition, acoustic fields can trigger periodic shapes and patterns coinciding with pressure nodes or antinodes (8–10), cluster configurations by the short-range acoustic attractions between particles (11), or arbitrary shapes using a prefabricated mask (12, 13). Greater shape complexity and programmability have also been achieved by using sophisticated hardware and processes, e.g., using tens of magnets (14), hundreds of sound emitters (15–17), and thousands of acoustic waveguides (18) or wave-modulating electrodes (19). On a different front, DNA origami enables the folding of a prefabricated DNA template into custom-shaped nanostructures by annealing it with staple strands (20), and capillary forces can deterministically drive nanoparticles into carefully designed shape-matching topographical traps (21). By controlling the temperature and chemical concentration, other structures and shapes can be self-assembled in a programmable manner (22, 23). Despite all the impressive achievements, those works are mostly built upon field- or template-based energy minimization and force balance. Thus, it remains a great challenge to mimic the natural process of shape formation using time-varying and spatially nonlinear external stimuli.

In this study, we introduce a method to programmatically form desired shapes by applying spatially nonlinear vibration fields in an iterative manner. Our method is inspired by natural shape formation processes, where the iteratively applied vibration fields mimic the time-varying stimuli. In contrast to other approaches using dynamic adjustable standing-wave fields (15–17), our assembly method is based on iterative control of particles out of traps or nodal lines. We used a centrally actuated Chladni plate to generate the vibration fields, which can simultaneously move from a few dozens to a hundred particles in a wide range of frequencies (see Fig. 1). The particle motion on the plate is spatially highly nonlinear because of the complexity of the vibration field at given frequencies (see fig. S1). In addition, the motion of particles also exhibits stochastic behavior (24). To programmatically form desired shapes from particles, we first trained neural networks that can predict the likely particle displacement in the vibration field for different frequencies. We then developed an assembly algorithm that iteratively selects and applies the vibration fields to minimize the difference between the desired shape and the actual particle distribution on the plate based on machine vision feedback. As a result, we can assemble up to a hundred particles into several two-dimensional recognizable shapes, examples of which are schematically illustrated in Fig. 1.

Fig. 1. Concept of the assembly method.

(A) Schematic of the experimental setup: The setup consists of a vibrating plate mounted on a piezoelectric actuator. Over the surface of the plate, particles (green discs) are dispersed and imaged by a camera from above. As a result of applying a sequence of vibration-induced displacement fields, the particles are assembled into a desired shape (represented by the violet letter A). During the assembly process, the particles tend to travel along complicated paths (red curved arrows). (B) General principle of the assembly algorithm: On the basis of machine vision feedback, the algorithm iteratively selects and applies the vibration-induced displacement fields that are likely to move the particles (green dots) toward a desired shape (represented by the red-bordered triangle). To simplify the process, the algorithm also dynamically adapts the position and orientation of the shape to the particle distribution.

RESULTS

Displacement fields on the vibrating plate

We used a centrally actuated Chladni plate as our experimental apparatus, which is schematically illustrated in Fig. 1A. The apparatus consists of a silicon plate with the dimensions of 50 mm by 50 mm by 0.525 mm mounted on a piezoelectric actuator, as in our previous reports (25, 26). Over the surface of the plate, particles, ϕ600-μm solder balls, are dispersed and imaged by a camera from above, where their positions and vibration-induced displacements are detected using machine vision. When the actuator excites the plate at a given frequency, the particles move in a spatially nonlinear manner following the vibration-induced displacement fields, as shown in Fig. 1B and fig. S1. Because the apparatus can generate displacement fields at almost any frequency between 1 and 30 kHz, Western musical notes n ∈ [1...52] are used as the frequency order. Each of the notes is attributed by a certain frequency and voltage amplitude (see Materials and Methods for details of the signal shape).

Thereafter, we quantitatively modeled the displacement field of each note using particle-tracking velocimetry (25, 26). Because the motion of particles on the plate is stochastic (24), we trained neural networks predicting the likely displacement of a particle as a function of its position

$$\mathbf{\Delta }\widehat{\mathbf{p}}={\mathit{u}}_{\mathit{n}}(\mathbf{p})$$ (1)

where p ∈ ℝ² is the initial position of the particle, $\mathbf{\Delta }\widehat{\mathbf{p}}\in {\mathrm{ℝ}}^2$ is the displacement of the particle, and u_n is the neural network fitted for note n. The complete set of all the displacement fields is presented in fig. S1. The details related to the training of the neural networks are explained in Materials and Methods.

The relationship between particle position p and displacement Δ$\widehat{\mathbf{p}}$ is highly nonlinear. In particular, the mean squared error (MSE) averaged over the notes of the neural networks is comparable to that of polynomials of up to the 11th order fitted to the same data (fig. S2). For higher-order polynomials, the average MSE grows increasingly large.

As shown in fig. S2, the average MSE of the neural networks is slightly lower than that of all polynomials. For this reason, the neural networks were chosen over the polynomials in this work. Nevertheless, polynomial models could be used as well.

Programmable assembly with displacement fields

We developed an algorithm that assembles particles into a user-specified two-dimensional shape by iteratively selecting and applying the displacement fields and by adapting the position and orientation of the shape to the particle distribution. At every iteration step, the algorithm detects N particle positions, $\mathbf{P}={\{{\mathbf{p}}_{\mathit{i}}\in {\mathrm{ℝ}}^2\}}_{\mathit{i}=1}^{\mathit{N}}$, using machine vision. Then, it converts the desired shape into a set of N shape-defining target points, or targets, $\mathbf{T}={\{{\mathbf{t}}_{\mathit{i}}\in {\mathrm{ℝ}}^2\}}_{\mathit{i}=1}^{\mathit{N}}$. The targets are evenly distributed within the inner area of the shape and represent the desired arrangement of the particles. Thus, the assembly problem is converted into the problem of aligning two point sets, particles P and targets T. This is formulated as the minimization of the following cost function

$${\text{min}}_{\mathbf{R},\mathbf{v},\mathit{n}}\mathit{J}(\mathbf{R},\mathbf{v},\mathit{n})$$ (2)

where

$$\mathit{J}(\mathbf{R},\mathbf{v},\mathit{n})=\frac{1}{\mathit{N}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\Vert \mathbf{R}{\mathbf{t}}_{\mathit{i}}+\mathit{v}-{\mathit{E}}_{\mathit{n}}({\mathbf{p}}_{\mathit{i}})\Vert }^2$$ (3)

p_i and t_i is a correspondence particle-target pair; ‖ ∙ ‖² is the squared Euclidean distance; N is the total number of correspondence pairs; E_n(p_i) = p_i + u_n(p_i) is the expected position of particle p_i as a function of playing note n; R ∈ ℝ^{2 × 2} and v ∈ ℝ^{2 × 1} are a rotation matrix and a translation vector, respectively; and J, the cost function to be minimized, is the sum of squared particle-target distances, which represents the difference between the desired shape and the particle distribution. The assembly algorithm iteratively minimizes the cost function (J) by establishing the correspondence pairs, by computing and applying optimal rigid transformation, i.e., rotation R and translation v, and by moving the particles with the selected notes at every iteration step. The correspondence pairs are established by matching the closest target for each particle, which is posed as an assignment problem and solved with the Hungarian algorithm (27). The optimal rigid transformation is computed with the Kabsch algorithm (28). The targets are transformed at every iteration step when the assembly algorithm finds such rotation R and translation v that further minimize the cost function (J). Otherwise, R and v become an identity matrix and a vector of zeros, respectively, leaving the target positions unchanged from the previous step. As a result, the distances between most of the particles and the targets are iteratively minimized, and the particles are eventually assembled into the desired shape, as schematically illustrated in Fig. 1B. A convergence criterion of the algorithm can be defined as a fixed threshold of the cost function. Alternatively, execution can be stopped on the basis of a visual evaluation of the result. For N particles, the time complexity of one iteration of the algorithm is O(N³). The details of the algorithm are thoroughly explained in Materials and Methods.

Using our algorithmic approach, we assembled up to 100 particles into several recognizable shapes on the Chladni plate (Fig. 2, fig. S3, and movies S1 to S4). Examples of the corresponding normalized cost function dynamics are shown in Fig. 3. The number of iteration steps performed by the assembly algorithm ranged from 19 to 1128, and the assembly time ranged from 38 s to about 38 min (table S1). The percentage of the particles collected within the shape boundaries by the end of the assembly process ranged from 63.2 to 88.5% (fig. S4). It is noteworthy that the assembly algorithm may ignore the outlier particles displaced unacceptably far from most particles and targets (see Materials and Methods for details of the algorithm).

Fig. 2. Assembly demonstrations.

(A) Assembly process from the initial particle distribution (left) to the final result (right). (B) Particles assembled into the letters forming the word “AALTO.” (C) The result of assembling particles (left) into one shape (middle) and reassembling them into another shape (right) thereafter. The number of particles involved in the assemblies ranges from 88 to 100. Scale bars, 5 mm. Photo credit: A. Kopitca, Aalto University.

Fig. 3. Cost function dynamics.

(A) Two different shapes (the corresponding assembly results are shown in Fig. 2, A and B, second column). (B) Fixed shape formed from a varying number of particles (the corresponding assembly results are shown in fig. S3).

We also demonstrated a method to arrest the assembly to form a permanent metallic object. An exemplary result of the fusion is shown in fig. S5, and the details of the fusion method are described in Materials and Methods.

Effects of the number of particles and shape complexity

We investigated the sensitivity of the assembly algorithm to the number of particles involved in the assembly process by forming a fixed shape, the letter “C,” from 25, 50, 75, and 100 particles (fig. S3 shows the final assembly results). As a result, all the corresponding normalized cost function curves exponentially decreased and reached the predefined convergence threshold of J/J_max = 0.08 in 53 ± 12 iteration steps (Fig. 3B), thus implying the negligible effect of the number of particles on the cost function dynamics. On the other hand, following the O(N³) time complexity of the assembly algorithm, the execution time of one iteration step increases cubically with the number of particles, reaching ~2 s for 100 particles. It is noteworthy that about 25% of the execution time is spent on signal generation, which is fixed at 500 ms. The rest is spent on other operations of the assembly algorithm, where the execution time depends on the computational resources available.

In a vibration field, the motion of particles becomes coupled and hard to separate as the center-to-center distance between the particles exceeds the critical threshold related to the wavelength of the vibration (25). Similar difficulties may occur in forming particular shapes requiring particle separation at small distances. These shapes have fully or partially enclosed areas of a width close to or smaller than the smallest node-antinode distance that can be achieved on the plate, which is 4.8 mm (for a frequency of 19.912 kHz). For example, the letter “A” (Fig. 2B, first and second columns) has one fully enclosed triangular area of a width up to 2.8 mm and one partially enclosed trapezoidal area of a width ranging from 4.3 to 6.8 mm. This may result in multiple outlier particles that have failed to separate (see the lower side of Fig. 2B, first column). Thus, a shape becomes more complex with more critical areas prone to producing outlier particles, which, in turn, degrade the quality of the final assembly result. In addition, failures in particle separation can also result in the nonuniformity of the particle distribution within the shape boundaries (e.g., see the upper right side of the letter “O” in Fig. 2B, last column). In practice, the highest shape complexity that we could achieve without producing an excessive number of outlier particles approximately corresponds to the complexity of Latin letters and basic geometric shapes (Fig. 2 and fig. S3), including simple disconnected shapes, as shown in movie S4.

Nonlinear and stochastic motion of particles

As a result of applying spatially highly nonlinear displacement fields, particles tend to travel along highly nonlinear paths during the assembly process. This property is illustrated in Fig. 4A by demonstrating the sorted ratios of the traveled distance to the displacement magnitude of all the individual particles assembled into the letter “L.” All the individual trails and displacements are displayed in fig. S6, and the corresponding assembly result is shown in Fig. 2B (third column). A displacement is a vector whose length is the Euclidian distance from the initial position to the final position of a particle, and a traveled distance is the sum of all distances between each two consecutive steps. The ratio ranges from 1.79 to 77.42, and the average ratio is 8.37, which means that the traveled distances of most particles exceeded their displacement magnitudes by several times. The nonlinearity of motion used in this work distinguishes it from many state-of-the-art assembly methods where particles tend to travel linearly along the spatial gradient of an external field, e.g., acoustic (12) or magnetic field (14), and are eventually trapped in a local or global energy minimum.

Fig. 4. Nonlinearity and stochasticity of the particle motion.

(A) Sorted ratios of the distance traveled to the displacement magnitude of the individual particles involved in the assembly process (the corresponding assembly result, the letter L, is shown in Fig. 2B, third column). (B) Position difference between the particles having identical initial positions (of 0.5-mm precision) and experiencing an identical sequence of notes. The curves are the mean Euclidean distances between the particle pairs, and the shaded areas are the SDs around the means.

The particle motion on a Chladni plate is known to exhibit stochastic behavior and resemble that of Brownian motion (24). To test the stochastic behavior, we repeated six experiments with pairwise identical initial positions (of 0.5-mm precision) of nine particles and an identical sequence of 52 notes and compared the results in pairs. The trails of each repetition are presented in fig. S7. These seemingly identical conditions resulted in the distance between particle pairs growing at the average rate of 57.3 μm per step and reaching a maximum value of 3.5 mm (Fig. 4B). We presume that the stochastic properties of the particle motion are mainly a consequence of two factors: the slightly irregular particle shape and the sensitive dependence of the particles on initial conditions (25). Using Lyapunov exponents, we approximated the rate of separation of trajectories between the particle pairs as ‖δ(k)‖ ≈ ‖δ₀‖e^λk, where δ₀ is an initial separation vector between particle pairs, k is a time step, and λ is a Lyapunov exponent. The Lyapunov exponents ranged from 0.028 to 0.06 (fig. S8); hence, the neighboring particle trajectories separated exponentially fast. It is noteworthy that the machine vision feedback is expected to compensate for these stochastic properties during the assembly process and thus to prevent the prediction error of the neural networks from accumulating over time.

DISCUSSION

We reported a method to programmatically assemble particles into a desired two-dimensional shape by applying a sequence of vibration fields. While similar processes of shape formation in nature are often directed by the external stimuli with stochastic characteristics, such as speed and direction of wind (1), we use a feedback loop to iteratively apply the vibration fields to adjust the assembly process. The assembly algorithm that we developed obtains a user-specified shape, converts the shape into a set of shape-defining target points, or targets, and then iteratively selects and applies the vibration-induced displacement fields to minimize the distances between the targets and the particles based on machine vision feedback. To predict the particle motion, the algorithm uses neural networks trained with particle-tracking velocimetry data.

Previously, we demonstrated independent manipulation of up to four particles toward fixed targets with the present hardware (25). However, using the method proposed in this paper, we can go far beyond this limit and assemble up to a hundred particles into recognizable shapes. Because of the high degree of nonlinearity of the vibration fields and the stochastic behavior of the particles, it is necessary to incorporate the highly flexible algorithms of the proposed method. Eliminating elements, e.g., the rigid transformation of the targets (rotation R and translation v from Eqs. 2 and 3) from the assembly algorithm, which keeps the desired shape fixed, generally results in unsuccessful shape formation; fig. S9 shows the results of forming a shape when the targets are fixed.

Using our method, we formed several two-dimensional recognizable shapes, such as Latin letters and basic geometric shapes, on a Chladni plate. Defining the exact limits of the achievable shape complexity is an open problem because it is related to multiple factors: the degree of similarity between the shape and one of the Chladni figures (fig. S1), the number of critical areas prone to producing outlier particles in the shape, and the prediction accuracy of the neural networks. In addition, the particle motion on a Chladni plate exhibits stochastic behavior, which further complicates the problem.

Although this work focuses on submillimeter particles on a vibrating plate in ambient air, the method is potentially applicable to smaller particles or in a fluid medium. Although the particle motion in those scenarios follows different physical mechanisms, e.g., acoustic streaming (29, 30), acoustic radiation force, and gravity (26), the motion across the whole plate could be modeled in a similar manner (26), and thus, our method can be applied. Because particles of different sizes and materials will move differently on the same vibrating plate, e.g., toward nodal lines or antinodes, our method can potentially be extended to simultaneously assemble two different types of particles. Beyond the Chladni plate, our approach may be applicable to similar systems exhibiting spatially nonlinear and stochastic dynamics, such as systems with turbulent flow fields (31, 32) or microfluidic systems with bulk acoustic waves (33). Furthermore, our method may inspire novel fabrication technologies and applications in cell and particle sorting based on sequences of external force fields rather than on field- or template-based energy minimization and force balance.

MATERIALS AND METHODS

Apparatus

A Chladni plate was fabricated by directly following the approach described in (25). In particular, the plate with dimensions 50 mm by 50 mm by 0.525 mm was diced from a silicon wafer with a mechanical wafer-dicing saw (DAD 3220). The plate was glued to a piezoelectric actuator (Piezomechanik/PSt 150/2 × 3/20, approximately 33 nm/V displacement unloaded, −30 to 150 V) with cyanoacrylate adhesive. To assure the horizontality of the plate, the piezoelectric actuator was mounted on a dual-axis goniometer (Thorlabs, GN2/M) for fine orientation adjustment. The plate was imaged by a camera (ImperX/IGV-B1621C-KC000 with Infinity/InfiniMite Alpha lens), which was mounted on a metallic experimental platform and connected to a control PC. To enhance the visibility of the particles by making them appear bright on a dark background, the plate was illuminated by a light-emitting diode ring, encircling the plate. To produce vibration, the plate was excited with the signals generated by a control PC. The output signal was amplified 20× (Piezosystems/EPA-104-230) before sending it to the piezoelectric actuator. The bias voltage of the driving signal was 60 V. A photograph of the experimental setup is shown in fig. S10.

Particles

In the modeling and assembly experiments, the particles that we used were solder balls (Martin Smt/VD90.5106, Sn96.5Ag3Cu0.5, Ø 600 μm). To make the originally spherical particles less prone to rolling motion on the plate, they were flattened under a mechanical press. A photograph of a representative sample of the particles is shown in fig. S11. Movie S5 shows a high-speed recording of the motion of a particle on the Chladni plate from the side. Alternatively, different particles with similar characteristics, i.e., low sphericity and high reflectivity and heat fusibility or ultraviolet curability, could be used for assembly and fusion.

Uncontrolled amount of pressure applied to flatten the particles led to variations in their height-to-radius ratio. Nevertheless, the motion direction of the particles of varying height and radius on the Chladni plate is generally preserved; movie S6 shows the motion of a disc-like particle and a less flattened solder ball with close initial positions.

Signal shape

Each note is a sinusoid with a triangular envelope of 500-ms duration. The frequencies of the notes correspond to the Western musical scale, from 1047 to 19,912 Hz (in total, 52 distinct frequencies). The voltage amplitudes of the notes vary from 1.49 to 90 V. The amplitudes were tuned for each note individually to produce approximately equal net particle displacements among the notes by directly following the approach described in (25).

Displacement fields

Solder balls (>100, average of 168) were evenly distributed over the whole plate, and 52 notes were played in a random order; each note repeated 50 times. After at least every 25 notes, the particles were redistributed over the whole plate. The position of each particle before and after playing each note was detected with machine vision, resulting in ~6500 data points per note and ~338 × 10³ data points in total. As a result, datasets (p, ∆p) representing the initial positions and displacements of the particles were collected for each note. On the basis of the datasets, we trained neural networks to model note-dependent displacement fields of the plate. Each neural network was a two-layer feedforward network, with a sigmoid transfer function in the hidden layer and a linear transfer function in the output layer. The hidden layer size varied between 14 and 20. The neural networks were trained with the Levenberg-Marquardt backpropagation algorithm.

Fusion of particles

After the desired shape was fully formed from solder balls, we first adhered the assembled particles to a flat piece of silicon with a sticky tape by pressing it down on the particles. Second, we placed the piece of silicon with the particles adhered to it on a hot plate. Third, we placed another flat piece of silicon (∼1.5 g) with an additional weight (∼16.1 g) on top of it such that the particles were sandwiched between the two pieces of silicon. Last, we turned on the hot plate at a set temperature of 315°C. As a result of heating up and experiencing a compressive force, after ∼6 min, the solder balls melted, expanded, and adhered to each other, thus merging and becoming a single metallic object.

Assembly algorithm

The algorithm assembles the particles into a user-specified shape by performing a series of operations in an iterative manner. The assembly problem is formulated as the problem of aligning two sets of points, particles $\mathbf{P}={\{{\mathbf{p}}_{\mathit{i}}\in {\mathrm{ℝ}}^2\}}_{\mathit{i}=1}^{\mathit{N}}$ and targets $\mathbf{T}={\{{\mathbf{t}}_{\mathit{i}}\in {\mathrm{ℝ}}^2\}}_{\mathit{i}=1}^{\mathit{N}}$. The particles are detected in the image delivered by the camera as bright circles with the specified radius using the circle detection method described in (34). The targets are created on the basis of the number of detected particles and the square binary image of the desired shape, where each pixel that is part of the shape is colored white and pixels outside the shape are colored black. The targets are evenly distributed within the inner area of the shape by the algorithm and represent the desired arrangement of the particles. To align the point sets, P and T, the algorithm establishes the correspondence particle-target pairs, optimally transforms the targets, and moves the particles with the selected notes at every iteration step.

The correspondence particle-target pairs are established by matching the closest target for each particle. Specifically, the algorithm builds the cost matrix of all the possible assignments, whose (i, j)th element represents the cost to assign the ith particle to the jth target. The algorithm then computes the set of optimal assignments such that the total cost is minimized. The assignment problem is solved with the Hungarian algorithm (27), and its solution is represented by a set of assigned targets, $\tilde{\mathbf{T}}={\{{\tilde{\mathbf{t}}}_{\mathit{i}}\in {\mathrm{ℝ}}^2\}}_{\mathit{i}=1}^{\mathit{N}}$.

The optimal rigid transformation consists of rotation matrix R ∈ ℝ^{2 × 2} and translation vector v ∈ ℝ^{2 × 1}. The transformation fits the set of assigned targets to the set of particles in the least-square sense because the sets are paired, the algorithm solves the equation below with the Kabsch algorithm (28).

$$[{\mathbf{R}}^{*},{\mathbf{v}}^{*}]={\text{arg} \text{min}}_{\mathbf{R},\mathbf{v}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\Vert \mathbf{R}{\tilde{\mathbf{t}}}_{\mathit{i}}+\mathbf{v}-{\mathbf{p}}_{\mathit{i}}\Vert }^2$$ (4)

The algorithm computes the correspondence particle-target pairs and the optimal rigid transformation for 2π/θ_inc > 0 different orientations of the targets with respect to the current one, where 0 < θ_inc < 2π, a rotation increment, is considered as a design parameter. The assigned and transformed targets of each orientation are then stored, along with the corresponding cost value, i.e., the sum of squared particle-target distances. Afterward, the algorithm extracts the targets associated with the least costly orientation and uses them to select the note.

To select the note, the algorithm calculates the expected positions of the particles for each note n ∈ [1...52] and then selects the note that is expected to minimize the sum of squared particle-target distances

$${\mathit{n}}^{*}={ \text{arg} \text{min}}_{\mathit{n}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\Vert {\tilde{\mathbf{t}}}_{\mathit{i}}-{\mathit{E}}_{\mathit{n}}({\mathbf{p}}_{\mathit{i}})\Vert }^2$$ (5)

where E_n(p_i) = p_i + u_n(p_i) is the expected position of particle p_i as a function of playing note n.

In addition, the particles displaced outside the zone of interest can be ignored by the algorithm. The zone of interest is defined by the current positions of the particles relative to the current positions of the targets. Specifically, the set of particles is converted into the following filtered set

$${\{{\mathbf{p}}_{\mathit{i}}:\Vert {\mathbf{p}}_{\mathit{i}}-{\widehat{\mathbf{t}}}_{\mathit{i}}\Vert \le \frac{\mathrm{\delta }}{{\mathit{N}}_{\text{new}}}{\displaystyle {\sum }_{\mathit{i}=1}^{{\mathit{N}}_{\text{new}}}}\Vert {\mathbf{p}}_{\mathit{i}}-{\widehat{\mathbf{t}}}_{\mathit{i}}\Vert \}}_{\mathit{i}=1}^{{\mathit{N}}_{\text{new}}}$$ (6)

where ${\widehat{\mathbf{t}}}_{\mathit{i}}$ is the target located closest to the particle p_i, δ > 0 is a design parameter, and 0 < N_new ≤ N is the number of particles after the outliers are removed.

The whole procedure is summarized in Algorithm 1. The design parameters used in the experiments were 2π/θ_inc = 8 and δ = 2.5. During the assembly experiments, execution of the algorithm was stopped primarily on the basis of a visual evaluation of the result. For N particles, the complexity of the assembly algorithm is dominated by the Hungarian algorithm, which is O(N³) and executed 2π/θ_inc times at every iteration. The assembly algorithm was designed and applied by using MATLAB R2020b.

Algorithm 1. Assembly algorithm.
Require: Image of the shape, δ, θinc, NNs (neural networks)
P ← detect particles; T ← create targets (P, image of the shape); align the centroids of the sets: $\mathbf{T}=\mathbf{T}-(\frac{1}{\mathit{N}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\mathbf{t}}_{\mathit{i}}-\frac{1}{\mathit{N}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\mathbf{p}}_{\mathit{i}})$; repeat     create empty sets for different orientations of the targets and cost values:     Ts = {};     Costs = {};     set the starting rotation angle and index:     θ = 0;     ind = 1;     repeat         rotate the targets around their centroid:         ${\mathbf{T}}_{\mathrm{\theta }}=\left[\begin{array}{cc}\text{cos}\mathrm{\theta }& -\text{sin}\mathrm{\theta }\\ \text{sin}\mathrm{\theta }& \text{cos}\mathrm{\theta }\end{array}\right](\mathbf{T}-\frac{1}{\mathit{N}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\mathbf{t}}_{\mathit{i}})+\frac{1}{\mathit{N}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\mathbf{t}}_{\mathit{i}}$;         establish the correspondence particle-target pairs:         $\tilde{\mathbf{T}}$ ← assign (Tθ, P);         compute and apply the optimal rigid transformation:         $[{\mathbf{R}}^{*},{\mathbf{v}}^{*}]={\text{arg} \text{min}}_{\mathbf{R},\mathbf{v}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\Vert \mathbf{R}{\tilde{\mathbf{t}}}_{\mathit{i}}+\mathbf{v}-{\mathbf{p}}_{\mathit{i}}\Vert }^2$;         $\mathbf{Ts} [\text{ind}]={\mathbf{R}}^{*}\tilde{\mathbf{T}}+{\mathbf{v}}^{*}$ ;         store the corresponding cost value:         $\mathbf{Costs}[\text{ind}]={\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\Vert {\mathbf{R}}^{*}{\tilde{\mathbf{t}}}_{\mathit{i}}+{\mathbf{v}}^{*}-{\mathbf{p}}_{\mathit{i}}\Vert }^2$ ;         θ = θ + θinc;         ind = ind + 1;     until θ ≥ 2 π     find the index of the least costly set of targets:     ind* ← find (Costs = min (Costs));     T = Ts [ind*];     select the note according to Eq. 5:     n* ← select note (T, P, NNs);     excite the plate with the selected note:     play (n*);     P ← detect particles;     $\mathbf{T}=\mathbf{T}-(\frac{1}{\mathit{N}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\mathbf{t}}_{\mathit{i}}-\frac{1}{\mathit{N}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\mathbf{p}}_{\mathit{i}})$ ;     create a filtered set of particles according to Eq. 6:     P* ← remove outliers (T, P, δ);     create new targets if the number of particles changes:     if length (P*)≠ length (P) then         T ← create targets (P*, image of the shape);         $\mathbf{T}=\mathbf{T}-(\frac{1}{\mathit{N}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\mathbf{t}}_{\mathit{i}}-\frac{1}{\mathit{N}}{\displaystyle {\sum }_{\mathit{i}=1}^{\mathit{N}}}{\mathbf{p}}_{\mathit{i}})$ ;     end if     P = P*; until convergence

Supplementary Materials

This PDF file includes:

Figs. S1 to S11

Table S1

Legends for movies S1 to S6

Other Supplementary Material for this manuscript includes the following:

Movies S1 to S6