Automatic Truss Design with Reinforcement Learning

1 Introduction

Truss layout design and optimization is a crucial and fundamental research topic in the building industry, as truss layouts can be found in a wide range of structures, including bridges, towers, roofs, floors [Stolpe, 2016; Alhaddad et al., 2020] and even in aerospace and automotive sectors [Wang et al., 2019]. As a basic component in building structures, a truss can support heavy loads and span long distances with a small amount of construction material. Efficient truss layout design can lead to significant cost savings and can also improve the physical performance and safety of the structure.

However, truss layout design is an NP-hard combinatorial optimization problem, which involves the optimization of node locations, topology between nodes, and the crosssectional areas of connecting bars [Fenton et al., 2015]. The possible search space for truss layouts is huge, nonlinear, and non-convex. There are also a number of constraints that must be satisfied, including material strength, displacement allowance, and stability of structural members [Luo et al., 2022b]. Traditionally, engineers design and optimize truss layouts using a combination of mathematical analysis and physical testing based on domain knowledge. They analyze the structural behavior and iteratively adjust the size and shape based on initial sketches [Dorn, 1964]. These methods rely heavily on subjective human expertise, resulting in a cumbersome and restrictive design process. An automated design approach is crucial for achieving greater efficiency and flexibility in the design process.

Previous studies have attempted to automate the design of truss layouts using heuristic algorithms, such as genetic algorithms [Permyakov et al., 2006], particle swarm optimization [Luh and Lin, 2011], simulated annealing [Lamberti, 2008], and differential evolution[Ho-Huu et al., 2016]. However, the size and the complexity of the search space impeded the achievement of optimal results. Note that the entire search space, including node positions, is continuous, and a tiny position change may drastically influence the physical performance of the entire truss layout. So, directly applying search-based methods can be particularly expensive. A low-resolution discretization over the search space may easily miss out on the optimal positions and lead to low-quality solutions [Luo et al., 2022b].

Reinforcement learning (RL) methods have achieved strong results in solving combinatorial optimization problems [Mazyavkina et al., 2021], such as the Traveling Salesman Problem (TSP) [Bello et al., 2016] and drug design [Jeon and Kim, 2020; Yoshimori et al., 2020]. These problems require the solver to find the optimal combination of a finite set of choices to maximize a certain objective function. Truss layout design is a similar combinatorial problem but it has the following differences. Unlike TSP problems where any order of the cities is feasible, truss layout design and optimization have tight physical constraints, making most truss layouts generated from random actions invalid. This in turn makes reward signals sparse for RL training. The objective function is also more complex than that in TSP, since there are more performance indices, like capacity and stability, beyond total mass. The settings of truss layout design are more similar to virtual screening in drug design, namely identifying potential drug candidates from large libraries of compounds [Yoshimori et al., 2020]. They both have complex constraints and performance indices. However, in drug design, there exists a large amount of data that can be used for pre-training [Jeon and Kim, 2020], whereas in truss layout design, little realworld data are available. These facts make it difficult to directly apply end-to-end RL training to truss layout design.

To sum up, the heuristic search methods can generate valid truss layouts, but with sub-optimal quality [Luo et al., 2022b]. On the other hand, RL can produce fine-grained refinement of truss layouts but suffer from sparse rewards. Therefore, we combine them as a two-stage search-and-refine algorithm named AutoTruss. In the search stage, we run a search-based method, Upper Confidence bounds applied to Trees (UCT), to derive diverse truss layouts under all physical constraints. In the refinement stage, we adopt the SAC algorithm to train an RL policy for refining the valid truss layouts from the search stage. We conduct experiments in both 2D and 3D cases, and results show that AutoTruss improved the SOTA performance by 6.8% on average in 2D test cases and as much as 25.1% in the more challenging 3D test cases.

2 RelatedWork

2.1 Truss Representation

A concise representation of a truss layout is fundamental for the truss layout design, which should capture both geometry and load conditions. There are mainly two types of representations: voxel-based [Li et al., 2022], and graph-based [Stolpe, 2016]. We adopt the graph-based method for its accuracy and flexibility. The voxel-based methods divide the design space into small, three-dimensional units called voxels, each assigned a value representing material density [Li et al., 2022; Klemmt, 2023; Du et al., 2018]. These methods cope well with boundary conditions, but cannot accommodate continuous variations in truss topology and is prone to discretization errors.

On the other hand, graph-based methods represent the truss layout as a graph, consisting of coordinates of nodes, bars connecting the nodes, and member area sizes [Fenton et al., 2015; Stolpe, 2016; Lieu, 2022], but often simplifying it to follow certain grids or only connecting neighboring nodes. Based on a graph-based approach, our method adopts a continuous additive method, allowing for greater flexibility in node connection and truss layout by adding nodes and connections freely from scratch. Furthermore, Graph Neural Network (GNN) [Scarselli et al., 2008] is well-suited for processing graph-structured data and complex relationships between elements, thus it is widely used in various real-world applications such as social networks [Li et al., 2021], chemistry [Fung et al., 2021; Yang et al., 2021], and recommendation systems [Guo and Wang, 2020; Wu et al., 2019a].

2.2 Truss Design and Optimization

There have been various methods for truss layout design and optimization over the years. Traditionally, engineers designed truss layouts based on sketches by experience and refined them with analytical math tools [Dorn, 1964]. This empirical method is time-consuming and far from accurate. With the advancement of technology, computer algorithms based on finite element analysis (FEA) have been adopted for faster and more efficient design [Mai et al., 2021]. These algorithms can be divided into two categories: gradient-based and nongradient-based. Gradient-based algorithms, such as steepest descent, are efficient in converging to a solution but can be complex to implement mathematically and often produces local solutions [Banh et al., 2021; Nguyen and Banh, 2018; Banh and Lee, 2019; Lieu, 2022]. On the other hand, nongradient-based algorithms, such as differential evolution (DE) and genetic algorithms (GA), do not require derivative calculations and are more flexible and robust in the presence of multiple local optima. As a relatively new entrant in this category, Monte Carlo Tree Search (MCTS) [Coulom, 2007] has shown to be highly effective in large search spaces with the success of AlphaGo [Silver et al., 2016], as it balances exploration and exploitation [Luo et al., 2022b; Luo et al., 2022a]. Different from previous works which simultaneously optimize truss topology and member sizes, we implement a two-stage search-and-refine approach to sequentially optimize topology and member sizes, which greatly reduces the search space and thus improves the training speed as well as the accuracy of the results. In this paper, we adopt UCT [Kocsis and Szepesvári, 2006], a variant of MCTS, as the search method for deriving various valid truss layouts in the search stage.

2.3 RL for Combinatorial Optimization

Recently, reinforcement learning (RL) has emerged as a powerful tool for solving challenging combinatorial optimization problems, such as virtual screening in drug design [Wu et al., 2019b; Deudon et al., 2018]. Various RL algorithms have been applied in this field, including value-based methods like Q-learning [Khalil et al., 2017], policy-based methods [Bello et al., 2016] and policy-gradient based methods [Kool et al., 2018]. One representative method in RL is Soft ActorCritic (SAC) [Haarnoja et al., 2018], which has been used in robotics [Taylor et al., 2021], autonomous vehicles [Guan et al., 2022], game playing [Zhou et al., 2022] and many others. In this study, we also leverage the power of RL to address a combinatorial optimization problem, which is fine-grain truss refinement. Specifically, we employ SAC algorithm for this task, as it has a high sample efficiency and a strong ability to explore the solution space.

3 Preliminary

3.1 ProblemFormulation

The truss layout design task is to minimize the mass of a truss layout by defining node locations, connections between the nodes, and cross-sectional areas of bars. Formally, a truss layout can be represented as a graph G " pV,Eq, where V is the set of nodes and E is the set of bars. A bar e P E can be defined as a tuple e " pu, v, zq, with nodes u, v P V , and cross-sectional area z P R. The mass can be written as

ÿ

In truss layout design, certain physical constraints need to be satisfied, to ensure displacement, stress, and buckle condition are within capacity, while the length, area, and slenderness of the bars are within the design limit. Constraint details can be found in Appendix A.1. We consider both 2D and 3D settings in this paper. The only difference is the calculation of the bar's cross-sectional area. In 2D settings, the cross-sectional area is only decided by the width of the bar. While in 3D settings, each bar is a hollow round tube, and the cross-sectional area is decided by the outer diameter and its thickness.

3.2 Upper Confidence Bounds Applied to Trees

Upper confidence bounds applied to trees (UCT) algorithm [Kocsis and Szepesvári, 2006] modifies Monte Carlo tree search (MCTS) method with Upper Confidence bounds, which searches for the best termination state s˚ with the highest reward RUCT ps˚ q with a balance between exploration and exploitation [Gelly and Silver, 2007]. Classical UCT is applied to finite states and actions. For each non-termination state s, UCT maintains an action-value function Qps, aq during tree search, which is calculated as Equ. (2):

Qps, aq " βWmeanps, aq ` p1 ´ βqWbestps, aq, (2) where Wmean denotes the average reward of all the termination states in the subtree rooted at state s, andWbest represents the highest reward in the subtree. β is a hyper-parameter to control the exploration preference between the average and the best reward [Kocsis and Szepesvári, 2006].

The policy of UCT πUCT psq selects the action that maximizes the upper confidence bound on the action value by

where npsq is the number of times that state s has been visited, and nps, aq is the number of times that action a has been taken from state s. Whenever a state s is visited, the counter npsq and nps, aq will be increased by 1.

When UCT begins, all the action values will be initialized to 0. In each UCT iteration, the search process starts from the root state s0 and expands the search tree according to Equ. (4). Simulation will be executed till a termination state is reached. Then the counters and the action values of visited state-action pairs will be updated accordingly. The process will be repeated within a given budget of search steps.

3.3 Reinforcement Learning

Reinforcement learning (RL) trains an agent to learn to make decisions by interacting with an environment and receiving feedback in the form of rewards. The agent's goal is to maximize its total reward over time. To apply RL training, we model the problem as a Markov Decision Process (MDP). MDP is parameterized by xS,A,R, P, γy, where S is the state space, A is the action space, R is the reward function, P ps1 | s, aq is the transition probability from state s to state s1 via action a, and γ is the discount factor. The goal is to find a policy πθ parameterized by θ that outputs an action πθpsq P A for each state s and maximizes the accumulative expected reward. The objective function is shown in Equ. (5).

«ÿ

Soft Actor-Critic

Soft Actor-Critic (SAC) is an off-policy reinforcement learning algorithm that combines the actor-critic framework with an entropy term to encourage exploration. SAC optimizes

«ÿ

where H pπq is the entropy of the policy at state st, and α is a temperature coefficient balancing exploration and exploitation. SAC maintains a data buffer D with all the transition samples and learns a soft Q-function Qψps, aq parameterized by ψ. Assuming the policy is parameterized by θ, SAC optimizes the policy by the following objective

The temperature α and the parameter ψ of the soft Q-network are also learned similarly.

4 AutoTruss: A Two-Stage Method

Truss layout design has a huge search space, which makes it extremely expensive for exhaustive search methods to achieve high performance. It is not feasible to apply end-to-end reinforcement learning (RL) methods either, since there are many restrictions on valid truss layouts, yielding highly sparse reward signals. Therefore, we proposed AutoTruss, a two-stage method consisting of a search stage and a refinement stage. In the search stage, AutoTruss uses a UCT search for diverse valid layouts. In the refinement stage, AutoTruss adopts deep RL to further improve the valid solutions. The overview of AutoTruss is shown in Fig. 1 with details described below.

4.1 Search Stage: UCT for Valid Designs

The purpose of the search stage is to find diverse valid truss layouts as a foundation for the refinement stage. We remark that diversity is important since similar topologies from the search stage will yield similar results from the refinement stage, while diverse inputs for the RL policy will improve the overall performances and robustness of AutoTruss.

We use UCT search to find valid truss layouts. We divide the generation process of a truss layout into three steps: node-adding step, bar-adding step, and cross-sectional areachanging step. The pipeline of UCT search is shown in Fig. 2. To be specific, given the initial truss layout G0 " pV0,E0q, our UCT search takes these three steps sequentially to produce a complete layout Gm " pVm,Emq from scratch. In the node-adding step, it adds new nodes to the layout until it reaches the maximum number of nodes, and then in the baradding step, bars with a random cross-sectional area will be added to the truss layout until it satisfies the structural constraints described in Sec. 3.1. Finally, we choose the appropriate cross-sectional area for each added bar in the crosssectional area changing step. Following [Luo et al., 2022b], for each complete truss layout Gm " pVm,Emq, the reward is defined by

κ is a scaling parameter, which is typically chosen to bound the maximum reward below 10 for numerical stability.

UCT Search with Continuous Actions

A challenge when applying classical UCT to truss layout design is that the actions are all continuous. Therefore, for any intermediate truss layout G, finding the optimal UCT action πUCT pGq according to Equ. (4) becomes non-trivial. In AutoTruss, we approximate the best action by drawing random samples and choosing the optimal action from the samples:

In our implementation, we choose N " 25.

Another issue for continuous actions is to compute the action value QUCT pG, aq since there are infinitely many such values to compute leading to an unbounded search tree size. In our implementation, we constrained the expansion size for each intermediate truss layout G such that we at most expand OpanpGqq children to compute the exact values [Yee et al., 2016]. For other state-action pair pG, a1 q without tree expansion, we approximate its QpG, a1 q and npG, a1 q via kernel regression [Nadaraya, 1964] based on the precise values of the expanded actions from G. Suppose there are M expanded actions, i.e., ¯ap1q, . . . , ¯apMq. The value QpG, a1 q can be approximated by

The counts npG, a1 q can be similarly approximated. Here Kp., .q denotes a kernel function. We simply adopt the Gaussian kernel in our implementation.

Diverse Layouts

To get diverse valid truss layouts for the refinement stage, we not only need to save the best truss layout, but also some other suboptimal valid truss layouts. Note that two truss layouts G1, G2 are topologically the same if and only if there exists a permutation σ over node indices such that

It is time-consuming to enumerate all the permutations, we relax the criterion and only adopt the identity permutation in practice for topology checking. Finally, we store the top 5 lightest valid layouts for each topology and use G to denote this set of diverse truss layouts we obtained.

4.2 Refinement Stage: RL for Adjustment

In the refinement stage, we adopt the SAC algorithm to refine those valid truss layouts G generated in the search stage.

Action Space

The RL policy needs to perform two types of actions, i.e., adjust a node position and the cross-sectional areas of a specific bar in a truss layout. For node position refinement, when given a specific node to change, the policy outputs a multi-dimensional vector denoting the change of node coordinates. In the 2-dimensional case, the policy outputs pδx, δyq indicating the change in the node's position. Similarly, in the 3-dimensional case, the policy outputs pδx, δy, δzq. Here all δi ă 0.5 such that the adjustment will be confined to a small zone with dimension no more than 0.5m. This is to ensure that the majority of actions taken by RL will not violate the constraints. For cross-sectional area changes, when given a specific bar to adjust, the policy outputs a single real value for area change in the 2-dimensional case. In the 3dimensional case, the policy outputs two continuous actions, namely changing the outer diameter and changing the thickness of the bar. Note that not all cross-sectional areas are valid in the 3-dimensional case, so the actual values are rounded up to the minimum legal value during execution.

Reward Function

The design principle of the reward function is to (1) penalize invalid layouts and (2) promote lighter layouts. Suppose an action a is taken on an intermediate layout G leading to a refined layout G1 , the reward function is defined as

Network Architecture

The network architecture of the RL policy is shown in Fig. 3. Inspired by the transformer architecture [Vaswani et al., 2017], we adopt (1) a self-attention encoder to extract the spatial relationship between nodes and bars, and (2) an action decoder to output high-precision refinement actions for the node or bar to be operated on in the current iteration. The nodes are represented using coordinates, loads, and whether or not they are supported. The bars are represented as the coordinates of the two end nodes, with (a) the cross-sectional area of the bar in 2D, or with (b) the outer diameter and the thickness of the bar in 3D. All the nodes and bars are passed through an embedding layer and then sent to the self-attention encoder for spatial relationship extraction. The node-bar adjacency matrix is also fed into a self-attention encoder to reflect the topology of the truss layout. Then, the results of the embedding layer for the node or bar being operated in the current iteration will be sent to the action decoder together with its embedding of the self-attention encoder. Finally, the policy outputs both the Q values and the multi-dimensional action. Full details can be found in Appendix A.4.

Rollout Generation for RL Training

In our approach, we employ a probabilistic initialization strategy for the initial state of RL. In particular, we keep maintaining the top-5 diverse layouts in G. When each episode starts, we uniformly sample from G with 50% probability. Otherwise, we alternatively start from the top-5 lightest truss lay-

Algorithm 1 AutoTruss

Inputs: Initial truss layout G0 " pV0,E0q, where V0 means fixed node set and E0 means the fixed bar set.

• 1: Diverse Truss Set G ÐH
• 2: while Tree Search Steps ă Limit do Ź search stage
• 3: Search from pV0,E0q w.r.t. Equ. (9)
• 4: Update counts and value for expanded nodes
• 5: Update G
• 6: end while
• 7: Initialize the policy π, data buffer D Ź refinement stage
• 8: while RL steps ă RL Limit do
• 9: Select initial state from G
• 10: Generate an episode τ w.r.t. the policy πθ
• 11: Update G
• 12: Add τ to D and update πθ via SAC
• 13: end while
• 14: return argminGPG MasspGq

outs found during training without considering topology diversity. The termination criterion of one episode is that the maximum number of 20 actions are performed. We also early terminate an episode if the policy generates 5 invalid layouts within a single episode. In addition, in each RL step, we randomly choose a node or a bar from the current layout for the policy to adjust. More details can be found in Appendix A.5.

. 4.3 Overall Algorithm

We summarize the overall process of AutoTruss in Algorithm 1. The input to the algorithm is the initial truss layout G0 " pV0,E0q as well as the constraints. V0 represents the support nodes and E0 represents the fixed bars. After applying the two stages, the algorithm finally outputs the lightest truss layout ever derived during the entire search process.

5 Experiments

We compare AutoTruss with 3 search-based baselines using both 2D and 3D test cases, where AutoTruss consistently produces the best truss designs. We also evaluate the effectiveness of each module in AutoTruss through ablation studies. We introduce test cases in Sec. 5.1, baselines in Sec. 5.2, and the experiment setup in Sec. 5.3. Main results and ablation studies are in Sec. 5.4 and Sec. 5.5 respectively.

5.1 Testbeds

2D Testbed

We choose two common 2D test cases in truss layout design [Fenton et al., 2015]: the 10-Bar Cantilever Truss (10Bar) and the 17-Bar Cantilever Truss (17-Bar), as shown in Fig.4. Both are common test cases in the field of structure generation and optimization [Assimi et al., 2017; Deb and Gulati, 2001; Tejani et al., 2018; Fenton et al., 2015] . There are 2 load cases in the 10-Bar case. The buckle constraint and the slenderness constraint are not applied to the 10-Bar case. In the 17-Bar case, all the constraints except the slenderness constraint are taken into consideration. The detailed settings are listed in Appendix A.1.

3D Testbed

We select the Cantilever Sundial Design (Sundial) as the 3D testbed, which follows [Luo et al., 2022a]. The test case was adapted from the sundial bracket truss located in Paternoster Square, London, UK [Shea and Zhao, 2004]. As shown in Fig. 5, the Sundial testbed is characterized by a higher dimension and a larger scale and complexity of the solution space when compared to the 2D testbed.

5.2 Baselines

We consider 3 competitors: AlphaTruss [Luo et al., 2022b], KR-UCT [Luo et al., 2022a], and SEOIGE [Fenton et al., 2015]. All the baseline methods can be applied to the 2D testbed, but only KR-UCT can be applied to the 3D testbed. Therefore, we compare 2D results with all three baselines and compare 3D results only with KR-UCT. We utilized the results of the baselines as reported in their original papers, as the test cases and evaluation methods used in those studies were consistent with those employed in our own research. The details of baselines are listed in Appendix A.2.

5.3 Experiment Setup

KR-UCT and SEOIGE use single-stage search while we use a two-stage search scheme. We balance the iterations in the search stage, and the environment steps in the refinement stage to keep a fair comparison. More specifically, we run 2e6 iterations in the search stage, which is half the number of iterations in KR-UCT, and 1.5e5 environment steps for RL training, so that the running time of the refinement stage is similar to the search stage with an RTX 3070 GPU. We remark that AutoTruss consumes substantially fewer trials (i.e., search iterations + RL steps) compared with baselines, and The details can be found in Appendix A.8. We run 3 seeds for each test case and report the best numbers with the mean numbers and standard deviations.

5.4 Main Results

2D Results

The mass of the solutions derived by AutoTruss and baselines are presented in Tab. 1, where p denotes the number of nodes in truss layouts. AutoTruss outperforms the baselines in all the settings by an average of 6.8%, demonstrating its ability to discover more lightweight truss layouts. Moreover, The illustration of the truss layouts is shown in Fig. 6. AutoTruss find the same truss layout topology with better refinement in 10-Bar load I case, and finds better topologies in other cases.

3D Results

Tab. 2 shows a comparison of AutoTruss and KR-UCT in the Cantilever Sundial truss layout design of the 3D testbed. Our method consistently outperforms KR-UCT by at least 25% under all settings. This highlights the effectiveness of AutoTruss in designing lightweight truss layouts within a larger search space. It is noteworthy that 3D truss design poses a greater challenge than 2D truss design, as the search space is substantially enlarged. Our approach exhibits a more significant improvement in the 3D case than the 2D counterpart.

The visualization comparison is shown in Fig. 7. The truss layouts derived by AutoTruss show a more elongated appearance compared with those derived by KR-UCT.

.

5.5 Ablation Study

In this section, we analyze the effectiveness of the two-stage scheme, the usage of diverse truss layouts in the search stage, as well as network architecture, all based on the 10-Bar truss layout design cases of 2D testbed through ablation studies. Results are reported as "mean (standard deviation)".

Search-Stage-Only v.s. Two-Stage

We present truss layouts only derived from the search stage and refined by the refinement stage separately in Fig. 8. In all cases, the refinement stage substantially reduces the total mass of the truss layout by 28% on average, demonstrating the importance of the refinement stage in AutoTruss for further performance improvement.

Usage of Diverse Truss Layouts

To investigate the advantages of the diverse truss layouts derived in the search stage, we use the lightest truss layouts derived in the search stage without different topologies, named AutoTruss w.o. Diverse. The results are presented in Tab. 3. AutoTruss outperforms AutoTruss w.o. Diverse by an average of 3% in all cases, which demonstrates the effectiveness of introducing diverse truss layouts.

Network Architecture

Transformer and GNN architectures are commonly employed to handle graphical inputs. The comparison between GNNbased Policy and AutoTruss is presented in Tab. 4. We utilized CGConv [Fey and Lenssen, 2019] as GNN module, which has been demonstrated to exhibit good performance in material property prediction tasks [Xie and Grossman, 2018]. The node positions, loads, and support information are embedded as nodes, and the cross-sectional area of each bar is recorded as an edge property. After GNN, we extract the embedding of the action node and then concatenate it with the action embedding for the final action. Empirically, we observe that a Transformer-based policy, as we used in AutoTruss, performs slightly better than a GNN-based Policy.

6 Conclusion

We propose a two-stage method AutoTruss that can automatically design truss layouts under various constraints. We use UCT search to find diverse valid truss layouts in the search stage and then use deep RL policy to refine the truss layouts derived in the search stage. AutoTruss outperforms the baselines by 6.8% on 2D testbed and 25.1% on 3D testbed. AutoTruss may perform poorly when generating large-scale spatial structures, and combining basic structural elements in the search stage could accelerate the search speed. We leave this as our future work.

Acknowledgements

The authors would like to acknowledge the financial support provided by the 2030 Innovation Megaprojects of China (Programme on New Generation Artificial Intelligence) Grant No. 2021AAA0150000 and Research on computer-generated intelligent design of buildings Grant No. SYXF0120020110.

Contribution Statement

Authors Weihua Du and Jinglun Zhao contributed equally to this work and should be considered co-first authors.