Towards a Resilient Intelligent Automation System

1 Introduction

Web Automation primarily involves the automatic control of web browsers to perform repetitive tasks such as data entry, content scraping, and interaction with web applications [Sugiura and Koseki, 1998; Little et al., 2007; Leshed et al., 2008; Le and Gulwani, 2014]. Early web automation tools were primarily developer-centric, requiring a good understanding of web technologies such as HTML, CSS, and JavaScript.

More recently, UI Automation has been evolving into an important solution for non-technical users overwhelmed with repetitive and ad-hoc tasks across enterprise digital platforms. Beyond just streamlining existing work, UI Automation plays a key role in unlocking the value of digital workers and transforming how businesses operate. Given recent advancements in Large Language Models (LLMs), could they potentially overcome the technical challenges and simplify no-code tools to better suit the skills of non-technical users?

Despite these recent advancements, commercial Intelligent Process Automation (IPA) solutions, including UIPAth [Tripathi, 2018], Automation Anywhere [Automation Anywhere, 2023], Microsoft Power Automate [Guilmette, 2020], and IBM RPA [IBM, 2022], encounter challenges in achieving autonomous resilience, particularly in adapting to dynamic UI implementations or layouts. While recent generative AI-based assistants, such as ChatGPT [OpenAI, 2024]

and Adept [Fuyu-Heavy, 2024], and LMM-based web agents [Zheng et al., 2024a; Wang et al., 2024; He et al., 2024; Koh et al., 2024], offer promising approaches by dynamically adjusting their automation strategies in response to UI changes, they often fall short in accurately grounding their actions for effective UI interaction [Zheng et al., 2024b]. These fundamental issues prevent the adoption of automation solutions by enterprises, limiting the fulfillment of a broader range of user needs [Schwartz et al., 2023].

1.1 The IPAChallenge Competition

The IPA Challenge Competition, jointly organized by industry and academia at IJCAI 2023, focused on enhancing the resiliency of IPA through AI. It required participants to develop solutions for automating UI tasks adaptable to changing conditions, a crucial aspect of robust IPA systems. Details of the competition can be found at http://iparesiliency-ijcai23. github.io/. Participants were tasked with creating an AI bot to execute four progressively challenging automation tasks, based on user data from an Excel spreadsheet

Task 1 - Beginner involved form-filling with data that changes position and labels (e.g., the input "Role in company" might be changed to "Job description") after each submission, testing the bot's semantic understanding of form fields. Task 2 - Advanced extended Task 1's complexity with variably positioned form labels and changes in the DOM architecture of the HTML (e.g., the label of the input "Manager" will be changed to "Supervisor" and will become a placeholder), challenging the bot to navigate structure changes. Task 3 - Pro required user lookup and form-filling for "Add" or "Remove" actions, with form elements and labels changing randomly, necessitating advanced adaptation techniques. Task 4 - Pro Max extended Task 3's complexity by introducing a "Reason" field with dynamically changing values and implementations.

Overall, eight teams participated in the challenge utilizing a mix of established IPA tools and custom AI technologies. Only two teams could navigate through the beginner and advanced tasks. The Pro, and Pro-max tasks, requiring nuanced UI understanding and element disambiguation, remained uncompleted by all, underscoring the existing gaps in IPA capabilities. Consequently, the challenge has been opened to the public, aiming to serve as a resilience benchmark and spur further innovation in the field.

2 IDA - AResilient IPA System

We introduce IDA (Intelligent Digital Apprentice) - an innovative no-code UI automation system for business users that successfully passes all the IPA challenges. While IDA includes many features to enable business users to teach it to perform UI automation, in this demo paper we focus mostly on the runtime components that enable IDA to be resilient to changes in the UI. IDA can perform automation flows by following instructions in natural language (Figure 1). Every instruction includes an action type, reference name of the UI element, and optional value parameter. IDA can be triggered to perform the automation on each row of an Excel file and leverage data in Excel as input parameters to the automation flow. In the IPA challenge each row corresponds to a user that needs to be added or updated in a system, and IDA needs to dynamically map column headers of the Excel to UI input elements which constantly change their names and underlying implementation. In the following subsections we describe the key algorithms that enable IDA to execute automation flows in a robust and resilient manner.

2.1 Grounded Element Selection

To automate natural language-based instructions, IDA must understand the UI semantically. For every UI element, IDA needs to understand its type and name (based on visible text). IDA employs several LLM-based pipelines which enables it to ground the instructions to semantic elements that exist in the application. The main steps of the algorithms are:

Step 1 - Discovering new heuristics for detecting UI elements. IDA uses heuristic rules to detect UI element types on the screen. For robustness of the solution, IDA uses LLMs to keep learning about new UI frameworks and implementations and keep its heuristics up to date for complete UI detection coverage. Given a snapshot of HTML from the DOM, the system first prompts an LLM with an HTML snippet of the current UI, and asks it to suggest several HTML heuristics in order to best collect UI elements. This includes identifying buttons, links, inputs, and semantic states of UI elements. The output of this step is a JSON file with the heuristics which is later will be used to extract all the elements. As the HTML DOM is super complex, the motivation behind this step is to use the power and knowledge of LLMs in HTML and web framework specifications.

Step 2 - Semantic UI understanding. In this step, the system applies the updated HTML heuristics to the current page and identifies all the candidate UI elements and their corresponding names. IDA uses CSS selectors to query the DOM (and the Shadow-DOM) based on expanded and refined HTML specification heuristics presented in Figure 2. To determine naming, it combines elements with their near-bylabels using reference attributes and the element's positions. This is done using both pixel-wise proximity algorithm and DOM-based distance. Finlay, the system constructs elements hierarchy such as lists, groups, and forms. For example, our system knows that some inputs are part of the "Add form" while the other belongs to the "Remove form". The output of this step is a full textual (and visual) representation of the current browser content with a unique name for each element.

Step 3 - Instruction to UI element grounding. In this step, IDA performs UI element grounding by matching the referenced element in the instruction to the list of UI elements performed by the Semantic UI Understanding step. IDA first filters all the UI elements that are not relevant to the instruction by using a two-step hierarchy semantic similarity. That is, it compares the cosine similarity score between the embeddings on both the group/form level and then on the elements level itself. In real-world web pages, when the number of elements on the screen can be huge, this is a super crucial step. Next, it prompts an LLM to match the instruction to the list of elements. The LLM task is to choose the right element ID to perform the action. Once grounded, the system then executes the action on the chosen UI element. The usage of multiple-step semantic meaning and an LLM approach enables the system to perform robustly.

2.2 Detection of Semantic Element State

In some cases, detecting the element type and name is not enough and IDA must detect complex UI elements and their internal states. For example in more complex semantic UI elements such as search results tables or attachments, IDA employs an innovative approach to recognize the elements, their names, and their state at a specific state within the automation flow. IDA includes a definition of possible semantic objects and states (e.g., a search result has "one record" or "multiple records" state, whereas file attachment may be "present" or "absent") and uses an LLM to parse HTML snippets and detect the semantic objects and their current states. The LLM task is also used to generate a user-friendly name for the semantic object, as well as to generate a JavaScript code that can evaluate the state of the object at runtime. As presented in the following task prompt (Figure 5) and response from the LLM (Figure 6), the LLM is able to detect the semantic element type (table), its user-friendly name ("Companies table"), and its semantic state ("has multiple records").

In the context of full automation flow, IDA has the ability to perform semantic action on these UI elements and apply a condition on the answer, resulting in different behavior of the system. Currently, this feature was not tested for resiliency as the CSS selector of "0 records found" was not changed in the challenge. Future work will have to tackle these types of issues.

3 Discussion

IDA stands out from state-of-the-art systems through its integration of LLMs for robust semantic understanding and dynamic element matching via grounding instructions to semantic UI elements. This enables IDA to handle complex resiliency issues and surpass competitors in the IPA Challenge. Beyond the IPA challenge, we have put IDA to the test in a wide range of additional real-world applications across diverse platforms such as SAP and Salesforce, where IDA demonstrated exceptional generalization without the need for extensive reconfiguration. This pivotal capability may mark a new era towards a more resilient and trustworthy IPA as well as set new adaptability benchmarks. Despite its innovative approach, IDA faces limitations in scenarios with complex or ambiguous UI elements, where reliance on semantic similarity may introduce inaccuracies. This points to the need for further research to improve semantic analysis and element matching. The IPA Challenge, serving as an open benchmark, underscores the importance of resilient automation solutions. IDA's performance encourages the pursuit of new benchmarks in the field, driving innovation and research toward more adaptive and robust IPA solutions.