Show HN: Give your AI agent on-screen guides that show users where to click | Hacker NewsHacker Newsnew | past | comments | ask | show | jobs | submitloginShow HN: Give your AI agent on-screen guides that show users where to click5 points by pancomplex 1 hour ago | hide | past | favorite | discussHey HN. I'm Christian, one of the founders of Frigade (YC W23). I've noticed that a lot of in-app AI agents struggle to actually understand the products they exist in.For instance, let's say a user asks an agent how to do something in a given SaaS product. In an ideal case, maybe that agent replies saying it has a tool to do the task and just automates that work entirely for the user. That's a great outcome.But often that's not the case. Maybe there is no tool call for that exact task, or maybe the user's question is best solved by a specific UI workflow or interface. In these cases, many agents tend to fall back on basic RAG on their help center, or sometimes even searching the internet for an understanding of their own product. This can be a very slow process and most of the time help center articles are outdated as products evolve faster than them today. Even worse, no one likes reading a long list of bullets and mapping that back to a UI.My tool (Assist API) solves this gap with a single tool call defined like this: const frigade_guide_tool = {
description: 'Call this tool to answer product questions or guide the user through a task.',
parameters: {
query: {
type: 'string',
description: 'What the user is asking or wants to do',
},
},
run: ({ query }) => frigade.assist({ query }),
}

Here's a demo I recorded on how to set it up with the Vercel AI SDK: https://www.youtube.com/watch?v=9WQ0UbLjC6IWhen called, the tool will do the following:1) Gather context on what the user is seeing on screen, their permissions, feature flags, and more. Then one of the following:2a) If solvable: Generate an on screen guide for how to fix a given problem2b) If conceptual: Return text describing to the parent agent how to solve the problem2c) Reject (i.e. unable to help)How does the tool know what to do?The tool learns a given application UI by using a browser-based agent. You provide a test account to your software (i.e. staging og preview), and a browser agent logs in and works its way through the entire product. It then builds its own map of how the application works which can the be queried about any product-related question or how to get from A to B in the UI. It also writes its own documentation from this map. The agent re-runs on a schedule or can be triggered through CI/CD.Docs and more details: https://frigade.com/assist-api help

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact
Search:

AI agents operating within software applications frequently encounter difficulties in genuinely understanding the products they interact with. A significant challenge arises when a user asks an agent how to perform a specific task within a given Software as a Service environment. Ideally, an agent could automate the task entirely by utilizing available tools; however, often, the required action necessitates understanding a specific user interface workflow rather than a direct tool call. When agents cannot find a direct tool for a request, they often rely on inefficient methods such as retrieving information from help center articles or conducting general internet searches. This fallback mechanism is slow, and help center documentation is often outdated, presenting users with long, unstructured text that is difficult to translate into actionable steps directly related to the user interface.

To address this gap, the author introduced a solution via a tool called the Assist API. This tool is designed to facilitate answering product questions or guiding users through specific tasks via a single tool call. When invoked, the tool performs a multi-step process. First, it gathers comprehensive context regarding what the user is currently viewing on the screen, including the user's permissions and feature flags. Based on this context, the tool can execute one of three outcomes: if the task is solvable, it generates an on-screen guide explaining how to resolve the specific issue; if the problem is conceptual, it returns descriptive text detailing the solution for the parent agent; or if the agent is unable to provide assistance, it rejects the request.

The intelligence of the Assist API stems from how the underlying tool learns the application's structure. This learning process is achieved through a browser-based agent that interacts directly with the application. This agent is provided with test accounts for staging or preview environments and navigates through the entire product. By doing so, the agent builds an internal map of how the application functions, specifically charting the paths between different elements on the user interface. This mapping allows the agent to accurately answer questions about the product or guide users through navigating from one point to another in the UI. Furthermore, the agent generates its own documentation based on this derived map, which can be updated through scheduled runs or integration with continuous integration and continuous deployment pipelines.