Achieving High-Level Goals with Autonomous Language Agents

As large language models (LLMs) continue to advance, they have enabled the creation of autonomous language agents capable of solving complex tasks in dynamic environments without task-specific training. However, these agents often face challenges when tasked with broad, high-level goals due to their ambiguous nature and delayed rewards. The impracticality of frequent model retraining to adapt to new goals and tasks further complicates the issue.

Illustration of an AI framework

Current approaches focus on two types of auxiliary guidance: prior task decomposition and post-hoc experience summarization. However, these methods have limitations, such as a lack of empirical grounding or difficulty in effectively prioritizing strategies.

Enhancing LLM capabilities

The challenge lies in enabling autonomous language agents to achieve high-level goals without training while overcoming these limitations consistently. Prior studies have explored various methods to mitigate these challenges; Reflexion enables agents to reflect on failures and devise new plans, while Voyager develops a code-based skill library from detailed feedback. Some approaches analyze both failed and successful attempts to summarize causal abstractions. However, the learnings from feedback are often too general and unsystematic.

Decomposing high-level goals

LLMs struggle with long-term, high-level goals in decision-making tasks, requiring additional support modules. Decomposition methods like Decomposed Prompting, OKR-Agent, and ADAPT break down complex tasks into sub-tasks or use hierarchical agents. Yet, these approaches often decompose tasks before environmental interaction, lacking grounded, dynamic adjustment.

The SELFGOAL framework

Researchers from Fudan University and Allen Institute for AI propose SELFGOAL, a self-adaptive framework for language agents to utilize both prior knowledge and environmental feedback to achieve high-level goals. The main idea is to build a tree of textual subgoals, where agents choose appropriate ones as guidelines based on the current situation.

The GOALTREE structure

SELFGOAL features two main modules to operate a GOALTREE: a Search Module that selects the most suited goal nodes, and a Decomposition Module that breaks down goal nodes into more concrete subgoals. An Act Module uses the selected subgoals as guidelines for the LLM to take actions.

SELFGOAL in action

This approach provides precise guidance for high-level goals and adapts to diverse environments, significantly improving language agent performance in both collaborative and competitive scenarios.

SELFGOAL results

SELFGOAL significantly outperforms baseline frameworks in various environments with high-level goals, showing greater improvements with larger LLMs. Unlike task decomposition methods like ReAct and ADAPT, which may provide unsuitable or overly broad guidance, or post-hoc experience summarization methods like Reflexion and CLIN, which can produce overly detailed guidelines, SELFGOAL dynamically adjusts its guidance.

SELFGOAL in the Public Good Game

In the Public Good Game, SELFGOAL refines its subgoals based on observed player behaviors, allowing agents to adapt their strategies effectively. The framework also shows superior performance with smaller LLMs, attributed to its logical, structural architecture. In competitive scenarios, such as auction competitions, SELFGOAL demonstrates a clear advantage over baselines, employing more strategic bidding behaviors that lead to better outcomes.

SELFGOAL in auction competition

In conclusion, SELFGOAL represents a significant advancement in enabling autonomous language agents to consistently achieve high-level goals without frequent retraining. By dynamically generating and refining a hierarchical GOALTREE of contextual subgoals based on environmental interactions, SELFGOAL significantly improves agent performance. The method proves effective in both competitive and cooperative scenarios, outperforming baseline approaches. The continual updating of GOALTREE enables agents to navigate complex environments with greater precision and adaptability. While SELFGOAL shows effectiveness even for smaller models, there remains a demand for improved understanding and summarizing capabilities in models to fully realize its potential.