Researchers from Soochow University, China, have introduced a novel framework known as Chain-of-Tools (CoTools) to enhance how large language models (LLMs) utilize external tools. CoTools aims to provide an efficient and flexible approach that allows LLMs to leverage vast toolsets directly within their reasoning processes, including tools they haven’t been explicitly trained on.
This capability offers a promising advantage over existing methods, making it possible for enterprises to build more powerful and adaptable AI agents without facing the typical limitations of current tool integration techniques.
While modern LLMs excel in tasks like text generation and reasoning, their ability to interact with external resources such as databases or applications is essential for expanding their practical utility. The challenge lies in equipping these models with external tools—essentially APIs or functions they can call—to extend their capabilities.
Current methods for enabling tool use often face trade-offs. One common approach involves fine-tuning LLMs with tool usage examples, but this restricts the model to only those tools. Additionally, this fine-tuning can compromise the general reasoning abilities of the LLM, negatively impacting its core strengths.
Overcoming ICL Limitations with CoTools to Enhance Tool Use in Frozen LLMs
An alternative approach, in-context learning (ICL), allows LLMs to be provided with descriptions of available tools and examples of how to use them directly in the prompt. This method offers flexibility, as it enables the model to potentially use tools it hasn’t encountered before.
However, constructing such complex prompts becomes cumbersome, and the efficiency of the model decreases as the number of available tools increases. This makes ICL less practical for scenarios involving large or dynamic toolsets, as the model’s performance suffers from the added complexity.
CoTools Framework Enhances Large Language Models’ Interaction with External Tools for Greater FlexibilityCoTools addresses the limitations of existing methods by combining aspects of fine-tuning and semantic understanding while keeping the core LLM “frozen,” meaning its original weights and reasoning capabilities remain unchanged. The framework trains lightweight, specialized modules to work alongside the LLM during its generation process, rather than fine-tuning the entire model.
CoTools uses the semantic representation capabilities of the frozen foundation model to determine when and which tools to call, tapping into the LLM’s internal representations or “hidden states” as it processes text and generates responses.
Three Key Components of CoTools: Judge, Retriever, and Calling for Efficient Tool Usage
The CoTools framework consists of three main components that function sequentially during the LLM’s reasoning process. First, the Tool Judge analyzes the hidden state associated with the next token and determines whether calling a tool is necessary. If a tool is needed, the Tool Retriever selects the most suitable tool by comparing the query’s embedding with available tools, including unseen ones.
Finally, the Tool Calling component uses an ICL prompt to fill in the tool’s parameters and executes the tool, integrating the result back into the LLM’s response. This approach separates decision-making and tool selection from parameter filling, achieving efficiency even with large toolsets.
CoTools was tested across two distinct application scenarios: numerical reasoning using arithmetic tools and knowledge-based question answering (KBQA). In both cases, CoTools showed promising performance. In numerical benchmarks, it matched or exceeded other methods, while in KBQA tasks, it demonstrated superior tool selection accuracy, especially in scenarios with massive or unseen toolsets.
For enterprises, CoTools offers a powerful framework to build adaptable LLM-powered agents, especially as standards like the Model Context Protocol (MCP) make tool integration easier. It could enable the development of AI assistants capable of interacting with diverse tools and systems with minimal retraining.
