Sandbox Methods Explained

Sandbox methods are the primary way for Nekro Agent plugins to provide functionality and interaction capabilities to AI. When AI executes code in its sandbox environment, it can call these methods to obtain information, perform operations, or request further processing. Understanding the mechanism, types, and writing specifications of sandbox methods is crucial for developing effective plugins.

What are Sandbox Methods?

When AI needs to perform operations beyond its own capabilities (such as accessing external APIs, reading/writing files, executing complex calculations), it generates a piece of code that usually calls specific functions provided by plugins. These functions exposed by plugins for AI to call are what we refer to as "sandbox methods."

Core Feature: RPC Execution Mechanism

A very important concept is that although these methods are called by AI in a "sandbox" environment, their actual execution occurs in the main service process of Nekro Agent, not in an isolated sandbox. This communication is achieved through RPC (Remote Procedure Call).

• AI initiates a call request in the sandbox.
• The Nekro Agent core system receives the request and finds and executes the corresponding plugin sandbox method in the main process.
• The execution result of the method is returned to the sandbox environment through RPC for AI's subsequent use.

The advantage of this mechanism is that plugin methods can access all resources of the main service environment (such as database connections, core service APIs, etc.) while maintaining the isolation and security of the sandbox environment. However, developers also need to pay attention to the differences between the two environments, especially when handling file paths and other issues (see the File Interaction chapter for details).

Registering Sandbox Methods

Sandbox methods are registered to the plugin instance through the @plugin.mount_sandbox_method() decorator.

from nekro_agent.api.plugin import SandboxMethodType
from nekro_agent.api.schemas import AgentCtx

@plugin.mount_sandbox_method(
    method_type=SandboxMethodType.TOOL,
    name="calculate_sum",
    description="Calculate the sum of two numbers."
)
async def my_sum_function(_ctx: AgentCtx, num1: int, num2: int) -> int:
    """Calculate and return the sum of two integers.

    Args:
        num1 (int): The first addend.
        num2 (int): The second addend.

    Returns:
        int: The sum of the two numbers.
    """
    return num1 + num2

Decorator Parameters:

• method_type (SandboxMethodType): Specifies the type of sandbox method, which determines how AI uses the method and how the framework handles its return value. See the next section for details.
• name (str): The method name displayed to users in the frontend. Naming should be concise and accurately describe the method's functionality. It doesn't have to be the same as the Python function name.
• description (str): A detailed description of the method, which will be displayed to users in the frontend.

Sandbox Method Types (SandboxMethodType)

SandboxMethodType is an enumeration type that defines different behavior patterns for sandbox methods. Choosing the correct type is crucial for smooth interaction between plugins and AI.

1. SandboxMethodType.TOOL (Tool Method)

• Purpose: Provide specific, directly usable tools or functions. After AI calls them, it can directly utilize their return results for subsequent thinking or generating responses. These methods usually perform calculations, data retrieval, simple operations, etc.

• Return Value: Can be any built-in serializable Python type (such as str, int, float, bool, list, dict, etc.), or custom objects that can be serialized by pickle. The framework will directly pass the return value to the code in the sandbox that called this method.

• AI Interaction: After AI calls it, it will wait for the method to finish executing and obtain the return value, then continue executing its task based on the return value. Usually does not directly trigger a new round of AI replies.

• Examples: Mathematical calculations, text translation, unit conversion, retrieving structured information from specific data sources, checking system status, etc.

  @plugin.mount_sandbox_method(SandboxMethodType.TOOL, "get_current_time", "Get the current date and time.")
  async def get_time(_ctx: AgentCtx) -> str:
      import datetime
      return datetime.datetime.now().isoformat()

2. SandboxMethodType.AGENT (Agent Method)

• Purpose: Used to execute operations that require further processing or interpretation by AI, or to provide information that requires AI to perform iterative thinking. These methods usually involve active interaction with the external world (such as web searches), generation of complex information, or scenarios where AI needs to generate new rounds of dialogue based on results.

• Return Value: Must be of string (str) type. This string is usually a description of the result of the executed operation or information that needs AI to understand and generate replies based on.

• AI Interaction: The string returned by the method will be added to the current conversation context and immediately trigger AI to perform a new round of thinking and reply. AI will treat this returned string as a new observation result or information input.

• Examples: Performing web searches and returning summaries, calling external knowledge bases, generating complex text content for AI to polish, operations that require further user confirmation, etc.

  @plugin.mount_sandbox_method(SandboxMethodType.AGENT, "search_knowledge_base", "Search the internal knowledge base based on keywords and return relevant information.")
  async def search_kb(_ctx: AgentCtx, query: str) -> str:
      # Assume search_internal_db is a function that performs the actual search
      results = await search_internal_db(query)
      if not results:
          return f"No information related to '{query}' was found in the knowledge base."
      return f"Knowledge base search results for '{query}' are as follows:\n{results}"

3. SandboxMethodType.BEHAVIOR (Behavior Method)

• Purpose: Used to execute certain operations or modify system state. AI needs to know the result of the operation, but this result itself should not directly trigger a new round of AI replies. Usually used to execute some "side effect" operations.

• Return Value: Must be of string (str) type. This string is a description of the execution result of the behavior.

• AI Interaction: The string returned by the method will be added to the current chat record as a system message for AI and user reference. However, unlike the AGENT type, it will not immediately trigger a new round of AI replies. AI may refer to this information in subsequent thinking.

• Examples: Sending messages/emails (returning send status), setting timers (returning success message), modifying certain configuration items (returning modification results), executing some background tasks without user perception and recording results.

  from nekro_agent.api import message
  
  @plugin.mount_sandbox_method(SandboxMethodType.BEHAVIOR, "send_channel_message", "Send a message to the current session.")
  async def send_chat_message(_ctx: AgentCtx, text_to_send: str) -> str:
      try:
          await message.send_text(_ctx.from_chat_key, text_to_send, _ctx)
          return f"Message '{text_to_send[:30]}...' has been successfully sent."
      except Exception as e:
          return f"Failed to send message: {e}"

4. SandboxMethodType.MULTIMODAL_AGENT (Multimodal Agent Method)

• Purpose: Similar to the AGENT type, but specifically used for processing and returning multimodal content (such as images, audio, etc.) and requiring AI to understand and respond to this multimodal content.

• Return Value: Must be multimodal message segments that conform to a specific structure (usually OpenAI-defined message formats, such as message objects containing image URLs or base64 data).

• AI Interaction: The returned multimodal message segments will be added to the current conversation context and immediately trigger AI to perform a new round of thinking and reply. AI should be able to process and understand this multimodal content.

• Examples: Generating an image and returning it to AI, converting text to speech and having AI play it, analyzing a user-uploaded image and returning analysis results in a combined text-image message.

  # Pseudo-code example, actual structure depends on Nekro Agent's handling of multimodal messages
  @plugin.mount_sandbox_method(SandboxMethodType.MULTIMODAL_AGENT, "generate_image_and_comment", "Generate an image on a specific topic with a comment.")
  async def generate_image_with_comment(_ctx: AgentCtx, topic: str) -> List[Dict[str, Any]]: # Return type might be a dict
      image_url = await generate_image_service(topic) # Assume this is an image generation service
      comment = f"This is an image about '{topic}'."
      # Return multimodal message structure conforming to OpenAI or Nekro Agent internal specifications
      return [
          {
              "type": "text",
              "text": comment
          },
          {
              "type": "image_url",
              "image_url": {
                  "url": "data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAUA..." # (Base64 encoded data of the image)
              }
          },
      ]

Choosing the Right Type:

Requirement Scenario	Recommended Type
AI needs to directly use the data returned by the method for subsequent processing	TOOL
AI needs to have a new round of dialogue based on the text output of the method	AGENT
After AI executes an operation, only the result needs to be recorded, no immediate dialogue	BEHAVIOR
AI needs to have a new round of dialogue based on the multimodal content returned by the method	MULTIMODAL_AGENT

Writing Sandbox Methods: Specifications and Best Practices

To ensure AI can correctly and efficiently use your sandbox methods, please follow these specifications:

1. Clear Function Signatures and Type Annotations:

   • All sandbox methods must be asynchronous functions (async def).
   • The first parameter must be _ctx: AgentCtx, which provides session context information.
   • Add clear Python type annotations for all parameters and return values. This not only helps with code maintenance but is also an important basis for AI to understand parameter types.

2. Detailed and Structured Docstrings: This is one of the most important sources of information for AI to understand and use your sandbox methods! Docstrings should follow a certain format (such as Google style, Numpy style) and at least include:

   • Concise summary line: Clearly explain the functionality of the method.
   • Detailed description (optional): If needed, further explain the working method, limitations, or notes of the method.
   • Args: section: List all parameters (except _ctx), explaining the name, type, and meaning of each parameter.
   • Returns: section: Explain the type and meaning of the return value.
   • Example: section (strongly recommended): Provide one or more Python call examples that AI can understand and imitate. This is particularly useful for complex methods.

   @plugin.mount_sandbox_method(SandboxMethodType.TOOL, "get_user_preference", "Get a specific preference setting for a specified user.")
   async def get_user_preference(_ctx: AgentCtx, user_id: str, preference_key: str) -> Optional[str]:
       """Get the user's preference setting value.
   
       Retrieve the corresponding preference setting value from plugin storage based on user ID and preference key name.
   
       Args:
           user_id (str): The unique identifier of the user.
           preference_key (str): The key name of the preference setting to query (e.g., "theme", "language").
   
       Returns:
           Optional[str]: If the preference setting is found, return its string value; if not found, return None.
       
       Example:
           ```python
           # Get the 'theme' preference setting for user 'user_123'
           theme = get_user_preference(user_id="user_123", preference_key="theme")
           if theme:
               print(f"User's theme is: {theme}")
           else:
               print("User's theme preference setting not found.")
           ```
       """
       # ... implementation logic ...
       stored_value = await plugin.store.get(user_key=user_id, store_key=preference_key)
       return stored_value

3. Appropriate Naming:

   • The name parameter of the sandbox method (the name exposed to AI) should use clear, concise naming conventions that align with AI calling habits (usually snake_case or camelCase, depending on AI preferences, but maintain consistency within the plugin).
   • Python function names can be more descriptive.

4. Error Handling:

   • Properly handle possible exceptions within the method (such as API call failures, file not found, data format errors, etc.).
   • Try not to let uncaught exceptions be directly thrown to AI. If possible, include error information in a friendly way in the return value (especially for TOOL types), or log detailed information.
   • For AGENT or BEHAVIOR types, if a serious error occurs, you can explain the error situation in the returned description string.

5. Return Value Handling:

   • Strictly adhere to the type requirements for return values of the selected SandboxMethodType.
   • Ensure the returned data is understandable and processable by AI.

6. Correct Use of Context (AgentCtx): The AgentCtx object (_ctx) contains key information about the current session, such as _ctx.from_chat_key (session identifier), ctx.from_user_id (user identifier), etc. Use this information as needed, for example, for data storage that distinguishes between different sessions, getting message senders, etc.

By following these specifications, you can create powerful sandbox methods that are easier for AI to understand, more willing to use, and less prone to errors.