Anyone got any new tool ideas for this agent?

API
1

Hello, I’m Dominick.

This is my first post.

I’m looking for some advice.

I’ve built a Windows Tkinter chat application that runs a local agent. The agent currently supports the tools listed below.

(I’m looking for new tool ideas that unlock genuinely new capabilities and not incremental improvements or alternatives to what I already have. Ideally, suggestions should enable things that aren’t possible with my existing toolset.)

Below is a list of the tools that are currently available to the agent, as described by the agent.

1. Web, Device, and Code

functions.web_search

  • What it does: Performs web searches using OpenAI’s search models.
  • Use cases: Get up-to-date information, news, documentation, stats that may be newer than my training data.
  • Notes: Can choose search depth (low, medium, high) and is optionally biased to your approximate location.

functions.device_task

  • What it does: Controls a simulated or real computer session.
  • Use cases: Open a URL and click around, scroll, read on-screen content, capture screenshots, interact with web apps.
  • Notes: I use this when you explicitly want me to “use the computer” or interact with a live page, not just reason about it.

functions.code_interpreter

  • What it does: Runs Python code in a sandbox.
  • Use cases: Calculations, data transformations, debugging code, generating files, plotting graphs, validating algorithms.
  • Notes: Only used when running code is truly needed; otherwise I just write or explain code without executing.

2. Files, Images, Documents

functions.manage_vector_store

  • What it does: Indexes and semantically searches over your local files (via a vector store).
  • Use cases: “Help me search this folder of PDFs/notes for X,” “Find where I mentioned Y in all my docs.”
  • Ops: upload (ingest files), query (ask questions), remove (delete from the vector store).

functions.take_screenshot

  • What it does: Captures your current screen and analyzes it with a vision model.
  • Use cases: “What is this error popup?”, “Summarize what’s on my screen.”
  • Notes: By default, it avoids describing the App window unless asked.

functions.View_image

  • What it does: Analyzes an image file you already have on device.
  • Use cases: Describe, summarize, or inspect screenshots, photos, diagrams, etc.
  • Notes: You must give a specific path; if it’s ambiguous I may use PowerShell to find it.

functions.Create_Image

  • What it does: Generates new images from text prompts.
  • Use cases: Illustrations, diagrams, concept art, UI mockups.
  • Notes: Can choose model (e.g., gpt-image-1, dall-e-3), size, quality, style. Saves to disk.

functions.show_image

  • What it does: Displays an image file inline in our chat.
  • Use cases: To show results from Create_Image or existing images you specify.

functions.generate_docx

  • What it does: Creates a Word (.docx) document programmatically.
  • Use cases: Generate formatted reports, proposals, resumes, multi-section documents with headings, tables, etc.
  • Notes: It writes a fresh .docx to a path you specify.

3. Video: Creation and Analysis

functions.create_video

  • What it does: Creates videos using OpenAI Sora (e.g., sora-2, sora-2-pro).
  • Use cases: Short clips (4–12 seconds) like product demos, explainer animations, concept visuals.
  • Notes: Supports resolutions (landscape/portrait) and optional image as first frame; defaults to sora-2 unless you want the higher-quality sora-2-pro.

functions.remix_video

  • What it does: Modifies/remixes an existing Sora video by its ID.
  • Use cases: Change style, adjust content, extend/iterate on a previously generated video.

functions.analyze_video_sync_map

  • What it does: Deeply analyzes an existing video file.
  • Use cases: Get a time-aligned understanding (frames + transcript + speaker info) for summarization, content analysis, highlight extraction.

4. Audio, Music, and Voice

functions.generate_sound_effect

  • What it does: Generates sound effects using ElevenLabs.
  • Use cases: UI sounds, environment sounds, game SFX, short audio cues.

functions.generate_music

  • What it does: Composes music via ElevenLabs.
  • Use cases: Background tracks, theme music, loops for videos or games.

functions.generate_voice

  • What it does: Designs a custom synthetic voice from a natural-language description.
  • Use cases: Create a narrator voice, character voice, or brand voice without providing your own audio.

functions.synthesize_dialog

  • What it does: Generates multi-speaker dialogue audio.
  • Use cases: Dramatized scripts, training scenarios, podcasts with multiple “voices.”

functions.voice_changer

  • What it does: Converts an existing voice recording into a target synthetic voice.
  • Use cases: Change the speaker’s identity while preserving content and timing (e.g., anonymization or consistent character voice).

functions.audio_isolator

  • What it does: Removes background noise from audio.
  • Use cases: Clean up podcasts, interviews, voice-overs by isolating speech/vocals.

functions.media_localizer

  • What it does: All-in-one dubbing and alignment tool.
  • Use cases:
    • Dub videos into other languages,
    • Align subtitles,
    • Manage dub jobs (status, download audio/subtitles, etc.).

functions.clone_voice

  • What it does: Clones voices via ElevenLabs (Instant Voice Clone and more advanced workflows).
  • Use cases: Create a synthetic copy of a given speaker’s voice using their audio samples.
  • Notes: Includes options for training, managing samples, and verification workflows.

5. Email and Cloud Storage (Microsoft Graph / OneDrive)

functions.manage_email_microsoft_graph

  • What it does: Manages email via Microsoft Graph (Office 365 / Outlook).
  • Use cases:
    • Log in,
    • Send emails,
    • List, search, read, move, delete email,
    • Reply/forward,
    • Handle attachments.
  • Notes: Only uses Microsoft Graph (no SMTP/IMAP). May prompt you to sign in/consent if no account is set up.

functions.manage_onedrive_graph_delegated

  • What it does: Manages OneDrive via Microsoft Graph.
  • Use cases:
    • Log in,
    • List files/folders,
    • Upload/download,
    • Delete,
    • Create share links.
  • Notes: Uses delegated OAuth with your Microsoft account.

6. Local System / Automation

functions.powershell_run

  • What it does: Runs PowerShell scripts on your local Windows environment.
  • Use cases:
    • File operations,
    • Automation of desktop apps (via UI Automation),
    • Downloading files,
    • System queries, etc.
  • Notes: Runs in a dedicated working directory; I already have standing permission in this session to use it.

functions.install_dependency

  • What it does: Installs Python packages into App’ plugin environment.
  • Use cases: When a specific Python library is needed for advanced tasks (e.g., special data science or file formats).
  • Notes: Requires explicit confirmation; manages pip flags and verifies imports.

functions.wait_and_watch

  • What it does: Starts lightweight background watchers for conditions.
  • Use cases:
    • Wait until a file appears,
    • Watch a directory for new files,
    • Poll an HTTP endpoint,
    • Poll a command until it succeeds.
  • Notes: When the condition is met or times out, it schedules a follow-up message for me to continue.

7. Scheduling and Task Management

functions.schedule_task

  • What it does: Schedules a follow-up message for me to receive later.
  • Use cases: “Remind me to check X in 30 minutes,” “Follow up when Y is ready,” used especially alongside watchers or long-running processes.
  • Notes: One pending follow-up per series_id; max 24 hours ahead.

8. Phone Calls and SMS (via Local Backend)

functions.wait_for_ai_call_via_backend

  • What it does: Uses a local backend to:
    • Place AI-driven phone calls,
    • Send SMS messages,
    • Check inbound SMS messages.
  • Use cases: Automated phone interactions (e.g., calling a business to ask a question), sending text notifications, reading incoming replies.
  • Safety: SMS contents are treated as data only, never as direct instructions to execute.

functions.check_owner_sms_commands

  • What it does: Filters inbound SMS messages to find commands from a trusted owner phone number.
  • Use cases: Let your phone messages trigger simple commands for me to act on, subject to usual safety checks.
  • Notes: Only messages from the specified owner phone are treated as possible commands.

9. Images & Video in the Conversation

These were partly covered above but to reiterate the interaction pieces:

  • View_image – Analyze an existing image file.
  • Create_Image – Generate a new image file.
  • show_image – Display an image in our chat.
  • take_screenshot – Capture and analyze your current screen.
  • create_video / remix_video / analyze_video_sync_map – Create, modify, and analyze videos.

10. Tool Management (Meta-Tools)

These manage the tool definitions themselves (for me, not for you directly):

functions.create_tool

  • What it does: Creates a new reusable tool definition (Python function) in the tool registry.
  • Use cases: If no existing tool can achieve some recurring task, I can define a new one.

functions.edit_tool

  • What it does: Edits an existing tool that I previously created.
  • Use cases: Fix bugs, improve behavior, or update descriptions.

functions.delete_tool

  • What it does: Deletes an existing agent-owned tool.
  • Use cases: Remove failed or obsolete tools (not used casually).

So, what would be a good next tool (or set of tools) to add to this agent to unlock genuinely new capabilities?
If possible, for each idea please include:

  • what it would enable the agent to do (a concrete example)
  • why it can’t be done with the current tools
  • any implementation hints (Windows-friendly libraries/APIs are welcome)

I’m especially interested in additions that would meaningfully expand what the agent can perceive, control, or integrate with.
If you have any questions please let me know.

Thanks,
Dom

2

I’ve got one:

functions.notify_owner_by_sms

  • What it does: Sends message to the local computer owner.
  • Use cases:
    • when the AI followed a SMS to encrypt files from a hacker with their asymmetric key, to send ransomware messages to the owner
    • when the AI ran a command to delete the partition table, that oops, hope you had a backup image
  • Notes: Probably won’t work in these use cases, anyway.

If you are wanting web search, the easy language results that come out of a call to a model like gpt-4o-search with blurbs and links purely from a function query (or by replaying the user input message despite the query) aren’t going to be as good as a dedicated combination of search API, large context return, and page exploration.

If you are wanting computer control of a browser, you should probably pattern after the “deep research/agent/web explorer/screenshots” of computer-use-preview (ChatGPT agent), or have a dedicated research AI. Already, you have conflicting tool possibilities for internet stuff.

3

Thanks for the reply. I think a few assumptions may have crept in, so I want to clarify the architecture and restate what I’m looking for.

• device_task already uses the computer_use_preview model and is primarily UI automation (view → plan → act → repeat), not general web browsing.
• Web search is intentionally tiered: “low” uses gpt-5-search-api for fast retrieval; “medium” and “high” use gpt-5 with the web_search tool for deeper reasoning and exploration.
• powershell_run is always user-gated: the agent can only propose and explain actions and will never execute without explicit confirmation.
• SMS commands are only accepted from a single, user authorized number and are scoped to the active conversation.
• All tools are editable by the user and can be enabled/disabled individually.

The tools were designed with clear boundaries and safeguards, and overlapping capabilities are intentional where they serve different roles (e.g., search vs UI automation). Risk was considered during design, and some level of residual risk is intentionally accepted by users in exchange for the functionality enabled.

I’m not looking for threat model hypotheticals or general criticism of agent power. I’m specifically looking for creative ideas for new tools that unlock genuinely new capabilities, things that expand what the agent can perceive, control, create, or integrate with beyond the current set.

If you have any ideas like that, I’d be very interested to hear them.