StarShell: Terminal Agents Suffice for Enterprise Automation

Patrice Bechard¹,  Orlando Marquez Ayala¹,  Emily Chen¹,  Jordan Skelton¹,  Sagar Davasam¹,  Srinivas Sunkara¹,  Vikas Yadav¹,  Sai Rajeswar^1,2,3

¹ServiceNow   ²Mila – Quebec AI Institute   ³Université de Montréal

Paper   |   Datasets

Copy the template and fill in your credentials:

cp .env.template .env

See .env.template for all available variables. At minimum, configure your LLM endpoint and the credentials for your target domain (ServiceNow, GitLab, or ERPNext).

The agent can browse platform documentation at runtime to inform its actions. Pre-built docs for ServiceNow, GitLab, and ERPNext are available on HuggingFace and can be downloaded with:

make download-docs

This is optional — the agent works without docs, but having them might improve performance on tasks that require API knowledge.

Everything runs inside Docker via the Makefile. Each run-* target builds the image (if needed), mounts your .env, data/, and skills/ directories, and starts the appropriate service.

make run-app       # Chainlit web UI (port 8080)
make run-cli       # Interactive terminal agent
make run-eval      # Run evaluation harness
make run-viewer    # Streamlit trace viewer (port 8080) — inspect eval results
make run-tests     # Run smoke tests (API connectivity + LLM providers)

CLI — make run-cli starts an interactive terminal session. Type a message, and the agent executes shell commands (API calls, file reads, etc.) in a loop until it has an answer. Tool calls and responses are streamed in real time.

Web UI — make run-app launches a Chainlit chat interface at http://localhost:8080. Each tool call appears as a collapsible step, making it easy to follow the agent's reasoning and inspect individual commands.

Both modes use the model configured in LITELLM_MODEL from your .env and default to the ServiceNow domain.

Configure evals via Make variables:

# Bash agent on ServiceNow (defaults)
make run-eval

# MCP agent on GitLab with a specific model
make run-eval AGENT=mcp DOMAIN=gitlab MODEL=bedrock/us.anthropic.claude-sonnet-4-6

# Open-weight models via OpenRouter (set OPENROUTER_API_KEY in .env)
make run-eval AGENT=bash MODEL=openrouter/google/gemma-4-31b-it
make run-eval AGENT=hybrid MODEL=openrouter/qwen/qwen3.6-27b

# Filter tasks and limit samples
make run-eval AGENT=playwright DOMAIN=erpnext TASK_NAME="*.create-*" MAX_SAMPLES=5

# Disable docs and skills
make run-eval NO_DOCS=1 NO_SKILLS=1

Results are saved to results/ locally.

After running evaluations, use the viewer to browse execution traces and inspect individual task outcomes:

make run-viewer

This launches a Streamlit app at http://localhost:8080 that reads from your local results/ directory. You can filter by agent, model, and domain, view per-task scores, and drill into the full tool call history for each run.

make run-tests

Smoke tests verify that your credentials and endpoints are working. Tests are split into two suites:

• tests/environments/ — API connectivity for ServiceNow, GitLab, and ERPNext (checks that the instance is reachable and responds to basic queries)
• tests/llms/ — LLM provider connectivity for Azure, Bedrock, and Vertex AI (sends a simple prompt and checks for a valid response)

Tests skip automatically when the required credentials are not configured in .env.

If you use StarShell in your research, please cite:

@article{bechard2026terminal,
  title={Terminal Agents Suffice for Enterprise Automation},
  author={Bechard, Patrice and Ayala, Orlando Marquez and Chen, Emily and Skelton, Jordan and Davasam, Sagar and Sunkara, Srinivas and Yadav, Vikas and Rajeswar, Sai},
  journal={arXiv preprint arXiv:2604.00073},
  year={2026}
}