Four games, one request.

simulate.experiment.sh provides game environments over HTTP for training and evaluating AI agents. It runs chess, Go, no-limit hold'em poker and Connect Four. You create a match, read its observation and legal actions, and send actions back. Rewards come back the moment a match (or a poker hand) finishes.

The batch endpoint is the main way to use it. One request can create matches, play moves in hundreds of them, roll them out to the end and fork them for search. That lets a training loop step a whole population of games in one round trip.

To measure an agent, run an evaluation session. A session schedules matches against other agents or built-in bots, rotates seats, replays the same cards or openings from both sides, and reports bb/100 or Elo with 95% confidence intervals.

• Base URL: https://simulate.experiment.sh
• Format: JSON in, JSON out. Send Content-Type: application/json.
• Authentication: none. A match id is the only key to a match, so treat ids as secrets if that matters to you.
• Interactive reference: /reference (OpenAPI at /openapi.json). This guide as Markdown: /llms.txt.

Quick start

Create a chess match:

curl -s https://simulate.experiment.sh/v1/matches \
  -H 'content-type: application/json' \
  -d '{"game": "chess"}'

{
  "id": "m_Q2b7sV1kz0xYp9aE",
  "game": "chess",
  "status": "active",
  "ply": 0,
  "current_player": 0,
  "num_players": 2,
  "result": null,
  "returns": [0.0, 0.0],
  "observation": {
    "fen": "rnbqkbnr/pppppppp/8/8/8/8/PPPPPPPP/RNBQKBNR w KQkq - 0 1",
    "board": ["rnbqkbnr", "pppppppp", "........", "........",
              "........", "........", "PPPPPPPP", "RNBQKBNR"],
    "turn": "white",
    "in_check": false,
    "last_move": null,
    "fullmove_number": 1,
    "halfmove_clock": 0
  },
  "legal_actions": ["a2a3", "a2a4", "b1a3", "b1c3", "..."],
  "config": {},
  "seed": null,
  "parent": null,
  "created_at": "2026-10-20T16:02:11.402Z",
  "updated_at": "2026-10-20T16:02:11.402Z"
}

Play a move:

curl -s https://simulate.experiment.sh/v1/matches/m_Q2b7sV1kz0xYp9aE/actions \
  -H 'content-type: application/json' \
  -d '{"action": "e2e4"}'

The response holds rewards, the reward each player earned from this request, and applied, the number of actions applied. The match's new state is in state.

Do the same for four games in one request:

curl -s https://simulate.experiment.sh/v1/batch \
  -H 'content-type: application/json' \
  -d '{
    "operations": [
      {"op": "create", "game": "chess",    "label": "c"},
      {"op": "create", "game": "go",       "label": "g", "config": {"size": 9}},
      {"op": "create", "game": "poker",    "label": "p", "seed": 7},
      {"op": "create", "game": "connect4", "label": "f"},
      {"op": "act", "id": "@c", "actions": ["e2e4", "e7e5", "Nf3"]},
      {"op": "act", "id": "@g", "actions": ["E5", "C3"]},
      {"op": "act", "id": "@p", "action": "call"},
      {"op": "rollout", "id": "@f", "seed": 1}
    ]
  }'

Concepts

Match. One game in progress, stored on the server under an id such as m_Q2b7sV1kz0xYp9aE. A poker match can span several hands. Matches untouched for 7 days are deleted.

Players. Seats are numbered from 0. Chess: 0 is white. Go: 0 is black. Connect Four: 0 is X. Poker: seats 0 to players - 1. current_player is the seat to act, or null once the match is over.

Agents. A match can name an agent for each seat with agents, for example ["me", "bot:greedy"]. Seats named bot:<name> are played by the server the moment it is their turn, so you only ever act for your own seats. Without agents your client plays every seat. See Built-in bots.

Ply. The number of actions applied so far. Pass expect_ply with an action to make the server reject the action if another client has moved in the meantime.

Observation. A game-specific JSON description of the position (see Games). In poker each seat sees only its own hole cards; see Viewing poker.

Legal actions. legal_actions lists every action the current player may take as strings. Poker raises take an amount; their bounds are in observation.raise_range.

Rewards and returns. Chess, Go and Connect Four pay +1 to the winner and -1 to the loser when the match ends, and 0 to both for a draw. Poker pays each seat its chip gain or loss when each hand ends. rewards in an act or rollout response is what this request earned. returns in the state is the running total for the whole match.

Result. null while status is active. Once finished, it is an object with winner (a seat, or null for a draw or tie) and reason, plus game-specific fields such as Go's score.

Batch API

POST /v1/batch runs a list of operations in order and returns one result per operation. Operations in a batch can touch any number of matches and any mix of games, up to 1,000 operations per request.

Request

{
  "operations": [ {"op": "...", "...": "..."} ],
  "atomic": false,
  "observation": true,
  "legal_actions": true
}

Operations

• id is a match id, or @name to refer to a match that an earlier create or fork in the same batch gave "label": "name".
• actions applies several actions in order, alternating players as the game dictates. It is all or nothing: if any action is illegal the match is left as it was.
• player makes act fail with not_your_turn unless every action in the operation is taken by that seat. agent does the same by agent name, which is easier in a session where seats rotate. Use one of them when an agent submits only its own moves.
• bot_actions lists the moves built-in bots made after your action, as {"player", "agent", "action"}. rewards includes what those moves earned.
• rollout plays uniformly random legal moves until the match ends or steps moves have been played. Go rollouts never fill the mover's own single-point eyes, and pass only when no other move is left. A seed makes the moves reproducible.
• fork copies a match, including its history and returns, into a new match with its own id (parent points at the source). Use it for tree search: fork, try a line, read the outcome, delete.

Response

{
  "results": [
    {"index": 0, "op": "create", "label": "c", "ok": true, "id": "m_...", "state": {"...": "..."}},
    {"index": 1, "op": "act", "ok": true, "id": "m_...", "state": {"...": "..."},
     "rewards": [0.0, 0.0], "applied": 3},
    {"index": 2, "op": "act", "ok": false, "id": "m_...",
     "error": {"code": "illegal_action", "message": "'e2e5' is not a legal move in this position"}}
  ],
  "summary": {"operations": 3, "succeeded": 2, "failed": 1, "committed": true,
              "steps": 4, "elapsed_ms": 3.1}
}

The batch itself returns HTTP 200 whenever the body is well formed, even if some operations fail. Check each result's ok. A failed operation changes nothing. Later operations still run unless atomic is set. summary.committed is false only when an atomic batch failed and nothing was saved.

Each batch runs inside one database transaction. Two batches that touch the same match run one after the other, never interleaved.

Example: stepping many games at once

A vectorised training loop keeps a list of match ids and sends one batch per step:

import random
import requests

API = "https://simulate.experiment.sh"
N = 64

def batch(operations, **options):
    r = requests.post(f"{API}/v1/batch", json={"operations": operations, **options})
    r.raise_for_status()
    return r.json()["results"]

states = [r["state"] for r in batch([{"op": "create", "game": "connect4"}] * N)]

while any(s["status"] == "active" for s in states):
    active = [s for s in states if s["status"] == "active"]
    ops = [
        {"op": "act", "id": s["id"], "action": random.choice(s["legal_actions"]),  # your policy
         "expect_ply": s["ply"]}
        for s in active
    ]
    by_id = {r["id"]: r["state"] for r in batch(ops) if r["ok"]}
    states = [by_id.get(s["id"], s) for s in states]

print([s["result"]["winner"] for s in states])
batch([{"op": "delete", "id": s["id"]} for s in states])

Example: Monte Carlo evaluation with fork

To score each candidate move from a position, fork the match once per move, play the move, roll the copy out several times, then clean up:

{"operations": [
  {"op": "fork", "id": "m_root", "label": "a"},
  {"op": "act", "id": "@a", "action": "3"},
  {"op": "rollout", "id": "@a", "seed": 1},
  {"op": "delete", "id": "@a"},
  {"op": "fork", "id": "m_root", "label": "b"},
  {"op": "act", "id": "@b", "action": "4"},
  {"op": "rollout", "id": "@b", "seed": 1},
  {"op": "delete", "id": "@b"}
], "observation": false, "legal_actions": false}

Each rollout result's rewards is the outcome of that playout. A match created and deleted in the same batch is never written to storage.

Evaluation

Single matches say little about an agent. The first mover in chess and the button in poker have built-in edges, card luck swamps skill over hundreds of hands, and one bust in a long poker match ends the measurement. An evaluation session handles all of this. You list the agents and how many matches to play. The server schedules them, rotates seats, plays any built-in bots, and keeps a report with confidence intervals.

Creating a session

curl -s https://simulate.experiment.sh/v1/sessions \
  -H 'content-type: application/json' \
  -d '{"game": "poker", "agents": ["mine", "bot:tight"], "matches": 4000, "seed": 1}'

Two agents can be two versions of your own model (["v7", "v6"]), your model against a bot, or two bots for a baseline. Client agents are just names. Your code decides which policy answers when current_agent is one of them.

Running a session

Start matches with next, play them like any other match, and repeat until next returns no matches. Bots move on their own, so a match is always waiting for one of your agents or already finished. Each match carries agents, current_agent, session and session_index. Matches started together can be played in parallel through the batch endpoint.

import requests

API = "https://simulate.experiment.sh"

def policy(state):  # replace with your agent
    legal = state["legal_actions"]
    return "call" if "call" in legal else legal[0]

session = requests.post(f"{API}/v1/sessions", json={
    "game": "poker", "agents": ["mine", "bot:tight"], "matches": 2000, "seed": 1,
}).json()

while True:
    started = requests.post(f"{API}/v1/sessions/{session['id']}/next",
                            json={"count": 200}).json()["matches"]
    if not started:
        break
    active = [s for s in started if s["status"] == "active"]
    while active:
        ops = [{"op": "act", "id": s["id"], "agent": "mine", "action": policy(s)}
               for s in active]
        results = requests.post(f"{API}/v1/batch", json={"operations": ops}).json()["results"]
        active = [r["state"] for r in results if r["ok"] and r["state"]["status"] == "active"]

report = requests.get(f"{API}/v1/sessions/{session['id']}").json()["report"]
for agent in report["agents"]:
    print(agent["agent"], agent["bb_per_100"], agent["bb_per_100_ci95"])

If your client stops part-way, GET /v1/sessions/{id} lists the unfinished match ids in active. Resume them, then carry on with next.

How matches are scheduled

With n seats, match i uses seat rotation i mod n. Agent k sits in seat (k − rotation) mod n, so every agent plays every seat equally often: white and black in chess, the button and the blinds in poker.

With duplicate on (the default), the n consecutive matches of each rotation cycle share one deal:

• Poker: every rotation gets the same cards, so when one agent is dealt aces, the other gets the same aces in the mirrored match. Each match is one hand with full stacks (the cash-game format used in poker AI research), so a big loss cannot end the evaluation early.
• Board games: every rotation starts from the same position after opening_plies random moves, played once with each colour. Use a few opening plies when an agent is deterministic. Otherwise every game between two fixed agents is the same game.

Built-in bots draw their randomness from the deal and the position, so the same seed replays the same session exactly.

Reading the report

GET /v1/sessions/{id} returns progress (matches, started, finished, status) and a report with one entry per agent:

{
  "scored_matches": 4000, "finished_matches": 4000, "groups": 2000, "group_size": 2,
  "agents": [
    {"agent": "bot:tight", "builtin": true, "illegal_actions": 0, "hands": 4000,
     "net_chips": 2265.0, "net_big_blinds": 1132.5,
     "bb_per_100": 28.31, "bb_per_100_ci95": [19.98, 36.65], "mbb_per_hand": 283.1,
     "by_position": {"button": {"hands": 2000, "bb_per_100": -23.62},
                     "big_blind": {"hands": 2000, "bb_per_100": 80.25}}},
    {"agent": "bot:calling_station", "...": "..."}
  ]
}

That is bot:tight against bot:calling_station over 4,000 hands with "seed": 1.

Poker results are in big blinds, so they compare across blind sizes:

• bb_per_100: big blinds won per 100 hands, the standard win rate. mbb_per_hand is the same number in milli-big-blinds per hand (1 bb/100 = 10 mbb/hand), the unit used in poker AI papers.
• bb_per_100_ci95: the 95% confidence interval. If it excludes 0, the result is unlikely to be luck.
• by_position: win rate from each seat (button, small_blind, big_blind, then seat_3 and up).

Board games are scored 1 for a win, ½ for a draw and 0 for a loss:

• wins, draws, losses, and score, the average.
• score_ci95: the 95% confidence interval for score.
• elo and elo_ci95: the rating difference against the other agent implied by score. Elo is null at a score of exactly 0 or 1, where it is unbounded.
• los: likelihood of superiority, the probability that this agent is genuinely stronger, from wins and losses.
• by_seat: results as white/black (chess), black/white (Go) or first/second (Connect Four).
• average_margin (Go): average points won by, negative when behind.

Every agent also gets illegal_actions, the number of actions it submitted that were rejected as illegal. This is often the first thing to fix in a language-model agent. Rejected actions change nothing, and the agent tries again.

Method. Statistics are computed over groups. With duplicate, a group is one deal or opening played in every rotation, and its value is the agent's average over those matches. Without it, each match is a group. Only complete groups are scored, so a running session's report is unbiased, and scored_matches can trail finished_matches by a few. Intervals are normal approximations: mean ± 1.96 × the standard error of the group values. GET /v1/sessions/{id}/results returns every match's raw outcome (deal, rotation, agents, returns, winner, reason, plies) if you want to compute your own statistics.

How many matches

The interval shrinks with the square root of the number of groups: four times the matches halves its width. Some reference points:

Duplicate scheduling removes card luck that both agents would play the same way. It narrows intervals most when agents play similar strategies, and only a little when they play very differently. Decide the number of matches before you start. Stopping the first time the interval excludes zero makes false positives much more likely.

Fair play

So that a report measures the agent and nothing else, session matches have these restrictions while they are running:

• An agent cannot act for a seat played by a built-in bot.
• Poker matches only show the acting seat's view (view must be current), and cannot be forked, since a copy would reveal the deck and the other hands.
• Session matches cannot be rolled out, and cannot be deleted until they finish. Fork a chess, Go or Connect Four match to search from a copy.

There is no authentication, so these rules keep honest clients honest rather than stop a determined cheat. Finished matches are unrestricted.

Built-in bots

GET /v1/bots lists these. Use them as fixed reference opponents. They are simple on purpose, so a learning agent should beat all of them.

Bots work in ordinary matches too: {"op": "create", "game": "chess", "agents": ["me", "bot:greedy"]} gives a training opponent that replies as soon as you move.

Endpoints

The single-match endpoints run one batch operation and return its result directly. Errors use the HTTP statuses listed in Errors.

Games

GET /v1/games returns the same information in machine-readable form.

Chess

"game": "chess". Standard rules, including castling, en passant, promotion and under-promotion.

• Actions: UCI (e2e4, e7e8q) or SAN (Nf3, O-O, exd8=Q+). legal_actions is sorted UCI. resign is accepted from the player to move.
• Ends on: checkmate, stalemate, insufficient material, threefold repetition, the fifty-move rule, resignation, or max_plies. result.reason is one of checkmate, stalemate, insufficient_material, threefold_repetition, fifty_moves, fivefold_repetition, seventyfive_moves, resignation, max_plies.
• Observation: fen, board (8 strings from rank 8 to rank 1, uppercase white, . empty), turn, in_check, last_move, fullmove_number, halfmove_clock.

Go

"game": "go". Area scoring with positional superko. Suicide is illegal.

• Actions: a point in GTP notation (D4, Q16). Columns run A to T without I, and row 1 is the bottom. Also pass and resign. Case does not matter.
• Ends on: two consecutive passes, resignation, or max_moves. The board is then scored by area: stones plus empty regions that touch only one colour, with komi added for white. Every stone on the board counts as alive, so capture dead stones before passing. result.score holds black, white and margin.
• Observation: size, board (rows from the top, X black, O white, . empty), to_play, komi, captures, consecutive_passes, last_move, move_number, and area_score, the score if the game ended now.
• Action indices: if your network wants integers, point (row, col) with row 0 at the top maps to col_letter + str(size - row).

Poker

"game": "poker". No-limit Texas hold'em for 2 to 9 players.

• Actions:
• fold (only when facing a bet), check, call.
• raise:<total>, where <total> is your whole bet for this street, not the amount added. Bounds are in observation.raise_range.
• raise:min, raise:half_pot, raise:pot and raise:max. Bare raise means raise:min.
• bet:<total> is the same as raise:<total>, and allin is a shortcut for the biggest legal raise or call.
• legal_actions names the action kinds (fold, call, raise or check, raise).
• Rules:
• Heads-up, the button posts the small blind and acts first before the flop.
• An all-in raise smaller than a full raise does not reopen raising for players who have already acted.
• Side pots are split among the best eligible hands. Odd chips go to the first winner left of the button.
• When no more betting is possible, the remaining board is dealt out automatically.
• The button moves to the next seat with chips after each hand.
• Rewards: each seat's chip gain or loss, paid when a hand ends. returns is the match total.
• Result: winner is the seat with the most chips (null on a tie), plus stacks, hands_played, and each seat's net_chips, net_big_blinds and bb_per_100. With reset_stacks the winner is the seat that won the most chips.
• Observation:
• Hand state: hand_number, street, board, button, blinds, pot, current_bet, to_call, raise_range, and actions (this hand so far).
• players: each seat's stack, bet (this street), committed (this hand), status (active, folded, allin or out), and cards.
• last_hand: the previous hand's board, actions, pots, payouts and showdown cards.
• Seeds: the deck for hand k is shuffled from seed and k, so the same seed and the same actions replay the same match. seed is echoed in the state only when you supplied it. A server-chosen seed stays private so agents cannot predict the cards.
• History: the state keeps the last 100 finished hands.

Viewing poker

view decides whose hole cards appear in observation.players[*].cards:

• "current" (default): the seat to act, which is what that agent should see. Once the match is over, every hand is shown.
• A seat number: that seat's view, whoever is to act.
• "all": every hand, for logging or a referee.

Anyone with the match id can ask for any view. The server does not hide information between clients, so keep views honest in your own training code.

For search over hidden information, fork with reshuffle_seed reshuffles the undealt deck in the copy, so rollouts from the fork see different runouts. Opponents' hole cards are not resampled.

Connect Four

"game": "connect4". Seven columns, six rows. Four in a row horizontally, vertically or diagonally wins. A full board is a draw.

• Actions: column 0-6, as a string or an integer.
• Observation: board (6 strings from the top row down, X, O, .), to_play, last_move.

Errors

Single-match endpoints return an HTTP error status. Batch results carry the same error object inside the failing result:

{"error": {"code": "illegal_action", "message": "column 3 is full"}}

Limits

• 1,000 operations and 20,000 actions plus rollout moves per request. Bot moves are not limited, since they always end with the game.
• Sessions schedule up to 100,000 matches, and next starts up to 1,000 at a time.
• About 25 seconds of work per request. A rollout still running at that point stops early with truncated: true; call it again to continue.
• Matches expire after 7 days without activity. Delete them when you are done.
• One host serves every client. For large jobs, prefer fewer, larger batches over many small concurrent ones.