# Orlo: production AI evals, evidence, and release guardrails

Markdown version of https://useorlo.com/, generated 2026-10-13 from the page itself. Illustrations and interactive examples are left out.

**Hold every AI answer and agent action to a standard your team wrote.**

Models you compare, answers your application returns, steps your agents take, and the AI tools your people use all follow the same examples, rules, and skills.

## Review close scores before switching models.

The inbox flags comparisons where the results are too close to separate the candidates. More examples can help you decide.

## Use the same rules in testing and production.

Keep your task’s examples and rules connected to the release your application uses.

### [Compare models on your own examples.](https://useorlo.com/evaluation.html)

Run the same examples through each candidate. Read the failed answers before you choose.

### [Require approval for selected actions.](https://useorlo.com/agents.html)

Let an agent look things up. Have a person approve the actions that need judgment.

### [Check answers while your task is live.](https://useorlo.com/deployment.html)

Review the request, response, and release together when an answer fails a rule.

## Turn reviewed corrections into evaluation examples.

A live answer can reveal a case your examples missed. Review the correction before adding it to the task’s dataset.

### Correct the ticket’s priority.

A reviewer catches an outage that the model treated as a routine question.

### Add the correction to your examples.

The example records the correct priority and whether a person should handle the ticket.

### Include it in the next comparison.

Run each candidate against the reviewed ticket to check whether it makes the same mistake.

## See how Orlo checks a live answer.

Explore a support answer with a missing field and inspect the warning Orlo returns with it.

## About Orlo

Orlo is evals, evidence, and release guardrails for production AI, made by fineSample. Teams use it to compare AI models on their own examples with confidence intervals, release the winner as a frozen version, check every live answer against their rules, decide what each agent step may do, and keep a record of what ran. It runs as a hosted service or on a team’s own infrastructure and calls the models a team chooses rather than hosting any. Orlo is not a model host, an agent framework, or a compliance system. Documentation is at [docs.useorlo.com](https://docs.useorlo.com/).

## More

Orlo is a fineSample product. Documentation: https://docs.useorlo.com/

Other pages as Markdown: [Evaluation](https://useorlo.com/evaluation.md), [Agent permissions](https://useorlo.com/agents.md), [In production](https://useorlo.com/deployment.md). Index: https://useorlo.com/llms.txt