BackLuke Scott Notes
IndexPractice operations
Practice operations

Measure AI by the work that gets finished

I want to know what our AI tools help us finish. That might be an email campaign that goes out, a website improvement that works, an internal tool the team actually uses, or a customer follow-up that happens when it should.

The spending is easier to count. Subscriptions and usage charges arrive on a bill. The work takes a little more attention.

Rippling's AI Spend Console is one example of connecting those records. The company describes breaking down AI costs by team or department and comparing token spending with pull requests and code reviews. Those comparisons can help a team investigate where the money goes. They still need someone to judge whether the resulting work was useful.

A small team can start with a much simpler record. The approach I would use is to measure one recurring job from preparation through its accepted result.

Include the software cost, the time spent preparing context, the time spent reviewing the output, and the corrections needed before anyone can use it. Decide in advance what a completed job means.

For an email campaign, a draft is one stage. A completed campaign has approved copy, working links, the right audience, and a confirmed send. For a website change, the work includes checking the result and making sure the relevant forms or integrations still function. For follow-up, the useful result is an appropriate message reaching the right person, with the next step recorded.

I would keep a short record for each workflow:

  • What job were we trying to finish?
  • What counted as an acceptable result?
  • What did the tools and human review cost?
  • How much correction did the result need?
  • What happened after we used it?

That last question depends on the job. A campaign may lead to inquiries or appointments. An internal tool may remove a repeated manual step. A support workflow may help the team resolve an issue sooner. Pick something that tells you whether the work served its purpose.

I would also separate the first attempt from repeated use. It can take a day to develop a good process. That day may be worthwhile if the next ten jobs become easier, but you need to look at those later jobs to find out. Saving the process gives the team a chance to reuse it; it doesn't prove the savings on its own.

This is where a shared repository helps. Keep the instructions and approved example next to the notes from the work. When you try a different model or change the process, you have a baseline to compare against. You can evaluate whether it produced acceptable work with less correction or lower cost.

A cheaper model that needs an hour of cleanup may be a poor choice for that job. A more expensive one may be worthwhile when it consistently completes the difficult part. The useful comparison includes what people have to do afterward.

Start with one workflow and record the effort for the next few runs. Review the finished work with the person responsible for it, including an example that needed substantial correction. Use that record to decide whether to keep the process, change the tool, or spend more time improving the instructions.

Luke Scott