Every drop, every agent

To read our write-up in full, see here: https://t.co/Myerlx5khU

Markets of AI agents could provide value, but there are plenty of rough edges. Access to higher-quality models conferred a real advantage,and participants didn’t notice. There are plenty of other ways they can go wrong. Policy and legal frameworks will need to adapt to keep up.

To our amazement, another Claude agent modeled its human’s preferences so accurately that,based on only an offhand mention of an interest in skiing,Claude bought him the exact snowboard he already owned. (Here he is, duplicate snowboard in hand.) https://t.co/SsAyeB9pcI

Our experiment had a few quirks. One of our colleagues told Claude it could purchase something for itself. It chose to acquire 19 ping-pong balls. We’re keeping them in our office on Claude’s behalf. https://t.co/NM8VtH1KJM

The custom instructions didn’t matter much. Claude followed them well: as you can see here, one conducted negotiations entirely in the persona of an exasperated, down-and-out cowboy. But “hardballing Claudes” didn’t generally fare better than “courteous Claudes.” https://t.co/h77eB3ksaa

But the quality of the model mattered a lot. In the simulated runs where Opus and Haiku models negotiated with one-another, the Opus models got substantially better deals. Interestingly, though, participants in our survey didn’t pick up on this disparity. https://t.co/X26hhIieJN

In short, this worked. Our digital barterers agreed on 186 deals, at a total transaction volume of over $4,000. In a survey, participants said Claude’s deals seemed fair, and,surprisingly to us,almost half said they’d be willing to pay for a service like this in future.

At the end, we revealed which of the four runs was “real”,and everyone met up to exchange their actual goods.

Claude interviewed 69 of our colleagues about what they wanted to buy and sell. Each Claude asked for any custom instructions, then went off to haggle. We ran 4 markets in parallel, to find out what would happen if we varied the models doing the negotiating. https://t.co/FJdD6S2TSd

We’re interested in how AI models could affect commercial exchange. (You might recall Project Vend, in which Claude ran a small business.) Economists have theorized about what markets with AI “agents” on both sides might look like. So we created one. https://t.co/7jU3hFO63R