1
00:00:14,940 --> 00:00:16,560
Okay.

2
00:00:16,940 --> 00:00:18,539
Let's get started.

3
00:00:19,480 --> 00:00:24,889
So, programming assignment, 4…

4
00:00:25,880 --> 00:00:31,870
is due in a couple days, right? So, it's, November 21st.

5
00:00:32,689 --> 00:00:36,010
so hopefully that's going well.

6
00:00:36,930 --> 00:00:41,130
That is the one where you are reproducing multi-agent debate.

7
00:00:41,600 --> 00:00:44,080
Actually, I've had a couple questions about it.

8
00:00:46,970 --> 00:00:54,650
So… A couple students have raised… An issue that,

9
00:00:55,630 --> 00:00:58,859
With, like, the strength of current models.

10
00:00:59,580 --> 00:01:04,510
They're having trouble, reproducing the improvement on these tasks.

11
00:01:04,879 --> 00:01:09,330
Right? And so… You know, there's…

12
00:01:09,930 --> 00:01:13,369
There's quite a few things you can… you can try to do.

13
00:01:15,020 --> 00:01:22,110
To compensate for that, Right? But if there's not much headroom for the
debate to improve, then…

14
00:01:22,700 --> 00:01:25,230
That's kind of a null result.

15
00:01:25,980 --> 00:01:32,829
But, so one thing you can do is you can use older, smaller models,
right? So,

16
00:01:33,610 --> 00:01:38,110
In this paper itself, you know, it's from 2023.

17
00:01:38,750 --> 00:01:42,730
So, obviously those models are quite old.

18
00:01:43,280 --> 00:01:51,649
You can also use no thinking, right? Or low thinking, low effort levels,
you know, all of that.

19
00:01:52,150 --> 00:02:01,060
But… Okay, so let me know if you have any… Particular questions about
this one.

20
00:02:03,290 --> 00:02:04,150
Okay.

21
00:02:04,590 --> 00:02:06,869
But, you know, of course, if you fixed a model.

22
00:02:07,470 --> 00:02:12,750
and you didn't see any improvement, that's also fine, like, for you to
report.

23
00:02:12,950 --> 00:02:14,210
In your report.

24
00:02:15,370 --> 00:02:22,390
Okay, so, we're mostly through the…

25
00:02:22,630 --> 00:02:26,449
the topics on the syllabus, right? So we're…

26
00:02:26,960 --> 00:02:29,319
As far as content goes in this class.

27
00:02:29,710 --> 00:02:36,410
We, we've covered pretty much everything we set out to.

28
00:02:37,020 --> 00:02:41,800
Today, I'm going to talk about agent benchmarks and evaluation.

29
00:02:42,150 --> 00:02:44,150
Which is an interesting topic.

30
00:02:45,610 --> 00:02:49,620
In and of itself, even for single agents, so we'll start there.

31
00:02:50,190 --> 00:03:00,700
And then we'll talk a little bit about multi-agent benchmarks, which of
course, you know, there's much fewer of, and have their own particular
problems, in addition to

32
00:03:00,910 --> 00:03:02,729
Whatever they might inherit.

33
00:03:03,320 --> 00:03:05,850
From the, single agent setting.

34
00:03:06,830 --> 00:03:09,619
Agent benchmarks and evaluation.

35
00:03:10,190 --> 00:03:12,370
Okay,

36
00:03:15,540 --> 00:03:20,470
So, first, like I said, we're gonna talk about the single agent setting.

37
00:03:20,730 --> 00:03:25,499
And all the issues… That have come up over the years.

38
00:03:26,230 --> 00:03:31,139
So, SWE-bench is a software engineering benchmark.

39
00:03:31,620 --> 00:03:32,980
Very famous.

40
00:03:33,120 --> 00:03:42,079
And it's had… So many, so many problems, okay?

41
00:03:42,600 --> 00:03:45,399
So it was launched in October 2023.

42
00:03:46,300 --> 00:03:56,069
This timeline has some of the… some of the attempts to fix it, and
problems that were discovered with it, right?

43
00:03:56,570 --> 00:04:03,910
So, different versions were released to try to, address some of the
issues, so the verified

44
00:04:04,060 --> 00:04:08,840
Like, a year later, OpenAI…

45
00:04:09,140 --> 00:04:12,349
stopped using it in February 2026.

46
00:04:12,900 --> 00:04:20,250
Then, like, a successor came out in July 2026, SWE-bench Pro. SWE-bench
Pro.

47
00:04:20,520 --> 00:04:22,939
That one also had issues. Okay.

48
00:04:23,310 --> 00:04:26,960
So, so let's go through some of the issues.

49
00:04:28,790 --> 00:04:38,520
that… that came up and continue to come up. And of course, it's not
just, specific to SWE-bench itself, right?

50
00:04:39,090 --> 00:04:42,100
So… Okay.

51
00:04:42,930 --> 00:04:51,409
So, let's ask, why did OpenAI stop reporting, on SWE-bench Verified
February 2026?

52
00:04:51,940 --> 00:04:53,540
So, they did an audit.

53
00:04:53,740 --> 00:05:00,119
Okay, so they took… One of their models, so o3, Their reasoning model…

54
00:05:00,390 --> 00:05:03,089
From a couple years, or a year or two ago.

55
00:05:03,650 --> 00:05:10,850
138 problems on SWE-bench that it was not able to solve, over 64 runs.

56
00:05:11,290 --> 00:05:16,109
They put 6 software engineers per problem, at least, on it.

57
00:05:16,680 --> 00:05:22,979
And they discovered… That, like, 60% of those instances had,

58
00:05:23,320 --> 00:05:27,629
have problems, right? So material issues. Okay, so…

59
00:05:28,170 --> 00:05:35,670
And those, you can break down into different categories. So, narrow
tests, we'll talk about, wide tests.

60
00:05:36,280 --> 00:05:38,729
And then other issues. Okay.

61
00:05:39,150 --> 00:05:45,920
So… So, that's, that's a problem, right?

62
00:05:46,660 --> 00:05:50,460
And then contamination is another… is another problem.

63
00:05:50,570 --> 00:05:56,649
Okay, so contamination means… That the benchmark was released.

64
00:05:56,820 --> 00:06:01,100
You know, and was probably in the training data, or, you know.

65
00:06:01,220 --> 00:06:07,350
artifacts from it could have been in the training data for the model
itself, right?

66
00:06:07,820 --> 00:06:15,290
So, for example, If the gold patches are in the training data, right,
then the frontier models

67
00:06:15,430 --> 00:06:17,210
And reproduce those.

68
00:06:17,600 --> 00:06:25,000
Without even, you know, without trying… without needing to, derive them
their selves, right?

69
00:06:25,460 --> 00:06:34,159
So, so that's also going to limit how effective the benchmark is, right?

70
00:06:34,560 --> 00:06:41,130
So, Epoch AI… Reviewed the benchmark as well. Verdict flawed.

71
00:06:41,550 --> 00:06:46,770
No longer measures stated capability due to contamination. Okay.

72
00:06:47,010 --> 00:06:54,740
But even aside from contamination… so contamination is pretty easy, or
relatively easy, to, to deal with.

73
00:06:55,660 --> 00:07:00,039
You know, and there's different methods of trying to do that. One thing,
of course, is just…

74
00:07:00,580 --> 00:07:09,020
testing on problems that were developed post the training cutoff, right?
So that automatically addresses that issue.

75
00:07:09,860 --> 00:07:15,289
But, right, so SWE-bench Pro.

76
00:07:15,410 --> 00:07:19,479
So, as I saw in the timeline, that was released in July 2026.

77
00:07:19,930 --> 00:07:24,269
So just… You know, a few months ago, like, 5 months ago or something.

78
00:07:24,920 --> 00:07:28,130
And that one didn't last very long either.

79
00:07:28,710 --> 00:07:34,479
Right, so Frontier models went from 23% to 80% in 8 months.

80
00:07:35,250 --> 00:07:38,480
Audited again.

81
00:07:38,940 --> 00:07:45,810
And maybe, like, 30% of it is… is still broken.

82
00:07:45,910 --> 00:07:46,590
Okay.

83
00:07:47,290 --> 00:07:53,669
So, okay, so the question is, why? Like, well, what's going on? Why… why
is it so hard?

84
00:07:53,960 --> 00:07:56,520
To come up with a benchmark that's not broken.

85
00:07:57,220 --> 00:08:03,980
And there's actually just… A large host of issues that can go wrong.

86
00:08:04,080 --> 00:08:04,830
Okay.

87
00:08:05,240 --> 00:08:08,500
So, one of those is overly strict tests.

88
00:08:09,370 --> 00:08:14,660
Right, so testing for things that weren't really specified in the prompt
itself.

89
00:08:15,700 --> 00:08:25,509
Right? Underspecified prompts, so the prompt itself that you give the
model

90
00:08:25,910 --> 00:08:31,550
they're actually, you know, they're very sensitive to that. You know,
like, if you just gave…

91
00:08:31,960 --> 00:08:35,099
Even beyond the task specification itself.

92
00:08:35,350 --> 00:08:40,140
Right? If you just tweak the prompt a little bit, you're gonna get
different behavior.

93
00:08:40,380 --> 00:08:45,769
Right, so that makes it really hard to, like, ask, what are we actually
measuring here?

94
00:08:46,580 --> 00:08:52,860
Low coverage tests, misleading prompts, Okay, so…

95
00:08:53,090 --> 00:08:58,560
Yeah, OpenAI retracted their recommendation to use SWE-bench Pro, okay?

96
00:08:59,100 --> 00:09:06,889
Evaluation flaws are easier to detect now.

97
00:09:07,460 --> 00:09:16,849
Right, but of course, this is starting to become circular, you know,
like, we're using AI to detect the evaluation flaws that we're supposed
to be measuring the AI with.

98
00:09:16,950 --> 00:09:21,110
Right, so… But… okay.

99
00:09:21,580 --> 00:09:27,429
But then the question is, you know, like, these numbers have been used

100
00:09:28,040 --> 00:09:30,839
To quantify how good these, like, you know, if you've…

101
00:09:31,260 --> 00:09:42,030
if you're interested in LLM models, you have almost certainly looked at
quite a few benchmarks, right? To try to compare models and be like,
well.

102
00:09:42,170 --> 00:09:44,079
The latest one is better.

103
00:09:44,530 --> 00:09:50,110
But what does that actually mean? You know, like, what are these numbers
even measuring?

104
00:09:50,620 --> 00:09:56,760
Given that there's all these problems, and we can sort of try to
investigate that

105
00:09:57,020 --> 00:10:00,810
You know, more formally and structured as well.

106
00:10:01,170 --> 00:10:05,569
Right, but are we really measuring capability, which is what we want,
right?

107
00:10:05,920 --> 00:10:09,230
Are we measuring, like, test suite quirks?

108
00:10:09,510 --> 00:10:11,230
Memorization…

109
00:10:11,740 --> 00:10:21,049
How does the scaffold, like, how does the scaffolding, or, like, the
harness that you're running the model in? So, like, how does it get its
tool results back?

110
00:10:21,580 --> 00:10:24,260
What tools is it even allowed to use?

111
00:10:24,860 --> 00:10:31,509
Is there any other, like, management in that process, right? So that's
called scaffolding.

112
00:10:32,210 --> 00:10:33,630
Right,

113
00:10:34,500 --> 00:10:41,520
Okay, so today, we're going to try to look at some of these things, some
of the problems that can happen.

114
00:10:41,910 --> 00:10:51,719
with, you know, just… Misspecification, testing things that aren't
really what you want to test.

115
00:10:52,340 --> 00:10:58,120
And then we'll look at… Measurement failures themselves.

116
00:10:58,320 --> 00:11:01,749
So this is just another dimension here, is, like.

117
00:11:02,410 --> 00:11:07,459
Well, if I run the model for longer, Right? Or if I…

118
00:11:07,780 --> 00:11:11,630
Give it different strategies to solve the problem.

119
00:11:11,980 --> 00:11:14,670
Like…

120
00:11:15,250 --> 00:11:24,250
then… then definitely cost starts to become a factor, like, well, how
much am I spending to solve this problem? Is that something we should
also measure?

121
00:11:24,950 --> 00:11:30,500
Right, then what about just, like, variance, right? Because the models
aren't deterministic.

122
00:11:31,270 --> 00:11:34,879
Even with temperature zero, they're not deterministic, so…

123
00:11:35,430 --> 00:11:43,809
Like, one single number, like, what does that even mean? Like, is… we
probably need to be measuring variance.

124
00:11:44,260 --> 00:11:46,040
Reward hacking?

125
00:11:48,690 --> 00:12:01,239
So, is the model even, you know, cheating, right? Is it cheating? Is
that possible? And do models cheat? Like, if we give them the
opportunity to… and the answer is yes.

126
00:12:01,980 --> 00:12:06,800
Then we'll start looking at multi-agent benchmarks.

127
00:12:07,490 --> 00:12:14,140
And then, you know, a lot of problems that we can just view from a game
theoretic

128
00:12:14,370 --> 00:12:22,029
perspective do arise, right? So, if you remember in Lecture 20,

129
00:12:22,870 --> 00:12:29,529
we… we already touched on that a little bit, like, in terms of, like,
the multiplayer rating, like the Elo.

130
00:12:29,990 --> 00:12:34,120
Versus exploitability, like, those are not measuring the same things.

131
00:12:34,680 --> 00:12:36,410
Right? And…

132
00:12:37,130 --> 00:12:43,949
when you do talk about something like Elo, you're really talking about
how is the age… how well does the agent perform?

133
00:12:44,400 --> 00:12:49,220
In the context of a population of other agents.

134
00:12:49,430 --> 00:12:56,309
Right, so if you remove that population, Or if you change it, then the
Elo changes, right?

135
00:12:57,010 --> 00:13:01,930
Okay, so… Yeah, so what makes a good benchmark?

136
00:13:02,190 --> 00:13:07,979
You know, to what extent can we, address some of these problems?

137
00:13:08,190 --> 00:13:10,319
Okay, so let's… part one.

138
00:13:10,660 --> 00:13:12,269
validity failures.

139
00:13:13,260 --> 00:13:19,189
Okay, so, the task validity.

140
00:13:19,560 --> 00:13:25,369
Right, so this is what we want, right? We want our task to be valid,
which means it should be solvable.

141
00:13:26,160 --> 00:13:32,189
The task is solvable if and only if the agent has the target capability
that we're trying to measure.

142
00:13:32,680 --> 00:13:33,580
Okay.

143
00:13:33,900 --> 00:13:36,230
So there's ways that this can fail.

144
00:13:37,460 --> 00:13:39,290
The ground truth is wrong.

145
00:13:39,500 --> 00:13:41,400
Task is impossible.

146
00:13:41,670 --> 00:13:43,579
There exists a shortcut.

147
00:13:43,910 --> 00:13:47,630
To just, you know… Directly get the answer.

148
00:13:48,430 --> 00:13:54,640
But then, once the task is, valid.

149
00:13:55,120 --> 00:13:57,520
We still have to grade the task.

150
00:13:57,880 --> 00:14:03,739
Right? Which means we need some way to assess If we actually solved.

151
00:14:03,940 --> 00:14:05,200
The task or not.

152
00:14:05,470 --> 00:14:08,500
Right, so the grader can maybe be fooled.

153
00:14:09,300 --> 00:14:17,629
Maybe there's, like… if you're trying to use automated procedural
things, like string matching, that's gonna give a lot of problems, like…

154
00:14:18,590 --> 00:14:26,470
you know… If you look at SWE-bench, for example, software engineering,
Right.

155
00:14:26,680 --> 00:14:32,259
Some of the problems are things like the tests are testing for, like,
names.

156
00:14:32,700 --> 00:14:42,499
names of functions and things like that, where, you know, the LLM could
have picked a different name, right? So that would be a string matching
problem.

157
00:14:42,840 --> 00:14:47,630
Or an LLM judge itself is unvalidated.

158
00:14:47,950 --> 00:14:57,440
And then now you have another component. Well, if you change the LLM
judge model out, what happens, right? Or what if you change the LLM
judge prompt?

159
00:14:57,810 --> 00:14:58,640
Right.

160
00:14:58,760 --> 00:15:03,849
Or what if the LLM judge isn't deterministic, which it won't be?

161
00:15:04,250 --> 00:15:06,549
So now do we have a variance there?

162
00:15:06,920 --> 00:15:14,740
Okay, so… ABC's audit of 10 agent benchmarks.

163
00:15:15,420 --> 00:15:18,710
7 of those had outcome validity flaws.

164
00:15:19,510 --> 00:15:22,390
7 had task validity flaws.

165
00:15:23,280 --> 00:15:26,579
80% do not acknowledge their weaknesses.

166
00:15:27,040 --> 00:15:35,260
Okay, so that's… That's not good, right? 7 out of 10. Overestimates?

167
00:15:36,510 --> 00:15:41,380
And underestimates here, so, like, overestimating correctness.

168
00:15:42,150 --> 00:15:46,390
You know, so this one, KernelBench, maybe by up to 31%.

169
00:15:46,840 --> 00:15:52,570
Underestimating these other… these other tasks, like OSWorld,

170
00:15:53,260 --> 00:15:55,230
Some of those were just broken.

171
00:15:55,480 --> 00:15:57,820
For example, by Website Drift.

172
00:15:58,090 --> 00:15:58,840
Okay.

173
00:15:59,530 --> 00:16:06,110
So… There's no way the agent's gonna be able to solve that.

174
00:16:06,270 --> 00:16:12,870
Right, so, wrong ground truth.

175
00:16:13,560 --> 00:16:23,050
Right? So that's, that's a problem, right? So… Algorithm-flagged errors.

176
00:16:23,510 --> 00:16:26,589
Only 51% of those were confirmed.

177
00:16:27,230 --> 00:16:28,750
Rankings flip.

178
00:16:29,610 --> 00:16:35,730
Okay, so this is… this is interesting, right? So let's say we just take…
ImageNet.

179
00:16:36,310 --> 00:16:43,390
Which is a compute… computer vision benchmark. So, you know, this isn't
just specific to LLMs, obviously, right?

180
00:16:44,100 --> 00:16:50,610
But let's take, ImageNet… ImageNet, and… you know…

181
00:16:51,730 --> 00:16:55,680
There's… there's, you know, at least 6%

182
00:16:56,300 --> 00:17:00,140
Of those were mislabeled already. Okay.

183
00:17:00,530 --> 00:17:03,400
So let's, let's adjust that a little bit.

184
00:17:03,590 --> 00:17:10,760
Let's, If the share of mislabeled test examples Changes a bit.

185
00:17:11,430 --> 00:17:18,450
then it flips the order of the rankings, right? So, like, ResNet-18…
Computer vision model.

186
00:17:18,620 --> 00:17:20,499
Beats ResNet-50.

187
00:17:21,020 --> 00:17:25,880
If we just tweak the mislabeled examples again, so…

188
00:17:26,330 --> 00:17:33,799
So it's not clear, you know, because you could say, well, all the… all
the models have the same environment, even if they're mislabeled.

189
00:17:34,500 --> 00:17:41,370
Right? So… do we even care that much about this? But yeah, it can… it
can flip the…

190
00:17:41,550 --> 00:17:47,369
flip the rankings, right? So… Right, so we might ask…

191
00:17:47,640 --> 00:17:49,610
What does, like, a half point…

192
00:17:50,290 --> 00:17:54,349
lead in a leaderboard? Like, what… what… if…

193
00:17:54,540 --> 00:17:59,579
There's errors in the labels, and if we adjust those a little bit, it
flips the ranking.

194
00:17:59,890 --> 00:18:04,629
then, then what does… what does that even mean? Okay.

195
00:18:07,980 --> 00:18:10,889
Okay, so let's talk about wrong ground truth.

196
00:18:11,520 --> 00:18:12,470
Okay.

197
00:18:12,940 --> 00:18:20,410
So here's a couple of benchmarks, so Humanity's Last Exam, and MMLU…

198
00:18:20,730 --> 00:18:23,999
Which are meant to, like, assess, like.

199
00:18:24,650 --> 00:18:28,920
Scientific expertise in a bunch of different domains.

200
00:18:29,090 --> 00:18:36,030
Right? And, so, 57%…

201
00:18:36,180 --> 00:18:42,410
Of a subset of virology questions on this one apparently contain errors.

202
00:18:42,740 --> 00:18:45,650
Okay, according to this study here, right?

203
00:18:46,220 --> 00:18:53,139
Estimated… 6.5% of all the questions are wrong on that one.

204
00:18:53,640 --> 00:18:55,449
Humanity's Last Exam?

205
00:18:56,200 --> 00:19:07,749
So… One study found, like, 30% of those So, text-only, chemistry and
biology.

206
00:19:07,960 --> 00:19:13,310
Answers were contradicted by peer-reviewed literature, Okay.

207
00:19:13,990 --> 00:19:18,570
Alright, so that's 30% of the… of the entire benchmark there.

208
00:19:19,450 --> 00:19:29,510
An independent follow-up tried to reproduce that, and they found only
18%. Okay, but only, only 18%, so that's,

209
00:19:30,280 --> 00:19:37,510
That's a very large… that's a very large error in the ground truths
here, you know, or at least we can't be confident.

210
00:19:37,720 --> 00:19:49,510
Right? So expert written does not mean correct, right? So these
benchmarks were written by experts, right? So they, you know, they hired
people in… they hired scientists in chemistry, biology.

211
00:19:50,230 --> 00:19:57,020
You know, the goal of producing, like, extremely hard, sort of, like,
graduate-level Questions.

212
00:19:57,360 --> 00:20:01,690
and, but just because Expert wrote those.

213
00:20:01,980 --> 00:20:03,939
Does not mean they are correct.

214
00:20:04,300 --> 00:20:08,189
The harder the benchmark, the harder it is to check the answer key.

215
00:20:08,600 --> 00:20:10,239
Alright, so that's,

216
00:20:10,740 --> 00:20:15,570
that's also interesting, right? So, like, if we're trying to come up
with problems that are…

217
00:20:15,980 --> 00:20:20,999
You know, hard graduate student, research questions.

218
00:20:21,730 --> 00:20:28,469
then who's gonna even be able to check that, right? So here, what
they're trying to do is they're trying to…

219
00:20:28,880 --> 00:20:29,560
Oops.

220
00:20:30,810 --> 00:20:38,560
They're trying to compare with peer-reviewed literature, right? So
that's one way to try to check here.

221
00:20:38,910 --> 00:20:40,100
But, of course.

222
00:20:40,450 --> 00:20:46,569
The harder the benchmark is, the harder it is for us to check the answer
key itself.

223
00:20:46,710 --> 00:20:50,850
Right, and detect those errors in the, the ground truth.

224
00:20:52,020 --> 00:21:00,990
As far as coding, So coding should be, like, one of the easiest… Right?
It feels…

225
00:21:01,620 --> 00:21:07,209
But, you know, the way we grade coding is often with, like, tests,
right? Like, unit tests.

226
00:21:07,690 --> 00:21:13,760
So… And there's… there's many ways that those can go wrong.

227
00:21:14,640 --> 00:21:21,440
Right, so narrow tests… is where, the…

228
00:21:22,040 --> 00:21:29,159
They're testing implementation… implementation details that were never
in the problem state.

229
00:21:30,100 --> 00:21:36,059
Right? So… You know, going beyond what the statement is asking for.

230
00:21:36,280 --> 00:21:39,140
That's… we can label that as a narrow test.

231
00:21:40,710 --> 00:21:45,160
A wide test is checking behavior of the task never asked for.

232
00:21:45,430 --> 00:21:48,370
Right, so in two different ways, right?

233
00:21:48,700 --> 00:22:01,399
So this is just, like, implementation details. Like, did you use a… did
you use a priority queue versus a heap, or whatever, or like, you know,
so that would be an example of, like, a narrow test.

234
00:22:01,910 --> 00:22:05,679
The task didn't tell you you had to use a priority queue.

235
00:22:05,880 --> 00:22:10,699
You know, you did it a different way, or you implemented it, from
scratch.

236
00:22:12,060 --> 00:22:17,320
So, and then the test was specifically looking for a priority queue, for
example.

237
00:22:18,190 --> 00:22:20,910
Wide tests is just more like…

238
00:22:21,130 --> 00:22:23,719
Is testing a feature that the task

239
00:22:23,860 --> 00:22:29,829
never even asked for, right? And so here's examples.

240
00:22:30,190 --> 00:22:34,009
from SWE-bench of two of the… two problems.

241
00:22:34,350 --> 00:22:36,940
That had each one of these issues, right?

242
00:22:37,310 --> 00:22:45,120
Yeah, so the test here… import a function named get_annotation.

243
00:22:45,390 --> 00:22:47,540
Which the issue never mentioned.

244
00:22:48,650 --> 00:22:59,139
Okay, this one… The pull request fixes three issues. The prompt just
describes only one.

245
00:22:59,490 --> 00:23:02,750
Okay, so two issues were not given.

246
00:23:02,890 --> 00:23:04,570
In the… in the prompt?

247
00:23:07,640 --> 00:23:22,309
But this story, you know, is not specific to this one assessment of
OpenAI by… of SWE-bench. So that was where they… that's where these
numbers, like 35%, 18%, are coming from. SWE-bench Verified.

248
00:23:22,450 --> 00:23:23,780
OpenAI.

249
00:23:23,880 --> 00:23:27,819
We did a, you know, the assessment we talked about earlier.

250
00:23:27,960 --> 00:23:28,820
Bonnet.

251
00:23:29,300 --> 00:23:37,600
Right, but, same story with the many different versions of SWE-bench
over the years.

252
00:23:39,570 --> 00:23:48,520
Okay, so, what about graders that can be fooled?

253
00:23:49,150 --> 00:23:56,459
Okay, so, like, what if there's a bug in the grading process or
something like that? Is, you know, one way that could happen?

254
00:23:57,240 --> 00:24:03,079
So here's… here's a few benchmarks in this table, of examples of this.

255
00:24:03,750 --> 00:24:06,350
So, like, tau-bench Airline.

256
00:24:07,430 --> 00:24:10,570
If you just return empty responses.

257
00:24:11,200 --> 00:24:18,470
That gets you, like, a 38% success score for whatever reason, you know,
just the way the grader is set up.

258
00:24:18,670 --> 00:24:20,329
An empty response.

259
00:24:20,530 --> 00:24:27,900
Just gives you, like, a 38%… Score?

260
00:24:28,350 --> 00:24:35,810
Which beats a GPT-4o agent. Trying to actually… Solve the…

261
00:24:36,460 --> 00:24:47,179
the, the thing. So, like, success in that case just means you didn't do
anything. Database left unchanged, right? So if GPT-4o actually…

262
00:24:47,290 --> 00:24:49,650
Try to change the database, then…

263
00:24:49,950 --> 00:24:52,750
They didn't get credit on this 38%.

264
00:24:53,760 --> 00:25:00,260
So tau-bench, dumping the whole… Database.

265
00:25:01,200 --> 00:25:06,390
Like, got you 2% success on airline, 3.6% on retail.

266
00:25:06,970 --> 00:25:16,070
Right, SWE-Lancer… Overwrites the test?

267
00:25:16,630 --> 00:25:21,130
With assert 1 is equal to 1, so apparently that was possible.

268
00:25:21,930 --> 00:25:26,709
So, in this case, the tests… We're in a zip.

269
00:25:26,850 --> 00:25:33,630
you know, that the agent can actually reach and modify. So a
password-protected zip.

270
00:25:34,000 --> 00:25:40,710
So, yeah, being able to hack the grader, or fool the grader.

271
00:25:41,130 --> 00:25:49,070
Or the grader being misconfigured, and, like, solution… like, edge case
solutions, like, dumping the whole database.

272
00:25:49,430 --> 00:25:54,240
Actually pass some of the, tests.

273
00:25:54,900 --> 00:26:03,870
Right, which then led, this study by ABC, so this Zhu et al.,

274
00:26:04,350 --> 00:26:08,760
To recommend always reporting a trivial agent baseline.

275
00:26:09,020 --> 00:26:09,710
Okay.

276
00:26:10,030 --> 00:26:14,430
Like, so an agent that does nothing, What does this score?

277
00:26:14,860 --> 00:26:18,249
Right, and most benchmarks don't do that… yet.

278
00:26:18,570 --> 00:26:21,970
But this is a recommendation.

279
00:26:25,390 --> 00:26:30,340
Right, but let's, let's go into this a little bit more, so graders that
can be fooled.

280
00:26:31,070 --> 00:26:39,339
so… An audit agent covered 13 agent benchmarks.

281
00:26:39,620 --> 00:26:40,400
Okay.

282
00:26:40,580 --> 00:26:42,330
Including all of these.

283
00:26:43,130 --> 00:26:45,579
Right? Some of which we've mentioned already.

284
00:26:45,770 --> 00:26:48,840
WebArena, tau-bench, OSWorld.

285
00:26:49,480 --> 00:26:55,170
Found 45 confirmed exploits across 16 attack types.

286
00:26:56,270 --> 00:27:01,590
Several reaching 100% with zero task solving.

287
00:27:02,160 --> 00:27:02,870
Okay.

288
00:27:03,510 --> 00:27:08,600
So… WebArena?

289
00:27:09,730 --> 00:27:14,159
Apparently, you know, the way that test environment is set up.

290
00:27:14,450 --> 00:27:18,650
The tests are actually sitting there somewhere in the file system.

291
00:27:18,770 --> 00:27:19,610
Okay.

292
00:27:19,770 --> 00:27:29,220
So if the agent can just read it and return the answer… You know? And
honestly.

293
00:27:29,660 --> 00:27:35,099
you know, I've, you know, if you do… if you do work with AI agents.

294
00:27:35,210 --> 00:27:43,310
and LLMs a lot, you do want to start assessing, you know, how the
performance is looking on the tasks that you care about.

295
00:27:43,880 --> 00:27:52,590
Right? And so you will set up your own benchmarks, which I've done many
times, and run into pretty much all these issues as well, you know?

296
00:27:52,790 --> 00:27:58,630
Things like the test being visible, Or,

297
00:27:59,010 --> 00:28:04,229
The test checks that a C extension exists, but never calls it.

298
00:28:04,580 --> 00:28:10,999
So you can just, for example, trivially make it exist, okay, and not
implement the thing.

299
00:28:11,450 --> 00:28:20,949
Frontier-CS, the submission walks up the Python call stack and replaces
the scoring function.

300
00:28:21,330 --> 00:28:24,510
So this is definitely, like, a hack, right?

301
00:28:24,680 --> 00:28:31,399
But, you know, if the agent just does that, they can actually get a
perfect score

302
00:28:31,690 --> 00:28:32,850
on this one.

303
00:28:33,000 --> 00:28:33,790
Okay.

304
00:28:34,310 --> 00:28:41,269
So, design principle… Isolate everything that scores from everything
being scored.

305
00:28:41,830 --> 00:28:47,300
feels like this should be obvious, right? But… You know?

306
00:28:47,490 --> 00:28:48,970
We have to…

307
00:28:49,310 --> 00:28:56,270
Sometimes things being obvious in theory is different from it actually
carrying out into the implementation, right?

308
00:28:56,590 --> 00:28:59,110
So, mount references read-only.

309
00:28:59,620 --> 00:29:01,230
Checksum them.

310
00:29:01,930 --> 00:29:03,500
Right, make sure…

311
00:29:03,790 --> 00:29:09,809
Make sure there's nothing that can be overwritten in the… in the tests
or in the graders.

312
00:29:11,970 --> 00:29:12,860
Okay.

313
00:29:13,380 --> 00:29:19,240
So let's switch a little bit to the contamination that we mentioned
earlier.

314
00:29:19,810 --> 00:29:27,580
Right, so remember, that's where the model's training data might have
leaked things about the test. It doesn't have to be.

315
00:29:27,720 --> 00:29:31,240
Thought the exact test questions were in…

316
00:29:31,450 --> 00:29:34,250
were in the, training data. Like, what if…

317
00:29:34,410 --> 00:29:45,559
There's a blog post that describes problems that the agent struggled
with, or a blog post describing ways to hack the grader, right? That's
in the training data.

318
00:29:45,730 --> 00:29:53,530
Right, then the model, like, information can just leak, right? So…

319
00:29:53,740 --> 00:29:56,170
Here's actually a clever test for that.

320
00:29:56,359 --> 00:29:59,990
So this is from Oren et al.

321
00:30:01,210 --> 00:30:03,659
The, the exchangeability test.

322
00:30:04,740 --> 00:30:11,150
Okay, so, and the idea is, if the model never saw the test set.

323
00:30:12,220 --> 00:30:15,660
Then, trying to get it to generate the test set.

324
00:30:16,240 --> 00:30:21,309
The ordering of the examples should be equally likely.

325
00:30:21,960 --> 00:30:23,930
Right? So what does that mean?

326
00:30:24,770 --> 00:30:28,979
So, you know, when you're generating with an LLM,

327
00:30:29,850 --> 00:30:33,349
You're always doing it one token at a time, right?

328
00:30:33,580 --> 00:30:39,800
And, so, for the next token to be generated.

329
00:30:40,910 --> 00:30:48,600
there's probabilities for that, right? So there's, like, a… what you get
out at the end is a probability function.

330
00:30:49,290 --> 00:30:53,430
over the tokens. So if you're doing greedy,

331
00:30:53,750 --> 00:30:56,419
inference, then you just take the top one.

332
00:30:57,420 --> 00:31:00,270
If you're doing temperature, you…

333
00:31:00,440 --> 00:31:10,689
the degree of the temperature kind of changes the shape of the
distribution, right? So it makes it flatter the higher the temperature,
so you're getting more of a random output.

334
00:31:11,110 --> 00:31:14,440
But what you can do is you can ask,

335
00:31:15,150 --> 00:31:18,970
You know, like, what's the probability that the…

336
00:31:19,150 --> 00:31:23,129
the LLM actually generates this test set.

337
00:31:23,300 --> 00:31:24,990
Or this test example.

338
00:31:26,450 --> 00:31:29,080
And then you can say, well, if I flip the order.

339
00:31:29,720 --> 00:31:31,950
How does that change the probability?

340
00:31:32,450 --> 00:31:37,180
So if the agent knows nothing about, like, if they've never seen the
test set.

341
00:31:38,330 --> 00:31:44,919
Then, they shouldn't be able to have any preference on, like, what order
they would…

342
00:31:45,270 --> 00:31:50,270
generate the test set in, right? So, contaminated model, though.

343
00:31:50,480 --> 00:31:56,830
would, right? They should prefer the canonical published order that
exists, right?

344
00:31:58,210 --> 00:31:59,160
So…

345
00:31:59,600 --> 00:32:08,080
So this is great, this gives us an actual tool that we can use to say,
like, was there some degree of contamination here?

346
00:32:08,440 --> 00:32:09,929
Right,

347
00:32:10,530 --> 00:32:19,849
And, you know, so this paper, they showed it worked. They showed it
worked for, like, a really small model, 1.4 billion, and 1,000 example
test sets.

348
00:32:20,380 --> 00:32:24,569
Right? They did an audit of 5 public models.

349
00:32:24,840 --> 00:32:29,929
And didn't find a lot of evidence for pervasive contamination, so that's
good.

350
00:32:31,080 --> 00:32:38,999
But, you know, obviously it's gonna depend on what benchmark you look
at, like, how old it is.

351
00:32:39,180 --> 00:32:46,060
how it could have showed up in the training data. And so here's some
other perspectives, and…

352
00:32:46,210 --> 00:32:50,270
findings on that, right? So…

353
00:32:50,580 --> 00:33:00,730
For this particular one, so this Deng et al. TS-Guessing, they tested,
can GPT… generate…

354
00:33:00,930 --> 00:33:06,569
an option that they, that they masked in the, in the MMLU.

355
00:33:06,840 --> 00:33:09,490
Right? And they found that it could.

356
00:33:09,630 --> 00:33:11,269
To a high extent.

357
00:33:12,440 --> 00:33:19,829
Now, what if we just totally put in, like, a fresh test set? You know,
like, very similar problems, different test set.

358
00:33:19,990 --> 00:33:25,890
And the accuracy dropped up to 8% in this study on a couple of different
benchmarks.

359
00:33:26,390 --> 00:33:31,990
For a couple of models here, so Phi and Mistral, if you remember the
Mistral model.

360
00:33:32,420 --> 00:33:36,259
Overfit at almost every scale.

361
00:33:38,190 --> 00:33:39,960
SWE-bench Illusion.

362
00:33:42,720 --> 00:33:49,520
Right, so, okay, so this is… this is another approach. It's like, well,
what if we just give the model the,

363
00:33:49,840 --> 00:33:54,760
The text from the problem. So just the… the issue text.

364
00:33:55,530 --> 00:34:03,679
And then ask it to name the buggy file, which it shouldn't be able to do
if it hasn't seen the repo, right? If it hasn't seen the repo.

365
00:34:03,860 --> 00:34:08,730
It shouldn't be able to name the buggy file, but they find that it can.

366
00:34:09,010 --> 00:34:12,019
Okay, up to, you know, 76%.

367
00:34:12,270 --> 00:34:13,909
On the verified set.

368
00:34:14,190 --> 00:34:16,720
53% of other repos.

369
00:34:19,040 --> 00:34:30,749
And then, of course, just plain gold patch recall. Like, does it… can it
actually just reproduce the full answer from memory exactly the way it's
written, you know, in the gold patch?

370
00:34:30,960 --> 00:34:38,859
And it turns out they can, you know, on different, on different
benchmarks, so OpenAI

371
00:34:39,120 --> 00:34:42,409
In that February, analysis.

372
00:34:43,730 --> 00:34:49,699
Okay, so this is a problem, Ben, that Benchmark,

373
00:34:50,060 --> 00:34:56,130
Benchmark designers are gonna have to… Always be thinking about, right?

374
00:34:56,320 --> 00:35:03,920
And then that's going to compound with the other problems that we saw
earlier in the lecture, right? Because if you're constantly coming up
with new instances.

375
00:35:04,170 --> 00:35:08,419
And those new instances are going to have issues with their tests.

376
00:35:08,630 --> 00:35:17,220
Right? Like, you're basically having to redesign the benchmark
periodically, right? But…

377
00:35:17,510 --> 00:35:23,490
But what you can do is you can, so LiveCodeBench, that's another,
software engineering one.

378
00:35:23,760 --> 00:35:28,449
You know, they only score models on problems released after they're cut
off.

379
00:35:28,670 --> 00:35:29,440
Okay.

380
00:35:29,990 --> 00:35:33,800
And we see that,

381
00:35:34,180 --> 00:35:36,139
You know, we see that having an effect.

382
00:35:37,020 --> 00:35:38,910
Refresh continuously.

383
00:35:39,620 --> 00:35:43,330
Right, so always be generating new, new tasks.

384
00:35:43,870 --> 00:35:45,960
Withhold answers.

385
00:35:46,940 --> 00:35:50,020
So this is actually an interesting strategy.

386
00:35:50,240 --> 00:35:53,369
It's just saying, okay, well, we're not even going to let

387
00:35:53,520 --> 00:35:59,489
the public or anybody see the actual answers, right? We're gonna keep
that out of the training data.

388
00:36:00,020 --> 00:36:05,529
Just by, just by keeping those private, which means…

389
00:36:05,900 --> 00:36:08,569
You have to trust us that they're right.

390
00:36:08,870 --> 00:36:13,630
Right? And you have to trust us, you know, that we're even evaluating
the models.

391
00:36:13,940 --> 00:36:19,500
Fairly, right? There's… So there seems to be problems with that as well.

392
00:36:20,340 --> 00:36:22,650
Test for it, like we were just saying.

393
00:36:23,750 --> 00:36:30,840
Right, so… a private split stops leakage of answers.

394
00:36:31,060 --> 00:36:33,929
Does it stop selection on the test set?

395
00:36:34,730 --> 00:36:36,930
Okay, so let's keep that question in mind.

396
00:36:37,340 --> 00:36:39,270
As we continue to go here.

397
00:36:39,820 --> 00:36:42,240
Okay.

398
00:36:43,430 --> 00:36:49,200
Okay, so another issue, as we mentioned in the early slide, shortcuts.

399
00:36:49,540 --> 00:36:57,810
Right, solving the benchmark, not the task. Okay, so a shortcut… Is, you
know, like, getting an answer…

400
00:36:57,940 --> 00:37:01,079
That works on the benchmark, but does not work…

401
00:37:01,280 --> 00:37:07,960
in, like, a different testing environment. Okay, so for example, Classic
example.

402
00:37:08,480 --> 00:37:11,709
Is only being able to recognize a cow.

403
00:37:12,030 --> 00:37:15,099
If there's also grass in the image.

404
00:37:15,240 --> 00:37:17,809
Right, so Geirhos et al., 2020.

405
00:37:18,610 --> 00:37:24,580
So, you know, if you're ImageNet, Mostly has cows on grass.

406
00:37:25,340 --> 00:37:30,360
And your model just… Detects cows only on grass.

407
00:37:31,190 --> 00:37:32,740
Then, you know…

408
00:37:33,270 --> 00:37:41,990
that's the shortcut, right? That's the shortcut. You're saying, if
there's grass, I'm gonna think it's a cow. If there's not grass, I'm not
going to…

409
00:37:42,320 --> 00:37:49,650
Right? But, you know, you could have a cow that's not occurring with
grass, Right, so…

410
00:37:50,340 --> 00:37:55,760
And so this problem itself seems actually quite deep. Right,

411
00:37:56,070 --> 00:38:00,260
You know, because we're asking, like, we're starting to ask questions
like.

412
00:38:00,900 --> 00:38:10,699
what is actually the model allowed to use for inference, right? Because
you might argue… If there's only…

413
00:38:10,810 --> 00:38:14,310
There's only grass with cows in the benchmark.

414
00:38:14,950 --> 00:38:18,790
You know, it seems fair, right, that the,

415
00:38:19,520 --> 00:38:21,690
You know, the model could use that.

416
00:38:22,220 --> 00:38:24,359
On this particular benchmark.

417
00:38:24,490 --> 00:38:31,430
So yeah, when it comes to questions like… You know, and there's…

418
00:38:31,620 --> 00:38:35,169
Like, you can… you can come up with examples of this sort of thing.

419
00:38:35,370 --> 00:38:40,320
that we clearly want to rule out. Like,

420
00:38:41,070 --> 00:38:43,399
like, if you have an MRI,

421
00:38:43,950 --> 00:38:48,389
Let's say you have an MRI of, like, a lung, patient.

422
00:38:49,030 --> 00:38:52,179
And, trying to diagnose a disease.

423
00:38:52,730 --> 00:38:59,899
And maybe there's… There's, like, a label of the machine that took the
image.

424
00:39:00,010 --> 00:39:01,259
In the corner.

425
00:39:01,960 --> 00:39:12,010
Right, and all the patients with one disease had that label from…
because they all came from one machine, other one didn't. That's
clearly, you know, then the model can just shortcut

426
00:39:12,540 --> 00:39:17,060
Take the label of that machine, right? Always get the right answer.

427
00:39:17,400 --> 00:39:23,149
And that's clearly not, not a desirable behavior.

428
00:39:23,920 --> 00:39:31,280
But the question is, like, how far down that rabbit hole do you go,
right? Like…

429
00:39:32,240 --> 00:39:46,410
are they allowed to use physical laws about the fact that we exist here
on Earth, right? You know, versus on the moon, right? Are they allowed
to use those kind of things? So, you know, it's sort of a… it's a
question that's not…

430
00:39:46,920 --> 00:39:48,499
Totally well defined.

431
00:39:49,700 --> 00:39:54,539
But, but, you know, there's different ways…

432
00:39:55,000 --> 00:39:57,650
Of this phenomenon showing up.

433
00:39:57,810 --> 00:40:11,380
Okay, so… So here's 3 different, benchmarks, where, shortcuts have been
found, right?

434
00:40:11,490 --> 00:40:14,780
So, for example… WebArena.

435
00:40:15,450 --> 00:40:17,610
If we just hard code…

436
00:40:17,710 --> 00:40:23,440
site policies. You know, so it's like, if we're on…

437
00:40:23,730 --> 00:40:31,570
reddit.com, apply this policy. If we're on Google.com or whatever, apply
this other policy, right?

438
00:40:32,120 --> 00:40:39,140
Right? So, So that seems like a shortcut.

439
00:40:39,390 --> 00:40:43,589
This agent was able to get more than double the best baseline.

440
00:40:44,340 --> 00:40:51,899
Multi-agent StarCraft, a policy that only conditioned on the time step.

441
00:40:52,770 --> 00:40:56,819
Got non-trivial win rates on many maps.

442
00:40:57,280 --> 00:41:01,939
Okay, so this is a multi-agent benchmark.

443
00:41:03,200 --> 00:41:04,880
Where they're getting…

444
00:41:05,100 --> 00:41:12,099
You know, they're getting win rates based on not even paying any
attention to what the other agents are doing.

445
00:41:13,370 --> 00:41:18,640
So, that doesn't seem like what that benchmark… Wanted to,

446
00:41:19,430 --> 00:41:24,120
to assess. There's no strategic reasoning there. Seems like a shortcut,
right?

447
00:41:24,890 --> 00:41:27,599
Okay, so here's what we've looked at so far.

448
00:41:29,420 --> 00:41:30,860
Four different ways.

449
00:41:31,360 --> 00:41:36,620
You know, so we talked about task validity versus outcome validity.

450
00:41:37,020 --> 00:41:40,809
Right, and we looked at wrong ground truth, that's a task issue.

451
00:41:42,140 --> 00:41:45,340
Foolable grader, that's an outcome issue.

452
00:41:46,200 --> 00:41:50,210
Contamination, that's a task issue, shortcuts, task issue.

453
00:41:50,500 --> 00:41:54,450
Okay, but these are all things.

454
00:41:55,110 --> 00:41:58,679
Well, yeah, yeah, we can take defenses against them.

455
00:41:58,820 --> 00:42:02,820
But those might be very expensive, they might not be scalable.

456
00:42:03,040 --> 00:42:10,669
And they're also, like, not totally solvable or checkable. You know,
like, if we did a human audit of all the failures.

457
00:42:11,330 --> 00:42:17,280
That doesn't mean if we do another human audit, we're not going to
continue to find more errors, right?

458
00:42:18,660 --> 00:42:23,369
The SWE-bench Verified suffered from at least 3 of those 4 issues.

459
00:42:23,840 --> 00:42:32,640
Right, so… Yeah, definitely.

460
00:42:34,040 --> 00:42:38,420
We are not restricted to just one showing up in a given benchmark.

461
00:42:38,790 --> 00:42:39,580
Okay.

462
00:42:42,600 --> 00:42:44,739
So now, part 2.

463
00:42:46,150 --> 00:42:54,230
We're gonna talk about not just issues with, like, the benchmark setup
itself, But failures in measurement.

464
00:42:57,520 --> 00:43:00,979
So, for example, Cost and variance.

465
00:43:02,750 --> 00:43:09,749
So this, Kapoor et al, 2025… Alright, they,

466
00:43:10,700 --> 00:43:15,720
They took, this, HumanEval benchmark.

467
00:43:16,950 --> 00:43:24,190
And they looked at different strategies to solve it with the same
underlying base model.

468
00:43:24,880 --> 00:43:25,640
Okay.

469
00:43:26,250 --> 00:43:28,860
And

470
00:43:29,340 --> 00:43:40,950
So, warming is this top row here. That was actually they came up with in
the paper itself, or it all, so that's their own baseline, which we're
going to talk about in a second.

471
00:43:41,810 --> 00:43:47,720
Language agent tree search. So this is this famous, or, like.

472
00:43:48,290 --> 00:43:53,600
somewhat famous, I guess, doing, like, a Monte Carlo tree search.

473
00:43:53,850 --> 00:43:55,659
Over the reasoning space.

474
00:43:55,910 --> 00:43:58,799
Okay, so extremely expensive.

475
00:44:00,650 --> 00:44:06,529
You know? So this is not just doing, like, greedy decoding, or, like,
not just letting the model run.

476
00:44:07,200 --> 00:44:12,409
But, like, actually exposing, like, imposing an external structure on
that.

477
00:44:13,380 --> 00:44:19,960
Which then, you know, is going to introduce, like, the cost confound.

478
00:44:20,640 --> 00:44:25,190
Right? So, like, if I'm doing a really expensive… you know…

479
00:44:25,570 --> 00:44:29,719
test time scaling of the model, which is what LATS is.

480
00:44:29,930 --> 00:44:32,489
Like, can you see it in this cost number here? Like, it…

481
00:44:32,730 --> 00:44:37,860
To evaluate it on this benchmark, it took $134.

482
00:44:38,100 --> 00:44:41,359
Versus, you know, just, like, a few dollars.

483
00:44:41,790 --> 00:44:45,569
And this is all the same model, right? So token prices are all the same.

484
00:44:46,400 --> 00:44:54,969
Versus… You know, well, okay, and then let's talk about what is the
warming strategy?

485
00:44:55,120 --> 00:44:55,830
Okay.

486
00:44:56,120 --> 00:45:04,639
And so the way it works is with HumanEval, This is a coding, coding
benchmark.

487
00:45:05,660 --> 00:45:12,559
so the strategy is… Generate an answer, generate a solution, a code.

488
00:45:13,010 --> 00:45:15,760
But then don't just submit it.

489
00:45:16,440 --> 00:45:21,570
Okay, then run it on the actual example test in the problem statement.

490
00:45:23,400 --> 00:45:26,460
If it passes those, submit it.

491
00:45:27,300 --> 00:45:31,030
If it fails… Retry.

492
00:45:31,560 --> 00:45:33,250
Up to 5 attempts.

493
00:45:34,000 --> 00:45:38,020
And gradually raise the model temperature with each retry.

494
00:45:38,460 --> 00:45:39,200
Okay.

495
00:45:40,230 --> 00:45:48,489
So… Okay, so that's just this strategy. You know, we can do it with
GPT-4.

496
00:45:48,730 --> 00:45:53,620
Right? And they saw, you know, they saw this…

497
00:45:53,960 --> 00:45:58,659
this difference, right? So that was their own baseline, this warming
thing.

498
00:45:59,100 --> 00:46:01,980
93% on HumanEval.

499
00:46:03,450 --> 00:46:05,230
Costs $2.

500
00:46:05,930 --> 00:46:12,050
Versus, the language agent tree search with the Monte Carlo tree search
over the reason space.

501
00:46:12,900 --> 00:46:17,660
Cost $134.

502
00:46:18,410 --> 00:46:22,790
so, a two-sentence strategy.

503
00:46:23,410 --> 00:46:30,960
be a published… Highly cited tree search, reasonings-based thing, right?

504
00:46:31,920 --> 00:46:37,159
That cost over 50 times what the, the two-step strategy cost.

505
00:46:38,090 --> 00:46:43,810
Right, and, you know, so, like… In 2026,

506
00:46:45,010 --> 00:46:53,810
You know, so this was, 2025. Things change. Things change fast, right,
in this space. In 2026,

507
00:46:54,300 --> 00:46:57,480
You know, we don't usually,

508
00:46:57,940 --> 00:47:06,699
we don't usually do things like the language agent tree search when
we're benchmarking models, right? We usually just put them in their own
tool loop.

509
00:47:06,880 --> 00:47:12,320
If they decide to run the example tests, That's probably a good
strategy.

510
00:47:12,540 --> 00:47:14,839
If they don't, we don't force them to.

511
00:47:15,660 --> 00:47:20,870
Right? But, you know, you don't have to just…

512
00:47:21,180 --> 00:47:30,039
test the model by decoding from it, right? You can… you can… and this
is… this is scaffolding, right? This is 100% scaffolding.

513
00:47:30,770 --> 00:47:34,160
But then, you know, it introduces…

514
00:47:34,550 --> 00:47:41,490
All of these different confounds, you know, like the cost, But then not
just cost.

515
00:47:43,710 --> 00:47:51,429
But also variance as well. Okay. But so what did they conclude in this
2025 Kapoor et al?

516
00:47:52,190 --> 00:47:59,530
So, no significant accuracy difference between the warming strategy And
the best performing architecture.

517
00:48:00,660 --> 00:48:06,430
For substantially similar accuracy, cost can differ by two orders of
magnitude.

518
00:48:07,280 --> 00:48:11,390
Accuracy alone?

519
00:48:11,660 --> 00:48:15,540
It's maybe not the best measure, because we can improve it.

520
00:48:15,770 --> 00:48:21,829
by just… Doing scientifically meaningless methods, like retrying.

521
00:48:21,960 --> 00:48:28,880
Okay, so, okay,

522
00:48:29,040 --> 00:48:32,640
So, to be clear, right, they're not retrying on the actual

523
00:48:32,910 --> 00:48:39,150
grading tests. They're retrying on the example tests here, right? So
nothing has been submitted to the grader.

524
00:48:39,410 --> 00:48:42,940
Until that process, completes.

525
00:48:43,540 --> 00:48:44,400
Okay.

526
00:48:44,730 --> 00:48:52,250
so… Yeah.

527
00:48:55,210 --> 00:49:05,449
And then, also, just reporting a single number, right? Like, let's say
we report the accuracy of 92.7%, which is what was reported.

528
00:49:06,120 --> 00:49:12,770
In the LATS paper for HumanEval. Okay, they reported 92.7.

529
00:49:13,230 --> 00:49:21,929
In this Kapoor et al.'s reruns… You know, it ranged from, like, 82% to
something close to that.

530
00:49:22,050 --> 00:49:22,820
Okay.

531
00:49:23,120 --> 00:49:30,230
But then if you're just running it, like, 50 times and reporting the
best thing, you know, that doesn't seem…

532
00:49:33,240 --> 00:49:34,559
Very valid.

533
00:49:35,700 --> 00:49:41,150
But then, now that we have two metrics, we've got cost and accuracy that
we can care about.

534
00:49:41,600 --> 00:49:44,629
then we can start talking about a Pareto frontier.

535
00:49:45,240 --> 00:49:50,330
Right, because we can say… This agent…

536
00:49:50,460 --> 00:49:54,239
You know, was outperformed by this other one.

537
00:49:54,410 --> 00:49:58,469
But it costs less, so it's not strictly dominated, right?

538
00:49:58,930 --> 00:50:02,410
And so here's, for example.

539
00:50:02,840 --> 00:50:12,540
from that paper, putting those strategies on a Pareto front, Right? So,
for example… Zero shot.

540
00:50:13,050 --> 00:50:15,770
Is not dominated.

541
00:50:16,180 --> 00:50:24,300
by, by retry and escalation, even though, you know, it scores above
escalation and below retry.

542
00:50:24,660 --> 00:50:34,519
Right? However… we can… we can make this thing convex, right? So we can
mix across strategies.

543
00:50:34,830 --> 00:50:41,899
Right, so I can do Agent A with probability P, Agent B with probability
1 minus P. So I can actually get…

544
00:50:42,110 --> 00:50:45,669
Presumably, I can actually get this full line, right?

545
00:50:46,250 --> 00:50:51,950
So, that actually dominates, things like zero-shot here.

546
00:50:52,380 --> 00:50:53,070
Okay.

547
00:50:53,520 --> 00:50:59,619
So, so, this is a little weird, because…

548
00:50:59,990 --> 00:51:04,680
You probably wouldn't do this in practice, you know? Like, take a
problem and be like.

549
00:51:04,830 --> 00:51:09,730
I have an expensive strategy that does well, a cheap strategy that does
less well.

550
00:51:09,980 --> 00:51:12,610
I'm gonna pick a probability and mix them.

551
00:51:12,790 --> 00:51:16,680
You know, but you could. You could do that.

552
00:51:18,230 --> 00:51:25,849
So, and of course that ties in a little bit with our… Mixed strategy
concept. It,

553
00:51:26,390 --> 00:51:29,380
That we've talked about game theoretically, right?

554
00:51:29,780 --> 00:51:35,509
But then, of course, the variance issue itself, right?

555
00:51:35,840 --> 00:51:38,840
Like, we should probably have error bars.

556
00:51:39,260 --> 00:51:47,700
with, with the numbers we get out of benchmarks. We need to do multiple
runs, like, we probably should do multiple runs.

557
00:51:47,880 --> 00:51:50,450
Temperature zero is not deterministic.

558
00:51:51,080 --> 00:51:55,500
And then I sort of mentioned this earlier.

559
00:51:56,450 --> 00:52:02,620
But we should probably talk about it a little bit more. So, like, the
prompt itself… Right?

560
00:52:02,820 --> 00:52:07,220
And by the way, this is not a thing that's specific to old models, okay?

561
00:52:07,830 --> 00:52:11,230
If we change the prompt a little bit.

562
00:52:11,600 --> 00:52:14,099
It can give a very large…

563
00:52:14,630 --> 00:52:17,700
You know, difference in performance.

564
00:52:17,920 --> 00:52:21,149
So, what does it really mean if I take HumanEval.

565
00:52:21,630 --> 00:52:24,529
I rephrased the prompt a little bit.

566
00:52:24,690 --> 00:52:31,439
And my results flip, you know? Like, that doesn't seem… That doesn't
seem very meaningful.

567
00:52:32,040 --> 00:52:32,710
Okay.

568
00:52:32,840 --> 00:52:36,050
Okay.

569
00:52:36,700 --> 00:52:38,230
And then…

570
00:52:38,770 --> 00:52:45,830
Yeah, where does scaffolding… where does scaffolding fit in into the
process? Like, how is that… how does that get reported?

571
00:52:46,040 --> 00:52:50,540
So you've probably seen, on…

572
00:52:51,120 --> 00:52:57,390
I'm pretty… I think it was Human's… Human… Human's Last… Humanity's Last
Exam, the HLE.

573
00:52:57,800 --> 00:53:07,389
Right? So, OpenAI and Anthropic, You know, their models, do not get…

574
00:53:07,580 --> 00:53:11,960
You know, with the harness that they use, to score.

575
00:53:13,630 --> 00:53:16,890
They perform, you know, say at, like, 85%.

576
00:53:17,330 --> 00:53:23,430
If they use… if OpenAI uses their own harness, They get 98%.

577
00:53:23,860 --> 00:53:26,239
Right? So that's a huge difference.

578
00:53:27,040 --> 00:53:31,429
And we're not even, like, we're not even talking about things like,

579
00:53:32,260 --> 00:53:38,369
like the Monte Carlo tree search and reasoning space. We're not talking
about that, we're just letting the models

580
00:53:38,510 --> 00:53:39,909
Do their own thing.

581
00:53:40,040 --> 00:53:42,760
But we're changing which harness it's in.

582
00:53:42,910 --> 00:53:46,369
Like, how does past context get represented?

583
00:53:46,540 --> 00:53:54,799
All of that, and, so even just switching that out on, like, Humanity's
Last Exam.

584
00:53:56,240 --> 00:54:00,650
Gives a large difference, so…

585
00:54:00,910 --> 00:54:09,220
So what harness should we use, or what scaffold should we use? Just the
best one for this particular benchmark? Probably doesn't seem great.

586
00:54:09,610 --> 00:54:17,189
So, and yeah, and this is that same phenomenon on several different
benchmarks here. So, like, SWE-bench Verified.

587
00:54:17,610 --> 00:54:22,490
Switching scaffolds worth up to 11 points, for example.

588
00:54:22,670 --> 00:54:25,090
Okay.

589
00:54:25,450 --> 00:54:29,690
reward hacking.

590
00:54:30,200 --> 00:54:41,010
Okay, so… This means, the model… The model cheating. Okay.

591
00:54:41,360 --> 00:54:43,960
And actually, ImpossibleBench…

592
00:54:44,400 --> 00:54:50,790
is a benchmark specifically designed to measure this. Okay, so this is
kind of a cool idea.

593
00:54:51,560 --> 00:55:00,200
they, they made instances where the tests actually always contradict the
spec. Okay, so any pass.

594
00:55:00,350 --> 00:55:02,850
Is some kind of cheat on this one.

595
00:55:03,030 --> 00:55:08,559
And they measured GPT-5 cheating 54% of the time.

596
00:55:08,970 --> 00:55:13,669
If instead, they let it flag the task for a human.

597
00:55:14,240 --> 00:55:19,900
That, instead of submitting something, then that went down to 9%.

598
00:55:20,790 --> 00:55:22,970
METR.

599
00:55:23,090 --> 00:55:31,519
you've probably heard of METR, right? An independent, sort of, like,
auditing, nonprofit.

600
00:55:31,800 --> 00:55:37,420
So they found that hacking on RE-Bench, 30… 30%, okay?

601
00:55:37,620 --> 00:55:40,020
When the scoring function was visible.

602
00:55:40,810 --> 00:55:47,339
Told, do not cheat, agent still hacks, 80% on one task.

603
00:55:47,700 --> 00:55:56,780
hacks include… Stubbing the evaluator, finding the grader's answer, etc.

604
00:55:56,900 --> 00:55:58,720
okay.

605
00:56:03,720 --> 00:56:12,270
Here's more examples. So in chess, let's look at chess specifically. So
here the task is to win against a chess engine.

606
00:56:13,000 --> 00:56:19,380
They found that o3 hacked in 88% of runs, so this is according to
Bondarenko et al.

607
00:56:20,380 --> 00:56:24,690
o1-preview attempted hacks in 36% of runs.

608
00:56:24,870 --> 00:56:31,439
Hacks being things like… Replace the chess engine, so replace Stockfish.

609
00:56:31,860 --> 00:56:46,350
Overwrite the board file, you know, like, things like that, right? So,
you know, if you can't beat Stockfish, just get rid of it, right? The
leader… leaderboard illusion…

610
00:56:47,310 --> 00:56:53,770
So, Meta tested 27 private variants on Chatbot Arena.

611
00:56:55,600 --> 00:56:58,809
Providers can, retract scores.

612
00:56:59,920 --> 00:57:06,740
google, OpenAI, right, so…

613
00:57:08,370 --> 00:57:16,199
The point is, best-of-n private submissions bias the reported maximum
upward with zero capability change.

614
00:57:16,760 --> 00:57:20,300
So what does that mean? That means, okay, so my Meta…

615
00:57:22,600 --> 00:57:32,779
wants to get the best… their best score that they can on Chatbot Arena,
so they submit 27 different private variants, right?

616
00:57:32,890 --> 00:57:37,699
And then they just… Report the best win.

617
00:57:37,860 --> 00:57:46,660
Right? So… that's… that's another example of reward hacking, right? So
these might differ by…

618
00:57:46,990 --> 00:57:50,659
Fine-tuning, you know, or whatever.

619
00:57:50,780 --> 00:57:54,989
Just to try to game this one, this one benchmark.

620
00:57:55,310 --> 00:57:58,550
Okay, okay.

621
00:58:00,390 --> 00:58:05,419
So, and then we need to talk about Goodhart's law.

622
00:58:06,340 --> 00:58:11,790
Okay, so what if… You now just take the benchmark grader.

623
00:58:12,470 --> 00:58:15,270
And you use it as an RL rewards signal.

624
00:58:16,230 --> 00:58:21,930
Right, so that's kind of what this, like, what Meta was doing here,
right? So, like.

625
00:58:22,470 --> 00:58:27,239
Take the, the actual score itself that you get out of the benchmark.

626
00:58:27,630 --> 00:58:37,630
And try to optimize for that, right? So try to train your model
Specifically to, to,

627
00:58:37,760 --> 00:58:40,000
To do better on the benchmark.

628
00:58:40,670 --> 00:58:42,689
Right, you know, I think in the…

629
00:58:42,960 --> 00:58:48,339
You know, in the LLM community, like, this sometimes is called, like,
benchmaxing.

630
00:58:49,770 --> 00:58:56,210
But it's also, even just… even if it's not a benchmark, like, let's say
you want your model to get good at,

631
00:58:56,390 --> 00:58:57,850
research taste.

632
00:58:58,720 --> 00:59:05,639
Okay, and you design a grader to score research taste, and you try to,
train against that.

633
00:59:05,820 --> 00:59:06,710
Right.

634
00:59:07,020 --> 00:59:12,219
then you're gonna run into Goodhart's law. Okay,

635
00:59:13,140 --> 00:59:19,129
And but the point is, we're now not just trying to exploit the grader.

636
00:59:19,540 --> 00:59:25,540
As a measurement error, we're actually… that's a… that becomes a
training objective now.

637
00:59:25,820 --> 00:59:29,040
Right? So…

638
00:59:29,950 --> 00:59:41,250
For example, you know, this has been admitted behavior in, like, model
system cards. The model occasionally resorts to special casing in order
to pass test cases.

639
00:59:41,570 --> 00:59:44,579
attributed to reward hacking during RL.

640
00:59:45,550 --> 00:59:46,340
Great.

641
00:59:46,710 --> 00:59:52,790
And so what that means is, so what is Goodhart's law? Is that…

642
00:59:52,950 --> 00:59:56,520
As soon as you stop,

643
00:59:56,760 --> 01:00:00,860
As soon as you start optimizing for a grader.

644
01:00:01,250 --> 01:00:06,089
It stops becoming a measurement, and it starts becoming something.

645
01:00:06,250 --> 01:00:13,079
to exploit, that's not necessarily the thing that you wanted to measure
in the first place, right?

646
01:00:13,540 --> 01:00:16,740
So, okay.

647
01:00:17,950 --> 01:00:18,870
Alright.

648
01:00:19,190 --> 01:00:22,780
So Part 3, Multi-Agent Benchmarks.

649
01:00:23,680 --> 01:00:29,900
So, as you can imagine, So far, we've just been talking about single
agent.

650
01:00:30,150 --> 01:00:38,020
But of course, as soon as you open it up to multi-agent, things get, you
know, Way more complicated.

651
01:00:38,260 --> 01:00:45,880
And there's all sorts of issues that just come from the multi-agent part
itself.

652
01:00:46,090 --> 01:00:53,720
And, you know, also, it's not even clear what you should be measuring
anymore. You know, just like in game theory, we talked about

653
01:00:54,250 --> 01:00:59,240
What is the actual solution to the game, right?

654
01:00:59,610 --> 01:01:03,419
But here's a list, so here's a table listing,

655
01:01:04,030 --> 01:01:11,050
Like, quite a few different benchmarks that exist out there for
multi-agent, systems.

656
01:01:11,400 --> 01:01:15,800
So these are organized by category here, so, like, cooperative ones.

657
01:01:16,680 --> 01:01:19,870
Mixed-motive social dilemmas.

658
01:01:21,910 --> 01:01:24,309
Competitive or hidden role.

659
01:01:24,940 --> 01:01:27,880
Dialogue, social, group reasoning.

660
01:01:28,100 --> 01:01:28,850
Good.

661
01:01:29,120 --> 01:01:31,750
So, there's a lot of,

662
01:01:32,720 --> 01:01:40,870
a lot of, different ones out here, and there's different metrics that
they have, right? So, like, win rate and competitive ones.

663
01:01:41,320 --> 01:01:46,490
You know, different ways of scoring LLM plus human judges.

664
01:01:46,710 --> 01:01:50,450
For this dialogue one here,

665
01:01:51,090 --> 01:01:55,450
For cooperative ones, I mean, I guess that seems pretty simple. That
should just be, like.

666
01:01:56,150 --> 01:02:00,939
combined, how well did they do? You know, like, so just sum up

667
01:02:01,220 --> 01:02:06,290
their… their scores, right? So, like, the return score accuracy.

668
01:02:06,830 --> 01:02:14,899
Mixed motive, you know, then things get a lot more complicated, but we
can ask things like social welfare.

669
01:02:16,580 --> 01:02:17,870
survival.

670
01:02:18,270 --> 01:02:21,380
Crap. Other comments?

671
01:02:22,420 --> 01:02:23,290
Okay.

672
01:02:27,040 --> 01:02:33,330
So, Lesson 1, the partner population is the result.

673
01:02:33,550 --> 01:02:34,250
Okay.

674
01:02:34,830 --> 01:02:37,079
So, what do we mean by that?

675
01:02:37,440 --> 01:02:49,740
Well, how your agent does, like, let's say you're trying to do
coordination, that's going to,

676
01:02:49,870 --> 01:02:59,039
that's going to depend a lot on what you're trying to coordinate with,
right? So that background population is extremely…

677
01:02:59,210 --> 01:03:01,990
relevant to this, right?

678
01:03:02,440 --> 01:03:10,220
So here's a few examples. Overcooked AI, self-play agents do
significantly worse when paired with a model of a human.

679
01:03:11,260 --> 01:03:12,040
Okay.

680
01:03:12,590 --> 01:03:21,760
Melting Pot… Right, so what does that mean? So that means agents learned
ways to coordinate and self-play

681
01:03:21,870 --> 01:03:26,860
But that only kind of works with each other, right? If you switch it out
for a different model.

682
01:03:27,200 --> 01:03:30,800
One that was trained by imitating humans, for example.

683
01:03:31,300 --> 01:03:35,460
Then, that coordination doesn't really survive.

684
01:03:36,360 --> 01:03:43,230
Okay, Melting Pot… Score a focal population against a held-out
background.

685
01:03:43,400 --> 01:03:44,800
Population…

686
01:03:45,340 --> 01:03:52,589
Right? So trying to, reveal failures that training reward hides, right?
So that's a way to try to measure that.

687
01:03:53,130 --> 01:03:59,520
CoopEval, recent models consistently defect.

688
01:03:59,680 --> 01:04:02,360
In single-shot social dilemmas.

689
01:04:03,060 --> 01:04:06,259
Right? But, again…

690
01:04:07,200 --> 01:04:13,499
If we have any score, so let's try to abstract this out, so any score
against a fixed population.

691
01:04:14,059 --> 01:04:19,240
It's only gonna be a measure of how that… How that agent, Pi.

692
01:04:20,190 --> 01:04:28,589
Right? If the held-out population is sigma minus i, Right? We're only
scoring how well

693
01:04:28,729 --> 01:04:38,829
pi best response to that population, right? It's not a property of pi
alone, right? So I can't just report… so what does this tell me? I can't
just give a score, I can't say, like.

694
01:04:39,360 --> 01:04:42,720
Pi, in this multi-agent setting.

695
01:04:42,830 --> 01:04:50,370
has a score of X. It doesn't make any sense, right? Because if I change
sigma minus i.

696
01:04:50,849 --> 01:04:56,010
then the ranking's gonna change, right? So, you know, this…

697
01:04:56,370 --> 01:05:02,590
Absolutely, you know, we should… we should be very comfortable with this
statement after this course, right?

698
01:05:06,440 --> 01:05:07,300
Okay.

699
01:05:07,889 --> 01:05:10,980
So actually, I have a lot more slides here.

700
01:05:11,849 --> 01:05:15,090
So, I think I should probably stop.

701
01:05:16,510 --> 01:05:19,780
The course evaluations are open.

702
01:05:20,059 --> 01:05:21,110
And…

703
01:05:21,330 --> 01:05:29,319
I would really encourage you to… to go ahead and do those. Like, if you
can take this time now and… and do it, that would be great.

704
01:05:29,889 --> 01:05:34,190
You know, this course, this course has evolved

705
01:05:34,580 --> 01:05:39,630
Considerably from where it started. So it started as an algorithmic game
theory course.

706
01:05:39,860 --> 01:05:47,850
Which is, you know… Totally theoretical, like, models of computation,
proof-based, mechanism design type things.

707
01:05:48,500 --> 01:05:50,409
And I've been trying to make it…

708
01:05:50,700 --> 01:05:53,660
Something that's more, like, broadly appealing.

709
01:05:55,120 --> 01:06:05,299
But, you know, tying those two things together, like multi-agent, LLM
systems, and game theory, like, I feel like there's more progress to be
made.

710
01:06:05,790 --> 01:06:11,009
on… on that, within this course, you know? So…

711
01:06:11,570 --> 01:06:15,549
Please give me… please give me feedback, and

712
01:06:16,080 --> 01:06:17,740
Help me try to improve it.

713
01:06:17,980 --> 01:06:19,310
Going forward…
