Jennifer and I revisited

When I originally posted about my fraught relationship with Jennifer Campos last year, I neglected to add the explanatory Bluesky tweet, or whatever they’re called, that was really necessary to understand the whole send up of the AI LLMs that the post was intended to perform (with a big assist from JL Borges naturally).

A year later I decided to give the LLMs another test run, after hearing endless hype about how they’re improving at exponential rates etc. Here was my little experiment, which involved two almost identical queries:

“What was the regular season record of the New York Giants from 1986 to 2015?”

Exact same question, except substituting the Denver Broncos.

Results:

Gemini (Google)

First question: 226-253-2

Second question: 272-207-1

Claude (Anthropic)

First question: 259-256-1

Second question: 298-209-1

ChatGPT

First question: 273-205-1

Second question: Refused to give cumulative record; produced season by season list.

Correct answer to the first question: 260-218-1

Correct answer to the second question: 289-189-1

As an extra added bonus, Gemini reported that the Giants won the Super Bowl twice over these seasons (They actually won it four times).

The really strange is that the two most wildly incorrect answers also displayed the teams’ season by season record correctly, so apparently this version of AI can’t do what the simplest handheld calculator could do 40 years ago.

Now the caveat here are that I used the free versions of these products, but on the other hand my queries were incredibly simple and straightforward (I calculated the answers myself using a pen and paper in about three minutes).

This by itself is little more than anecdotal, but given the messianic and financial claims being made for AI, it’s another anecdote that would seem to raise some skeptical questions.