In Part 3, we rebuilt TNNS HEAD’s video coaching around the iPhone camera. We could record at 60 frames per second, follow the player we wanted to coach, and give the model a better idea of where they were on the court. It could still look at a forehand and tell us how to improve the overhead.
We’d spent months trying to give models a better account of what happened. Then Elorian released Glimpse, a model built to look closely at video. At the published rates, it was also about a third of the price of Sol per token. How much of that work could it do itself?
I gave Glimpse a 33-second tennis clip and asked it to coach the player in blue shorts. It came back with a ball-by-ball account, timestamps down to tenths of a second, and a description of a little girl patting the ball over from the net.
The little girl was a grown man at the far baseline.
When I told it that, Glimpse went back through the footage and corrected both who it was looking at and where he was standing. It pointed to perspective and the net obscuring part of his body. The guy looked small because he was far away. The second pass was impressive in its own right. Of course, I had to know the answer before I knew to ask it to look again.
The mistake mattered for the tennis. A soft feed from somebody standing at the net and a serve from the far baseline give the player two very different problems to solve. Glimpse had written a detailed account of the first one. The player was dealing with the second.
Why we got excited in the first place
I gave it a different clip, about 30 seconds of tennis shot through a chain-link fence. It came back with a timeline, advice tied to specific moments, and a gallery of frames it had pulled to examine the player’s position. It knew the fence and the distance made the far player harder to see. On a first read, there was plenty to like. Then you’d replay a timestamp and ask whether that really was a forehand, whether the opponent was standing where it said he was, or whether the point had actually ended.
We tried giving it more of what we knew
Up to this point, I’d been running clips by hand in Elorian’s interface. Then we put Glimpse into our test harness and started comparing runs on the same fenced clip. With video alone, it had read serves from the far court as balls being played from near the net.
We started by giving it SPI primitives. Our first pass used saved court projections, and one of those put the near player on the wrong side of the net. We tried again with reviewed landmarks and approximate positions for both players. Glimpse now put the opponent back near the far baseline. It still called everything after about 22 seconds ball collection, which meant it skipped the late exchange when the near player came forward.
Then we gave it what we called the disposition in Part 3. That was our first pass over the clip to work out its overall flow and put the events in order. We used Sol’s saved read of the point, kept the timestamps and uncertainties, and stripped out the suggested coaching.
That worked better on this clip. Glimpse picked up the far player’s serve attempts, including one around nine seconds where the receiver never swung. It picked up the high backhand return on a later serve and kept following the exchange as the player came in for a low ball.
The stretch it had written off as ball collection became the place it most wanted to coach. We still had to see if the lesson it picked held up.
Same account, different coaching
We gave Glimpse and Sol the same written sequence and approximate spatial notes for that clip. Glimpse also had the original video and audio. Sol worked from the written material. We wanted to see what each would make of the exchange once it had an account to work from.
Sol picked the low forehand as the player came forward. Give yourself room, play a controlled ball, then get ready for the next one. Its cue was “Low ball: control, then ready.” It offered that as something to practice. It could see a situation worth preparing for without claiming the player had botched it. I could imagine taking that cue onto a court, though there’s a limit to what it tells you about this particular point.
Glimpse picked the scramble at the net. It said the player had popped the ball up to himself and should have backed up sooner to get under it. That’s more specific. If that’s what happened, you can see exactly what you’d work on. But when we went back to the evidence, we still had a question about who touched the ball and whether it was live. Glimpse acknowledged some of that uncertainty and still kept the advice about retreating under your own pop-up. That was the part of the story we needed to establish before we could give the advice.
I wouldn’t pick a winner from one clip. The comparison did show us where to look when an answer sounded good. Take away the claim about the pop-up and ask what’s left of the lesson. Sometimes you’re left with useful advice. Sometimes you’re left with a drill for something that never happened.
Where we landed
By this point we’d already scaled back our ambitions to one good coaching insight. We wanted more than 90 percent of reports to contain at least one piece of advice we’d stand behind. A report could get plenty right and still miss that bar if the one thing it told the player to change came from a bad read of the point.
That was the pattern across our testing. The first results could be damn impressive. Then we’d watch the point, check the timestamps, and see if the advice still made sense. Once we started doing that consistently, we found enough errors that we couldn’t count on the reports to clear our bar. Glimpse could go back and look again, which was something we’d wanted all along. We still had to know when to send it back.
We got to the point where we could show someone a clip and give them a good idea of what this could be. We couldn’t yet give a player that experience often enough to rely on it. So we paused.
In the fifth and final post, I’ll explain where we’ve left video coaching at TNNS HEAD and what we learned along the way. What would we keep if we started over? What would we do differently? And what would have to change for us to take another run at it?