Gemma 4: Strong on Basic Tasks, Weak on Real Research

+

Google recently dropped Gemma 4, its most powerful open model ever. Gemma 4 excels at common tasks like summarizing documents. But for deeper research, it falls short.

This morning, I ran Gemma 4 through a three round test. I had it perform real tasks I do every day as an investor.

Let me show you what Gemma does well and where it struggles…

Round #1: Prepping for a Founder Meeting

Next week, I’m meeting the founder of a really interesting robotics company. Can Gemma prepare me?

Gemma couldn’t find any information on the founder. I wondered — is this just because she’s very private?

So, I asked Grok the exact same question. Grok pulled everything in seconds!

I’m giving Gemma an F on this round.

Round #2: Finding New Investments

Gemma had a tough time researching people. Maybe it will do better with companies?

I asked Gemma to find companies in the three areas I mentioned on yesterday’s post.

American-made photovoltaics
Robots for rare earth mining/refining
Fiber optic wire guided drones

I asked Gemma to return specific information about each company. Let’s see what this model can do…

Gemma found a nice list of companies in the areas I’m looking for. It also gave me some useful info about each one.

I would’ve liked Gemma to focus on companies that are a little earlier stage. Still, it found some fascinating startups.

I’m giving this round a B+.

Round #3: Summarizing a Deal Memo

For the final round, I had Gemma summarize a deal memo in a startup I invested in last year, Cortex AI.

I often refer back to my deal memos to inform follow-on investments. If Gemma can do a good job summarizing the memo, that would be super helpful.

Gemma gave me a great summary, calling out key info like the founder’s background and early traction.

For the first two rounds, I used Gemma 4 31B IT, which Google calls its “flagship open-weight dense model.” For the this final round, I tried Gemma 4 26B A4B IT, a mixture of experts model designed to be extremely compute efficient.

If Gemma 26B can summarize a complex document like this, it should do well on pitch decks, news articles, and a lot more. And the cost will be minimal.

I’m giving this round an A.

Wrap-Up

I’m giving Gemma 4 a C+ overall.

Gemma does a good job at basic tasks like summarizing a document. But ask it to go a little deeper, and the model struggles.

I suggest using Gemma for simple tasks in the applications you build. It handles those competently and cheaply (especially 26B).

For more complex tasks, stick to one of the frontier models.

Gemma 4: Strong on Basic Tasks, Weak on Real Research

Share this:

Leave a comment Cancel reply