DECODED: Battle of the Bots
I asked ChatGPT, Claude, Gemini, and NotebookLM to summarize a quarterly report. Here's how they did.
This is Decoded: AI tools for getting straight answers from complex investment documents.
AI is a powerful tool for making sense of investment documents. Let’s compare the output of ChatGPT, Claude, Gemini, and NotebookLM when given the same quarterly report and the same command.
The Process
I downloaded the report from the investor portal. The provided pdf was a well-written and concise one and a half-page narrative.
I opened a new window in Chrome for each of the AI tools, so I had four tabs running simultaneously.
I fed each AI tool the same information: I uploaded the same document, and gave it the same command:
"Summarize the attached report to give the investor a summary of the investment's performance for the quarter reported.”I waited for the results, which took less than 30 seconds.
The Results
ChatGPT
Analysis Grade: B-
A well-presented but very long report ending with an excellent 4-point conclusion, which would be more useful at the beginning of the response than at the end.
However, the response included non-annotated conclusions, which raises questions about what data was used in the analysis. This is a prime example of how AI can provide a confident answer based on potentially flawed interpretation. So, I challenged Chat on its analysis by asking:
What source(s) were used to form the following conclusion: “Leasing activity …”
Chat responded with a very detailed answer, but the most valuable insight was in the closing:
If you want a stricter, investor-grade version, I can rewrite that section to clearly distinguish reported facts vs inferred interpretation, which is typically how institutional investors expect it to be presented.
AI is getting better at “teaching” the user, but users must realize that AI is not a mind-reader and the results are only as good as the instructions.
Confidentiality Grade: F
ChatGPT is not secure. By sharing investment information with ChatGPT, you may be violating your confidentiality agreement.
Claude
Grade: A
Excellent, condensed version of the report’s contents, presented in a very logical and easily digestible format. Nothing extra - just a shorter version of the original that captured all the important details.
If you want the 10-second read that doesn’t leave you with questions, Claude will deliver that.
Confidentiality Grade: B-
Claude is not secure by default. By sharing investment information with Claude, you may be violating your confidentiality agreement. Check your settings and subscriptions details.
Gemini
Grade: C
Not a terrible summary, but the most difficult to digest. The information was condensed into wordy sentences in a jumble that jumped from topic to topic. On the plus side, each point was annotated. Nothing added, but not much gained either. The original report was easier to read.
Confidentiality Grade: F
Gemini is not secure. By sharing investment information with Gemini, you may be violating your confidentiality agreement.
NotebookLM
Grade: B
True to form, NotebookLM provided an accurate and concise summary. There was no data interpretation or additional information woven in that was not provided in the report itself, and every point was annotated. A bit long, but got the job done on the first pass.
Confidentiality Grade: A*
The underlying model for NotebookLM is web-based, but the source information you upload remains within your NotebookLM account; it is not used to train Google’s models or shared elsewhere.
*Use of the feedback buttons will automatically and irrevocably share the entire chat: all the documents, prompts, etc. To maintain confidentiality, don’t use the feedback function.
Conclusion
If AI is the bridge between reading and not reading your reports, then any of these tools would be a meaningful improvement. However, confidentiality is an obligation; share accordingly.
If you already have a favorite AI tool, it is probably getting the job done but be aware of its limitations - and maybe shop around.
If you want the best tool for the job - look to NotebookLM.
Your Turn
What AI tools, tricks, or limitations are you finding? Share in the comments below.