A Product Analysis on the custom GPT Store
Deconstructing what worked and what didn't
Being able to understand and analyze the competition is an important aspect of being a good Product Manager. When a competitor launches something interesting and noteworthy, it’s helpful to take a moment to dive deep into the launch to understand and extract any key learnings. This includes reading press reactions, user feedback, and testing things out yourself. I then summarize my findings in a doc, pointing out noteworthy aspects of the launch, and adding my own product commentary on top of that. I find it helpful to include references, data, and direct quotes - then I share it out with my team for others to add their own thoughts on top of.
It’s been 8 days since OpenAI has launched their custom GPT store - so naturally, I wanted to better understand exactly what that means from a Product PoV, given its close relation to the product I own: Google AI Studio & Gemini API.
The following is my analysis on this launch. It’s something that I usually write up and share internally, but this time I’ll start by sharing openly with you all. The only section missing from this doc, is how I further absorb these takeaways and apply them to my current thinking on my own product (that part I need to keep internal). I hope you find this useful both for the content itself, and also as a reference for something you can do in your own product roles.
Details
Custom GPTs were announced on November 6 and the store went live on January 10
There are are “over 3 Million GPTs” that have been built → Compare this to the 180.5M users (August 2023) who have created a ChatGPT account, 100M of which are active weekly
You must be a ChatGPT plus subscriber to both build and use custom GPTs
OpenAI promises a revenue share, but still have not disclosed what this will be or how it will work
Positives
Anyone who has built something interesting that they want to share with the world, can now do so for free.
The end-user has to pay for their own consumption, rather than the developer paying (and then setting up billing to charge their users → this is non-trivial to build for general consumers)
This is similar to the “open source workaround” that we’ve been seeing where developers can build something, then the person downloading and running the open source application needs to enter their own API key to make it work (and thus pay for the service directly themselves)
OpenAI now gets more data to better understand user and developer patterns
Negatives
Lack of Discoverability: of the 3M GPTs, only 72 are shown on the store – the rest are left for users to try to “discover” (fun fact - there are custom GPTs to help you find GPTs)
No Moat: It’s easy to clone a custom GPT
Quality is Lacking: The functionality of a custom GPT isn’t strong - RAG implementation was lacking, no context management, no tools to help with testing, Instruction following still isn’t good enough
Poor Developer Experience: The store itself is simplistic and lacking in basic features: no review process for submission, no download counts or ratings, they allow duplicate names making it confusing to search; quota is too low, growth / sharing is restricted to other ChatGPT paid users
Takeaway
Either OpenAI continues to invest in the custom GPT store, enabling a more robust “building” experience in addition to a full featured store…or this is a moment in time that will enable OpenAI to further understand what users are building and using - so they can better prioritize those use cases for future model development.
Deeper Dive
OpenAI has chosen to focus on custom GPTs in the following categories:
Writing
Productivity
Research & Analysis
Programming
Education
Lifestyle
Which likely means that either (i) of all the custom GPTs that have been built, these are the categories that apply to the most (or at least the most noteworthy), and/or (ii) these are the categories that OpenAI feels like custom GPTs can best service.
Within each of these categories, they have listed 12 custom GPTs:
Revenue sharing is still not implemented, and details on how this will work have not yet been disclosed.
Looking at 3rd party GPT rankers, we see the following trends:
There is a very long tail of custom GPTs → #500 has 1k conversation and #1,000 has 900
From the top 100 GPTs, the top 7 GPTs account for 50% of conversations and the top 2 GPTs account for over 1/4 of all traffic - and one of those is literally GPT-4
A Few Notable Quotes
“Very cool concept in theory.”
“They are too janky in their current form to be actually be truly useful.”
“your moat is just a text prompt”
“ it currently functions more like a giant directory of tweaked ChatGPTs”
“the fact I don't have to monetize just to support OpenAI API costs is still very appealing”
Sources
Press: Business Insider, Mashable, Wired
Reddit Threads [1, 2, 3]