neptune.ai
Neptune.ai is an experimentation management platform that was used by hundreds of data scientists building the world’s foundational models. It has since been acquired by OpenAI and is used as their internal tool for continuing to build the world’s most powerful foundational models.
400~
million acquisition
2b+
Experiments tracked
92%
roadmap delivery
I was hired as the head of design and quickly transitioned to covering both product and design. I was brought in the solve everything UX and transition the company away from emergency mode fixes and into a cohesive platform worthy of the data scientists who are building the world’s most powerful foundational models.
Where did I start
Early challenges
Emergency mode was killing an incredible product
When joining neptune there was so many opportunities and the most difficult part was actually where to start. The product had evolved over 8 years by solving emergency after emergency and no clear or cohesive product or design vision. The company also had pivoted right before joining from targeting smaller companies developing AI products to purely focusing on foundational model teams. This changed a lot of the focus’ and when I was brought in we needed a cohesive plan and fast, as the foundational teams we recently signed deals with had complaints with the legacy and over heavy UX. We quickly set up a plan for tackling some low-hanging fruit, some obvious medium term UX issues and some longer term features that were sorely needed for the new use cases we were trying to solve.
New and very private sector
One of the biggest issues faced throughout my time before sale was that this whole sector is so new, so complex and so private. It was a lot to get up to speed on, even for someone who had always worked in very technical fields with very technical profiles. I spent a lot of time with our CEO/CTO, our support team and our customers getting up to speed on everything I could possibly learn. Also, since the competition within foundational model companies is so fierce, there was almost nothing we could see directly and almost no data we could pull from our customers. We often had users describing what they were seeing and we would reinterpret that in a demo environment and edit from there so we could see problems or common use cases. This was such a unique issue and was a real challenge to identify opportunities, we often worked hand in hand with users to iron our priorities and workflows.
The Features
Where did we focus
My role at neptune originally began with running design but very quickly transitioned to covering design and product. I focused on creating the product vision and roadmap with the CEO/CTO and customers, ironing out PRDs, setting up sprints, assigning work and also, almost all the actual design within the product. It was a busy time but well worth it when our largest detractors became our biggest supporters and eventually the company was acquired by OpenAI.
Our main focus’ were to fix as many of the big issue, low hanging fruit as possible, while support medium term UX changes and longer term features that were needed by our new customer focus. This broke out into a number of major efforts but I will detail only a few below.
New IA changes
When I joined Neptune’s navigation was structured around objects (e.g., views, charts, dashboards) rather than around the user’s actual jobs-to-be-done, due to how the product evolved, in a state of constant emergency mode and without much UX experience on the team. As a result, users often struggled to understand where to begin, how to move between tasks, or what the appropriate next step is in a workflow. It is was not immediately obvious how to use Neptune to actually accomplish the goals of the user. We set out to completely change the information architecture of the product to simplify and streamline the product to the users actual goals.
We spoke to our users and any foundational model developing data scientists we could and go to work streamlining. I would be happy to share the PRDs and full designs for this individually but because this is now a proprietary system used internally by OpenAI I can’t necessarily share much here.
The changing of the IA had to be rolled out in pieces that made sense and didn’t break the experience simply due to how much of the underlying product architecture would need to change. It was a massive chunk of work and something we were still working on releasing when the company was acquired.
Charts & Legends
With most developer tools or products for technical users the joke is that it is 99% tables and charts, and this product realistically is no different, only that environments frequently needed to have charts for 10+ experiments across 12+ attributes on just a single legend, that the name of these would be over 124 characters, users would need to select them from a list of half a billion objects, they would need to toggle specific viewing criteria and see the legend when hovering over extremely specific points in the charts. The charts, legends and interactions had to be finetuned so heavily to be usable it is something that truly can never be all the way perfect. There were frequently 50+ charts on a single screen and the legend had to be there when needed and get out of the way when it isn’t. It had to be configurable and predict a users needs and somehow it also had to be simple. I could write a book at this point about the UX of complex charts and legends. I would love to walk you through them but since there are pages and pages of specs, I’ll just include a quick screenshot for you below.
Complex names and tons of attributes
As I mentioned, the names of experiments and attributes were extremely lengthly, users needed to include a ton of information in them, often they were automatically generated and they needed to skim these easily. We needed to anticipate what a user needed to know at a certain point in their workflow and display them easily at that point.
We did this by setting up a highlighting structure when searching for attributes, created a way to simplify experiment names with smart shortening in subheader rows in the legend and using a smart shortening and highlighting in experiment names across the product, most importantly in the primary experiments/runs table. This was all particularly challenging without being able to SEE the actual names our users had within the product. We had to create similarly structured names, using randomised or anonimised specifics while maintaining the naming structures and variables often used. We relied a lot on our customers feedback here but just starting with that base was a challenge.
DESIGN SYSTEM
Building the foundation
When I got to the company there was no design system. The majority of the design file was screenshots from the product with rectangles and text boxes laid on top. We had to create a design system with the current state of the product so developers could develop features and make fixes without confusing new work as well as one with where we needed to be. As you can imagine this was a bit of a challenge, especially while moving forward. However, it was also a great opportunity for a full audit of the product, identifying the most important and low hanging fruit UX fixes. It was a great first task for myself and a new designer to take on. We focused on creating a design system for the current state that was only the basics of what was needed and focusing on the evolution going forward. I was very lucky to have a ton of buy-in from the CEO/CTO as well as the development team who had been suffering without clear components in the product so adoption was swift. There was some major challenges with a product so heavy, like complex dropdowns and action menus, legends etc but surprisingly the biggest challenge may have been with colors. When users need to see upwards of 20 lines on a chart with colors, highlighting AND often line style differences, the colors that are used are critically important, as well as line styles. The algorithm for assigning, reassigning and which colors to use took myself and a developer upwards of a month of research and fine tuning. It still remains something the entire industry struggles with.
Want to talk?
I’m currently open to consulting, advising and speaking opportunities. I might be open to a full time position with the right team and product fit.
Amber Rucker
LinkedIn↗