Two days to build something meaningful
A colleague and I had the opportunity to attend a two-day hackathon in London, organized by HG Catalyst under the auspices of Anthropic. The brief was fairly open-ended: come up with a product or solution with potential business value that we might not normally have time to explore, and try to turn the idea into a working prototype in just two days.
We worked in a team of five and decided to take advantage of an interesting opportunity. The team.blue group, which we represented at the hackathon, has a broad portfolio of digital products and services. We asked ourselves: what if customers didn’t need to know which specific products they needed?
A hairdresser starting a business, for example, needs a website, a booking system and perhaps other services. A tradesperson needs a way to showcase their work and send invoices. Our idea was to create an AI portal where customers could simply describe what they wanted to do, and the system would help them not only choose services but also configure them.
Why an ordinary chatbot isn’t enough
From previous experience, we knew that customers often can’t articulate exactly what they need. Presenting them with an empty chat window and the question “How can I help you?” might not work. We chose a combination of a structured form and AI. First came a few general questions, then the AI generated follow-up questions to clarify the details. In the background, an agent called the necessary tools and assembled a selection of products based on the answers.
The idea itself was interesting, but for me, everything surrounding it was even more so. How would we know whether the agent was working correctly? How would we find out where users were getting stuck? How would we evaluate feedback? How would we prevent abuse or prompt injection attempts? And above all, how would we ensure that the system could be improved over time?
Observability is more than an add-on
This was where the most interesting part began for me. The other team members brought experience in areas I hadn't focused on as much before, such as AI application observability and security mechanisms. We used Langfuse to track individual conversations, sessions, and interactions with the AI. We added a way to rate the relevance of responses in the application, and this feedback was linked to specific interactions. That gave us a foundation for analyzing the agent's behavior later on.
The solution also included versioning system prompts and gradually improving them based on evaluations. Using LiteLLM, we also set up request routing between different AI models, so we could more easily compare their suitability for specific operations and switch between them without making major changes to the application.
It's fair to add that, over the two days, we mainly prepared the infrastructure for future evaluation and iteration. We had no real customers and primarily tested it ourselves as a team. These were not conclusions drawn from extensive real-world use, but rather foundations we could build on. I realized that creating a functional AI agent is one thing, but being able to measure, evaluate, and systematically improve its behavior is an entirely different discipline.
AI writes code, but we have to manage quality
We had little time for implementation, so we created a lot of the code with the help of AI. That made it all the more important to set up the development environment, automated tests, static analysis, and automated code review right from the start. We didn't have the capacity to manually check every line, so we needed reasonable guardrails for rapid development.
I focused on the backend and AI API integration, while other team members focused on areas where they had experience. We then brought the individual parts together into one system. Our diverse experience proved to be one of our greatest advantages.
Communication can't be automated
Not everything went smoothly. At one point, we realized we hadn't made it clear enough how the frontend and backend were supposed to communicate. When we connected the parts, it became apparent that our ideas weren't quite aligned. This resulted in a minor delay and a need to quickly agree on the details. In the end, we managed it, but it was a useful reminder: the faster we develop, the faster poorly communicated decisions make themselves felt. AI can speed up implementation, but it can't create mutual understanding for us.
Five perspectives, one shared result
Looking back today, the most valuable thing wasn't the amount of code written or any particular technical solution, but the opportunity to work with people who approached the same problem from different perspectives. Together, we designed a system that was more than just a collection of API calls: it was something we could monitor, evaluate, and gradually improve.
The biggest lesson for me was that as AI makes writing code cheaper, the ability to design a good system, collaborate, and understand the bigger picture becomes more important.
And how did it turn out?
Forty-seven teams took part in the hackathon, and our project ultimately won the audience vote. It was a great way to end two intense days, but the greatest value for me was seeing how quickly a well-coordinated team can create a complex solution with the help of AI, and how much one can learn from colleagues with different experiences.
