Vinted Engineering

TL;DR: Vinted operates a billion-scale search system - time to reflect and celebrate.

Ever wondered what goes into making the search experience seamless at Vinted? In this post, our Search Platform team take you through their latest improvement. Discover how the team’s hard work and expertise allowed us to transition from Elasticsearch to Vespa, a cutting-edge search engine that’s set to elevate how our members find their next second-hand gem.

In today’s digital landscape, recommender systems have become ubiquitous, curating user experiences across a wide array of online platforms, including Vinted - Europe’s largest online second-hand fashion marketplace. In this blog post, we outline our journey of adopting the Vespa search engine to serve personalized homepage listing recommendations, helping our members find deals they will enjoy. We are excited to share our story as we have found Vespa to be a great solution combining the now trendy vector search with more traditional sparse search techniques, as well as offering a great engineering experience.

At Vinted, we’ve implemented a 3-stage recommender system that leverages both explicit and implicit user preferences to offer users a curated list of items presented on the homepage. Explicit preferences are inputted by users on the app, allowing them to specify details such as the clothing sizes they are interested in. Meanwhile, implicit preferences are extracted from historical user interactions on the platform, including clicks and purchases, via the use of machine learning models. This system distills a tailored selection from millions of available listings, presenting users with options most aligned with their tastes and behaviors.

A tiny disclaimer: When I say mock, I will be referring to a Mock Test Double. The mocking framework used to create this Test Double can be known as dynamic mock libraries, as defined by Mark Seemann. All code I will show here will be using Kotlin programming language.

A robust and reliable test suite helps us be more confident during refactors, preventing a code breach from going unnoticed; it also could improve the speed of code reviews, where the tests can act like documentation about the happy path, error handling, and edge cases. Lastly, it could highlight design problems if the code is difficult to test. Using mocks and stubs has become standard practice to isolate components and ensure reliable tests (London School). However, it’s imperative to recognize the fragility and potential pitfalls associated with excessive dynamic mock libraries and stub usage.

Building an effective and efficient data ingestion pipeline is a challenging task. Let’s cover the migration from scheduled Sidekiq background jobs to real-time indexing built using Apache Flink.

As containerisation continues to revolutionize the way software applications are developed and deployed, ensuring the security of container environments has become an utmost priority. Security in a containerised environment requires a different approach than traditional security mechanisms. It involves continuously monitoring container activities, identifying security threats, and ensuring compliance.

In this blog post, we want to share our experience with Falco, an open-source tool designed specifically for securing containerised environments.

Here at Vinted, we adhere to the continuous deployment principle, meaning that each merge to the main branch of the code repository initiates an automatic deployment process. As a result, the merged code goes live in a short period of time. Merge small and merge often are two practices that are instrumental to our day-to-day engineering work.

This approach mitigates friction when working on a shared code base, shortens the feedback loop, and reduces the anxiety of breaking the production environment with a gigantic change. But for it to actually work, one must have very good code test coverage, deployment automation tools with solid safeguards in place, a reliable observability stack to catch problems in the deployed products, battle-tested rollback, recovery procedures, and much more.

In this post I’ll touch on the testing phase. Specifically, I’m going to explain how we make use of the Vitess database to run backend tests of our core application.

This is the fourth in a series of chapters sharing our Vitess Voyage story. After a rough ‘Vinted Autumn’, this time we came to the conclusion that vertical sharding was no longer an option.

This is the third in a series of chapters sharing our Vitess Voyage story. With the plan ready and wounds healed, we made the move.

This is the second in a series of chapters sharing our Vitess Voyage story. After a busy ‘Vinted Autumn’, the real work took hold.

“Winter is coming” - Ned Stark, Game of Thrones

Au contraire, autumn is the season to warn us of upcoming challenges - the most extreme growth and workload period of the whole year. In fact, we coined the term - ‘Vinted Autumn’. Our main MySQL databases were taking a massive beating every autumn despite sharding them vertically multiple times. While managing a dozen physical servers is an ok task, managing 42 servers manually is quite cumbersome. Additionally, the process of vertical sharding itself was increasingly hard to orchestrate and there was no way of turning back. Hacking around a horizontal sharding solution was not an option either. Naturally, we wanted to bring in tools to efficiently manage MySQL.

Vinted is taking a significant step to strengthen our presence in Lithuania - we’re officially expanding to Kaunas this year. We have a number of software engineering and engineering management roles that were previously only located in Vilnius which are now open for applicants living in or near Kaunas as well.

In this blog post I will explain our reasons for this expansion, paint a picture of Vinted today and share my personal reasons for joining Vinted more than seven years ago. Let’s find out if my reasons aged well.

The Software Asset Inventory keeps track of all the software that is operating in an organisational environment. These assets might range from business applications operating on an organisational level, to the personal applications installed on the end-user devices.

These software assets include:

• Code repositories - to identify vulnerabilities in project dependencies.
• Software installed in servers - to identify vulnerabilities in packages.
• Software packaged inside Docker container images - to identify vulnerabilities in packages.
• End-user devices - to identify vulnerabilities in third-party software.

Faceted search is a powerful feature that allows users to quickly refine their search results when browsing through the vast selection of items on Vinted. With almost 600 million unique items, the ability to narrow down search results is crucial for our users to find the perfect item quickly and efficiently. Additionally, faceted search allows for exploration, providing users with the ability to discover new items they may not have otherwise found. In this blog post, we’ll take a closer look at how we’ve implemented faceted search using Elasticsearch in Vinted.

On a daily basis, the automation tasks that engineers deal with are usually pretty straightforward; often something as simple as “send an email reminder after 2 days of inactivity”. These automations are very valuable. However, from time to time we’re faced with some really tough automation challenges.

Here’s one of them.

(Spoiler alert: Engineers had to work together with data scientists to solve it!)

The biggest challenge starting out in today’s job market is getting your foot in the door when you have limited experience. To help more people gain valuable ‘first job’ experience, we’re inviting future tech talents to apply for the Vinted Engineering Academy and kick-start their careers.

Vinted is Europe’s biggest pre-loved fashion marketplace, with over 75 million registered members in 18 countries. It runs on Ruby on Rails. Ruby was the perfect choice for building applications in a rapidly changing business environment. But Ruby has its limitations, both in language design and in finding talented developers experienced with this programming language. As Vinted grows, we know we have to expand our technology stack.

At Vinted the teams are structured in different ways. We have a unified organisational hierarchy across the company, but once we get to a team level, everything changes. Teams are free to organise themselves as they please. This is a story of how the Quality Assurance role in the Stardust team - the one responsible for the launch of the locker network - evolved.

My name is Mangirdas, I’m an Engineering Manager here at Vinted, and I’ve just participated in a miracle project. Our team (Stardust) coped with a big technical challenge to bring a new project to life: the Vinted Go lockers network.

What happened is remarkable. Vinted is already a big company by Lithuanian standards, with well over 1000 employees in Vilnius alone. It’s not a secret that the bigger the company, the harder it is to launch new products and move fast. However, in this case, our team, responsible for the engineering part of the project, had the complete freedom to make decisions and move as fast as a brand new startup. Until now, I’ve only read books about these ‘startup within a company’ projects. This time I got to experience it, and it was incredible.

Vinted was founded during the peak of Ruby popularity (2008). Choosing Ruby was as common and fashionable as Rust, NodeJS, or GoLang would be today. Many years have passed and we still stick with Ruby among other big companies like Airbnb, Shopify, GitHub, Instacart, Stripe, Twitch, Delivery Hero, StackShare, Accenture, Groupon. And here is why…

My name is Malachi and I work as a backend engineer as part of the shipping platform team in Berlin.

In this post I will explain my experience from the first 6 months of working at Vinted explaining how onboarding was like and how we work and collaborate.

At Vinted, our long-term strategy is to build an international marketplace where users can trade between countries.
Initially, we had separate legal entities, core service instances and databases to serve our users in the countries we operate in. We called them portals. For example, www.vinted.lt and www.kleiderkreisel.de were distinct portals.
To achieve our goal of building an international marketplace, we needed to merge them into a single entity. This led us to solve several challenges here at Vinted engineering:

• How to create a seamless flow to help our users migrate
• How to migrate the data between our portals

Migrating data between our portals was an interesting engineering challenge.
In this post, I will share some details on how we did it and what we have learned.

An old fact from 2014 December 12th was significant for Vinted: the company switched from Sphinx search engine to Elasticsearch 1.4.1. At the time of writing this post, we use Elasticsearch 7.15. Without a doubt, a lot has happened in between. This chapter will focus on Elasticsearch metrics, we will share our accumulated experiences from four generations of collecting metrics.

Elasticsearch is really fast out of the box, but it doesn’t know your data model or access patterns. It’s no secret that Elasticsearch powers our catalogue, but with the ever-increasing amount of listings and queries, we’ve had a number of problems to solve.

Managing DNS infrastructure at scale comes with its challenges.

Sometimes, when infrastructure expands, things that worked fine for years can start to become an inconvenience, or even blockers. It’s also possible that human error can cause service downtimes.

In this post, we will share with you our approach to automate DNS services inside datacenters, and the solution we took to eliminate the possibility of human error when configuring DNS services.

Vinted is growing – fast. To keep up with this increase in use, its Elasticsearch infrastructure constantly needs to be tuned. However, the most important requirement for any Elasticsearch performance tuning is to benchmark its production-like data. To collect this data, we’ve built a query logging solution that records Elasticsearch search requests at HTTP level and stores them to, well, Elasticsearch! Having a detailed query log not only allows us to tune the performance with the proper data, but it also allows us deeper visibility of our query traffic, which in turn means we can even perform speculative experiments.

Milda Mitkutė and Justas Janauskas have founded Vinted in 2008. Since then, all the software engineers were co-located in Vilnius: from our founder, CEO, and the first software engineer Justas to the more than 240 people working in engineering today. This year, 2021, we will expand our engineering team outside of Vilnius. We’ve already started hiring for engineering (and product, design, analytics) roles in Berlin. It’s a big and essential step. In this blog post, I will explain the why, what, and how of this move.

One of the key features of a good web service is user input validation. External input cannot be trusted and application must prevent malicious data from being processed.

It’s not only a matter of security but as well as for service usability and development experience. When something goes wrong, at the very least service should respond with 400 Bad Request, but good service will also respond with exact details of what went wrong.

In order not to invent its own error response format, a service should use RFC7807. RFC 7807 provides a standard format for returning problem details from HTTP APIs.

Managing Elasticsearch indices at scale isn’t easy.

Index settings were part of our core application monolith logic. Making the slightest change to them meant that the whole application had to be redeployed, lots of unrelated tests had to run. Also, it was built to be only aware of a single language. We needed to be more flexible and have fine-grained control over our Elasticsearch clusters.

At the same time, we were planning to break our single Elasticsearch cluster into multiple ones for better load distribution and the ability to upgrade smaller clusters. Managing all of that from a single application was difficult. See how we managed to manage search traffic across multiple Elasticsearch clusters in second chapter of these series.

We’ve decided to decouple search setup and ingestion logic into a separate application, very obviously called Elasticsearch Index Management, or EIM for short. It would turn out to be a set of CLI utilities, automated jobs, and webhook listeners to automate our routine work. I’ll share how we solved these problems and what we learned.

It began with Elasticsearch cluster upgrades. Every major Elasticsearch version brings new features, free performance, compatibility issues, and uncertainty. In addition, we had an itch to dissect requests, conjoin request with a response for analysis, record queries for stress testing, and account for each request per second. With these requirements in mind, we have built a sidecar component next to primary application.

The leftmost component is the core application that is not aware that search requests are being handled by multiple Elasticsearch clusters. Next component is sidecar service written in Rust programming language, we call srouter.

Data ingestion into Elasticsearch at scale is hard. In this post, I’m going to share a success story of leveraging the Kafka Connect Elasticsearch Sink Connector to support the ever-increasing usage of the Vinted platform and to help our search engineers to ship features quickly all while having an operational and reliable system.

Logs flow until they don’t. One year ago our centralised log management (LM) system became quite unstable. This often caused LM service outages and human intervention was required, sometimes several times a week.
In this post I’m going to give an overview of the improvements made to the log management system over the course of one year. As a result of the improvements, we now have quite a simple LM system design which is fault-resistant and easy to scale.

Around one year ago I was considering applying for an open position as a Backend Engineer at Vinted. At the time I was still in my first ever IT job, and had very little experience with job interviews. So I decided to do some research. I couldn’t find out what the Vinted hiring process looked like, but I did come across a Hacker News post featuring System Design Interview tips - and luckily discovered that the github account also had tips for coding interviews. The latter mentioned algorithms (which I wasn’t so good with) so I found a course on Coursera and decided to go through it before. At this point you can probably tell that my self-doubt was spiralling. Thankfully, after a few weekends in the library I realised that I may have been going a bit overboard with prep for just daring to send over a CV.

Fast forward to today, and I’m now a proud member of Vinted’s Engineering team. And I’ve since even had the chance to participate in the other side of the hiring process! As I’ve experienced first-hand the fear of the unknown when it comes to interviewing, I decided to share how the Engineer Hiring @ Vinted actually works. I’m going to focus on the Backend Engineer hiring process, but the general steps (intro meeting, homework assignment, technical interview, try-out) are the same for all candidates. Now let’s go into more detail on each of these steps.

During the last year, Vinted SRE team reimplemented the whole data center network starting with new equipment, finishing with the topology and architecture change. The project had its ups and downs: things that seemed complex at the start became trivial with a change in point of view, and the seemingly trivial tasks revealed unanticipated, hidden complexities. We hope it is an interesting story to read.

TL,DR: We migrated from several large plain subnets spanning three physical locations to Clos topology with eBGP for routing and AS per rack with anycast service addresses and EVPN for multi tenancy. We deployed with Ansible and conducted integration testing with Kitchen-InSpec.

And here comes the longer version.

After being a PHP developer for 10+ years, I’ve switched programming languages twice in the past two years. During all of my years of web development, I assumed I knew many programming languages. I used JavaScript relatively often. I knew jQuery (it was cool at the time). Obviously, Java, C++, and Pascal were always in my portfolio.

I always assumed I had a broad view and was absolutely fine…

Every day Vinted gets a few billion events from our customers via mobile phones or web clients. Every time a user clicks somewhere, views an item, views another user’s profile, buys something, leaves feedback, etc., a JSON representation of that action is sent. And it’s a long time (at the time of writing, about an hour) before that same JSON from plain text arrives in our hard drives, compressed and available for our analysts. Only then can we start querying and drawing more insights from this valuable information.

This blog post is about how we approach Engineering at Vinted. It’s a high-level overview, which describes how we think and what principles we apply. By design, it doesn’t go into details, so we would love to hear from readers of this blog on social media. Which parts have you found the most interesting? What would you like us to expand on in the future blog posts?

Most Android developers use SharedPreferences. It’s a great way of storing something simple across app screens or even sessions. As this simple tool may be perfect for many app developers, it may not be that handy and powerful to use when the app becomes more complex. Therefore, at Vinted, our SharedPreferences evolved into something we really liked and deemed worth sharing with the whole community - PrefeRx - a reactive SharedPreferences library for Kotlin. It allows not only to manage primitives, enums and other serializable objects with ease, but adds all the RxJava goodness. Let’s go through some examples of what this library can actually do for you.

Managing physical hardware is a diligent process of bookkeeping. Tracking everything from an email message to added/removed hardware components was tedious, for each request to the provider, one had to not to forget to fill what was changed and leave remarks.

Yeah, every Android developer knows what it means. And it doesn’t stop there. Slow builds drain the battery very fast and fingers suffer from burning keyboard keys. Even colleagues around are distracted by your noisy CPU fan. We are dreaming off better hardware during a build. Even the best MacBook still stuffed into 4C/8T CPU + 16 GB RAM. And Xeon based laptops hardly fit into backpacks. What if we could move the build process to a remote machine?

There are many recipes describing how much resources to allocate for Elasticsearch, but there are none about using non-uniform memory access^[1]. This post is about leveraging NUMA^[2] for efficient resource usage.

Finding the right settings of indices is a relentless process of monitoring and experimenting. I want to share key methods of detecting scalability problems of Elasticsearch using metrics and automation.

We use Elasticsearch for ranking product listings and full-text search for products, forum posts and user messages. Most complex queries include scoring functions, various filters and weight boosters. Before Elasticsearch we have used Sphinx, but due to demand for more dynamic querying logic we migrated to ES 1.7.x, now 5.4.0.

I meet a lot of new people and talk Vinted to them. Over a few weeks I realized there are many things known to insiders, which can easily make an outsider raise their eyebrows. I thought I’ll share some of them with you, our readers.

Vinted runs 11 portals in 10 countries - Lithuania, Germany, France, USA, Poland, Czech Republic, Spain, United Kingdom, Austria and the Netherlands. We serve ~ 16 million users.

Everyone in Vinted top management - Justas Janauskas, Mantas Mikuckas and Thomas Plantenga - has an engineering background.

Vinted has a fully equipped professional kitchen in its Vilnius office where most of us have lunch every day. You can eat lunch with the CEO, ask our CFO about economic trends or learn why aggressive marketing works for us with our Head of Marketing pretty much any day.

At Google I/O ‘17 the Android team announced first-class support for Kotlin. Since then, the language has received a lot more interest from Android developers.

This is great news for Android developers and the whole Kotlin community! Many developers are trying out Kotlin, reading up on best practices or figuring out how to introduce it in their projects.

While we at Vinted have been using it in production since September 2015!

One with a backbone will do more than a thousand with a mere wishbone.

Usually, having a backbone is a good thing.

At Vinted we started using Backbone.js, a framework to help build web user interfaces, more than 5 years ago, in late 2011. The JavaScript framework map looked completely different. “Backbone vs. Knockout vs. JavascriptMVC” was a hot discussion back then. Who even remembers JavascriptMVC now? 😅

Backbone.js has served us well over the years. We’ve built many great things with it. I found more than 800 instances of the word “Backbone” in our JavaScript, representing a bunch of views and models we created.

Two years ago I had an interesting challenge: explain to a junior developer why he should prefer interactor pattern instead of Rails Active Record Callbacks.

Arguing against callbacks is easy - thanks to the community there is a lot of experience shared about their drawbacks: callbacks pain, 7 Patterns to Refactor Fat ActiveRecord Models.

The second part is not that easy - explain to a developer why do I believe that interactor is a better alternative than, e.g. the commonly referred-to service object?

Three months ago I started leading Vinted’s new SRE (Site Reliability Engineering) team. What’s new about it you ask? Well, the previous lead left and the team was merged with two developers who work on our data warehouse - which now became SRE responsibility. Our head of engineering put his trust in me - a backend engineer in Vinted at the time - to lead this new team.

The first thing we got to do was to set OKR’s for our next quarter. Then we set to work.

Three months have passed.

I’m sharing an outline of the speech I gave in our off-site in Labanoras. We went there to reflect on the past and plan for the future.

Data warehouses are typically assembled from a variety of data sources with different formats and purposes. As such, ETL (Extract, Transform and Load) is a key process to bring all the data together into a standard, homogeneous environment. The ETL process is composed of:

• Extraction – data is extracted from its sources
• Transformation – data is transformed for storage in the proper format or structure, for querying and analysis
• Loading – it is loaded into the final target (i.e. database)

Managing big ETL process might be a challenge because of:

• overall and ever-growing complexity
• transformation dependency management hell (when final transformations depend on other transformations)
• inaccurate, outdated or missing ETL process documentation
• a lack of regression tests that can prevent unexpected errors after a framework upgrade or unintentional changes in seemingly unrelated areas

and many more. There is a simple solution: unit/integration tests.

New platforms have and will keep bubbling up - be it smartphones, tablets, smartwatches or embedded devices. Both startups and established organisations encounter a need to enter and support them. We went through different stages of transformation to support multiple platforms. This story can be useful in understanding how to best do that.

The PolyConf conference took place in the end of June. It unites polyglot programmers focusing on new as well as functional programming languages.

This is my overview of its most interesting ideas.

Sublime Text is amazing editor. It shines at its speed, file search and multiple cursors. If some functionality is missing, you can easily extend it via plugins.

I have written TestRSpec plugin for running RSpec specs. Give it a try and provide me feedback via GitHub issues.

The React Europe conference took place in the beginning of June. It unites people who shape or are interested in the future of web as well as mobile applications.

Let me share the most interesting ideas from the conference.

On the 26th of May, some of us took a day off from our everyday work and went to learn new things at Devops Pro Vilnius. Here are some things we found interesting.

Last Friday marked the end of my journey as an iOS developer intern at Vinted.

It all started with my future mentor Mindaugas teaching an introductory iOS course at Vilnius University. Developing iOS apps has been a dream of mine, which is why I signed up for the additional course. My interest in Vinted was sparked by Mindaugas, who taught his class using real examples from the app they were building. One day, mid-semester, I approached Mindaugas with a question, ‘Maybe you are looking for a guy who wants to become an iOS dev?’

At Vinted, we have Guilds. No, not the World of Warcraft type - if this was based on computer games, I’d push, oh, so hard for us to have factions (go figure). The Spotify type. At Vinted, a Guild unifies people from different product teams who own and develop a certain area - backend, iOS, analysts, design - to name a few.

Every other thursday we have a Guild day. That’s a whole day dedicated to Guild efforts. People in a Guild conceive their own initiatives, form teams, make a backlog of tasks and work away.

Quite predictably, the iOS Guild will be working on something new and cool, like Vinted for Apple TV, the design Guild might be working on something they call “posh feed”, while we in the backend Guild work mostly on making our life easier via automation of reoccurring tasks and give engineering love to parts of our backend system that no team can reasonably own. Say, the background jobs.

We rely on background jobs for a lot of things - from sending CRM emails and push notifications, to processing transactions. In all our portals combined, we process hundreds of jobs per second under normal circumstances and thousands if we have an intensive CRM campaign going on.

It’s a critical piece of our infrastructure. I was quite surprised when on my first weeks at Vinted I took at look at our Resque admin panel and asked my fellow backenders:

What happens when your code reaches the end-user just once in a while?

Imagine. It’s a late Friday evening. A developer (still on his 3 month trial period) just finished manually deploying changes to production. He’s alone in the office, with everyone gone for the weekend. Checks the home page. Everything seems fine. Opens up Hacker News, skims a couple of articles and gets ready to head home. But before this developer goes out the door, “All Along the Watchtower” starts playing. That’s his phone’s ringtone and it’s the COO calling.

Big Data Strategy 2015 conference took place in Vilnius on October 5th, 2015. Vinted engineers gave two talks about our Hadoop -based data warehouse, metric computation, and lessons learnt while implementing it.

It was the last day of my internship as a front-end developer. Or, more like two days after the last day of my internship. The office in Žirmūnai street was in a festivity mode. At Vinted, celebrating colleagues’ birthdays is as typical as having a cup of coffee in the morning. Such simple yet profound traditions makes everyone feel valuable here at one of the most successful Lithuanian startups.

The post was written by Elena Lapinskaitė after her internship at Vinted.

When you’re 18, there are so many ‘firsts’ to experience. Your first driving ticket, your first holiday abroad on your own (and getting lost in Warsaw in the middle of the night with a dead phone, fun), your first morning not remembering what happened the night before, your first diagnosis of arthritis and first dentures. And your first proper work experience.

At Vinted, we use a data structures server Redis for many things including Resque, news feed, application, etc. We are not able to restart or upgrade Redis instances without having zero downtime. High availability is critical for us. Therefore, we decided to try database services like Redis Sentinel or Redis Cluster.

The engineers at Vinted began the year working on the improvement of the caching layer. We managed to significantly improve the performance of our Rails application and substantially reduce the number of database queries. But this additional load to our caching layer brought up some new challenges.

At Vinted, new code is deployed to production machines all the time with zero downtime for members. Deployment changesets are small. So we can move fast and fearlessly.

To keep the speed of development from decreasing we have an extensive test suite for easy refactoring.

But in cases where it’s difficult to reproduce a complete dataset from a production environment there are other approaches to testing changes safely.

Git is one of the most wonderful tools developers can use to make their lives better. As Linus himself put it best: “If you actually like using CVS, you shouldn’t be here. You should be in some mental institution.”

However, Git is often underused and abused with badly written messages, junk commits and weird branching. You should not treat Git, or any other SCM tool for that matter, as a way to “Save” your code, just like you would save your progress in a game.