Vinted Engineering

An old fact from 2014 December 12th was significant for Vinted: the company switched from Sphinx search engine to Elasticsearch 1.4.1. At the time of writing this post, we use Elasticsearch 7.15. Without a doubt, a lot has happened in between. This chapter will focus on Elasticsearch metrics, we will share our accumulated experiences from four generations of collecting metrics.

Elasticsearch is really fast out of the box, but it doesn’t know your data model or access patterns. It’s no secret that Elasticsearch powers our catalogue, but with the ever-increasing amount of listings and queries, we’ve had a number of problems to solve.

Managing DNS infrastructure at scale comes with its challenges.

Sometimes, when infrastructure expands, things that worked fine for years can start to become an inconvenience, or even blockers. It’s also possible that human error can cause service downtimes.

In this post, we will share with you our approach to automate DNS services inside datacenters, and the solution we took to eliminate the possibility of human error when configuring DNS services.

Vinted is growing – fast. To keep up with this increase in use, its Elasticsearch infrastructure constantly needs to be tuned. However, the most important requirement for any Elasticsearch performance tuning is to benchmark its production-like data. To collect this data, we’ve built a query logging solution that records Elasticsearch search requests at HTTP level and stores them to, well, Elasticsearch! Having a detailed query log not only allows us to tune the performance with the proper data, but it also allows us deeper visibility of our query traffic, which in turn means we can even perform speculative experiments.

Milda Mitkutė and Justas Janauskas have founded Vinted in 2008. Since then, all the software engineers were co-located in Vilnius: from our founder, CEO, and the first software engineer Justas to the more than 240 people working in engineering today. This year, 2021, we will expand our engineering team outside of Vilnius. We’ve already started hiring for engineering (and product, design, analytics) roles in Berlin. It’s a big and essential step. In this blog post, I will explain the why, what, and how of this move.

One of the key features of a good web service is user input validation. External input cannot be trusted and application must prevent malicious data from being processed.

It’s not only a matter of security but as well as for service usability and development experience. When something goes wrong, at the very least service should respond with 400 Bad Request, but good service will also respond with exact details of what went wrong.

In order not to invent its own error response format, a service should use RFC7807. RFC 7807 provides a standard format for returning problem details from HTTP APIs.

Managing Elasticsearch indices at scale isn’t easy.

Index settings were part of our core application monolith logic. Making the slightest change to them meant that the whole application had to be redeployed, lots of unrelated tests had to run. Also, it was built to be only aware of a single language. We needed to be more flexible and have fine-grained control over our Elasticsearch clusters.

At the same time, we were planning to break our single Elasticsearch cluster into multiple ones for better load distribution and the ability to upgrade smaller clusters. Managing all of that from a single application was difficult. See how we managed to manage search traffic across multiple Elasticsearch clusters in second chapter of these series.

We’ve decided to decouple search setup and ingestion logic into a separate application, very obviously called Elasticsearch Index Management, or EIM for short. It would turn out to be a set of CLI utilities, automated jobs, and webhook listeners to automate our routine work. I’ll share how we solved these problems and what we learned.

It began with Elasticsearch cluster upgrades. Every major Elasticsearch version brings new features, free performance, compatibility issues, and uncertainty. In addition, we had an itch to dissect requests, conjoin request with a response for analysis, record queries for stress testing, and account for each request per second. With these requirements in mind, we have built a sidecar component next to primary application.

The leftmost component is the core application that is not aware that search requests are being handled by multiple Elasticsearch clusters. Next component is sidecar service written in Rust programming language, we call srouter.

Data ingestion into Elasticsearch at scale is hard. In this post, I’m going to share a success story of leveraging the Kafka Connect Elasticsearch Sink Connector to support the ever-increasing usage of the Vinted platform and to help our search engineers to ship features quickly all while having an operational and reliable system.

Logs flow until they don’t. One year ago our centralised log management (LM) system became quite unstable. This often caused LM service outages and human intervention was required, sometimes several times a week.
In this post I’m going to give an overview of the improvements made to the log management system over the course of one year. As a result of the improvements, we now have quite a simple LM system design which is fault-resistant and easy to scale.

Around one year ago I was considering applying for an open position as a Backend Engineer at Vinted. At the time I was still in my first ever IT job, and had very little experience with job interviews. So I decided to do some research. I couldn’t find out what the Vinted hiring process looked like, but I did come across a Hacker News post featuring System Design Interview tips - and luckily discovered that the github account also had tips for coding interviews. The latter mentioned algorithms (which I wasn’t so good with) so I found a course on Coursera and decided to go through it before. At this point you can probably tell that my self-doubt was spiralling. Thankfully, after a few weekends in the library I realised that I may have been going a bit overboard with prep for just daring to send over a CV.

Fast forward to today, and I’m now a proud member of Vinted’s Engineering team. And I’ve since even had the chance to participate in the other side of the hiring process! As I’ve experienced first-hand the fear of the unknown when it comes to interviewing, I decided to share how the Engineer Hiring @ Vinted actually works. I’m going to focus on the Backend Engineer hiring process, but the general steps (intro meeting, homework assignment, technical interview, try-out) are the same for all candidates. Now let’s go into more detail on each of these steps.

During the last year, Vinted SRE team reimplemented the whole data center network starting with new equipment, finishing with the topology and architecture change. The project had its ups and downs: things that seemed complex at the start became trivial with a change in point of view, and the seemingly trivial tasks revealed unanticipated, hidden complexities. We hope it is an interesting story to read.

TL,DR: We migrated from several large plain subnets spanning three physical locations to Clos topology with eBGP for routing and AS per rack with anycast service addresses and EVPN for multi tenancy. We deployed with Ansible and conducted integration testing with Kitchen-InSpec.

And here comes the longer version.

After being a PHP developer for 10+ years, I’ve switched programming languages twice in the past two years. During all of my years of web development, I assumed I knew many programming languages. I used JavaScript relatively often. I knew jQuery (it was cool at the time). Obviously, Java, C++, and Pascal were always in my portfolio.

I always assumed I had a broad view and was absolutely fine…

Every day Vinted gets a few billion events from our customers via mobile phones or web clients. Every time a user clicks somewhere, views an item, views another user’s profile, buys something, leaves feedback, etc., a JSON representation of that action is sent. And it’s a long time (at the time of writing, about an hour) before that same JSON from plain text arrives in our hard drives, compressed and available for our analysts. Only then can we start querying and drawing more insights from this valuable information.

This blog post is about how we approach Engineering at Vinted. It’s a high-level overview, which describes how we think and what principles we apply. By design, it doesn’t go into details, so we would love to hear from readers of this blog on social media. Which parts have you found the most interesting? What would you like us to expand on in the future blog posts?

Most Android developers use SharedPreferences. It’s a great way of storing something simple across app screens or even sessions. As this simple tool may be perfect for many app developers, it may not be that handy and powerful to use when the app becomes more complex. Therefore, at Vinted, our SharedPreferences evolved into something we really liked and deemed worth sharing with the whole community - PrefeRx - a reactive SharedPreferences library for Kotlin. It allows not only to manage primitives, enums and other serializable objects with ease, but adds all the RxJava goodness. Let’s go through some examples of what this library can actually do for you.

Managing physical hardware is a diligent process of bookkeeping. Tracking everything from an email message to added/removed hardware components was tedious, for each request to the provider, one had to not to forget to fill what was changed and leave remarks.

Yeah, every Android developer knows what it means. And it doesn’t stop there. Slow builds drain the battery very fast and fingers suffer from burning keyboard keys. Even colleagues around are distracted by your noisy CPU fan. We are dreaming off better hardware during a build. Even the best MacBook still stuffed into 4C/8T CPU + 16 GB RAM. And Xeon based laptops hardly fit into backpacks. What if we could move the build process to a remote machine?

There are many recipes describing how much resources to allocate for Elasticsearch, but there are none about using non-uniform memory access^[1]. This post is about leveraging NUMA^[2] for efficient resource usage.

Finding the right settings of indices is a relentless process of monitoring and experimenting. I want to share key methods of detecting scalability problems of Elasticsearch using metrics and automation.

We use Elasticsearch for ranking product listings and full-text search for products, forum posts and user messages. Most complex queries include scoring functions, various filters and weight boosters. Before Elasticsearch we have used Sphinx, but due to demand for more dynamic querying logic we migrated to ES 1.7.x, now 5.4.0.

I meet a lot of new people and talk Vinted to them. Over a few weeks I realized there are many things known to insiders, which can easily make an outsider raise their eyebrows. I thought I’ll share some of them with you, our readers.

Vinted runs 11 portals in 10 countries - Lithuania, Germany, France, USA, Poland, Czech Republic, Spain, United Kingdom, Austria and the Netherlands. We serve ~ 16 million users.

Everyone in Vinted top management - Justas Janauskas, Mantas Mikuckas and Thomas Plantenga - has an engineering background.

Vinted has a fully equipped professional kitchen in its Vilnius office where most of us have lunch every day. You can eat lunch with the CEO, ask our CFO about economic trends or learn why aggressive marketing works for us with our Head of Marketing pretty much any day.

At Google I/O ‘17 the Android team announced first-class support for Kotlin. Since then, the language has received a lot more interest from Android developers.

This is great news for Android developers and the whole Kotlin community! Many developers are trying out Kotlin, reading up on best practices or figuring out how to introduce it in their projects.

While we at Vinted have been using it in production since September 2015!

One with a backbone will do more than a thousand with a mere wishbone.

Usually, having a backbone is a good thing.

At Vinted we started using Backbone.js, a framework to help build web user interfaces, more than 5 years ago, in late 2011. The JavaScript framework map looked completely different. “Backbone vs. Knockout vs. JavascriptMVC” was a hot discussion back then. Who even remembers JavascriptMVC now? 😅

Backbone.js has served us well over the years. We’ve built many great things with it. I found more than 800 instances of the word “Backbone” in our JavaScript, representing a bunch of views and models we created.

Two years ago I had an interesting challenge: explain to a junior developer why he should prefer interactor pattern instead of Rails Active Record Callbacks.

Arguing against callbacks is easy - thanks to the community there is a lot of experience shared about their drawbacks: callbacks pain, 7 Patterns to Refactor Fat ActiveRecord Models.

The second part is not that easy - explain to a developer why do I believe that interactor is a better alternative than, e.g. the commonly referred-to service object?

Three months ago I started leading Vinted’s new SRE (Site Reliability Engineering) team. What’s new about it you ask? Well, the previous lead left and the team was merged with two developers who work on our data warehouse - which now became SRE responsibility. Our head of engineering put his trust in me - a backend engineer in Vinted at the time - to lead this new team.

The first thing we got to do was to set OKR’s for our next quarter. Then we set to work.

Three months have passed.

I’m sharing an outline of the speech I gave in our off-site in Labanoras. We went there to reflect on the past and plan for the future.

Data warehouses are typically assembled from a variety of data sources with different formats and purposes. As such, ETL (Extract, Transform and Load) is a key process to bring all the data together into a standard, homogeneous environment. The ETL process is composed of:

• Extraction – data is extracted from its sources
• Transformation – data is transformed for storage in the proper format or structure, for querying and analysis
• Loading – it is loaded into the final target (i.e. database)

Managing big ETL process might be a challenge because of:

• overall and ever-growing complexity
• transformation dependency management hell (when final transformations depend on other transformations)
• inaccurate, outdated or missing ETL process documentation
• a lack of regression tests that can prevent unexpected errors after a framework upgrade or unintentional changes in seemingly unrelated areas

and many more. There is a simple solution: unit/integration tests.

New platforms have and will keep bubbling up - be it smartphones, tablets, smartwatches or embedded devices. Both startups and established organisations encounter a need to enter and support them. We went through different stages of transformation to support multiple platforms. This story can be useful in understanding how to best do that.

The PolyConf conference took place in the end of June. It unites polyglot programmers focusing on new as well as functional programming languages.

This is my overview of its most interesting ideas.

Sublime Text is amazing editor. It shines at its speed, file search and multiple cursors. If some functionality is missing, you can easily extend it via plugins.

I have written TestRSpec plugin for running RSpec specs. Give it a try and provide me feedback via GitHub issues.

The React Europe conference took place in the beginning of June. It unites people who shape or are interested in the future of web as well as mobile applications.

Let me share the most interesting ideas from the conference.

On the 26th of May, some of us took a day off from our everyday work and went to learn new things at Devops Pro Vilnius. Here are some things we found interesting.

Last Friday marked the end of my journey as an iOS developer intern at Vinted.

It all started with my future mentor Mindaugas teaching an introductory iOS course at Vilnius University. Developing iOS apps has been a dream of mine, which is why I signed up for the additional course. My interest in Vinted was sparked by Mindaugas, who taught his class using real examples from the app they were building. One day, mid-semester, I approached Mindaugas with a question, ‘Maybe you are looking for a guy who wants to become an iOS dev?’

At Vinted, we have Guilds. No, not the World of Warcraft type - if this was based on computer games, I’d push, oh, so hard for us to have factions (go figure). The Spotify type. At Vinted, a Guild unifies people from different product teams who own and develop a certain area - backend, iOS, analysts, design - to name a few.

Every other thursday we have a Guild day. That’s a whole day dedicated to Guild efforts. People in a Guild conceive their own initiatives, form teams, make a backlog of tasks and work away.

Quite predictably, the iOS Guild will be working on something new and cool, like Vinted for Apple TV, the design Guild might be working on something they call “posh feed”, while we in the backend Guild work mostly on making our life easier via automation of reoccurring tasks and give engineering love to parts of our backend system that no team can reasonably own. Say, the background jobs.

We rely on background jobs for a lot of things - from sending CRM emails and push notifications, to processing transactions. In all our portals combined, we process hundreds of jobs per second under normal circumstances and thousands if we have an intensive CRM campaign going on.

It’s a critical piece of our infrastructure. I was quite surprised when on my first weeks at Vinted I took at look at our Resque admin panel and asked my fellow backenders:

What happens when your code reaches the end-user just once in a while?

Imagine. It’s a late Friday evening. A developer (still on his 3 month trial period) just finished manually deploying changes to production. He’s alone in the office, with everyone gone for the weekend. Checks the home page. Everything seems fine. Opens up Hacker News, skims a couple of articles and gets ready to head home. But before this developer goes out the door, “All Along the Watchtower” starts playing. That’s his phone’s ringtone and it’s the COO calling.

Big Data Strategy 2015 conference took place in Vilnius on October 5th, 2015. Vinted engineers gave two talks about our Hadoop -based data warehouse, metric computation, and lessons learnt while implementing it.

It was the last day of my internship as a front-end developer. Or, more like two days after the last day of my internship. The office in Žirmūnai street was in a festivity mode. At Vinted, celebrating colleagues’ birthdays is as typical as having a cup of coffee in the morning. Such simple yet profound traditions makes everyone feel valuable here at one of the most successful Lithuanian startups.

The post was written by Elena Lapinskaitė after her internship at Vinted.

When you’re 18, there are so many ‘firsts’ to experience. Your first driving ticket, your first holiday abroad on your own (and getting lost in Warsaw in the middle of the night with a dead phone, fun), your first morning not remembering what happened the night before, your first diagnosis of arthritis and first dentures. And your first proper work experience.

At Vinted, we use a data structures server Redis for many things including Resque, news feed, application, etc. We are not able to restart or upgrade Redis instances without having zero downtime. High availability is critical for us. Therefore, we decided to try database services like Redis Sentinel or Redis Cluster.

The engineers at Vinted began the year working on the improvement of the caching layer. We managed to significantly improve the performance of our Rails application and substantially reduce the number of database queries. But this additional load to our caching layer brought up some new challenges.

At Vinted, new code is deployed to production machines all the time with zero downtime for members. Deployment changesets are small. So we can move fast and fearlessly.

To keep the speed of development from decreasing we have an extensive test suite for easy refactoring.

But in cases where it’s difficult to reproduce a complete dataset from a production environment there are other approaches to testing changes safely.

Git is one of the most wonderful tools developers can use to make their lives better. As Linus himself put it best: “If you actually like using CVS, you shouldn’t be here. You should be in some mental institution.”

However, Git is often underused and abused with badly written messages, junk commits and weird branching. You should not treat Git, or any other SCM tool for that matter, as a way to “Save” your code, just like you would save your progress in a game.