Vinted Engineering

As containerisation continues to revolutionize the way software applications are developed and deployed, ensuring the security of container environments has become an utmost priority. Security in a containerised environment requires a different approach than traditional security mechanisms. It involves continuously monitoring container activities, identifying security threats, and ensuring compliance.

In this blog post, we want to share our experience with Falco, an open-source tool designed specifically for securing containerised environments.

Empowering Kubernetes runtime security

Falco, originally developed by Sysdig, is an open-source runtime security tool purpose-built for Kubernetes. Leveraging kernel instrumentation, Falco monitors system calls and events within containers and provides real-time insights into container activity.

The main factors why we chose Falco to enhance Vinted Kubernetes security are:

• Container native approach
• Real-time threat detection through rule-based monitoring
• Robust community support
• Extensibility

Falco deployment in Vinted

Within Vinted’s Kubernetes clusters, Falco is deployed as a DaemonSet, utilizing the official Falco Helm Charts. This ensures Falco operates within containers across every Kubernetes node. Additionally, to enhance operational efficiency, ensure a contemporary security posture, and mitigate dependencies on external connections, we have implemented a mechanism that retrieves Falco Helm charts to our internal Harbor Helm charts repository. Deployments management is executed through ArgoCD, adhering to GitOps best practices.

Instead of utilising the conventional approach of building, compiling, and maintaining Falco probe drivers, we opted for Falco’s modern BPF (Berkeley Packet Filter) probe. This decision has streamlined the instrumentation of our Kubernetes clusters for enhanced runtime security monitoring. It is imperative, however, to acknowledge that harnessing BPF’s capabilities demands specific requirements. To maximise the potential of this advanced probe, the custom Linux kernel must be eBPF-compatible and compiled with BTF (BPF Type Format) support.

Rules update mechanism

Our approach to distributing our custom Falco security rules is complex and highly effective. There’s a lot of moving parts. At its core, we have an index file placed in a private AWS S3 bucket, which acts as a guide for Falco, directing it to the specific location where it can access our security rules. These rules are securely stored within the internal Harbor OCI (Open Container Initiative) repository.

The effectiveness of this system unfolds when updates are required. Whenever rules, maintained in a private Git repository, undergo modifications, our continuous integration (CI) pipeline takes charge. First of all, it mounts rules and runs validation tests:

rules-validity-check:
    image: falcosecurity/falco-no-driver:0.35.1
    volumes:
    - ./rules:/etc/falco/rules
    command: /usr/bin/falco -o load_plugins[0]=k8saudit -o load_plugins[1]=json --validate /etc/falco/rules/vinted_rules.yaml

After successful validation, CI infrastructure encapsulates the custom rules in the OCI format and pushes them to the internal Harbor OCI repository:

rules-registry-push-oci:
image: falcosecurity/rules-registry:v0.1.0-1-876e81b
environment:
    - REGISTRY_USER
    - REGISTRY_TOKEN
    - GITHUB_REPO_URL
    - OCI_REPO_PREFIX
volumes:
    - .:/etc/falco
    - ./rules:/app/rules
command: /app/rules-registry push-to-oci /etc/falco/registry.yaml $OCI_TAG

To accommodate Falco security rules in the OCI package, we have developed a custom container image with Falco’s rules-registry CLI.

Furthermore, while Falco scans for rule updates hourly, within this interval, all Falco pods autonomously retrieve the revised rules. This ensures that our security policies remain consistently up-to-date.

Integration with security incident management process

The integration of Falco with Vinted’s security incident management process involved leveraging Falco’s runtime security capabilities to monitor, detect, and alert on anomalous behaviour within Vinted Kubernetes infrastructure. By integrating Falco, we can automatically detect and respond to security threats in real-time, ensuring the safety and integrity of our systems.

It is not just about detecting threats using Falco but also about promptly notifying the Security Engineering team when something suspicious occurs. When Falco detects a breach of its rules, it generates an alert, which Falcosidekick delivers to the security incident management platform. This integration creates a dynamic and efficient feedback loop, ensuring the security team is aware of potential threats and equipped to respond swiftly and effectively.