This is the story of how we built a highly available data pipeline that processes terabytes of network data every day, making it available to security researchers for assessment and threat hunting.
Building this kind of stuff in the cloud is not that complicated, but if you have to make it near real-time, fault tolerant and 24/7 available, well… that’s another story.
In this talk, we will tell you how we achieved this ambitious goal and how we missed a few good nights of sleep while trying to do that!
Spoiler alert: contains AWS, serverless, elastic search, monitoring, alerting & more!