The economics come before the code: architecting a real-time pipeline