Handling large-scale audio/video file transcription in real-time requires a resilient architecture that doesn't block the main application thread. Multi-tenant data isolation and secure storage were critical requirements.
Implemented a NestJS backend with Redis-backed BullMQ for background jobs. Used Amazon S3 for secure file handling and implemented a custom authentication layer for tenant isolation.
Chose a queue-based architecture to decouple ingestion from processing, ensuring high availability even during traffic spikes. Relied on structured data extraction to provide immediate value to downstream systems.
Optimizing memory usage during heavy processing and managing transient network failures during long-running tasks.