Secure URL Project

I designed and implemented a distributed system architecture optimized for large object transfers in B2B data exchange scenarios. At the heart of this system is a robust Apache Kafka-based messaging backbone, which I configured to handle real-time streaming and high-throughput data transfer workloads. By leveraging Kafka’s inherent scalability, fault tolerance, and durability, I enabled secure and efficient file exchange across business boundaries. I also engineered a dedicated internal network layer ("kafkanet") to facilitate seamless broker communication and data replication, ensuring reliable and performant transmission of large data payloads.

To support this architecture, I integrated a suite of backend services, including a PostgreSQL instance for managing metadata and transactional information associated with each transfer. I configured pgAdmin to serve as the operational interface for managing database activity. All components were containerized and deployed via Docker Compose, a setup I maintained to ensure consistent local development and reproducibility. For production-grade orchestration, I introduced and configured Kubernetes, enabling dynamic scaling across nodes to meet variable demand from multiple concurrent B2B clients. This ensured reliable handling of simultaneous large object transfers across the system.

I applied Terraform to manage infrastructure as code, allowing for repeatable, version-controlled environment builds that support both DevOps automation and disaster recovery. I also incorporated Prometheus and Grafana for real-time observability and alerting, giving stakeholders visibility into system health and performance. To extend system capabilities, I developed and integrated interfaces using HTTP (urlhttp) and SMS APIs (surlsms) to enable flexible external triggering and confirmation of transfers. These integrations improved usability and enabled smooth partner onboarding. The result is a secure, scalable, and resilient platform tailored for high-volume, high-value B2B data exchanges.