I’ve been trying to figure out whether moving our data engineering workflows to a public cloud makes sense or if we should keep investing in our private setup. Right now, we’re hitting capacity limits on storage and compute, especially with real-time ETL jobs. Public cloud seems easier to scale, but I’m worried about security and unpredictable costs. Anyone been through this transition and can share what worked (or didn’t) for them?
We actually faced the same decision earlier this year. After months of testing, we ended up adopting a hybrid approach — critical workloads stay on our private cloud for compliance reasons, while burst processing happens on public cloud nodes. It took some work to get the pipelines communicating smoothly, but the flexibility has been worth it. What really helped us was getting external guidance; the folks at devops consulting had some excellent resources and consultations on hybrid infrastructure and data scaling strategies. They’re very practical about balancing cost efficiency with reliability. My advice: start small, run stress tests on your most demanding workflows, and compare performance metrics before deciding where to invest long term.
You must be logged in to reply to this topic.