I’ve exhausted Google searches and we’re doing our own hands-on evaluations on this topic. But wanted to see this channel’s thoughts on BigQuery versus Snowflake as a DW. Our data volume is relatively small and we have <10 users on it. We just need a serverless DW to make our data ops as efficient as possible.
3 Likes
My opinion:
Since you have a small amount of data, it might not be worth it to set up a full DW like BigQuery. Even though they are “serverless”, your team will still spend a lot of time working to get the datasets you need into them.
It would be recommended to setting up a small Postgres database with something like RDS or Google Cloud SQL. They will host the instances for you, it’s easier to get set up, easier to get the data into, and there’s much more existing tooling for working with Postgres than Snowflake or BigQuery.
Most of the companies I work with start with Postgres and eventually mature to a real data warehouse.
3 Likes
We are in some sense shooting from the hip here, not knowing what your data stack is like or its level of maturity. With that said, any data pipelining tools you have that are off the shelf (e.g. Fivetran) should well support input and output to Postgres (Google Cloud SQL).
3 Likes
“Despite the fact that they are “serverless”, your group will, in any case, invest a great deal of energy attempting to get the datasets you need into them.”
Would you be able to explain what you mean by that? We as of now have all our promoting/deals/item/web use information adjusted to both Snowflake and BigQuery.
3 Likes
Perhaps the particular instruments we use end up having better environment reconciliation with BQ/Snowflake. We would have to accomplish more information fighting and custom coding to get our information into uncovered metal Postgres (basically from our stack)
3 Likes
I didn’t know you previously had something set up. In the event that you as of now have Snowflake and BigQuery, there isn’t a lot of motivation to likewise present Postgres.
I would say, it’s a bit trickier, to begin with, Snowflake. We needed to fabricate a custom pipeline to move information from S3 - > Snowpipe - > Snowflake.
Concerning the contrasts between BigQuery and Snowflake, I don’t believe there’s that enormous of a distinction between them. I figure you would be glad in the event that you went with both of them.
2 Likes
Bodes well. We conceptualized simply utilizing an uncovered metal or oversaw data set occurrence yet the cloud DWs are so modest since they didn’t appear to bode well from a TCO outlook.
1 Like
This topic was automatically closed 180 days after the last reply. New replies are no longer allowed.