Hey there, let me share with you my journey at incident.io. About six months ago, during my interview process, Jack, my future manager, mentioned that the Data team had been managing without a dedicated Data Engineer for the past two years. Jack and other product engineers had been shouldering this responsibility.
As the company grew, so did the data ecosystem, leading to a potential bottleneck due to the current data platform’s limitations. This challenge excited me as I thrive on solving technical problems, so I decided to join the team.
In this post, I want to highlight three key areas I focused on in my first three months: improving our development platform, exploring different data ingestion tools, and reducing our platform costs.
### Development Platform
Our data stack includes Fivetran for data ingestion, BigQuery, dbt for transformation, CircleCI for CI, and Omni for BI. One of my early wins was uploading dbt artifacts to Google Cloud Storage after each production run. These artifacts are crucial for our workflow, and storing them in GCS allowed us to access them easily from anywhere.
By doing this, we streamlined our dbt development workflow and improved our CI process. Additionally, I implemented a custom Docker image for our dbt repository, making our CI and production runs smoother and faster.
### Data Ingestion Tool Exploration
Most of our data sources were being ingested using Fivetran when I joined. While exploring alternatives, we found that Fivetran was the most cost-effective option for us, considering the high number of upserts in our database. Despite looking into other tools like Google Datastream, Stitch, and Airbyte, we stuck with Fivetran due to its efficiency and pricing structure.
### Platform Cost
As our company expanded, our platform costs grew as well. I analyzed our costs and made significant reductions by optimizing our Fivetran BigQuery costs and implementing changes in dbt to reduce scanning costs.
In conclusion, my time at incident.io has been challenging and fulfilling. Working in a dynamic environment has allowed me to make impactful contributions and continuously improve our data workflows. I thrive in this kind of setting, where I can tackle complex problems with freedom and creativity. Join me on this exciting journey at incident.io!
