Tech Blog
Jun 21, 2022
[Tech Blog] Cloud Composer (Airflow) for Machine Learning Data Pipeline
Hi, I'm Naoki Komoto (河本 直起) working as Machine Learning Engineer in AnyMind.
In AnyMind, we’ve just started introducing machine learning functions to our products. So we are developing our MLOps environment, including the data pipeline from scratch.
In this article, I'd like to introduce our data pipeline using Cloud Composer (Airflow) including its current set up and future plans.
Problems
The dataset for the machine learning (ML) model training is originally stored in the product application’s RDB. When I joined the team, model training applicatio