[Tech Blog] Cloud Composer (Airflow) for Machine Learning Data Pipeline

Hi, I'm Naoki Komoto (河本 直起) working as Machine Learning Engineer in AnyMind. In AnyMind, we are developing our MLOps environment, including the data pipeline from scratch. In this article, I'd like to introduce our data pipeline using Cloud Composer (Airflow) including its current set up and future plans. Problems The dataset for the machine learning (ML) model training is originally stored in the product application’s RDB. When I joined the team, model training application directly fetches dataset from the RDB. As RDB is chosen due to its suitabil