Creating an E2E ML Pipeline
Explore building end to end machine learning pipelines in Azure by integrating multiple jobs step by step. Learn to run jobs sequentially or in parallel, pass outputs between dependent jobs, and visualize pipeline workflows using Azure Machine Learning studio. This lesson helps you master pipeline creation and management for scalable model training.
We'll cover the following...
What is a pipeline job?
Often, building ML models requires stitching multiple jobs together. For example, we might want to run two jobs in sequential order. We can stitch multiple steps into a single Azure pipeline in such a case. We will define a new job type pipeline and the sequential steps in the YAML file.
We can see a pipeline command example below.
We have two jobs: hello_job and world_job, which run in sequence as a single pipeline. Let’s run this job.
$schema: https://azuremlschemas.azureedge.net/latest/pipelineJob.schema.json
type: pipeline
jobs:
hello_job:
command: echo "hello"
environment:
image: library/python:latest
compute: azureml:ComputeEdu
world_job:
command: echo "world"
environment:
image: library/python:latest
compute: azureml:ComputeEdu