In the first article about CI, I discussed CI concepts and steps, automated build vs manual build among other things. Here, I will focus mainly on the build agent which is the main player that is responsible for carrying all the work in the CI pipeline.
Table of Contents
What is CI Pipeline?
The CI workflow is called pipeline indicative of its sequential nature as each step leads to the next. As a refresher, this diagram represents the main step in any CI pipeline:
The pipeline Trigger:
Continuous Integration (CI) pipelines can be triggered either manually or automatically. Typically, these pipelines triggered by code commits to a specific repository. However, the configuration can be tailored so that only particular pipelines are triggered when code is pushed to certain predefined branches.
This feature is useful in large-scale projects with a microservices architecture, where it’s often necessary to build or update a specific microservice independently, rather than deploying the entire application.
In this table I summarized the different automatic triggers that are available in Azure DevOps platform:
| Automatic Trigger Type | Description | Repository Types |
| Continuous Integration Trigger | Initiates automated builds upon code commits to integrate changes regularly. | Azure Repos Git, GitHub, Bitbucket Cloud |
| Pull Request Trigger | Activates builds for code validation during pull request reviews. | Azure Repos Git, GitHub |
| Gated Check-in Trigger | Ensures code changes pass validation tests before merging with the main branch. | TFVC |
| Comment-Based Trigger | Starts builds in response to specific comments in pull requests. | Azure Repos Git, GitHub |
| Scheduled Trigger | Schedules automated builds at predefined times or intervals. | Azure Repos Git, GitHub, Bitbucket Cloud |
| Pipeline Dependency Trigger | Triggers a pipeline following the completion of a preceding one, creating dependencies. | Azure Repos Git, GitHub, Bitbucket Cloud |
| Build Completion Trigger | Initiates a build process when a specified build pipeline is completed. | Azure Repos Git, GitHub, Bitbucket Cloud |
But what/who exactly is doing all this work involved in the CI pipeline?
All this work is performed by a build agent. A build agent consists of two main components:a software and an execution environment as explained below. The choice of build agent (software and environment) depends on the project’s requirements, like the technology stack and the infrastructure consideration.
The build agent plays a crucial role in the build and deployment process, acting as the executor of tasks outlined by a build server or a CI/CD system. When it receives instructions, it initiates a range of actions: compiling code, executing tests, and generating deployable packages. These actions collectively form what is known as a ‘build agent job‘, which consists of a sequence of tasks executed in a specific order on the same target environment.
Self-Hosted vs Provider-Hosted Agent:
Build agents software need an execution environment in which the software will reside. This can be either hosted (Microsoft run cloud agent with ready made images ) or self-hosted (private).
In this table there is a comparison between hosted agents against self-hosted agents.
| Self-Hosted Agent (on premises or in the cloud) | Provider-Hosted Agent (cloud) |
| You manage the agent completely. You can customize it as you wish. (Some jobs will need certain software/dependencies that are not available in Microsoft-hosted agents) | You consume it as is (provided as saas) and you can’t customize beyond what is offered by the service provider. (i.e. you can’t increases the machine’s RAMs, CPU cores etc) |
| You build your own configuration (OS+Apps) | Ready-made images (OS/Apps) |
| You manage all the patches and updates. | You don’t deal with patches or updates. |
| It needs time to build your configuration. | Desired configuration ready immediately. |
| You can implement caching. | You can’t have caching implemented as everytime you build you will get a new machine. |
| Works well for certain jobs where Microsoft-hosted agents does not have line of sight access (i.e an on-premises company have firewall) |
Each method (either hosted or self hosted) has its strengths and limitations. There is no “better” approach here. Each method has its use cases. You will need to consider lots of aforementioned factors before deciding which one to use.
I would like to take a moment here and reflect on this. In IT and I will dare to say in life in general when you are presented with two options it’s alway wise to think about the trade off. Each solution/option/service/product has its strengths and limitations. What is crucial is to understand these strengths and limitations and consider which one makes more sense to your use case.
By the way, the question of “when to use/when not to use hosted agent and self hosted agents” is an interview question that measures the depth of the interviewee knowledge on this topic, and you can expect it to be heard during a devops position job interview, so let’s make sure we get this right!
Hosted vs self-hosted agents use cases:
| Self-Hosted (On-premises/Cloud) | Hosted (Cloud) |
| Large size code base and long compile time. | Small size code base and short compile time. |
| Intensive CI/Intensive build or Intensive CD | Optimized CI/not intensive build nor intensive CD. |
| Limited accessibility/security | |
| Redundant second line support for mission critical application |
Which one to choose?
Mohamad Radwan, a Microsoft MVP, prefers to start with a microsoft hosted agent first, and if the need arises you can switch to self hosted. The idea is to conserve resources. It’s similar to “least privilege principle” in security, you start with the least resources consuming option and you go to the more advanced/sophisticated/powerful option.
What is the difference between windows service and windows process?
Build agent can run as a service or a process. When it’s run as a service it start when the host machine boots up. This setup is ideal for use cases that don’t require user interaction or the use of Graphical User Interface (GUI).
In the other hand, running the build agent as process comes in handy when you deal with user interface tasks. The following tables provide more details:
| Service | Process (Interactive) |
| It used for build that doesn’t require run UI automation. | The process is used for builds that require running UI automation (interactive) and can also be used for non-interactive builds. |
| Runs and appears in Windows service management. | Doesn’t run nor appear in windows services management instead It appears in task management as a process. |
| The service login using an independent account from the user account login to the machine which is more secure and limited in its permission. | The process login using the same account login to the machine. |
| The service automatically starts with the machine. | The process needs to starts manually |
| The service has a recovery mode if it’s crashed. | No Built-in Recovery. If you want this feature you need to write a script or use a monitoring tool. |
What are agent capabilities?
Agent capabilities determine what task a build agent can run. For instance, if your build includes tasks that need .Net core then you will need to have .Net core run time for it to run. Agent capabilities consist of two types:
- System defined capabilities: such as the operating system version, installed software, system settings, environment variables, and hardware specifications.
- User defined capabilities: An example of it would be selecting a version of Java or Python installed, the amount of RAM available, presence of specific tools like Docker, etc.
This concludes my article about build agents. I am hoping you learned something new from it!
Reference:
- Radwan, Mohamed. “Managing Version Control.” Fundamentals of Modern Software Engineering and DevOps course, Module 7. DevOps Visions. https://github.com/MohamedRadwan-DevOps/devops-step-by-step/tree/main/source/advanced-introduction-to-devops. Accessed on 10th January 2024
- Microsoft. (n.d.). Windows agent – Azure Pipelines | Microsoft Docs. Learn Microsoft. https://learn.microsoft.com/en-us/azure/devops/pipelines/agents/windows-agent?view=azure-devops Accessed on 14 th January 2024
- Microsoft. (n.d.). Triggers – Azure Pipelines | Microsoft Docs. Learn Microsoft. https://learn.microsoft.com/en-us/azure/devops/pipelines/build/triggers?view=azure-devops Accessed on 10th January 2024
