Perforce and Jenkins

While Git and SVN are widely used, Perforce is also a common choice for version control at larger game studios, especially for Unreal Engine projects with large art assets.

Jenkins is an open-source CI/CD tool used by many companies.
As a result, it's common to see Perforce and Jenkins used together in game development environments.

Perforce supports a variety of APIs, so you can use it directly in a Jenkins build pipeline.
However, an official Perforce plugin is also available to integrate it seamlessly with Jenkins' SCM (Software Configuration Management) system and pipelines.
GitHub - jenkinsci/p4-plugin: Perforce plugin for Jenkins

The Perforce plugin provides features such as:

• Viewing Perforce changelists directly on the build page
• Automatically triggering builds through polling
• Automatically creating Perforce workspaces for builds

The Problem

Simple configurations generally work without major issues, but conflicts can arise as pipelines become more complex and the number of build node slots increases.

Riot Games has a helpful article on this topic that's worth reading first:
Using Perforce in a Complex Jenkins Pipeline | Riot Games Technology
Building on that article, I'll provide some additional explanations and alternatives.

Perforce Workspaces

To understand the problem, you first need to understand Perforce workspaces.

Unlike Git, Perforce uses a centralized architecture.
To download files from Perforce, you must register a workspace.
Each workspace has a unique name and contains mappings between Perforce paths and local directories.

For example, if a workspace named kjs104901_depot is registered as shown below, files from the //depot path on the Perforce server are downloaded to D:\p4work\kjs104901_depot.

Understanding Jenkins Workspaces

Next, let's look at how Jenkins workspaces are structured.

Jenkins creates a unique directory for each build to work in.
For example, if the Jenkins root workspace directory is C:\Jenkins\workspace, the build workspace for a project named blog would be:

C:\Jenkins\workspace\blog

When multiple instances of the same build run concurrently on a build node, Jenkins automatically appends @ followed by a number to the workspace name to avoid conflicts:

C:\Jenkins\workspace\blog@2
C:\Jenkins\workspace\blog@3

The Perforce Plugin Problem

A Perforce workspace is needed to download files from Perforce and save build artifacts.
You can create this workspace automatically using the Jenkins Perforce plugin.

The Perforce workspace's local path can simply be set to the Jenkins workspace directory.
The question is how to name the Perforce workspace.

The official Perforce plugin documentation suggests generating names like this:

jenkins-%{NODE_NAME}-%{JOB_NAME}-${EXECUTOR_NUMBER}

This name combines the node name, job name, and executor number.
It seems reasonable because it should ensure that workspace names are unique.

The problem, however, is that Jenkins workspace suffixes such as @2 and @3 may not match EXECUTOR_NUMBER.
JENKINS-48882 Build Executor number is not consistent with EXECUTOR_NUMBER

These two values are managed independently. As a result, a Perforce workspace named jenkins-node-blog-2 could actually be used in the @1 or @3 Jenkins workspace.

This can be a serious problem because of how Perforce works.
Perforce caches information about which files have been downloaded to a local directory.
For example, if all files have been downloaded in @2 and the workspace then moves to @3, Perforce may decide that the files are "already up to date" and not download them again.

In other words, sharing the same Perforce workspace name across different workspace paths can cause various problems, including missing build artifacts and cache errors.

Solutions

Option 1 - Riot Games' Approach

To address the EXECUTOR_NUMBER mismatch, Riot Games checked the actual workspace path generated by Jenkins, parsed it using the @ symbol to extract the slot number, and used that number in the Perforce workspace name.

In addition, they separated Jenkins workspaces by stage, then further separated Perforce workspaces by slot number within each stage.
(Since stage names can be too long, they used the first four characters of a hash value instead.)

def riotP4Sync(Map config = [:]) {
  def humanReadableName = safePath("${JOB_NAME}-${STAGE_NAME}")
  def jenkinsWorkspaceName = safePath("${JOB_NAME}-") + workspaceShortname(env.STAGE_NAME)
  
  ws(jenkinsWorkspaceName) {
    echo "[INFO] [riotP4Sync] Running in ${pwd()}"
    def workspaceNumber = 1
    if (pwd().contains('@')) {
      workspaceNumber = pwd().split('@').last()
      if (!workspaceNumber.isInteger()) {
        error("${workspaceNumber} is not an integer! Something went wrong in riotP4Sync. PWD is ${pwd()}")
      }
    }
    def p4WorkspaceName = safePath("${humanReadableName}-${workspaceNumber}")
    echo "[INFO] [riotP4Sync] Using P4 Workspace ${p4WorkspaceName}"
}

Option 2 - A Simplified Approach

Riot's example code doesn't include NODE_NAME, and in many cases there's no need to separate workspaces by stage.

A simple function like the one below, which safely extracts the slot number, can be enough to replace EXECUTOR_NUMBER.

private int getSlotNumber() {
    if (pwd().contains('@')) {
        workspaceNumber = pwd().split('@').last()
        if (!workspaceNumber.isInteger()) {
            error("${workspaceNumber} is not an integer. PWD is ${pwd()}")
        }
        return workspaceNumber as int
    }
    return 0
}

Option 3 - Limit Concurrent Builds

If the same build doesn't need to run in multiple places at once, you can keep Perforce workspace names unique without extracting a slot number.

If each build runs exclusively on a node, you can use a configuration like this:

How do you ensure that each build is unique on a node?
The Jenkins Lockable Resources plugin lets you acquire a lock for each node and job combination.
Lockable Resources | Jenkins plugin
For example, you can ensure that only one instance of the build-A job runs on the agent1 node at a time.

If you use a fixed node for each build rather than distributing builds across nodes, there's a simpler approach.
Disable concurrent builds with the following build property to ensure uniqueness across all Jenkins nodes.

properties properties: [ disableConcurrentBuilds() ]

The Problem of Tracking Changes

Normally, the Jenkins Perforce plugin can automatically show changes like this:

However, as explained above, when workspaces are separated by stage or slot number, the Perforce plugin may treat each workspace as newly created or changed.
As a result, changes can be omitted even when builds complete successfully.

In other words, even for the same job, the Perforce plugin may treat a different workspace name as having no continuity with the previous build.

Solution: Track Changes Manually

If you use build changes to drive Slack notifications, release notes, deployment logs, or similar integrations, you'll need to determine the changes yourself.

How to track changes:

• When a build succeeds, save the current changelist number as a Jenkins artifact. archiveArtifacts
• In the next successful build, use a Perforce command or API to retrieve the changelists between the previously saved number and the current number.