Some of our Jenkins agents are powerful bare metal machines with a lot of CPU cores but too less memory.
In the specific case: 32 Cores / 64gb RAM.
Approximately 4gb have been used by the Windows OS and other stuff, this make in average less than 2gb per CPU.

When your configure your project to build in parallel, per default MSVC uses all available CPUs.
So when your compiler has a lot to do, for instance a lot of templating or constexpr constructs, a single compile process may require more than 1gb RAM.

So lets set the parallel builds in our main CMake-File:

add_compile_options(
    $<$<CXX_COMPILER_ID:MSVC>:/MP>
)

This can also be written as

if (MSVC) 
  add_compile_options(/MP)
endif(MSVC)

But I prefer the generator expressions from the first snippet, this allows more specific control with less branches.

However, this allows parallel builds of all projects. The build time reduces dramatically, while the memory and cpu usage explodes.
As long as you have no problems, everything is fine.

But if your machine has a lot of cpus with too less memory, you may soon get an out of heap error (C1060).
This is because MSVC uses all available CPUs with possibly multiple builds, which may consume too much memory.
In this case you can restrict the parallel builds:

add_compile_options(
    $<$<CXX_COMPILER_ID:MSVC>:/MP2>
)

Now your build takes more time, but also consumes less memory.
Theoretically! If you have a mixed project setting of C++ projects, managed projects and dotnet stuff, the problem may still persist.
This is due to cmake doesn’t set /MP2 to non C or C++ projects. \

But there is also a workaround. If you build your projects using msbuild (for instance on CI/CD nodes), you leave /MP in your cmake file and pass the value on command line:

msbuild projects.sln /property:CL_MPCount=2 ...

This sets the /MP switch to the value of CL_MPCount, so in the example above to /MP2.
Now you can make this decision depending on the node your running on, and developers using the same cmake file locally can go on using the full parallel build speed.

If you set the number of parallel executors on your node / agent as environment variable, you can handle this a bit more dynamically:

:: use a default of 2
SET CL_MPCount=2
:: this is normally defined on every windows machine
IF DEFINED NUMBER_OF_PROCESSORS (
  :: this is your count of executors on this node
  IF "%NUMBER_OF_EXECUTORS%"=="" (
    :: if it is not defined, lets assume `2`
    SET /A CL_MPCount=%NUMBER_OF_PROCESSORS%/2
  ) ELSE (
    SET /A CL_MPCount=%NUMBER_OF_PROCESSORS%/%NUMBER_OF_EXECUTORS%
  )
)

msbuild projects.sln /p:CL_MPCount=%CL_MPCount%

In this snippet CL_MPCount is set to count of processors / count of executors (while the latter defaults to 2).