F# Compiler Server: making it faster

Per-compilation and whole-build experiments on the compiler server branch (repos/server), 2026-10-18, 11-core Apple M-series. Plan: plan.html · first results: results.html · code: compiler_server (diff).

Summary

Your result reproduces. Per compilation, a warm worker is much faster than stock fsc + JIT flags, in both wall time and CPU. Summed over 5 projects, the wall time falls by 25–30% and CPU by 35–45%. Small and medium projects take 0.2–0.6× of the flags' time; FCS takes 0.85–0.9×.
Whole builds needed the workers tuned. The first version was only level with the flags on whole builds. Two one-line worker settings fixed that, and both are now the branch default:
- DOTNET_TC_CallCountingDelayMs=0, keeping Tiered PGO on. A new worker used to compile its first large project at stock speed (FCS 26–28 s, against 14 s with the flags). Now it tiers up as fast as a JIT-flagged process, and steady state is unchanged.
- Non-concurrent GC: about 13% less CPU per compilation and 12% faster FCS-solution builds.
Whole builds with the new defaults:
- Fantomas: 19.4 s, against 25.3 s for stock + flags (−23%) and 30.2 s for stock (−36%). CPU −44%. It repeats to within about 0.3 s.
- FSharp.Compiler.Service.slnx: 39.0 s mean over 8 builds (36.3–41.4 s), against 45.7 s (−15%), CPU about −35%. This solution is dominated by the FCS compilation, where the JIT is only 2–3% of CPU and GC is 31–39%.
Not worth it:
- Reusing assembly caches between compilations: a warm worker's imports already take about 0.05 s.
- Tiered PGO off in workers: 11% slower at steady state.
- All cores as GC heaps per worker: 28% slower than the flags on parallel builds (47% slower than 5 heaps), as you expected.
- Skipping the idle-time GC: no effect.
Outputs: every server compilation and build produced the same assemblies as stock (main and reference assemblies, compared by hash).
Occasional slow builds: explained as MSBuild scheduling.
- About 4 of 40 warm FCS-solution server builds took 53–55 s instead of about 37–45 s; none of the 12 stock / stock + flags builds did.
- The per-compilation timing now in the build log caught one. FCS's two target frameworks were built one after the other on the same MSBuild node: the netstandard2.0 compilation started 23 s after net11.0, instead of together. Everything that depends on FCS then waited.
- The compilations themselves were faster than usual (11.9 and 11.3 s, against about 15.5 s when they overlap), and the client added at most 0.05 s per compilation.
- Whether the server makes this ordering more likely is not established.

Findings, ranked by benefit for the effort

#	Change	Measured effect	Complexity	Verdict
1	Keep the compiler warm: compile in long-lived workers (the server)	Per compilation (5 projects summed): wall -25%, CPU -36% vs stock + JIT flags (vs stock: -58%); FSharp.Build 0.21×, FCS 0.89× of flags. Whole builds with the tuned workers (rows 2–3): Fantomas -22% wall / -44% CPU vs flags (-35% vs stock); FSharp.Compiler.Service.slnx -12% wall / -29% CPU vs flags.	built (this branch)	Done.
2	Workers: DOTNET_TC_CallCountingDelayMs=0, Tiered PGO kept	Cold first FCS compilation 27.6 s → 17.5 s (-37%); steady state unchanged (18.8 vs 18.7 s). Build with fresh workers (FCS slnx warm-up): 58.2 s → 47.3 s, on par with stock + flags.	1 line (client)	Done: now the default.
3	Workers: non-concurrent GC (DOTNET_gcConcurrent=0; GCSettings.LatencyMode=Batch per compilation is equivalent)	Per compilation: CPU -13%, wall -5%. FCS slnx build: 44.4 s → 39.3 s (-12%).	1 line (client)	Done: now the default.
4	Workers: GC heap count (cores/2 vs all cores vs DATAS)	Alone, 11 heaps make FCS -12% faster, but in a parallel build they are a disaster: 58.6 s vs 40.0 s (+47%). DATAS: FCS slnx 37.7 s (more memory: up to 7.0 GB vs 4.5 GB), Fantomas 20.1 vs 19.7 s with more CPU.	1 line	Keep max(1, cores/2); DATAS is a reasonable alternative, not clearly better.
5	Worker affinity (a project goes back to the worker that compiled it)	Per compilation a dedicated worker is no faster than a shared warm one (19.4 vs 18.7 s): most compiler code paths are shared between projects. In builds it avoids sending a big compilation to a worker that has never run it.	~30 lines (client)	Kept; cheap. Fast tier-up (row 2) matters more.
6	Keep assembly caches between compilations (IL readers, shared imported CCUs)	No gain (18.5 vs 18.7 s): a warm worker spends ~0.03–0.05 s importing references.	small (opt-in switch exists)	Leave off: not stock-exact, and nothing to gain on these projects.
7	JIT flags in workers (TieredPGO=0)	Steady state +11% (both flags), +11% (TieredPGO=0 alone).	1 line	Rejected: Tiered PGO pays off in long-lived workers.
8	No compacting GC when idle	No effect (18.8 vs 18.7 s).	1 line	Keep the idle GC: it returns memory.
9	Reduce allocations in the compiler (large projects)	FCS allocates ~33 GB per compilation; GC is 31–39% of its busy CPU samples, JIT 2–3%: for FCS the server can only remove JIT/startup.	large (compiler work)	Biggest remaining lever for big projects, with or without the server.

Per-compilation results

Each compilation is replayed from the response file captured in a real build (outputs redirected). Process configs start dotnet fsc.dll per compilation; server configs send it to a worker over its socket, as the Fsc task does. 1 warm-up round, then 5 measured rounds of the 5 projects; cells are medians in seconds, with the ratio to stock + JIT flags. "Cold" columns are the warm-up round: the first time a fresh worker (or process) compiles that project.

config	FCS	FSharp.Core	ComponentTests	Test.Utilities	FSharp.Build	sum wall s	sum CPU s	cold FCS s	cold round s	FCS GC pause s
`stock` stock fsc process	23.95 1.76×	10.19 2.26×	6.55 1.67×	3.07 1.54×	1.02 1.15×	44.8 1.79×	170 1.53×	24.5	45.5	–
`flags` stock fsc process + JIT flags (TieredPGO=0, CallCountingDelayMs=0)	13.64 1.00×	4.51 1.00×	3.93 1.00×	2.00 1.00×	0.89 1.00×	25.0 1.00×	111 1.00×	13.7	25.0	–
`srv` worker: Server GC, 5 heaps (default)	12.17 0.89×	3.02 0.67×	2.50 0.64×	0.87 0.43×	0.19 0.21×	18.7 0.75×	72 0.64×	27.6	37.1	4.1
`srv_dedicated` worker per project (perfect affinity)	12.30 0.90×	3.20 0.71×	2.56 0.65×	0.99 0.50×	0.34 0.39×	19.4 0.78×	69 0.62×	27.1	49.9	3.9
`srv_reuse` worker: keep assembly caches between compilations	12.12 0.89×	3.01 0.67×	2.36 0.60×	0.81 0.40×	0.19 0.22×	18.5 0.74×	71 0.64×	26.4	34.5	3.8
`srv_noidlegc` worker: no idle compacting GC	12.33 0.90×	3.00 0.67×	2.39 0.61×	0.84 0.42×	0.20 0.22×	18.8 0.75×	72 0.64×	27.1	35.9	4.0
`srv_flags` worker + both JIT flags	13.31 0.98×	3.59 0.80×	2.75 0.70×	0.95 0.48×	0.21 0.24×	20.8 0.83×	77 0.69×	14.6	21.9	4.0
`srv_pgo0` worker: TieredPGO=0	13.36 0.98×	3.46 0.77×	2.80 0.71×	0.92 0.46×	0.21 0.24×	20.8 0.83×	77 0.69×	21.2	29.3	3.9
`srv_ccd0` worker: CallCountingDelayMs=0 (PGO kept)	12.32 0.90×	2.90 0.64×	2.61 0.66×	0.81 0.41×	0.20 0.22×	18.8 0.75×	72 0.65×	17.5	24.4	4.1
`srv_datas` worker: DATAS (no fixed heap count)	11.45 0.84×	3.16 0.70×	2.42 0.62×	0.91 0.46×	0.21 0.24×	18.2 0.73×	83 0.74×	26.8	35.6	3.1
`srv_heaps_all` worker: 11 heaps	11.27 0.83×	2.90 0.64×	2.38 0.61×	0.84 0.42×	0.20 0.22×	17.6 0.70×	78 0.70×	26.2	34.8	2.9
`srv_nonconc` worker: non-concurrent GC	11.63 0.85×	2.91 0.65×	2.19 0.56×	0.82 0.41×	0.23 0.25×	17.8 0.71×	62 0.56×	26.7	35.4	4.0
`srv_batch` worker: GC latency Batch while compiling	11.58 0.85×	2.87 0.64×	2.18 0.56×	0.86 0.43×	0.19 0.22×	17.7 0.71×	62 0.56×	27.2	35.4	4.0
`srv_heaps_all_nonconc` worker: 11 heaps + non-concurrent GC	10.28 0.75×	2.84 0.63×	1.90 0.48×	0.81 0.40×	0.19 0.21×	16.0 0.64×	69 0.62×	25.7	34.0	2.7
`srv_heaps_all_batch` worker: 11 heaps + Batch latency	10.30 0.75×	2.92 0.65×	1.95 0.50×	0.77 0.39×	0.20 0.22×	16.1 0.65×	69 0.62×	26.4	34.1	2.7
`srv_ccd0_nonconc` worker: CallCountingDelayMs=0 + non-concurrent GC	11.58 0.85×	2.91 0.65×	2.14 0.54×	0.78 0.39×	0.19 0.21×	17.6 0.70×	61 0.55×	17.4	23.9	4.0

Outputs: every server configuration produced assemblies identical to stock for every compilation (hashes compared per run).

Where the time goes in a large compilation (FCS)

Native sampling with macOS sample (1 ms, every thread), self samples of busy threads grouped by what runs: managed compiler code (JIT'd or ReadyToRun, unsymbolized), GC (gc_heap functions in libcoreclr), JIT (libclrjit), other runtime code, other native code. Sampling slows the compilation, so read shares, not times.

profile	managed	GC	JIT	runtime	other
flags	46.6%	38.8%	3.0%	6.0%	5.5%
srv	49.8%	30.6%	1.6%	7.4%	10.6%
srvh11nc	48.1%	36.2%	1.8%	9.6%	4.3%

flags = stock process with both JIT flags; srv = warm worker (default); srvh11nc = warm worker, 11 heaps, non-concurrent GC. For FCS the JIT is already small with the flags, so a warm worker can only win a little there; GC is the dominant cost.

Whole builds

dotnet build -c Release --no-restore --no-incremental, MSBuild node reuse off, 1 warm-up and 3 measured builds per config, configs run one after another. Server configs include worker affinity; their worker settings go to the workers only (FSHARP_COMPILER_SERVER_WORKER_*). CPU = build process tree + live workers.

FSharp.Compiler.Service.slnx (17 compilations, up to 9 at once)

config	warm-up (cold workers) s	measured builds s	mean s	vs flags	CPU s	CPU vs flags	workers	worker RSS MB
`flags`	46.6	46.1 · 45.3 · 45.6	45.7	1.00×	345	1.00×	0	0
`server`	67.1	45.3 · 45.4 · 42.6	44.4	0.97×	273	0.79×	8	3931
`server_nonconc`	58.2	38.4 · 39.1 · 40.5	39.3	0.86×	250	0.72×	8	4250
`server_ccd0_nonconc`	47.3	41.8 · 39.6 · 38.5	40.0	0.88×	246	0.71×	9	4111
`server_ccd0_nonconc_datas`	50.0	41.1 · 36.4 · 35.7	37.7	0.83×	254	0.73×	9	5926
`server_ccd0_nonconc_heapsall`	59.2	62.3 · 60.1 · 53.4	58.6	1.28×	307	0.89×	9	4166

Fantomas (fantomas.slnx, many projects, mostly dependency chains)

Built with this repo's main compiler and targets, so -p:LangVersion=11 is passed to every config: main's targets no longer default preview SDKs to F# 11 (the SDK's do), and Fantomas.Tests does not compile with main's newer default language version.

config	warm-up (cold workers) s	measured builds s	mean s	vs flags	CPU s	CPU vs flags	workers	worker RSS MB
`baseline`	29.2	29.2 · 31.7 · 29.7	30.2	1.20×	95	1.01×	0	0
`flags`	25.2	25.3 · 25.5 · 25.0	25.3	1.00×	94	1.00×	0	0
`server_ccd0_nonconc`	22.5	19.7 · 19.7 · 19.7	19.7	0.78×	53	0.56×	3	4381
`server_ccd0_nonconc_datas`	22.2	20.2 · 19.9 · 20.3	20.1	0.80×	61	0.65×	3	2367

Final: the branch's new defaults, no overrides

config	warm-up (cold workers) s	measured builds s	mean s	vs flags	CPU s	CPU vs flags	workers	worker RSS MB
`server` FCS slnx	48.3	44.7 · 55.3 · 41.9	47.3	–	239	–	9	4290
`server` FCS slnx, repeat	46.2	45.6 · 38.0 · 40.8 · 37.5 · 37.5	39.9	–	239	–	9	4591
`server` FCS slnx, repeat 2	43.9	40.8 · 41.4 · 37.7 · 36.3 · 38.1 · 39.8 · 38.3 · 39.9	39.0	–	228	–	8	4707
`server` Fantomas	21.7	19.8 · 19.4 · 19.0	19.4	–	52	–	3	3702

Stock + JIT flags for reference: 45.7 s (FCS slnx, table above) and 25.3 s (Fantomas). The first FCS run contains one of the unexplained slow builds (55.3 s); the repeat's first build started with 2.5 cores of outside load.

Not done yet, worth trying next

Slow-build ordering: the occasional slow builds come from MSBuild running FCS's two target frameworks one after the other. Next: more stock builds to see whether it happens there too at a lower rate, and whether -graph builds avoid it.
Pre-warm new workers: start one spare worker ahead of need, or let a new worker compile a small embedded sample before its first request. Fast tier-up already removed most of the cold cost, so this is worth it only if cold builds still matter.
Allocation profile of FCS (33 GB per compilation): the largest remaining cost for big projects, with or without the server.
Linux x64 with many cores (your work setup): the heap count and the non-concurrent GC choice should be checked there, where more compilations run at once.
Ship: dotnet build-server shutdown integration (dotnet/sdk), a version handshake and worker logging.