Bicep Spotter - Standards That Only Move Forward - A tool that already existed

Building Spotter, a ratchet-based standards tool for Azure Bicep

We are going on a bit of a storytelling journey here, so bear with me.

You know how an idea can get stuck in your head, sometimes for years? You don't have time to do anything with it. Or you don't yet understand it well enough to start. Or, honestly, you're a little afraid of how it might turn out. This is a story about one of those ideas. Mine has been rattling around since December 27, 2021, the day the Legacy Code Rocks episode "Betterer with Craig Spence" came out.

The episode was about Betterer, which had a simple but powerful idea: help a codebase improve incrementally, while stopping it from moving further away from the goal.

That idea stuck with me. Not because it was clever (it is), but mainly because it was realistic. Most codebases are not perfect (believe me), and most infrastructure repositories are not either. Standards change, teams grow, patterns evolve, and security expectations keep going up. What was acceptable two years ago might not be acceptable today.

The hard part is usually not defining the standard. The hard part is introducing it into an existing codebase, because as soon as you add a new rule you tend to get the same answer:

Great rule. But we already have 400 violations.

And then what? Do you block every pull request until everything is fixed? Ignore the rule? Block deployments? Write the standard in a wiki and hope people remember it? None of those options feel great, and that is the problem I wanted to explore.

The problem with "all-or-nothing" standards

Infrastructure-as-code has the same problem as application code, sometimes with higher consequences. You may want every storage account to disable public blob access, every Key Vault to have diagnostic settings, required tags on every resource, App Services that enforce HTTPS only, and sensible rules for SQL servers, network security groups, managed identities, private endpoints, and monitoring. None of these are strange requirements. They are ordinary platform standards.

But if you introduce them into an existing Bicep/infrastructure repository, you may quickly discover that the current state does not match the goal. That does not mean the team is bad (even if it can be experienced that way... sadly..). It usually means the standard arrived after the system had already grown.

This is where strict enforcement becomes painful. A traditional quality gate says that everything must be fixed before anything can move forward. That sounds good in theory. In practice it can stop delivery completely.

So I wanted a different workflow, one that could express; we accept where we are today, but we do not allow things to get worse. That became the core idea behind Spotter.

Azure Policy is the floor

In Azure, many of these standards can already be expressed with Azure Policy. It evaluates Azure resources and actions against business rules, and several rules can be grouped together into initiatives. Depending on the policy effect, Azure Policy can audit, deny, append, modify, or deploy related configuration.

I want to be clear about this part: Azure Policy is not something Spotter should replace (or any other tool TBH) . Azure Policy is the enforcement floor. It protects the real platform. If a deployment tries to create something dangerous, deny policies can stop it. If resources drift from desired standards, audit policies can bring that to the surface. If configuration can be remediated, effects like modify or deployIfNotExists may help. Azure Policy is where governance becomes real, the guardrails.

For developers working with Bicep, though, Azure Policy can be too late in the feedback cycle. By the time a deny policy rejects a deployment, the developer has already written the Bicep, opened a pull request, waited for CI, gone through review, and started a deployment. Only then does the error come back from Azure. That is slow feedback.

When a policy only audits, the problem gets even less visible. The deployment succeeds, the audit finding appears somewhere else, and the connection back to the Bicep code is easy to miss.

That was the gap I wanted to explore.

The Boy Scout Rule

The Boy Scout Rule is usually phrased as: leave the campground cleaner than you found it. This is something that i've heard the my entire career. A team should be able to introduce a new rule without being forced into a giant cleanup project. They should be able to say: these old violations exist, we know about them, they are tracked, but from now on new code must meet the standard. And when someone fixes an old violation, that improvement should stick. The standard should move forward, not backwards.

The ratchet

A ratchet only moves in one direction, and that became the mental model. Instead of treating the current state as a failure, Spotter treats it as a baseline. Existing violations are recorded, known, and visible, but they do not block every pull request. New violations fail the build. Fixed violations can be removed from the baseline, and over time the baseline shrinks.

The important part is that improvement becomes incremental. You do not need a giant cleanup project before introducing standards. You can start with the codebase you already have.

Building Spotter for Bicep

Spotter is my first version of this idea for Azure Bicep. The goal was a small, Bicep-focused tool that scans templates, applies standards, compares the result against a committed baseline, and fails only when a pull request introduces new violations.

The basic workflow is intentionally simple:

spotter init
spotter baseline
spotter check

spotter init creates an empty baseline. spotter baseline records the current violations. spotter check runs the rules and compares the current result to the baseline. If there are no new violations, the check passes. If a developer introduces a new violation, the check fails. And if the developer fixes something that was previously baselined, the check passes and Spotter tells the team that the baseline can now be ratcheted down.

That last part is what I like most. The baseline is not just a suppression file. It becomes a reviewed artifact in git history, so you can see whether the repository is moving in the right direction. Not by looking at a dashboard somewhere far away, but by looking at the same repository where the infrastructure code lives.

This approach is litteraly a direct idea-steal from Betterer.

Standards closer to the developer

The idea is to bring standards into the developer loop: in the editor, in CI, and in pull request review, rather than after deployment or only in a compliance report.

Spotter is not literally changing the Bicep compiler, but it sits close to the compilation flow by using Bicep output and evaluating what can be known statically from the template.

The visibility question pulled me in another direction too: a Spotter VS Code extension. It would give developers feedback in real time while they write the infrastructure code, showing new violations as warnings and baselined violations as lower-severity hints, and close the feedback loop even further. This lead me to explore how to programatically examine bicep templates. And it took me three tries to get it "okayish".

Getting Spotter to read Bicep took three tries

The VS Code extension route is also what forced the hardest technical decision in the project: how Spotter reads Bicep in the first place.

My first attempt was a hand-written parser that read Bicep syntax directly, just to get something to work from. It worked until it didn't, the solution was to naive. Bicep is a moving target, and every time Microsoft changed the syntax my parser would break. I was signing up to chase someone else's language forever, which is not a fun way to spend weekends.

So I threw that away and just shelled out to bicep build, once per file, once per save (for my VS Code Linter). The official compiler gives me correct output and I stop caring about syntax changes. That is great on the command line and terrible in an editor. Every file save spawned a fresh .NET process, and that startup alone was around 500ms. Type, save, wait half a second for the linter, repeat. No developer would keep that turned on. But the Developers now have a faster feedback cycle.

The third try i investigated @azure/bicep-rpc-client and initialized a jsonrpc session to remediate from starting/stopping Bicep over and over. Now Spotter bootstraps one Bicep process and talks to it over JSON-RPC for the whole coding session. You pay about 760ms once, when the extension wakes up or spotter check starts, and after that each file comes back in roughly ~200ms against the already-warm process. That is the difference between feeling instant and making the editor lag.

The current parser.ts is what fell out of that third attempt.

Rules owned by the platform team

Another thing I wanted was a simple rule model. Spotter does not need strong built-in opinions, because different organizations have different standards. Some care most about security, some about cost allocation or naming conventions, some about observability or architecture boundaries.

So Spotter rules are JSON files owned by the team. They can live in .spotter/rules/**/*.json or in a root-level spotter.rules.json file. A rule can be simple, such as:

{
  "id": "SPX001-storage-no-public-blob",
  "description": "Storage accounts must not allow public blob access",
  "severity": "error",
  "resourceTypes": ["Microsoft.Storage/storageAccounts"],
  "checks": [
    {
      "property": "properties.allowBlobPublicAccess",
      "operator": "notEquals",
      "value": true
    }
  ]
}

This is not meant to be a universal standard. Above is an example of the kind of thing a platform team might want to express. The important part is that the rule becomes part of the development workflow.

Using Azure Policy as input

One of the more interesting parts for me was exploring Azure Policy import. Most organizations that care about Azure governance already have policies or initiatives, and those policies represent organizational intent. So instead of asking teams to manually recreate every standard in another tool, I wanted to see whether some of that policy intent could be reused.

Spotter now got capabilities to generate rules from exported Azure Policy definitions and can filter through initiatives. In the current version, simple Audit and Deny policies with field-based conditions can be converted, and some AuditIfNotExists-style checks can become child-resource presence checks.

To me this is the strongest idea in Spotter: Azure Policy defines the floor, and Spotter helps developers move toward that floor before deployment. It is not about replacing governance. It is about shortening the feedback loop.

The limitations

There are obvious limitations. Spotter can only evaluate what is statically decidable from the Bicep template and the compiled ARM representation. It cannot fully understand live Azure state, fetch every external dependency, or evaluate relationships that only exist after deployment. It cannot replace remediation effects, and it cannot handle every Azure Policy construct.

Effects like DeployIfNotExists, Modify, and Append are not simple static checks. Some AuditIfNotExists policies at subscription or management group scope are not easy to map to a per-template rule either. More complex Azure Policy logic such as anyOf, not, count, and array traversal aliases is also be hard to convert directly.

I can live with that. A developer-loop tool does not have to understand everything. It has to give fast, useful feedback for the things it can know early. Azure Policy still protects the platform; Spotter helps developers reach the standard sooner.

Aaaand then I discovered Checkov

At this point I was feeling pretty good about the idea. I had a Bicep-focused tool, rules, a baseline, a ratchet workflow, Azure Policy import, and even a slogan: standards that only move forward. I had also survived a long (and very frustrating) prompting session with ChatGPT trying to get a cool logo.

Then, while researching something else entirely, I stumbled on Checkov.

Checkov, it turns out, already exists. And it:

• scans Bicep files and evaluates them against Azure best-practice policies. Fine.
• supports custom policies, both YAML-based and Python-based. Okay. Sure.
• has a baseline mode, where a .checkov.baseline file can be created and later used so that only new failed checks are reported. Oh, come on.

So yes. There is overlap. A LOT of overlap. Roughly the size and shape of my entire project, definitely more than I wanted to discover after building something.

That took some of the momentum out of it for me (approximatly 1.5 month between building and this post). There is a specific kind of developer-pain in thinking you have found a neat, focused idea, building a first version, and then discovering that a mature tool already covers a large part of the same problem space. It was humbling, and honestly slightly annoying, but mostly useful, because it forced my question to change. The question is no longer whether I invented a new category of tool. I did not. The better question is whether I learned something by applying the Betterer-style ratchet idea specifically to Bicep and Azure Policy-driven standards. I think I did.

So was this pointless?

I don't think so, but the positioning has to be honest. Spotter should not be described as if no other IaC scanning tool exists. Checkov is real, PSRule is real, and so are other scanning and policy-as-code tools. The value of Spotter is not that it is the first tool to scan infrastructure code, and not even that it is the only tool with a baseline concept.

The interesting part is narrower: what would a small, Bicep-first, Azure Policy-aware ratchet workflow feel like? That is still a useful experiment. Checkov may well be the right answer for most teams, and Spotter may remain a learning project. Or it becomes useful precisely because it is smaller, more focused, and easier to reason about for a specific Azure/Bicep workflow. The Azure Policy import idea might be the part worth developing further. And even if none of that happens, I now understand the problem much better than I did before. All of those are acceptable outcomes.

What I still like about the idea

I still like the philosophy. Tools that accept reality instead of pretending every repository starts clean. Standards that can be introduced without blocking every team. Developer feedback that happens before deployment, an improvement curve you can see in git history, and the idea that fixing one old violation should matter, and that a pull request should not make the codebase worse.

That last part is what I care about most. The scanner matters less to me than the direction.

Standards that only move forward

The slogan still works for me. The DevOps feedback loop is not about perfection, or full compliance from day one. It is a way to say: this is where we are, this is the standard we want, and from now on we move toward it. In a controllable and measurable way.

Azure Policy still remains the enforcement floor. Spotter was my attempt to build a helper in the developer loop. And then I discovered that Checkov overlaps with the idea much more than I expected. So there is that.

But I still think building it was worth it. Sometimes you build a tool because the world (read: me) desperately needs it. Sometimes you build a tool because you need to understand the idea properly. This one might be more of the second kind, and to be honest, that is still a pretty good reason to build something.

(And yes, the same LLM that failed to give me a decent logo also helped me build parts of bicep-spotter. The ideas, the decisions, the mistakes and most importantly the mild embarrassment are all mine.)

Still intrested?

Ok, Congratz and thank you! You made it this far down, i'm soon done rambling..

To get started using these extensions you can simply head to the github project and follow the instructions there.

Please connect with me on LinkedIn if you are intrested in this or another topic.

Keep on learning!