Three months offline

Friday’s supplementary estimates ask Parliament for almost $1.6 billion this year for satellite communications at National Defence.¹ The money’s for the Enhanced Satellite Communications Project, Polar, and in August the Defence Investment Agency awarded Telesat about $2.3 billion to carry military Ka-band traffic over the Arctic on its Lightspeed constellation.² On October 30 the Prime Minister told Politico he didn’t want Canada’s defences hooked to “a satellite system that a billionaire, almost trillionaire, decides to turn off.”³

Planning for a link you can’t count on is what the Cyber Centre has been asking of critical infrastructure since April. On October 26 it told owners and operators they “must prepare for the possibility that important systems, suppliers or communications will become unavailable or cannot be trusted.”⁴ For systems, its answer is to rebuild from trusted offline copies.

Three months on the shelf

The Cyber Centre launched CIREN, its Critical Infrastructure Resilience and Escalated Threat Navigation initiative, in April. At minimum, it says, all operators of essential services should “be prepared to isolate systems for up to 3 months”, plan to run on their own, and plan to rebuild.⁵ For the rebuild it asks organizations to “keep offline copies of firmware, configurations and documentation” and to “plan for full system rebuilding using trusted offline copies and immutable backups”.

The October 26 bulletin repeats the advice. It points to a campaign made public that week by the group tracked as Storm-0940, where in some cases the activity went from picking a target to privileged access in cloud systems in under three hours, and it tells operators to “maintain trusted offline copies of essential software, configurations, instructions and recovery material”.⁴

Neither document says what makes a copy trusted. CIREN gets as far as “validated firmware images” without saying validated against what.⁵ That’s fair for guidance written to every critical infrastructure sector at once, water utilities and pipelines included, and it leaves the definition with whoever has to do the rebuilding. We’d start with where the trust comes from.

The newest backup

An immutable backup can’t be changed after it’s taken. That protects it from an intruder who shows up later and does nothing about one who was already there. Access can now move in hours, and a careful intruder can sit in a system for a long time before anyone notices, so the newest backup of a compromised system is a faithful, unchangeable copy of the compromise. The one before it might be too.

So the trust has to come from outside the system being restored. For software and firmware, that’s the supplier’s release, the artifact as it was built and signed, checked against a key you pinned before anything went wrong. For configuration, it’s the version somebody approved, kept somewhere other than the machine that runs it. You still need the backup for the data, since that’s the one thing you can’t get anywhere else. A rebuild done this way installs the release fresh, applies the approved configuration and restores the data last. Only that last step depends on a backup, and the data still needs checking, because a directory or a database can carry an intruder’s accounts as easily as its records.

We’d worry most about configuration. Software at least comes from a supplier with some packaging discipline. Configuration gets edited in place on a running system and exported from it later, so often the only copy is a snapshot of a machine you’re no longer sure about. The bulletin lists configurations and instructions right beside software, and they need the same handling, a reviewed and versioned copy that’s signed when someone approves it.

Checking a copy with the network gone

The check has to work isolated, with only what’s already on the shelf, so anything in it that calls out will fail, whether that’s a vendor’s verification service or a licence server. Looking something up in a transparency log online fails too, though an inclusion proof shipped with the artifact still checks offline. Live revocation checks are gone for the duration, which means you won’t hear about a key revoked mid-isolation, possibly because of the incident you’re recovering from. Put the latest revocation list on the shelf and write down its date.

Three months is also long enough for the calendar to break things. Signing certificates expire and short-lived licences lapse. A check that starts failing in week six will either stop the rebuild or teach somebody to skip it. A signature with a trusted timestamp can still verify after its certificate expires, if the verifier honours the timestamp and has the timestamp authority’s chain on the shelf, and trust roots with their own expiry dates need the same look. Licences for anything you might rebuild offline should be the kind that don’t phone home. Ask suppliers about both while the network’s still up.

Then there’s the checker, which sits on the same shelf. If the supplier ships its own verifier, check that separately, against its own provenance and a key you pinned for it, or it’s vouching for itself. Stock tools you already trust skip that step.

Pull the cable

The bulletin asks operators to test how systems “will be restored and checked before they are reconnected”,⁴ and CIREN lists “patch and configuration verification” among the checks to run before reconnecting.⁵ The cheapest test is a rebuild with the network physically disconnected, and it’s where the hidden dependencies show up: an install script that fetches one more package, a container build that pulls its base image, a time sync, a licence check, a DNS name somebody hard-coded years ago. Each of those is a switch somebody else holds, and it’s better to find them on a Tuesday afternoon with the cable out than in week two of an isolation.

Classified networks start from the assumption that nothing calls out, and software crosses into them on purpose, checked on the way in. CIREN is asking critical infrastructure to be able to live like that for three months.

None of this makes a copy safe. A release that was bad when the supplier built it passes every check here, and the last note covered what rebuilding from source does and doesn’t catch. Keep a record of what went where during the rebuild, so there’s somewhere to start when something turns up.

Before you need it

If you run something CIREN covers, or ship software into something that does, pick one system you’d need in the first week and rebuild it with the cable out. Whatever breaks goes on a list, next to what’s on the shelf for that system, what checks it, which key the check trusts and where that key lives. That list takes a while to fill in.

Northfleet is a Canadian-incorporated vendor building the sovereign supply chain that wraps a customer-operated classified cluster: a deploy-time bundle protocol, attestation chain, and tamper-evident audit trail. The customer’s cleared engineering teams operate the cluster under their own keys. Northfleet holds no customer data and no customer signing keys. Every release Northfleet ships carries provenance binding it to the source it was built from, which the customer can check without Northfleet’s participation. The architecture assumes the vendor can be compromised or compelled, so compelling Northfleet doesn’t produce a silent backdoor, though catching one depends on someone actually checking releases against their source.

If part of what you’d have to rebuild runs on Kubernetes inside a classified boundary, talk to us.