Configuring mTLS Between Registration and Print Services

Symptom Statement Link to this section#

The print service listens over TLS, so traffic is encrypted — but it never checks who is on the other end of the socket. Any process that can reach the port and speak TLS can submit a print job, which means a laptop on the event LAN, a misconfigured test harness, or a spoofed client can inject badges that were never registered and never paid for. The encryption gives a false sense of safety: one-way TLS authenticates the server to the client, not the client to the server, so the print segment trusts every caller equally. The mirror-image symptom shows up after a certificate renewal: the print service suddenly refuses connections from the registration service with certificate verify failed or tlsv1 alert unknown ca, and jobs pile up in the outbound queue because the two ends no longer agree on trust. Both symptoms are the same missing control — mutual authentication — seen from opposite sides. This page is the service-to-service implementation of the Security Boundary Configuration stage, extending its deny-by-default posture from the HTTP ingress down to the transport between registration and print. Once mTLS is in place, the downstream failover and queue mechanics it protects live in on-site print failover and print queue orchestration.

Root Cause Analysis Link to this section#

The “anyone can print” and “renewal broke the link” symptoms share a small set of concrete causes. Identify which before editing any TLS config.

• No client-certificate verification. The server’s SSLContext runs with verify_mode=CERT_NONE (the default for a server context), so it completes a handshake with any client that offers a valid cipher. Encryption is on; authentication of the caller is off.
• Missing or wrong CA trust chain. The server has no CA bundle loaded (load_verify_locations was never called), or it trusts the public system CA store instead of your private issuing CA. Legitimate client certs signed by your internal CA are then treated as untrusted, and after a renewal that re-chains to a new intermediate, every client fails with unknown ca.
• Certificate or SAN mismatch. The client presents a cert whose subject alternative name does not match what the server pins, or the server’s hostname does not match its cert. Verification technically passes the chain but fails identity, so the connection is refused even though both certs are individually valid.
• Expired client certificate. Short-lived service certs lapse and no one rotated them; the handshake fails closed with certificate expired. This is the single most common renewal-time outage and looks identical to a trust-chain break until you read the alert.
• No certificate rotation process. Certs are issued once by hand and never tracked, so expiry is a surprise and rotation is a manual scramble that itself drops connections — the transport-layer version of the same hard-cutover problem that afflicts shared-secret rotation.

Symptom-to-Resolution Matrix Link to this section#

Server accepts any client (no mutual auth) Link to this section#

Symptoms

• Any host on the event network can POST a print job and get a badge produced.
• Packet captures show a TLS handshake with a CertificateRequest absent — the server never asks the client for a cert.

Root cause. The server context uses verify_mode=CERT_NONE, so it authenticates itself to clients but never authenticates callers.

Fix

1. Build the server context with ssl.create_default_context(ssl.Purpose.CLIENT_AUTH) and set context.verify_mode = ssl.CERT_REQUIRED.
2. Load your private issuing CA with context.load_verify_locations(cafile="ca.pem") so only certs chaining to it are accepted.
3. Reject the connection when no client cert is presented — CERT_REQUIRED does this at the handshake, before any job bytes are read.

Trust-chain / unknown-CA failures after renewal Link to this section#

Symptoms

• After a cert renewal the print server rejects the registration client with tlsv1 alert unknown ca.
• The client cert is valid but the server does not recognize its issuer.

Root cause. The server’s CA bundle does not include the new issuing (or intermediate) CA the renewed cert chains to.

Fix

1. Distribute the full chain — leaf plus any intermediates — to the client, and the issuing CA to the server’s verify locations.
2. When rotating the CA itself, load both the old and new CA into the server’s trust store during an overlap window so in-flight clients on either chain verify.
3. Confirm the chain with openssl verify -CAfile ca.pem client.pem before deploying.

SAN / identity mismatch Link to this section#

Symptoms

• The chain verifies but the connection is refused with a hostname or identity error.
• Works with check_hostname=False, fails when identity is enforced.

Root cause. The presented certificate’s SAN does not match the expected service identity, or the server cert’s SAN does not match the address the client dials.

Fix

1. Issue certs with an explicit SAN matching the service name each side dials, not just a legacy Common Name.
2. On the client, keep check_hostname=True and connect using the exact SAN value via wrap_socket(..., server_hostname=...).
3. On the server, read the peer cert with getpeercert() and assert the client SAN belongs to your allowlisted service identities before dispatching the job.

Expired client certificate Link to this section#

Symptoms

• Handshakes fail with certificate expired starting at a precise timestamp.
• A freshly issued cert restores service immediately.

Root cause. A short-lived client cert lapsed with no rotation in place.

Fix

1. Reissue the client cert and deploy it before the old one expires, overlapping validity windows.
2. Alert on notAfter well ahead of expiry rather than discovering it at handshake time.
3. Automate issuance (an internal CA or ACME-style renewal) so expiry stops being a manual event.

Minimal Working Implementation Link to this section#

A self-contained mTLS harness using only the standard library ssl module. build_server_context creates a CLIENT_AUTH context with verify_mode=CERT_REQUIRED and a loaded CA, so the print server refuses any client whose cert does not chain to the private CA. build_client_context loads the registration client’s own cert and key and the CA it uses to verify the server. The verification block performs a real in-process TLS handshake over a socket pair and asserts that the server sees the client’s certificate — proof that mutual authentication actually happened rather than being merely configured. Certificates are generated at runtime so the block runs anywhere.

import ssl
import socket
import threading
import datetime

from cryptography import x509
from cryptography.x509.oid import NameOID
from cryptography.hazmat.primitives import hashes, serialization
from cryptography.hazmat.primitives.asymmetric import rsa


def build_server_context(certfile: str, keyfile: str, cafile: str) -> ssl.SSLContext:
    """Print-server context: authenticate the CLIENT, deny by default."""
    ctx = ssl.create_default_context(ssl.Purpose.CLIENT_AUTH)
    ctx.load_cert_chain(certfile=certfile, keyfile=keyfile)
    ctx.load_verify_locations(cafile=cafile)
    ctx.verify_mode = ssl.CERT_REQUIRED  # no client cert => handshake fails
    return ctx


def build_client_context(certfile: str, keyfile: str, cafile: str) -> ssl.SSLContext:
    """Registration-client context: present a cert AND verify the server."""
    ctx = ssl.create_default_context(ssl.Purpose.SERVER_AUTH, cafile=cafile)
    ctx.load_cert_chain(certfile=certfile, keyfile=keyfile)
    ctx.check_hostname = True
    return ctx


def _issue(common_name: str, ca=None) -> tuple:
    """Return (key, name, cert). Pass ca=(key, name, cert) to sign a leaf."""
    key = rsa.generate_private_key(public_exponent=65537, key_size=2048)
    name = x509.Name([x509.NameAttribute(NameOID.COMMON_NAME, common_name)])
    issuer = ca[1] if ca else name
    signing_key = ca[0] if ca else key
    is_ca = ca is None
    now = datetime.datetime.now(datetime.timezone.utc)
    ski = x509.SubjectKeyIdentifier.from_public_key(key.public_key())
    builder = (
        x509.CertificateBuilder()
        .subject_name(name).issuer_name(issuer)
        .public_key(key.public_key())
        .serial_number(x509.random_serial_number())
        .not_valid_before(now).not_valid_after(now + datetime.timedelta(days=1))
        .add_extension(x509.SubjectAlternativeName([x509.DNSName(common_name)]), critical=False)
        .add_extension(x509.BasicConstraints(ca=is_ca, path_length=None), critical=True)
        .add_extension(ski, critical=False)
    )
    if is_ca:  # a CA needs certificate-signing key usage
        builder = builder.add_extension(
            x509.KeyUsage(
                digital_signature=False, key_cert_sign=True, crl_sign=True,
                content_commitment=False, key_encipherment=False, data_encipherment=False,
                key_agreement=False, encipher_only=False, decipher_only=False,
            ),
            critical=True,
        )
    else:  # leaf: usable for both server and client TLS auth, chained to the CA
        ca_ski = ca[2].extensions.get_extension_for_class(x509.SubjectKeyIdentifier).value
        builder = builder.add_extension(
            x509.AuthorityKeyIdentifier.from_issuer_subject_key_identifier(ca_ski),
            critical=False,
        ).add_extension(
            x509.ExtendedKeyUsage(
                [x509.oid.ExtendedKeyUsageOID.SERVER_AUTH,
                 x509.oid.ExtendedKeyUsageOID.CLIENT_AUTH]
            ),
            critical=False,
        )
    cert = builder.sign(signing_key, hashes.SHA256())
    return key, name, cert


def _write(path: str, key=None, cert=None) -> None:
    data = b""
    if cert is not None:
        data += cert.public_bytes(serialization.Encoding.PEM)
    if key is not None:
        data += key.private_bytes(
            serialization.Encoding.PEM,
            serialization.PrivateFormat.TraditionalOpenSSL,
            serialization.NoEncryption(),
        )
    with open(path, "wb") as fh:
        fh.write(data)


# --- Verification: a real mTLS handshake proves the client is authenticated ---
if __name__ == "__main__":
    import tempfile, os
    d = tempfile.mkdtemp()
    ca = _issue("print-ca")
    ca_cert = ca[2]
    s_key, _, s_cert = _issue("localhost", ca=ca)
    c_key, _, c_cert = _issue("registration.svc", ca=ca)
    paths = {n: os.path.join(d, n) for n in ("ca", "s", "c")}
    _write(paths["ca"], cert=ca_cert)
    _write(paths["s"], key=s_key, cert=s_cert)
    _write(paths["c"], key=c_key, cert=c_cert)

    server_ctx = build_server_context(paths["s"], paths["s"], paths["ca"])
    client_ctx = build_client_context(paths["c"], paths["c"], paths["ca"])

    lsock = socket.socket()
    lsock.bind(("localhost", 0))
    lsock.listen(1)
    port = lsock.getsockname()[1]
    peer_cn: dict = {}

    def serve():
        raw, _ = lsock.accept()
        with server_ctx.wrap_socket(raw, server_side=True) as tls:
            cert = tls.getpeercert()  # populated only under CERT_REQUIRED
            peer_cn["subject"] = cert
            tls.send(b"job-accepted")

    t = threading.Thread(target=serve)
    t.start()
    with socket.create_connection(("localhost", port)) as raw:
        with client_ctx.wrap_socket(raw, server_hostname="localhost") as tls:
            assert tls.recv(64) == b"job-accepted"
    t.join()

    assert peer_cn["subject"] is not None  # server actually saw a client cert
    assert server_ctx.verify_mode == ssl.CERT_REQUIRED
    print("OK: mutual TLS handshake verified the client certificate")

The assert on getpeercert() returning a non-empty structure is the load-bearing check: under one-way TLS the server would see None, so a passing assert proves the print server genuinely authenticated the registration client and would have rejected a certless spoofer at the handshake.

Memory & Performance Constraints Link to this section#

mTLS adds a bounded, one-time cost per connection; the constraints that bite are handshake CPU and trust-store freshness, not steady-state throughput.

Incident Triage & Rollback Link to this section#

Fast path when print jobs stop flowing after a cert change — target under fifteen minutes. Read the TLS alert first; it names the cause.

1. Classify the alert. Tail the print server: journalctl -u print-svc --since '-5min' | grep -iE 'alert|verify'. unknown ca is a trust-chain break, certificate expired is a lapsed cert, handshake failure with a SAN error is identity mismatch.
2. Inspect the presented cert. From the registration host, dump what the server sees: openssl s_client -connect print-svc:8443 -cert client.pem -key client.key -CAfile ca.pem </dev/null. Read the Verify return code line.
3. Check expiry directly. openssl x509 -in client.pem -noout -enddate — if notAfter is in the past, reissue and deploy the client cert before touching anything else.
4. Re-widen trust if the CA rotated. Load both old and new CA into the server verify store and reload:

cat ca-old.pem ca-new.pem > ca-bundle.pem && systemctl reload print-svc

Rollback. The safe rollback is to restore the previous cert/CA pair the two services last agreed on, never to disable verification. Do not “fix” an outage by setting verify_mode=CERT_NONE — that reopens the injection hole this control exists to close. Overlap the old and new trust anchors instead, then retire the old one once traffic on the new chain is confirmed.

Post-rollback validation. Confirm a clean mutual handshake and that the peer cert is seen:

openssl s_client -connect print-svc:8443 -cert client.pem -key client.key \
  -CAfile ca.pem </dev/null 2>&1 | grep 'Verify return code: 0 (ok)'

Frequently Asked Questions Link to this section#

Isn’t regular TLS enough since the traffic is already encrypted? No. One-way TLS authenticates the server to the client, so the registration side knows it is talking to the real print service — but the print service still accepts any client that completes a handshake. Encryption protects the bytes in transit; it says nothing about who is allowed to submit a job. Only mutual TLS, with CERT_REQUIRED and a loaded CA, authenticates the caller and closes the injection path.

How is rotating a client certificate different from rotating a shared secret? The discipline is the same — overlap the old and new so neither end is ever stranded — but the moving parts differ. With certs you overlap validity windows and, when the CA changes, overlap trust anchors by loading both CAs on the server. Deploy the new client cert before the old expires, confirm traffic on the new chain, then retire the old trust anchor.

Should I pin the client’s SAN, or is a valid CA chain enough? Pin the SAN. A valid chain only proves the cert was issued by your CA; it does not prove which service is calling. Read getpeercert() after the handshake and assert the client SAN is in your allowlist of print-submitting identities, so a leaked-but-valid cert for an unrelated internal service still cannot inject print jobs.

Related Link to this section#

• Security Boundary Configuration — the parent stage whose deny-by-default perimeter this mTLS layer extends from the HTTP ingress down to service-to-service transport.
• On-Site Print Failover — the downstream stage that reroutes jobs to a backup printer; its failover peers must present the same client certs this gate requires.
• Print Queue Orchestration — the stage that dispatches authenticated jobs across printers once mutual TLS has established which callers are trusted.
• Rotating HMAC Webhook Signing Secrets Without Dropping Events — a sibling guide applying the same overlap-window discipline to shared signing secrets instead of certificates.