Integrating Certificate Transparency Validation into TLS Analysis - ScanMalware Blog

Introduction

Since 2018, Google Chrome has required all publicly trusted TLS certificates to be logged in Certificate Transparency (CT) logs. This requirement has fundamentally changed how we verify certificate legitimacy and detect mis-issuance.

In this post, we'll explore what Certificate Transparency is, why it matters, and how we implemented CT validation in ScanMalware's TLS analyzer.

What is Certificate Transparency?

Certificate Transparency (RFC 6962) is an open framework for monitoring and auditing TLS certificates. It creates publicly auditable, append-only logs of all certificates issued by participating CAs.

The Problem CT Solves

Before CT, if a CA mistakenly (or maliciously) issued a certificate for your domain, you might not know until attackers used it:

Historical incidents:

DigiNotar (2011): Compromised CA issued fraudulent certificates for Google, detected only after widespread use
Comodo (2011): Unauthorized certificates issued for major sites
TURKTRUST (2013): Accidentally issued intermediate CA certificates to customers

The impact: Man-in-the-middle attacks, data theft, loss of trust in the CA system

How CT Works

Certificate issuance: CA wants to issue a certificate
Submission to logs: CA submits certificate to multiple CT logs
Signed Certificate Timestamp (SCT): Log returns a cryptographically signed promise to include the certificate
Certificate delivery: CA includes SCTs in the certificate or delivers them separately
Browser validation: Browser checks for valid SCTs before trusting the certificate
Public monitoring: Anyone can query CT logs to find certificates for their domains

SCT Delivery Methods

SCTs can be delivered in three ways:

1. Embedded in Certificate (X.509 Extension)

The SCT is embedded directly in the certificate during issuance.

Advantages:

Always available with the certificate
No additional network requests
Most reliable method

Example OID: 1.3.6.1.4.1.11129.2.4.2

2. TLS Extension

The server sends SCTs during the TLS handshake.

Advantages:

Can update SCTs without reissuing certificate
Useful for OCSP stapling scenarios

Disadvantages:

Server must support TLS extension
Additional handshake data

3. OCSP Stapling

SCTs are included in the OCSP response.

Advantages:

Works with existing OCSP infrastructure
Can be updated independently

Disadvantages:

Depends on OCSP responder availability
More complex validation

Browser Requirements

Google Chrome

Requirement (since April 2018):

All publicly trusted certificates must have SCTs from CT logs
Certificates without valid SCTs show security errors

Policy details:

Minimum 2 SCTs from different log operators
Logs must be Google-approved and monitored

What happens without SCTs?

This site can't provide a secure connection
Certificate Transparency required
ERR_CERTIFICATE_TRANSPARENCY_REQUIRED

Apple Safari

Requirement (since macOS 10.13, iOS 11):

Similar to Chrome, requires CT for all publicly trusted certificates
Enforces minimum number of SCTs based on certificate lifetime

Mozilla Firefox

Current status:

CT support present but not enforced by default
Working toward enforcement in future versions
Can verify SCTs when present

Our Implementation

We implemented comprehensive CT validation in the TLS analyzer.

SCT Extraction

Our implementation extracts SCTs from the X.509 certificate extension (OID 1.3.6.1.4.1.11129.2.4.2). The SCT data uses a specific binary format with a length prefix, followed by individual SCT entries. Each SCT is parsed to extract the log ID, timestamp, and cryptographic signature.

SCT Validation

We validate each SCT by:

Extracting log ID: Identifies which CT log issued the SCT
Verifying timestamp: Ensures the timestamp is reasonable
Checking signature: Cryptographically validates the SCT signature
Log lookup: Verifies the log is recognized and trusted

Fingerprint Calculation

Critical implementation detail: We initially used SHA1 fingerprints but discovered CT logs use SHA256. Using the wrong hash algorithm meant our Elasticsearch queries wouldn't match CT log entries. This was a subtle bug that affected our ability to verify certificates against the public CT logs and was fixed in our latest update.

Risk Scoring Integration

We integrate CT validation into our risk assessment:

CT Compliance (✅ Good)

Conditions:

Certificate has 2+ SCTs from different logs
All SCTs are from recognized CT logs
Timestamps are valid

Risk Score: -5 points (security bonus)

Missing CT (⚠️ Warning)

Conditions:

Certificate has 0 SCTs
Modern browser requirement not met

Risk Score: +15 points (medium severity)

Impact:

Chrome/Safari will reject the certificate
Users see security warnings
Site may be inaccessible

Insufficient SCTs (⚠️ Warning)

Conditions:

Certificate has 1 SCT (needs 2+)
Doesn't meet browser requirements

Risk Score: +10 points

Real-World Testing Results

GitHub.com (Compliant)

{
  "has_scts": true,
  "sct_count": 2,
  "ct_compliant": true,
  "scts": [
    {
      "log_id": "7s3QZNXbGs7FXLedtM0TojKHRny87N7DUUhZRnEftZs=",
      "timestamp": "2024-02-01T12:34:56.789Z",
      "log_operator": "Google 'Argon2024' log"
    },
    {
      "log_id": "SLDja9qmRzQP5WoC+p0w6xxSActW3SyB2bu/qznYhHM=",
      "timestamp": "2024-02-01T12:34:57.123Z",
      "log_operator": "Cloudflare 'Nimbus2024' Log"
    }
  ],
  "risk_score_impact": -5
}

Analysis: GitHub's certificate is fully compliant with CT requirements, has SCTs from two different log operators (Google and Cloudflare), and earns a security bonus.

Non-Compliant Certificate Example

{
  "has_scts": false,
  "sct_count": 0,
  "ct_compliant": false,
  "scts": [],
  "findings": [
    {
      "severity": "medium",
      "finding": "Missing Certificate Transparency",
      "description": "Certificate has no SCTs. Modern browsers require CT compliance for publicly trusted certificates.",
      "remediation": "Reissue certificate from a CA that supports Certificate Transparency",
      "impact": "Chrome and Safari will reject this certificate with ERR_CERTIFICATE_TRANSPARENCY_REQUIRED"
    }
  ],
  "risk_score_impact": 15
}

Elasticsearch Integration

We store CT data in Elasticsearch for advanced searches:

Index Structure

Index: ct-domains-*
Fields:
- d (text): Domain name
- d.keyword: Exact domain match
- t (date): Certificate timestamp
- h (keyword): SHA256 fingerprint
- s (text): Subject Alternative Names

Example Queries

Find all certificates for a domain:

GET /ct-domains-*/_search
{
  "query": {
    "term": {
      "d.keyword": "github.com"
    }
  }
}

Find certificates by fingerprint:

GET /ct-domains-*/_search
{
  "query": {
    "term": {
      "h": "ABC123...SHA256_HASH"
    }
  }
}

Benefits of Certificate Transparency

For Domain Owners

Detect unauthorized certificates: Monitor CT logs for unexpected certificates for your domains
Incident response: Quick discovery of CA compromise or mis-issuance
Compliance verification: Ensure your CAs are following proper procedures

For Security Researchers

Threat intelligence: Track certificate patterns for phishing detection
CA monitoring: Audit CA practices and reliability
Historical analysis: Study certificate issuance trends

For the Ecosystem

Accountability: CAs can't issue certificates secretly
Rapid detection: Mis-issuance discovered within hours, not months
Public trust: Transparent audit trail increases confidence

Monitoring Your Domains

Several services monitor CT logs for new certificates:

crt.sh: Free CT log search and monitoring
Facebook CT Monitoring: Free alerts for your domains
Censys: Commercial CT monitoring with advanced features
ScanMalware: Validates CT compliance as part of TLS analysis

Common Issues and Fixes

Issue 1: Missing SCTs

Symptom: Certificate works in Firefox but fails in Chrome/Safari

Cause: Certificate issued before 2018 or CA doesn't support CT

Fix: Reissue certificate from CT-compliant CA (Let's Encrypt, DigiCert, etc.)

Issue 2: Insufficient SCTs

Symptom: Certificate has 1 SCT but browser rejects it

Cause: Browsers require minimum 2 SCTs from different log operators

Fix: Reissue certificate or contact CA to provide additional SCTs

Issue 3: Revoked CT Log

Symptom: Certificate has SCTs but browser rejects them

Cause: CT log was disqualified or removed from browser trust list

Fix: Reissue certificate with SCTs from currently trusted logs

Future of Certificate Transparency

Shorter Log Lifetimes

Google is moving toward annual log rotation:

New logs every year
Older logs become read-only
Reduces impact of compromised logs

Integration with Other Systems

Abuse detection: Automatic flagging of suspicious certificates
Phishing prevention: Real-time detection of typosquatting certificates
CAA validation: Cross-reference CT data with CAA records

Expanded Coverage

Code signing certificates: Extending CT to software signing
S/MIME certificates: Email certificate transparency
Internal CAs: Optional CT for private certificate authorities

Conclusion

Certificate Transparency has fundamentally improved the security of the web's PKI infrastructure. By making all certificate issuance public and auditable, CT provides early detection of mis-issuance and increases accountability for Certificate Authorities.

ScanMalware's CT validation ensures that scanned sites meet modern browser requirements and helps identify certificates that may cause connection issues for users.

Recommendations

If your scan shows CT compliance issues:

Verify the finding: Test in Chrome/Safari to see if users are affected
Check certificate age: Pre-2018 certificates may not have CT
Reissue if necessary: Get a new certificate from a CT-compliant CA
Monitor your domains: Set up CT log monitoring for unauthorized certificates
Plan for automation: Modern CAs like Let's Encrypt handle CT automatically

Check your certificate's CT compliance at ScanMalware.com - we'll verify your SCTs and ensure browser compatibility!