!-->

ISP Escalation Report: Selective AWS S3 Connectivity Failure

Incident summary

• A specific home WAN path shows persistent HTTPS timeout behaviour to AWS S3 (s3.eu-west-1.amazonaws.com) while other AWS services such as STS remain reachable.
• The issue reproduces consistently from the same endpoint over home WAN and disappears when the same endpoint uses a mobile hotspot path.
• Investigation scope included endpoint stack, local filtering, OPNsense filtering/security engines, and controlled WAN/LAN packet captures.

Business impact

• S3-dependent operations fail (including deployment/upload flows that require S3).
• Partial AWS reachability (e.g., STS success) creates false-positive connectivity assumptions and delays root-cause isolation.

Environment and scope

• Affected endpoint: single Mac endpoint on home network path.
• Firewall/router: OPNsense (with API access validated and used for controlled tests).
• Comparison path: same endpoint over mobile hotspot (5G), where S3 succeeds.
• Service region pattern observed: EU West (eu-west-1) endpoints.

Methodology

1. Validate and remove potential endpoint-level blockers.
2. Perform A/B path testing with same host and same destination endpoint/IP.
3. Apply temporary explicit OPNsense pass rule to rule out rule-order/standard filtering behaviour.
4. Bypass and fully stop Zenarmor to rule out that security engine.
5. Perform fixed-destination (--resolve) captures on OPNsense LAN/WAN for S3 and STS and compare packet progression.

Key findings

1) Endpoint-local causes were ruled out

• Legacy Network Extension artifacts were cleaned up (including historical SonicWall residue).
• Local PF firewall was verified disabled.
• /etc/hosts overrides were not present for AWS endpoints.
• MTU reduction test (to 1280) did not resolve S3 timeouts.
• IPv4/IPv6 variants were tested without resolving S3 issue.

Result: local host stack is unlikely to be root cause.

2) DNS and endpoint randomness were ruled out

• Same destination IP was forced with curl --resolve.
• Outcome remained path-dependent:
  • Success over hotspot path.
  • Timeout over home WAN path.

Result: issue is not explained by DNS variance or random endpoint selection.

3) Standard OPNsense firewall rule path was ruled out

• OPNsense API auth was validated and used to add/apply a targeted temporary LAN pass rule for the affected endpoint on TCP/443.
• Rule was confirmed active in firewall API search results.
• No change in S3 timeout behaviour.

Result: not a normal PF rule-order/pass-list issue.

4) Zenarmor and IDS were ruled out

• Zenarmor bypass was enabled and tested.
• Zenarmor engine was fully stopped and tested.
• No change in S3 timeout behaviour.
• Suricata status check returned not running.

Result: issue did not depend on these local security engines.

Decisive packet-capture evidence

Controlled fixed-destination tests were executed with API-driven OPNsense captures:

• LAN capture filtered to affected client + TCP/443.
• WAN capture filtered to exact tested destination IP + TCP/443.
• Requests made with curl --interface <home-iface> --resolve ....

S3 fixed destination test (failing flow)

• Curl result:
  • TCP connect succeeds.
  • Timeout after ~12s.
• WAN capture metrics:
  • total_packets=3
  • syn=1, synack=1, ack=1
  • push=0
  • payload_bytes=0

Interpretation: TCP 3-way handshake completes, but no TLS/application payload exchange occurs on WAN for S3 flow.

STS fixed destination test (healthy control flow)

• Curl result:
  • Successful TLS/HTTP exchange with HTTP/1.1 302 Found.
• WAN capture metrics:
  • total_packets=47
  • syn=1, synack=1
  • push=19
  • payload_bytes=6702

Interpretation: normal encrypted payload exchange occurs on same WAN path/time window for control endpoint.

Comparative conclusion

• Same client, same local network equipment, same TCP port (443), same time window.
• One AWS endpoint class (STS) carries normal payload.
• S3 reaches only handshake stage and then stalls (no payload on WAN capture).

This strongly indicates selective upstream path handling outside local endpoint and ordinary local firewall policy logic, such as:

• ISP-side traffic classification/policy filtering for specific destination ranges,
• ISP-managed CPE behavior,
• transit/peering path-specific impairment toward S3 ranges.

Request to ISP (technical actions)

Please investigate the subscriber line for selective TCP/443 payload suppression to AWS S3 destination ranges:

1. Inspect edge/BNG/CGNAT logs for subscriber flows during test window.
2. Compare S3 vs STS handling for same source and same destination port.
3. Check for DPI/policy enforcement or anti-abuse controls affecting S3 ranges.
4. Validate transit/peering health and asymmetry for relevant AWS prefixes.
5. Temporarily bypass shaping/classification for this subscriber and re-test.
6. Provide ISP-side packet/log confirmation of where payload packets are dropped or stalled.

Evidence artifacts (collected)

• s3fixed_wan.zip — WAN capture bundle for fixed S3 test.
• stsfixed_wan.zip — WAN capture bundle for fixed STS test.
• s3fixed_lan.zip — LAN capture bundle for fixed S3 test.
• stsfixed_lan.zip — LAN capture bundle for fixed STS test.
• s3fixed_curl.out — verbose curl output for fixed S3 test.
• stsfixed_curl.out — verbose curl output for fixed STS test.

Capture/job references

• S3 LAN capture UUID: 6d34ee46-18e1-4828-8677-c57e34404d16
• S3 WAN capture UUID: e0dd22ad-d1d9-4f7c-8bd5-5bfb7486731f
• STS LAN capture UUID: a12fb3c9-5a22-4d82-b952-eb193843187b
• STS WAN capture UUID: 62881186-8fb3-48d6-bec5-fe0ae33f6edb

Final statement

Based on controlled A/B path tests and WAN packet evidence, this incident is not attributable to endpoint-local configuration or standard local firewall rules. The remaining fault domain is upstream network handling on the ISP path.