Day 4 - AWS CLI and SDK: From Credential Chaining to SigV4

💡 Related theory: The credential provider chain is essentially the Chain of Responsibility pattern (GoF, 1994). Each provider checks "can I handle this?" and, if not, passes to the next. Thanks to this pattern, a new provider (e.g., Pod Identity) can be slotted into the chain without changing existing code. If you look at credentials.py in the boto3 source, the CredentialResolver class implements exactly this pattern.

Named Profiles: Handling Multiple Accounts

In practice, almost every developer works with multiple AWS accounts simultaneously. Named profiles in ~/.aws/config and ~/.aws/credentials solve this.

source_profile is the key. When you use the dev profile, the CLI first uses the default profile's credentials to call sts:AssumeRole, then makes the real API calls with the returned temporary credentials. If mfa_serial is present, a token code is required, so you can enforce MFA on dangerous accounts like prod.

⚠️ Trap: The access keys in ~/.aws/credentials are stored in plain text. If a laptop without disk encryption is lost, that's a straight-up key leak. AWS strongly recommends IAM Identity Center (SSO) + aws configure sso instead of IAM User access keys. SSO stores credentials encrypted in the OS keyring (macOS Keychain, Windows Credential Manager, Linux libsecret), and when the token expires, it prompts re-authentication via the browser.

📚 Case study: In the 2019 Capital One incident, the attacker stole the IAM Role's temporary keys from EC2 metadata — but had permanent keys from a laptop's ~/.aws/credentials been there instead, they would have been discovered months later, unrotated. The 2021 Twitch source code leak also had .aws/credentials pushed to GitHub as one of its causes. AWS has since partnered with GitHub so that Push Protection automatically blocks pushes containing AKIA-pattern keys.

SDK Retry Behavior: AIMD and Exponential Backoff

When an API call returns throttling or a transient error, the SDK retries automatically. This retry algorithm doesn't appear directly on the exam, but it underlies the answers to scenarios like DynamoDB ProvisionedThroughputExceededException, Lambda 429s, and S3 SlowDown.

Standard mode uses exponential backoff with jitter. The first retry waits 0-1 seconds, the second 0-2 seconds, the third 0-4 seconds, at random. Without jitter, all clients retry at the same moment and get throttled again all at once — the "thundering herd" problem. This algorithm is detailed in the AWS Architecture Blog's Exponential Backoff and Jitter.

Adaptive mode uses a token bucket algorithm for the client to dynamically limit its own call rate. It slows down preemptively before being throttled by the server. The downside is throughput can be constrained, so standard is the default.

💡 Related theory: AIMD (Additive Increase, Multiplicative Decrease) is an algorithm originating in TCP congestion control (Jacobson 1988). Increase throughput slowly, but cut it quickly on detecting loss. The AWS SDK's adaptive retry applies a similar philosophy to client-side rate limiting. RFC 7567 (IETF AQM Working Group, 2015) covers standardization work on similar mechanisms. Even more interesting is that the fact that omitting jitter re-creates the "thundering herd" was reconfirmed in datacenter traffic analysis [Polly Vavilala et al., SIGCOMM 2017].

🔍 Going deeper: SDK retry decisions hinge on HTTP status codes and error codes. (1) 5xx, 429, 502, 503, 504 are retried. (2) ThrottlingException, Throttling, RequestLimitExceeded, RequestThrottled, ProvisionedThroughputExceededException are retried. (3) Among 4xx, 400 BadRequest, 403 AccessDenied, and 404 NotFound are not retried (permanent errors). However, some 4xx errors are retryable, like DynamoDB's TransactionConflictException, so per-service exceptions exist. The full list of retryable errors is in retries/special.py in the boto3 source.

⚠️ Trap: Leaving SDK retries as-is inside a Lambda function collides with the function timeout. Inside a Lambda with the default 3-second timeout, if DynamoDB returns throttling, the SDK waits up to 0-1s + 0-2s = 3 seconds and Lambda dies first. In such cases, the right approach is to disable SDK retries with AWS_MAX_ATTEMPTS=1 and retry at the function's caller (Step Functions, EventBridge Pipes, SQS DLQ).

SigV4: The Signature on Every API Call

Almost all AWS API calls go through SigV4 signing (exceptions: query-string signing for presigned URLs, IoT's MQTT, etc.). If you don't know how SigV4 works, you can't debug things like "why does a timestamp off by more than 15 minutes produce a 403?" or "why is this presigned URL only valid for 5 minutes?".

SigV4 signing steps:
1. Build the Canonical Request
   - HTTP method, canonical URI, canonical query string
   - canonical headers, signed headers, body hash (SHA256)

2. Build the String to Sign
   - "AWS4-HMAC-SHA256"
   - timestamp (X-Amz-Date)
   - credential scope (date/region/service/aws4_request)
   - SHA256(Canonical Request)

3. Derive the Signing Key (5-stage HMAC)
   kDate    = HMAC("AWS4" + SecretAccessKey, Date)
   kRegion  = HMAC(kDate, Region)
   kService = HMAC(kRegion, Service)
   kSigning = HMAC(kService, "aws4_request")
   
4. Final signature = HMAC-SHA256(kSigning, String to Sign)

5. Include it in the Authorization header, or embed it in a presigned URL's query string

🔍 Going deeper: SigV4's 5-stage key derivation is a deliberate security design. Separating kDate, kRegion, and so on lets you create and use intermediate keys "valid only for this date, this region, this service" without exposing the SecretAccessKey. A pattern of handing such a derived key to an external service (e.g., CloudFront → Lambda@Edge) when delegating partial permissions becomes possible. The 15-minute timestamp skew limit prevents replay attacks — even if an attacker intercepts the packet, it's useless after 15 minutes. A client clock drifting off NTP is the cause of the commonly seen SignatureDoesNotMatch error.

📚 Case study: In late 2024, AWS released SigV4a (Signature Version 4 Asymmetric). Based on ECDSA asymmetric signing, the same signature is simultaneously valid against multiple regional endpoints. It lets Multi-Region Access Points (S3) and cross-region traffic be handled in one shot without regenerating the signature per region. SigV4 was HMAC symmetric-key based, requiring a different signature per region — a limitation ECDSA resolved.

💡 Related theory: HMAC is a message authentication code defined in RFC 2104 (Krawczyk et al., 1997). Its structure, HMAC(K, m) = H((K ⊕ opad) || H((K ⊕ ipad) || m)), is safe against the length extension attacks that plague naive hash chaining. The reason SigV4 uses a 5-stage HMAC chain is that it is the standard pattern for a key derivation function (KDF), providing security properties similar to NIST SP 800-108 (KDF in counter mode).

Presigned URLs: Packaging Temporary Permission into a URL

An S3 presigned URL is a time-limited link the SDK creates by embedding a SigV4 signature into the URL's query string. The recipient can access the object with just that URL, no AWS credentials required.

The key constraint: a presigned URL's validity cannot exceed the validity of the issuer's credentials. If credentials issued via an EC2 instance profile expire after 1 hour, then even with ExpiresIn=86400 (24 hours), the URL dies after 1 hour. A staple exam question.

⚠️ Trap: If the issuer's IAM permissions are reduced after the presigned URL is created, the URL is invalidated even before it expires. The SDK signs with only the permissions at issuance time, but at actual usage time AWS also evaluates the current permissions. So the debugging scenario "we revoked permissions right after generating a presigned URL and the URL suddenly died" is entirely possible.

CloudShell: EC2 Inside the Browser

CloudShell is a browser-based shell launched in December 2020. You can immediately type aws commands from the console without spinning up a separate EC2. Internally, an Amazon Linux 2-based container is isolated per user, and credentials are automatically inherited from the console login session.

CloudShell's credentials are temporary credentials derived from the console session, so they expire just like an IAM Role's. That's why typing aws sts get-caller-identity shows an assumed-role ARN. CloudShell rarely appears on the exam, but in practice it's very handy for "running one urgent SQL line".

Debugging: What Broke, and Where

The golden tool for CLI/SDK debugging is the --debug flag.

Running this shows (1) which credential provider the credentials came from, (2) which endpoint was called, (3) the full canonical request of the SigV4 signature, and (4) the HTTP response code and headers. In boto3, boto3.set_stream_logger('', logging.DEBUG) yields the same information.

The most common debugging scenario in practice is "why did the credentials get resolved differently than expected?". Find a line like Found credentials in environment variables. or Found credentials in shared credentials file. in the --debug output and you immediately know which provider won. Next, verify the region resolution with the Endpoint: https://s3.ap-northeast-2.amazonaws.com line. Finally, capturing the StringToSign: block and reproducing the same signature in your own code helps when implementing SigV4 directly outside the SDK.

SDK Comparison: Key Differences by Language

Compared to v1, the v2/v3 SDKs share (1) async-first design, (2) module separation (import only the clients you need), and (3) a middleware system (retry, signing, logging as chainable handlers).

The JavaScript SDK v2 → v3 migration is a particularly big issue: v3 split npm packages per client (@aws-sdk/client-s3, @aws-sdk/client-dynamodb, etc.), and with tree shaking, Lambda package sizes dropped by more than half. v2 imported every service at once, directly impacting Node.js Lambda cold starts, whereas v3 pulls in only the needed clients, so cold-start reductions of 30-50ms are typical.

💡 Memorization tip: SDK version labels follow a confusing pattern. Python → boto3 is v1.x (boto was v1, and boto3 effectively plays the v2 role), for JavaScript, v3 is the modular one, for Java/Go/.NET, v2 is the latest. When AWS official docs say "SDK v2", it's not JavaScript — it's the other languages.

Wrapping Up

Today's picture is that the CLI and SDKs ultimately operate on the same mechanisms — the credential provider chain, region resolution, SigV4 signing, and retries. The patterns of handling multiple accounts with named profiles, using SSO without long-term keys via IAM Identity Center, and tracing the flow with --debug work identically across every SDK.

In the next article, we consolidate this Week 1 material into exam questions and pinpoint weak spots.