标签：#aws

This article provides a step by step comparison of three Gemma 4 deployments on a single AWS hosted GPU enabled system. A suite of Python MCP tools is built to simplify management of each deployment, and one benchmark harness is shared across all three so that the runtime is the only variable. https://github.com/xbill9/gemma4-dev The whole exercise cost under three dollars, and that is the part worth keeping. Nineteen instances and about four and a half instance-hours bought three serving sweeps, nine timed boots and a handful of A/B restarts. It also bought five wrong claims, each caught by measuring instead of reasoning. On hardware where a run is expensive, the cheapest of those five would have shipped with a caveat attached. What is this project trying to Do? Three rigs in this monorepo serve google/gemma-4-E2B-it on an AWS G5g instance. One runs vLLM, one runs a pure JAX port, one runs PyTorch with transformers. The hardware is identical and only the runtime slot moves, so this should be the cleanest A/B available. For months it was not, because each rig measured itself with its own harness and quoted its own number. Three harnesses computing three statistics is not a comparison. Prerequisites An AWS account with G-family quota in us-east-1 . Each g5g.2xlarge is 8 vCPU, so 16 vCPU of spot quota runs two at once. A subnet, a security group opening TCP 8000, and an instance profile carrying AmazonSSMManagedInstanceCore plus read on the Hugging Face token secret. A Hugging Face token in Secrets Manager. It is fetched at boot into a root-only EnvironmentFile and never appears in user data. boto3 and the standard credential chain. No AWS CLI shell-outs, no inbound SSH rule, and no private key anywhere in the flow. AWS EC2 G5g Instance g5g.2xlarge — 8 vCPU, 16 GiB host Host CPU AWS Graviton2, aarch64 GPU 1x NVIDIA T4G, Turing, SM 7.5 GPU memory 15,360 MiB per nvidia-smi ; AWS lists 16,384 nominal G5g is the only family AWS ships that puts an NVIDIA GPU behind a Gravi

To add the AWS AppConfig Agent Lambda extension to an AWS Lambda function, you can open the documentation, scroll through a table of ARNs, find the one that matches your AWS Region and architecture, copy it, and paste it into your template. However, a few months later, AWS publishes a new version and now your deployment is silently using an older one. There’s a better approach that uses public parameters in AWS Systems Manager Parameter Store (Parameter Store). What are public parameters in Parameter Store? Many AWS services use Parameter Store to publish read-only public parameters with names that start with aws/service/{service-name} . These public parameters contain up-to-date metadata about AWS services. You've probably seen them used for AMI lookups for fetching the latest Amazon Linux AMI ID without hardcoding it. The same mechanism is available for Lambda layer ARNs, ECS-optimized AMIs, and other resources that AWS updates regularly. The key idea is instead of looking up a value in documentation and pasting it into your code, you query Parameter Store at deploy time and get the current value. The only IAM permission that's required is ssm:GetParameter . The parameters are public and readable from any AWS account. The problem with hardcoded layer ARNs The AWS AppConfig Agent Lambda extension is distributed as a Lambda layer. To attach it, you need the layer's ARN, which includes a version number at the end: arn:aws:lambda:us-east-1:027255383542:layer:AWS-AppConfig-Extension:128 That version number changes every time AWS releases an update. If you hardcode it, you get a working deployment, but you also get silent drift. A few months from now, you'll be running an older version without realizing it. Imagine a team that's managing dozens of functions in multiple AWS Regions, and you can see how this can become a maintenance problem. Someone has to regularly check the documentation, update the ARN, and redeploy. It's not difficult work, but it's the kind of thing

AWS Lambda vs. Traditional Servers : When Serverless Actually Makes Sense "Serverless" is one of the more misleading names in cloud computing — there are still servers, you just don't manage them directly. AWS Lambda lets you run code in response to events without provisioning or maintaining a server yourself, and it genuinely changes how certain kinds of applications get built. But it's not a universal replacement for traditional servers, and understanding exactly where each one wins is more useful than treating this as "the new way vs. the old way." What Actually Changes With Lambda A traditional server — whether it's a physical machine, a VM, or a container running continuously — is always on, always consuming resources, and always your responsibility to patch, scale, and monitor, whether or not it's actively doing anything useful at a given moment. AWS Lambda flips this model: your code runs only in response to a specific trigger (an HTTP request, a file upload, a scheduled event), runs for as long as it takes to complete, and then stops. You're not paying for idle time, and you're not managing an operating system, patching, or server-level scaling — AWS handles all of that underneath the function itself. Where Lambda Genuinely Wins Event-driven, intermittent workloads — a function that runs occasionally in response to specific events (a file upload triggering image processing, a scheduled nightly job) is often dramatically cheaper on Lambda than paying for a server that sits idle most of the time Automatic scaling with zero configuration — Lambda scales from zero to many concurrent executions automatically, without you provisioning capacity in advance Reduced operational overhead — no patching an operating system, no managing server-level security updates, no capacity planning for a specific function Fast setup for simple, isolated tasks — a single-purpose function can go from idea to deployed in a genuinely short amount of time Where Traditional Servers Still

A lesson in overcorrection, from an AI-assisted builder who is scared stiff of shipping spaghetti and...

Paste a long document into ChatGPT and hit enter. Nothing happens for a second or two. Then the answer starts appearing, word by word, at a steady pace until it finishes. You have seen this hundreds of times. Most people never think about it. But those are two completely different things happening inside one request, running on two different parts of the same GPU, limited by two different bottlenecks. Once you know what they are, a lot of confusing things about serving AI models stop being confusing. Including why a faster GPU sometimes makes no difference at all. This is part two of a series. Part one covered the split between VRAM and cores. This post covers what each of them does when a request arrives. The example we will follow I am going to trace one question the whole way through: what is the capital of france Simple, short, and it produces a short answer. Perfect for watching the machinery. Before anything happens, the model server has already loaded the model's weights into VRAM. That happened at startup and they stay there. Nothing about your request loads a model. Your words get chopped into tokens, which are just pieces of words turned into numbers, and placed in VRAM alongside the weights. Now the work begins. Phase one: prefill, or "the pause" Before the model can write anything, it has to read everything you gave it. Here is the important part, and it is the opposite of what most people assume: it reads your entire prompt at once. Not word by word. All six words go to the cores together, in a single pass. That is possible because your whole prompt is already there. Nothing has to wait for anything else. So thousands of cores fire simultaneously and chew through all of it in one burst. flowchart LR A["what is the capital of france<br/>all 6 tokens together"] --> C W["16 GB of model weights"] --> C C["GPU CORES<br/>every core busy"] --> O1["First word: 'The'"] C --> O2["Notes saved to VRAM<br/>the KV cache"] Two things come out of that single pass: The

Verified students can earn a free $100 AWS certification voucher through Student Rewards. Here is the exact path, the badge tricks, and the mistakes to avoid.

Amazon recently announced Kiro Crew, an open-source system for running multiple Kiro coding agents across sessions, tools, and tasks. The new workspace lets developers assign asynchronous coding tasks to AI agents, allowing work such as incident investigation, ticket triage, migrations, and PR monitoring to continue without active supervision. By Renato Losio

Line 267 of a 427-line Terraform plan: # aws_rds_cluster.reporting must be replaced - /+ resource "aws_rds_cluster" "reporting" { ~ arn = "arn:aws:rds:us-east-1:842910557412:cluster:reporting" - > ( known after apply ) ~ cluster_resource_id = "cluster-D85642F9611A" - > ( known after apply ) ~ engine_version = "14.9" - > "15.4" ~ id = "reporting" - > ( known after apply ) ~ storage_encrypted = false - > true # forces replacement # (29 unchanged attributes hidden) } The merge request says "bump reporting Postgres to 15.4." The plan does exactly that. It also destroys the reporting database and creates an empty one in its place. Underneath the known-after-apply churn, two attributes are changing. One is the version bump, the thing your MR is about. The other is storage_encrypted flipping from false to true , and it isn't yours. Someone on another team that shares this repo merged it earlier in the week. You're just the one deploying. You review other people's Terraform MRs and have a feel for what each stack normally does; most weeks someone else shepherds the deploy. Today it's you. Your change goes out next, so you're carrying everything merged since the last deploy, including work you never reviewed and had no reason to know about. Nobody was negligent. The queue simply had someone else's change in it. It's a good change, by the way. You want encrypted storage. But there's no in-place path from unencrypted to encrypted on an RDS cluster. Terraform's only move is destroy and create. That's what -/+ means, and the comment at the end of the line says it in plain English: forces replacement . And the version bump alone would have failed. Going from 14 to 15 is a major version upgrade, and Aurora refuses those unless the config sets allow_major_version_upgrade = true . This one doesn't. That MR by itself would have died at apply, loudly, with an error naming the exact problem. A replacement doesn't upgrade anything. It creates a new cluster at 15.4 from scratch, so the f

I'm fairly new to SRE/DevOps, and one of the topics I recently spent time studying properly was EKS cluster upgrades . My first instinct, like most people starting out, was: "it's just a version bump, click upgrade in the console, done." That's basically what most beginner blog posts say too. But the more I read and the more I dug into real-world postmortems and discussions, the more I realized — the actual Kubernetes control plane upgrade is the easy part. Almost everything that can go wrong seems to happen around it, not because of it. Sharing what I learned here, mainly for my own notes, but hoping it's useful for anyone else early in their journey too. Learning #1: There's No "Undo" Button This was the first thing that surprised me. I assumed upgrades work like most software — if something breaks, you roll back. But with EKS, you cannot downgrade the control plane version once you upgrade it. So the plan can't be "upgrade, and if it breaks, revert." It has to be "test enough beforehand that breaking isn't really an option," and if something does go wrong, the fix is always moving forward, not backward. That single fact changes how you're supposed to approach the whole thing — testing has to happen before the button is clicked, not after. Learning #2: APIs Get Deprecated, and It's Usually Not Your Own Code That Breaks Kubernetes removes old API versions on a schedule. I already knew this conceptually, but what I didn't realize is that the risk usually isn't your own YAML files — it's the Helm charts and third-party tools you installed a while back and forgot about , which might still be using an older API version internally. There are tools built exactly for catching this before it becomes a problem: pluto detect-helm -owide pluto detect-files -d ./manifests kubent (kube-no-trouble) does something similar. I hadn't heard of either tool before researching this, and it made me realize how much of "being good at Kubernetes" is really just knowing which small tools e

This article provides a step by step deployment guide for serving Google's Gemma 4 on an AWS EC2 G5g instance using pure JAX. The code is here: github.com/xbill9/gemma4-dev What is this project trying to Do? This project aims to serve a modern open model on the cheapest whole CUDA GPU AWS will rent you, and to measure honestly what that costs. Aren't You Using The Wrong GPU? Probably! The T4G is a Turing chip from 2018. It has no bfloat16 and no fp8. But it is cheap, it is available when nothing else is, and it is attached to a Graviton2 host — which makes G5g the rare hardware axis that almost nothing in the ML ecosystem targets: aarch64 and CUDA together . So let's give pure JAX a shot on G5g! AWS EC2 G5g G5g instances pair an AWS Graviton2 (64-bit Arm) processor with NVIDIA T4G Tensor Core GPUs. At g5g.xlarge they are the cheapest EC2 instance carrying a whole NVIDIA GPU , and the only Arm-based GPU family AWS offers. Two GPU instances are cheaper per hour and neither can serve this model (us-east-1, Linux, on-demand, checked against the Pricing API on 2026-08-28): g6f.large at $0.2020 is genuinely NVIDIA and genuinely CUDA — but it is one eighth of a GPU with 3 GB , and the weights alone are 6.155 GB. The first g6f that fits is g6f.4xlarge at $0.9500, which is 1.7x this rig's g5g.2xlarge . g4ad.xlarge at $0.3785 carries an AMD Radeon Pro V520 — no CUDA at any price. Among whole NVIDIA GPUs, G5g is the floor: g5g.xlarge at $0.4200, and the next one up is g4dn.xlarge at $0.5260. More information is available here: https://aws.amazon.com/ec2/instance-types/g5g/ The default in this rig is g5g.2xlarge — 1 GPU, 8 vCPU, 16 GiB RAM. Note- the T4G reports 15,360 MiB of device memory, not the nominal 16 GB. Budget against the measured number. Gemma 4 Gemma is Google's family of open models built from the same research as Gemini. This rig serves google/gemma-4-E2B-it , the instruction-tuned reference release. JAX JAX is Google's array computing library — NumPy semantics, c

--- title : " AI Harness: the worst and the best buzzword in the industry" published : false tags : [ ai , harness , middleware , finops , aws , bedrock , opensource ] series : " TokenOps on AWS" cover_image : # TODO: circuit-breaker / middleware diagram --- AI Harness: the worst and the best buzzword in the industry "El mercado habla de 'AI Harness' como si fuera magia. El verdadero arnés de un LLM es un Proxy Inverso y un Middleware Transaccional determinístico. Es el código tradicional (styrr-llm y sayay-guard) el que confina, audita y presupuesta la inferencia probabilística antes de que toque tu infraestructura en la nube." — TokenOps raw research, Turno 8 The Hook "Harness" is the most polarizing word in AI engineering right now. Depending on who you ask it's either the industry's worst buzzword or the best technical concept ever packaged badly. It's both — and the difference is whether you can name the actual engineering underneath. Why It's the WORST Buzzword (the smoke) It's a wrapper. 90% of the time, "we built an Enterprise AI Harness" means someone wrote a Python requests script or an Express server that wraps the OpenAI or Bedrock API. Language appropriation. "Harness" literally means arnés — a tether. Marketing sells it as "an intelligent structural armor that tames the wild energy of AI." In systems engineering it's a middleware, or a glorified try/catch with JSON schema validation. No standard. No rigorous CS definition exists, so anyone calls anything "harness" — a log interceptor, a proxy, a YAML config file — inflating expectations without delivering real value. Why It's the BEST Buzzword (the engineering) Strip the LinkedIn marketing and the original test harness metaphor becomes genuinely powerful for generative AI: electrical isolation of uncertainty. An LLM is a highly unstable, probabilistic component. You cannot wire it directly into a bank's production database. You need a physical code "harness" that isolates it. When the model goes crazy

California residents have a legal right to access the data that companies collect about them. Actually exercising that right is a burdensome nightmare.

Everybody's cloud cost journey has the same first chapter: someone writes a Lambda that stops the dev instances at night and starts them in the morning. It works. It saves real money. And then the environment grows, and one morning the script that ran fine for a year quietly causes an outage. The shutdown script that works on one instance breaks at three hundred, and it breaks in four specific ways. Here is each one, because knowing them is the difference between saving money and writing a postmortem. The script that works on one instance # stop_dev.py, EventBridge at 20:00 import boto3 ec2 = boto3 . client ( " ec2 " ) ids = [ i [ " InstanceId " ] for r in ec2 . describe_instances ( Filters = [{ " Name " : " tag:env " , " Values " :[ " dev " ]}])[ " Reservations " ] for i in r [ " Instances " ]] ec2 . stop_instances ( InstanceIds = ids ) At small scale this is fine. At scale, here is what goes wrong. Break 1: dependency order Your app instance depends on a database. Stop them in a random order and starting back up, the app comes alive before the database is ready and lands in a crash loop. On one box you get away with it. Across an environment with app tiers, databases, and caches, ordering is not optional: databases up before apps, apps up before the things that call them. A flat list of instance IDs has no concept of "start this after that." Real scheduling needs dependency-aware sequencing (storage, then compute, then application), with delays between tiers. Break 2: timezones The script fires at 20:00. Whose 20:00? As you add teams in different regions, a single UTC cron either shuts down someone's environment in the middle of their afternoon or leaves it running all night. At scale, schedules have to be timezone-aware per environment or per team, not one global time that is wrong for most of the world. Break 3: no overrides, so people disable it The night QA needs staging up late for a release, the script kills it at 20:00 anyway. This happens twice, and then s

CPU rightsizing is a solved, well-documented practice. GPU rightsizing is where the real money is now, and almost nobody writes about it, because GPU instances are expensive enough that people are scared to touch them and unsure how. Given how much a GPU box costs per hour, an over-provisioned one is the single most expensive rightsizing mistake in your account. Here is how to rightsize AWS GPU instances without breaking the workload, including the compatibility check that quietly bites people. Know what each GPU family is for Rightsizing starts with using the right family, not just the right size. On AWS: G5 / G6 (NVIDIA A10G / L4): inference, graphics, smaller training. The workhorses for serving models and lighter ML. Cheaper per hour. P4 / P5 (A100 / H100): large-scale training and heavy inference. The expensive tier, built for jobs that genuinely need the horsepower and interconnect. The most common GPU waste is running a training-class P-family instance for an inference workload that a G-family instance would serve fine at a fraction of the cost. Wrong family is a bigger error than wrong size. Rightsize on the binding resource, and it is usually not CPU GPU workloads have several resources that can be the bottleneck, and CPU utilization, the thing you would check for a normal instance, is often the least relevant: GPU utilization: is the GPU actually busy, or idle between requests? (CloudWatch does not report this by default; you need the CloudWatch agent with GPU metrics or nvidia-smi telemetry.) GPU memory: many inference workloads are GPU-memory-bound, not compute-bound. A model that fits in less VRAM can move to a smaller GPU. Host CPU and RAM: sometimes the GPU is fine but the instance is over-sized on host resources. The rightsizing signal is a GPU sitting at low utilization or using a fraction of its VRAM over a sustained window (a 90-day-style baseline, same idea as CPU rightsizing). That is your candidate to move down a size or across to a cheaper fam

AI’s most fervent and optimistic promoters promise a future where AI is innovating its way out of...

If you've provisioned a VPC from a Terraform module without fully internalising what each piece is doing, that's fine — right up until something breaks. An instance that should be reachable isn't. A private instance can't pull a package update. And you're left checking five different resources with no clear mental model of how they connect. This post builds that mental model from the ground up. Not just definitions — the why behind each piece, so troubleshooting becomes deduction instead of guesswork. CIDR math you actually need A CIDR block is IP address / prefix length . The prefix length fixes the network portion; the remaining bits are your host space. Formula: 2^(32 - prefix) = total addresses . AWS reserves 5 per subnet (network address, VPC router, DNS, reserved, broadcast). CIDR Total addresses Usable /16 65,536 65,531 /20 4,096 4,091 /24 256 251 /28 16 11 To reverse-engineer a prefix from a required host count: round up to the next power of two, subtract the exponent from 32. Need 300 hosts? Next power of two is 512 (2⁹), so prefix = 32 - 9 = /23 . Run this before sizing any subnet that will host an autoscaling group or EKS node group. Start with /16 for the VPC itself. VPC CIDR is difficult to resize after the fact — once you have subnets, peering connections, or Transit Gateway attachments built against it, renumbering becomes a migration project. /16 costs nothing up front and avoids that corner. Subnet allocation: carving up the VPC A practical three-AZ production layout from 10.0.0.0/16 : Tier AZ-a AZ-b AZ-c Size Typical use Public 10.0.0.0/24 10.0.1.0/24 10.0.2.0/24 /24 ALB, NAT gateway, bastion Private/app 10.0.16.0/20 10.0.32.0/20 10.0.48.0/20 /20 EKS nodes, ECS, EC2 Data 10.0.64.0/24 10.0.65.0/24 10.0.66.0/24 /24 RDS, ElastiCache Reserved 10.0.128.0/17 /17 Future tiers, Transit Gateway, VPN The jump from /24 in the public tier to /20 in the app tier is intentional. ALBs and NAT gateways consume very few IPs; the app tier is where consumption scales

When I first looked at AWS, it felt unnecessarily complicated. EC2 runs something. S3 stores something. RDS manages something. Lambda does something “serverless.” I understood the definitions individually. But I still didn't understand AWS. The breakthrough comes when you stop learning these services separately and ask one simple question: How would I use EC2, S3, RDS and Lambda together to build one real application? That's when AWS starts making sense. So instead of another article explaining AWS services like dictionary definitions, let's build something. Imagine we're creating a simple job portal where users can create accounts, upload resumes and apply for jobs. Nothing extraordinary. But this small application is enough to understand some of the most important ideas in cloud architecture. First, Forget AWS for a Minute Before choosing any AWS service, think about what our application actually needs. Someone visits our website. They create an account. They upload their resume. They browse available jobs. They submit an application. When a resume is uploaded, perhaps we want to automatically process it and extract some basic information. Already, we can identify four different technical problems. We need somewhere to run our application. We need somewhere to store uploaded files. We need somewhere to store structured information such as users and applications. And we need something that can automatically react when certain events happen. Now AWS becomes easier. Because instead of memorizing services, we're matching problems to solutions. Our architecture starts with four pieces: EC2 → Application S3 → Files RDS → Structured Data Lambda → Event-Driven Processing Let's see what that actually means. EC2: Where Our Application Lives Our job portal needs backend code. Maybe we're building it using Python, Node.js, Java or another backend technology. That code needs somewhere to run. This is where Amazon EC2 enters the picture. Think of EC2 as renting a computer insid

A few weeks ago I wrote about hitting ReAct in the coursework and having a record scratch moment, because I had already met it without knowing its name. That post ended on a section called "Building Ahead of Understanding," which was me making peace with shipping things before I fully understand them. This week I shipped my first chatbot. It passed on the first attempt, on deadline day, on a project where the rubric was grading a product AWS had already discontinued. And I spent most of that day quietly worried that it did not count. Let me be clear about what the worry was, because it was not about cheating. Using AI agents to build a coding project is allowed here. I asked before I started, I got a yes, and I disclosed the whole arrangement in my README, including a section that names what each tool did and what I did. Nobody was misled about how this got built. The worry was smaller and more personal than that. I still did not type the code. My agents did. I directed, I validated, I decided, and underneath all of it was a small voice asking whether directing is the same as knowing. Whether a person who cannot write a Bedrock call from memory gets to say they learned Bedrock. Here is what I found out. The starter files were a generation behind the instructions Some context on where this came from. AWS AI & ML Scholars is a program AWS runs with Udacity, open to anyone 18 or over with no prior experience required. Everyone starts in a Challenge phase built on the AWS Certified AI Practitioner material, and the top 4,500 finishers get a fully funded nanodegree in one of three tracks: AI Programmer, Agentic AI Business Professional, or Agent Developer. I am in Agent Developer, the Bedrock AgentCore and multi-agent systems path. This chatbot is the first of its three projects. The project is a customer support chatbot on the Amazon Bedrock AgentCore managed harness. Three routes, one system prompt. A bug report gets collected across turns and filed to DynamoDB through

The deploy worked. The bill doubled. The blue-green cutover went perfectly. Traffic shifted to green, health checks passed, the team signed off, and moved on. It was one of those rare deployments that goes exactly as planned. Six weeks later, a cost anomaly surfaced in the monthly AWS review. Infrastructure spend had been running at roughly double what it should have been since the deployment date. Every EC2 instance, every RDS node, every load balancer from the blue environment was still running. Serving zero traffic. Billed at full price. For six weeks. Nobody had decommissioned it because nobody owned it after cutover. The team that ran the deployment assumed operations would clean it up. Operations assumed the team that deployed it would tear it down. The blue environment sat in a perfect ownership gap, healthy and idle and expensive, while both teams closed their tickets and moved on. This is the part blue-green deployment guides don't emphasize enough. The strategy is excellent for zero downtime releases and instant rollback capability. The rollback window is the dangerous part. It's open-ended by default, which means the old environment stays alive until someone makes a deliberate decision to shut it down. That decision requires ownership, and ownership requires someone to be responsible for it after the deployment is considered done. The fix is treating decommissioning as part of the deployment itself, not cleanup that happens afterward. Tag every blue environment resource at launch with a TTL: aws ec2 create-tags \ --resources i-1234567890abcdef0 \ --tags Key = DeploymentColor,Value = blue \ Key = CutoverDate,Value = 2026-01-14 \ Key = TTL,Value = 2026-01-21 Then wire Cost Anomaly Detection to alert when a specific environment tag is still generating spend past its TTL. The old environment doesn't get to become invisible just because traffic moved away from it. The deeper issue is that blue-green deployments create a window of parallel infrastructure that m

Today's two tasks are both about a new base. A feature branch that needs to sit on top of a master that has moved. A database instance that needs to come back from a snapshot taken when things were fine. In each case, the interesting question is the same: what carries over, and what do you have to say out loud? One Git task, one AWS task. Rebase a feature branch onto master without creating a merge commit, then snapshot an RDS instance and restore it into a new one. The tasks come from the KodeKloud Engineer platform. Rebase: not moving commits, replaying them The requirement was specific, and the specificity is the lesson. A developer's feature branch was behind master. Bring it up to date without losing any feature work, and without a merge commit. That second clause rules out git merge master . Merge joins two histories and records the join, which is the merge commit. Rebase does something else entirely. cd /usr/src/kodekloudrepos/media git branch git log --oneline --graph --all --decorate git checkout feature git rebase master git log --oneline --graph --decorate Git's own documentation describes what happens under git rebase master : it lists the commits on your branch that are not on master, checks out master, and then replays each of your commits on top of it, one at a time, in a way it compares to running git cherry-pick for each one. Replays. Not moves. Every commit that comes out the other side has a new hash, because a commit's identity includes its parent, and the parent is different now. Your work is preserved, the commits carrying it are not the same objects they were. That is exactly why there is no merge commit. Rebase does not join two histories, it rewrites yours so it looks like it was always based on master's current tip. You get a straight line, at the cost of a history that is no longer a record of what actually happened. Two things I had to be deliberate about. Direction. Rebase applies to the branch you are standing on and takes the branch yo