AI-Powered Calorie Counting: Mastering GPT-4o Vision and SAM for Automated Nutrition Tracking

Let’s be honest: manual diet tracking is a chore that almost nobody finishes. We start with good intentions, but typing "150g of grilled chicken" and "half a cup of brown rice" into an app every day is a recipe for burnout. But what if you could just snap a photo and let Multimodal AI do the heavy lifting? 📸 In this tutorial, we are building a production-ready automated nutrition logging system. We will combine the surgical precision of the Segment Anything Model (SAM) with the reasoning power of GPT-4o Vision . By the end of this post, you'll know how to transform raw pixels into a structured JSON of calories, macros, and portion sizes using FastAPI and Pydantic . We'll cover key concepts in Image Segmentation , Computer Vision , and LLM Structured Outputs . The Architecture: From Pixels to Proteins To get accurate results, we can't just toss a messy photo at an LLM and hope for the best. We need a pipeline that identifies individual food items, isolates them, and then performs a multi-step inference. graph TD A[User Uploads Food Image] --> B[FastAPI Backend] B --> C[SAM: Segment Anything Model] C --> D[Generate Individual Food Masks] D --> E[GPT-4o Vision: Multi-crop Analysis] E --> F[Pydantic Validation] F --> G[Structured Nutrition Report] G --> H[User Dashboard] Prerequisites To follow along, you'll need: Python 3.10+ OpenAI API Key (with GPT-4o access) FastAPI & Uvicorn (for the web layer) Segment Anything Model (SAM) weights (or a hosted inference API) Step 1: Defining the Nutrition Schema The secret to a reliable AI system is Structured Output . We don't want a "chatty" response; we want data our database can consume. We'll use Pydantic to define exactly what a "Meal" looks like. from pydantic import BaseModel , Field from typing import List class FoodItem ( BaseModel ): name : str = Field ( description = " Name of the food item " ) estimated_weight_g : float = Field ( description = " Weight in grams " ) calories : int = Field ( description = " Total calorie