标签：#data

We maintain a corpus of 456 role-specific resume examples in TypeScript. Someone asked me what a good bullet point actually looks like, and rather than answer from taste I decided to measure the thing I already had. Fifteen minutes later we had a script, a set of numbers, and a conclusion. The conclusion was wrong, because the script had silently read about twelve percent of the data. This is a post about that failure mode, and then about the numbers I got once the script worked. The corpus Thirty-one TypeScript files, each exporting an array of role objects. One role looks roughly like this: { slug : ' cloud-architect ' , title : ' Cloud Architect Resume ' , category : ' Information Technology ' , sampleData : { summary : ' ... ' , experiences : [ { company : ' Amazon Web Services ' , position : ' Senior Cloud Architect ' , description : ' - Designed multi-region architecture... \n - Led migration of... ' , }, ], skills : [...], }, tips : [...], } The interesting field is description . It holds a newline-delimited list of bullets as a single string, so the whole corpus of bullets is sitting there in source, greppable, without a database or an export step. Version one const descs = [... text . matchAll ( /description: ' ((?:[^ ' \\] | \\ . ) * ) '/g )]. map ( m => m [ 1 ]); Nothing exotic. Match description: , then a single-quoted string, allowing escapes so an apostrophe inside the text does not terminate the match early. It found 199 description strings. I did not question that, because I had no prior for what the number should be. 199 sounded like a lot of text. We computed medians off it, looked at the opener distribution, and started writing. The number that saved me was on a different line of the same output: roles 456 . The slug count was fine. So 456 roles between them had 199 job descriptions, which would mean the overwhelming majority of roles had no work history at all. I knew that was false, because I had rendered these pages. Why it read twelve percent

The short version NextAuth (now Auth.js) creates 4 tables in your database: users , accounts , sessions , and verification_tokens . The users and accounts tables have a one-to-one relationship via accounts.user_id . Sessions link to users via sessions.user_id . Verification tokens are short-lived and self-cleaning. The 4 tables users Column Type What it means id text / UUID Primary key. Generated by NextAuth. name text Display name from the OAuth provider (Google, GitHub, etc.) email text User's email. May be null if the provider doesn't share it. email_verified timestamp When the email was verified. Null if never verified. image text Profile picture URL from the provider. created_at timestamp When the user first signed in. updated_at timestamp Last profile sync from the provider. accounts This table links a user to an OAuth provider. One user can have multiple accounts (e.g., Google + GitHub). Column Type What it means id text / UUID Primary key. user_id text Foreign key → users.id . type text Always "oauth" or "oidc" . provider text "google" , "github" , "discord" , etc. provider_account_id text The provider's unique ID for this user. refresh_token text OAuth refresh token (encrypted in production). access_token text OAuth access token (encrypted in production). expires_at integer When the access token expires (Unix timestamp). token_type text Usually "Bearer" . scope text Permissions granted by the provider. id_token text OIDC ID token (if using OIDC). session_state text Provider-specific session state. sessions Active sessions for each user. NextAuth creates a new row here on every sign-in. Column Type What it means id text / UUID Primary key. session_token text The session token stored in the user's cookie. user_id text Foreign key → users.id . expires timestamp When this session expires. verification_tokens Short-lived tokens for email verification, password reset, etc. Self-cleaning old tokens are deleted automatically. Column Type What it means identifier te

I was testing out CH-Ops - an admin GUI for self-hosted ClickHouse - on a simple setup: 1 shard, 2 replicas. Stumbled onto the replication queue almost by accident. Here's what I did: I stopped one of the nodes (let's call it Node B), then inserted some data through the other one (Node A). Just wanted to see what would happen. Then, while Node B was still down, I checked it in CH-Ops. It had stuff sitting in its replication queue. My first assumption was: okay, this must be showing what's left to replicate across the cluster - the total pending replication work. So I switched over and checked Node A, the one that was actually up and had just received the insert. Its queue was empty. That didn't match what I expected at all. If the queue was a cluster-wide "here's what still needs to replicate" view, Node A should've shown something too - it was the one that had the fresh data now waiting to reach Node B. Instead it was Node B, the down one, sitting there with pending tasks. That mismatch is what sent me digging. Turns out the queue isn't cluster-wide at all - it's specific to each ClickHouse instance. Once I brought Node B back up, its queue drained in seconds and the data showed up. That whole experiment is basically the entire post in miniature. Here's the mental model I ended up with. A Queue Belongs to a Replica, Not to the Table This is the first thing to get straight. With a ReplicatedMergeTree table, you can have multiple replicas holding copies of the same data. It's tempting to think of replication as one shared pipe between them. It isn't. Each replica keeps its own local replication queue . So if you see: Replica 1 → queue_size = 0 Replica 2 → queue_size = 25 that doesn't mean 25 operations are waiting somewhere in the middle for both replicas to pick up. It means Replica 2, specifically, has 25 tasks it hasn't finished yet. Once that clicked for me, the rest of the system made a lot more sense. So Where Do These Tasks Come From? Replication in ClickHouse

Introduction: Turning Jumia Product Data into Business Insights E-commerce platforms generate a lot of product data, but raw numbers become useful only when they can support better decisions. For Jumia sellers, prices, discounts, ratings and customer reviews can provide clues about product performance, customer engagement and possible pricing strategies. For this project, I worked with a dataset of 112 Jumia products to explore these relationships using Microsoft Excel. I wanted to find out whether higher discounts are associated with more customer reviews, whether highly rated products receive stronger engagement, and whether product price is related to rating. I also wanted to identify the products performing best and those that may require a different pricing or marketing approach. I followed a complete data-analysis workflow: Raw Data → Cleaning → Transformation → Analysis → Visualization → Insights → Recommendations The project uses Excel Tables, Power Query, formulas and functions, PivotTables, PivotCharts, slicers and dashboard techniques. This article documents that process and shows how the raw Jumia data was transformed into an interactive dashboard and, ultimately, evidence-based business recommendations. Understanding the Dataset and Its Initial Problems Before cleaning the data, I first needed to understand what I was working with. The dataset contains 112 Jumia products and six main fields: Product, Current Price, Old Price, Discount, Review and Rating. Current Price and Old Price represent product pricing, Discount captures the promotional percentage, Review represents the number of customer reviews, while Rating records the average customer rating out of 5. I treated this stage as a data-quality audit rather than immediately changing anything. The purpose was to identify issues that could affect calculations and visualizations later. The raw dataset contained formatting and consistency issues that needed attention, particularly around numerical field

When multiple companies are behind one project, who bears responsibility for problems?

When multiple companies are behind one project, who bears responsibility for problems?

Author: Ignacio Paz An AI agent is only as useful as what it can remember - and how well it can connect the things it remembers. Most teams hand their agent a memory by reaching for a vector store: embed everything, retrieve by similarity, hope the relevant context comes back. It works, until you notice the agent keeps surfacing things that are near the question but not actually connected to it. Cobrainer , a skills-intelligence company based in Munich, took a different route. They gave their AI agent a memory that lives in the database as a graph, where the agent builds the relationships between nodes as it goes. They did it without adding a graph database, a vector engine, or a search engine to their stack. It all runs on SurrealDB, alongside a Rust-native agentic graph RAG built on the same store. Here's how, and why a single engine made the difference. The problem with flat memory Cobrainer runs a skills-intelligence platform - the kind of system that reasons about how people, roles, skills, and capabilities relate to one another. That's an inherently graph-shaped problem. But their first retrieval setup wasn't graph-shaped at all. It pulled context through flat vector retrieval over an S3-and-OpenSearch pipeline, which carried two recurring costs: Accuracy . Flat vector matches returned context that was loosely related - semantically near, but not necessarily connected in any meaningful way. The team wanted the agent to follow real relationships between entities, so its answers were grounded rather than approximate. Tokens . Broad vector matches meant stuffing a lot of marginally relevant context into every prompt - expensive, and more so with every call. The team wanted to fetch only the context that mattered. The obvious fix - adding a graph database on top of the vector and search systems they already ran - would have meant more infrastructure to operate. For a startup moving fast, that fragmentation was the thing to avoid, not embrace. What they wanted inst

COUNT(*) looks like a trivial operation: SELECT COUNT ( * ) FROM orders ; The query asks for a single number, but that doesn't mean PostgreSQL can produce it with a constant-time read from some internal counter. When we need an exact count, PostgreSQL has to determine how many rows are actually part of the visible result set for that query. On large tables, that work can become a meaningful chunk of total execution time. And the problem doesn't just go away by throwing an index at it. The useful question isn't "do I have an index?" It's: How many rows does PostgreSQL actually need to examine to compute this count — and can that work be reduced? Why COUNT(*) Can Be Expensive in PostgreSQL PostgreSQL uses MVCC — Multi-Version Concurrency Control — to manage concurrent access to data. That's what lets multiple transactions work at the same time while each sees a consistent view of the database. But it also means row visibility depends on the snapshot the query is running under. That's why PostgreSQL can't answer: SELECT COUNT ( * ) FROM orders ; by simply reading an exact counter stored somewhere in the table's metadata. To return an exact result, it has to process the rows — or an index structure representing those rows — and determine which ones are part of the visible result. On a small table, that cost is invisible. On a table with millions of rows, the amount of work starts to matter. Which leads to an important distinction: returning a single row from COUNT(*) does not mean processing a single row. How to Analyze a COUNT with EXPLAIN ANALYZE Before reaching for an index, it's worth looking at what PostgreSQL is actually doing. Say we have this query: SELECT COUNT ( * ) FROM orders WHERE status = 'completed' ; We can analyze it with: EXPLAIN ( ANALYZE , BUFFERS ) SELECT COUNT ( * ) FROM orders WHERE status = 'completed' ; The goal isn't to hunt for an Index Scan by default. Worth checking instead: the scan type estimated rows vs. actual rows processed rows discard

Ever woke up feeling like a truck hit you, only to realize your Apple Watch had been screaming "Warning!" via your data for the last 24 hours? Heart Rate Variability (HRV) is the "canary in the coal mine" for our bodies. It's a powerful metric that tracks the variation in time between each heartbeat, serving as a direct window into your Autonomic Nervous System. In this guide, we are going to build a real-time HRV anomaly detector using wearable data analysis , Scikit-learn , and AWS Lambda . By applying machine learning to time-series health data, we can identify physiological stress, potential infections, or overtraining before physical symptoms even manifest. If you’ve been looking to dive into anomaly detection in time-series or want to master health data engineering , you’re in the right place! The Architecture: From Heartbeat to Alert 🛠️ To achieve real-time monitoring, we need a pipeline that moves data from your wrist to a cloud-based inference engine. Here is the high-level flow: graph TD A[Apple Watch / Wearable] -->|Sync| B(Apple HealthKit) B -->|Webhook/Hook| C[AWS API Gateway] C --> D[AWS Lambda - Inference] D -->|Fetch History| E[(DynamoDB / S3)] D -->|Isolation Forest| F{Anomaly?} F -->|Yes| G[Push Notification / Alert] F -->|No| H[Log & Silent] Prerequisites 📋 Before we start coding, ensure you have the following: Python 3.9+ Scikit-learn & Pandas for data crunching. AWS Account (for Lambda deployment). An app to push HealthKit data (like Health Auto Export or a custom Swift hook). Step 1: Understanding the Data 📊 HRV data is tricky because it’s highly personalized. What is "low" for an athlete might be "high" for someone else. This is why we use Isolation Forest , an unsupervised learning algorithm that excels at detecting outliers in multi-dimensional datasets without needing labeled "sick" vs. "healthy" days. Step 2: Building the Anomaly Detection Logic Let's write the core logic using Scikit-learn . We’ll use the Isolation Forest algorithm becaus

Originally published on angeo.dev . Full tables, p-values and the sealed plan are there. Most claims about AI visibility are untestable by design: publish the signals, wait, attribute anything good that happens to the signals. I wanted a version I could not fudge, so I wrote the analysis plan first, hashed it, and sent the hash to the other party before I had any data. The question Do businesses AI assistants name repeatedly differ, on observable technical signals, from businesses the same assistants name once ? Every business in the corpus was named at least once, so this says nothing about how to enter an answer. It compares repeat against one-off mentions inside a named-business corpus. Four signals, all externally observable: Signal Check Crawler access Does robots.txt block any of 8 AI crawlers Content map Does the site serve /llms.txt Structured data Does a product page emit JSON-LD Product Buyability Does that node carry offers.availability Study setup The answers came from a partner (connexion.me), who ran 44 product-level home-decor buying questions across ChatGPT, Gemini and Perplexity, twice, in two arms — 264 answers per arm. Blinding was deliberate. I did not write the questions and did not see their store list until my plan was sealed; they never saw my frame, my scan results or my thresholds. Roster rows 669 no resolvable domain -186 resolved to a different company -3 marketplaces and listing surfaces -12 duplicate rows collapsed -10 Unique domains analysed 458 scanned successfully 455 Cases: 3+ mentions across both runs and present in both. Controls: exactly one mention across both runs. Head excluded first — anything in 53+ of 264 answers (Amazon, Etsy, Wayfair, Target, Home Depot). The pre-registration Sealed 10 August, SHA-256 9b4ccf12629e… : Under 15% of named businesses would be Magento No signal would separate the groups by more than 15 points Refutation condition: any signal differing by 20+ points with the named group higher Result — generic

Overview In this article, I'll introduce the concept of Multi-Version Concurrency Control (MVCC) and explain how Postgres implements this protocol across different isolation levels. I'm assuming you already have a basic understanding of isolation levels, database locks, and concurrency in general. I won't cover those concepts here, so if you're not familiar with them, I highly recommend checking out A Straightforward Guide for Isolation Levels first before continuing. The goal of this article is to help you understand: What Multi-Version Concurrency Control is How Postgres implements MVCC across different isolation levels Multi-Version Concurrency Control High-Level Concept The idea behind MVCC is simple: it's a protocol designed to accomplish one goal — when two or more transactions run concurrently on the same data, the end result should look as if those transactions ran one after another, in sequence. Take a look at the diagram above. Two transactions are running concurrently, and we want the end result to look as if either the first transaction ran and committed before the second one started, or vice versa. MVCC guarantees there are only two possible outcomes — never a third. But in reality, these transactions are running at the same time, so this is exactly the core idea of MVCC: it's a protocol that gives us this guarantee even though the transactions genuinely overlap in time. Note that other protocols aim for the same goal, like Two-Phase Locking and Optimistic Concurrency Control. They all take different approaches, but they're all working toward the same thing. The core idea of MVCC is that whenever a transaction updates a row, it doesn't mutate the value in place. Instead, it creates a new record as the latest version and links it back to the old version. After the update, the row has a new version, and the old version is never changed. This version chain exists per record — every update to a row creates a new version, and since each version is linked to

Polls show that overwhelming majorities of Americans hate data centers. China makes a perfect scapegoat for tech leaders and their allies—the only problem is a lack of evidence.

We shipped a filter that threw away bad GPS readings. Months later somebody asked whether it was working, and I could not answer. The evidence was gone. That question changed how I build anything that rejects data. The obvious version, and why it rots Mileage tracking depends on trustworthy distance, and GPS lies constantly. So the first version of our cleanup did what everyone's first version does: if (! fix . isPlausible ( previous )) return // drop it, move on accumulateDistance ( fix ) Clean data comes out the other end. It feels responsible. It is also a trap, because that return destroys the only record that could ever tell you whether the rejection was correct. Six months in, someone asked the reasonable question: is the filter right? I could not say how many readings we had dropped, on which journeys, or whether any of them had been a genuine drive through a tunnel rather than a glitch. We had built a thing that made a judgement call thousands of times a day and kept no record of any of it. Persist, then classify The rebuild flipped the default. Rejection stopped being a return and became a label. Only two cases are still deleted, because they cannot physically be real: // impossible coordinates if ( fix . lat ! in - 90.0 .. 90.0 || fix . lng ! in - 180.0 .. 180.0 ) return null // impossible accuracy: too precise to be true, or useless if ( fix . accuracyM <= 0.1f || fix . accuracyM >= 250f ) return null That is the entire delete list. Everything else is persisted and sorted into named accumulators: originalDistanceM += displacement // every metre we ever saw when { fix . isMock -> mockDistanceM += displacement abnormal -> { abnormalDistanceM += displacement if ( isHardSpike ) spikeDistanceM += displacement } accuracyGated -> { /* recorded, deliberately not counted */ } else -> cleanedDistanceM += displacement } Five numbers instead of one. The UI shows cleaned . The rest live beside it. And the row itself keeps its provenance: accuracy, provider, bearing, a

If you're prepping for a Business Intelligence (BI) Engineer, Data Analyst, Data Engineer, or Data Scientist role at Amazon, you probably already know SQL matters. It's a core part of the hiring bar. But Amazon isn't just checking your syntax. They want to see if you can think in sets, write clean queries under pressure, and reason about data the way the business actually uses it. Here's exactly how to prepare based on what the interview actually rewards. What the Interview Really Tests Amazon's SQL rounds usually show up in one of two ways. It's either a technical screen using a shared coding tool, or a whiteboarding case-study during the main loop. Either way, the interviewer is watching for a few specific signals. For starters, correctness always beats speed. A working query is far better than a clever one that fails. Communication is also huge. Do you talk through your logic before you even touch the keyboard? You'll often get a vague ask, like finding the "best" customers. You are completely expected to define what "best" means out loud before you start writing Common Table Expressions (CTEs). And watch out for messy data. Nulls, duplicates, and mismatched grain are almost always baked into the problem on purpose. The Core Topics to Master Focus your prep time on a few specific areas. Actually, it turns out this is where almost all the interview questions live. Joins Inside and Out You need to know your INNER, LEFT, RIGHT, and FULL OUTER joins cold. Be ready to explain exactly why row counts change after each one. A classic Amazon-style question is finding customers who placed orders but never left a review. That's just a LEFT JOIN with a NULL check. The interviewers want to see you reach for it right away. Window Functions Functions like ROW_NUMBER() , RANK() , DENSE_RANK() , and LAG() or LEAD() show up constantly. You might see a common pattern—like finding the second-highest order value per customer, or calculating month-over-month growth. If you're shaky he

Background : In migration projects involving different databases, incompatibility of SQL syntax is often encountered. Question : If there is a large amount of code that needs to be rewritten, manual processing would be time-consuming and prone to errors. Is it possible to achieve automatic conversion of code syntax in large quantities through tools? Solution : The open-source tool ZGLanguage can be utilized to perform automated conversion of SQL code in large batches. For example： Suppose SQL PIVOT function is as follows : SELECT * FROM ( select country , state , yr , qtr , sales , cogs from table111 ) PIVOT ( SUM ( sales ) AS ss1 , SUM ( cogs ) AS sc FOR qtr IN ( 'Q1' AS Quarter1 , 'Q2' AS Quarter2 , 'Q3' AS Quarter3 , 'Q4' AS Quarter4 ) ) tmp ; Using the ZGLanguage conversion rule, execute the conversion to obtain the result : SELECT * FROM ( select ### , ### , ### SUM ( case when qtr = 'Q1' then sales else null end ) AS Quarter1_ss1 , SUM ( case when qtr = 'Q2' then sales else null end ) AS Quarter2_ss1 , SUM ( case when qtr = 'Q3' then sales else null end ) AS Quarter3_ss1 , SUM ( case when qtr = 'Q4' then sales else null end ) AS Quarter4_ss1 , SUM ( case when qtr = 'Q1' then cogs else null end ) AS Quarter1_sc , SUM ( case when qtr = 'Q2' then cogs else null end ) AS Quarter2_sc , SUM ( case when qtr = 'Q3' then cogs else null end ) AS Quarter3_sc , SUM ( case when qtr = 'Q4' then cogs else null end ) AS Quarter4_sc from ( select country , state , yr , qtr , sales , cogs from table111 ) where qtr IN ( 'Q1' , 'Q2' , 'Q3' , 'Q4' ) group by ### , ### , ### ) tmp ; The conversion rule is as follows : __DEF_FUZZY__ Y __DEF_DEBUG__ N __DEF_CASE_SENSITIVE__ N __DEF_LINE_COMMENT__ -- __DEF_LINES_COMMENT__ /* */ __DEF_STR__ __IF_KW__ <1,100> [1,1]ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz [0,100]ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789_ [NO] XXX __DEF_PATH__ __FROM_PIVOT_1_1__ 1 : frm @ %__IF_KW__ | from : tab @ | __TABLE_NAME__ : ssl @

pg_anon found 1 of the 8 personal-data columns in my PostgreSQL database. The one it caught was email , and only because "email" is spelled the same in Spanish and English. The other seven — nombre , apellido , telefono , direccion , fecha_nac , tarjeta_ult4 and rut — walked straight through, unmasked. pg_anon is the open TantorLabs tool that masks personal data in PostgreSQL: it scans the database, flags the sensitive columns, and dumps a masked copy. Like pg_dump , but covering the sensitive parts on the way out. Exactly what you want before handing a colleague a copy of production. So I fed it a Chilean database and watched it miss almost everything — no error, no warning. It finished successfully and handed me a dump with names and national IDs still in cleartext. What the scan actually does Two filters, in order. First it reads each column's name against a set of regexes ( ^email$ , ^phone$ , ^ssn$ …). To the columns left over, it opens the data and tries patterns on the value (an email's @ , a card's 16 digits). Whatever no filter catches passes through. The rules in the demo meta-dict it ships with are written for English schemas. Mine aren't. 1 of 8 Column Holds Stock rules email email ✅ nombre first name ❌ apellido surname ❌ telefono phone ❌ direccion address ❌ fecha_nac birth date ❌ tarjeta_ult4 card digits ❌ rut national ID ❌ email got caught by name (it's an English word) and confirmed by its @ . Everything else has a Spanish name no stock rule looks for. The rut is the clearest miss. A RUT looks like 7917183-2 : seven or eight digits, a dash, and a mod-11 check digit that can be the letter K . The only national-ID rule pg_anon ships with is ssn . It has no idea what a RUT is — and it won't know a cpf (Brazil), dni (Spain, Argentina), curp (Mexico), nif (Portugal) or aadhaar (India) either. If your schema isn't American, the defaults miss your most sensitive column. The fix: a few lines of Spanish You teach it. Column names in your language, plus a conte

Most guides for connecting Spark to Amazon MSK assume the two live in the same cloud and authenticate with IAM. That covers a lot of cases. It does not cover the one that keeps showing up in large enterprises: the Kafka cluster is in AWS, the compute is Azure Databricks, IAM is off the table because the identity system is a corporate PKI, and the traffic never touches the public internet. This walks through that setup end to end. Certificates from a private CA, a private network path between the two clouds, and a Structured Streaming job that actually keeps running on a multi-node cluster instead of only on the driver. Why mTLS instead of IAM MSK offers four authentication modes: plaintext, TLS with client certificates, SASL/SCRAM, and IAM. IAM is the easiest and the best choice when your consumers run in AWS. Cross-cloud, IAM stops being convenient. Azure Databricks executors have no AWS identity. You can bolt one on with OIDC federation and assumed roles, but in most regulated enterprises the decision has already been made elsewhere: there is a corporate PKI, every service-to-service hop uses client certificates issued from it, and the security architecture review is going to ask why this one connection is different. mTLS is the path of least resistance, not the clever choice. One important constraint before you start. MSK will only accept client certificates issued by an AWS Private CA (ACM PCA) that is associated with the cluster. If your corporate PKI is not that CA, you have two options: stand up an ACM PCA subordinate signed by your corporate root, or issue the Databricks client certificate from a dedicated ACM PCA and treat it as a separate trust domain. The subordinate route is usually what security wants, and it takes longer to get approved than everything else in this article combined. Start that conversation first. Network path Three ways to get private connectivity between an Azure VNet and an AWS VPC: Site-to-site VPN. IPsec tunnel between an Azure VPN

Introduction Jumia is one of Africa's leading e-commerce platform that manages millions of transcations with a diverse products from electronics,beauty products and many more categories.Therefore,tracking key perfomance indicators is essential for supply chain operatios and profit optimization Objective My project aim is to build an interactive excel dashboard using Jumia transactional data.I aim to convert disorganized data into an interface that can help in decison making,identify trends and monitor products. Dataset description Data Cleaning and preparation process Raw data mostly contains inconsistency and errors that may occur that may interfere or give the wrong output. An example of a raw dataset In the example above we can see inconsistent and missing data that we need clean in order to have an effective output. First step is to format the prices from text to currency format and replace the before since excel will in order to calculate the discount eg below The image above is the discount price which was obtained by finding the difference between the old price and the new price. The image below is an example of the formula to categorize the prices whether high,low or medium.I used the IF,AND functions.Another example of a logical combination would be the us of OR . The difference when using the IF(AND function is that all the conditions must be met while in the IF(OR ,only one condition has to be met. In the image below i used logical combination of that are IF and AND for the discount category. In the image below i also used the IF AND functions to in the ratings category. After removing duplicates,removing inconsistent data eg texts in numbers columns.Below is an image of the cleaned version of the Jumia dataset. An example of a clean dataset Descriptive Analysis To calculate the average current price of products i used the average formula and highlighted the cells eg =AVERAGE(B2:B113) .The average old price of products was obtained by the same formula but

1. Project Introduction and Objective In this project, I used Microsoft Excel and Power Query to clean and analyze a Jumia product dataset and then built an interactive dashboard to summarize pricing, discounts, ratings and customer engagement. The main objective was to turn a small raw e-commerce dataset into useful business information. I wanted the final dashboard to answer practical questions such as: Do products with higher discounts receive more customer engagement? Do higher priced products have better ratings? Is there a relationship between product rating and number of reviews? Which products have the highest review engagement? Which products may require further investigation because they have high discounts but low ratings? The project also gave me practical experience in data cleaning, excel formulas, PivotTables, PivotCharts, slicers, correlation analysis and dashboard design. 2. Dataset and Business Questions The original dataset contained 115 rows and 6 columns: Product Current price Old price Discount Review Rating The dataset was small but it contained several realistic data quality problems. This made it useful for me to practice the complete analytics process rather than going directly to visualization. I structured the workbook into the following sheets: Raw_Data Cleaned_Data Analysis Pivot_Tables Dashboard Data_Dictionary As we have always been taught in class,I kept the Raw_Data sheet unchanged so that I always have a copy of the original source data. 3. Initial Data-Quality Audit Before cleaning the data, I profiled the dataset in Power Query using Column Quality, Column Distribution and Column Profile. The audit identified several issues: Data-quality check Result Original rows 115 Original columns 6 Blank Review values 58 Blank Rating values 58 Populated Review values stored as negative numbers 57 Current Price ranges 1 Old Price ranges 1 Exact duplicate rows removed 3 Discount values outside 0 to 100% 0 Rating values outside 0 to 5 after cle

Same query. Same table. Same million rows. One day it takes 4 seconds . The next day, 4 milliseconds . Nothing changed in the data. The only thing that changed was one line — you added an index . Four seconds to four milliseconds is a thousand times faster, from one line of SQL. But here's the part nobody tells you: half the indexes people add do nothing. The query stays slow, the writes get slower, and they can't figure out why. By the end of this you'll know what an index actually is — and the one rule that decides whether yours even gets used. Prefer to watch? Full walkthrough with the B-tree lookup animation: With no index: a full table scan You ask the database for one user by email. With no index, what does it do? It reads the first row. Not a match. The second row. Not a match. It keeps going — every single row — until it finds yours or runs out. A million rows, a million checks. SELECT * FROM users WHERE email = 'vlad@stack.dev' ; With no index, that WHERE line has only one way to run: look at all of them. The work grows with the table — ten times the rows, ten times the wait. That's a full table scan , and that's your four seconds. What an index actually is Most people picture an index as a copy of the table, or some kind of cache. It's neither. An index is a sorted map — just the column you search on, kept in order, with a pointer back to the full row. And the shape it's sorted into has a name: a B-tree (the default index in both Postgres and MySQL — technically a B+ tree). At the top, one node — the root . It splits into a few branches . Each branch splits again, down to the leaves , where the pointers to the rows actually live. Every node is sorted. The root doesn't hold your data — it holds signposts . Emails before "M"? Go left. "N" and after? Go right. Each step throws away half the tree, or more. You're never reading rows. You're following signs. The walk: three hops, not a million rows Watch what the lookup actually does: The root — one hop. A branc