All the ways Amazon’s AI brought me the socks I am wearing right now

My socks arrived at my home in Washington, DC, as so many things do these days, in a brown box sealed with Amazon Prime tape. I ripped it open and pulled out the cotton athletic socks I’d ordered just two days before for $19.65. Big deal, you may think. You bought socks on Amazon. That’s news?

No, it isn’t. What’s newsworthy is just how much artificial intelligence went into selecting those socks, completing the order and delivering them. You’ve probably never given it much thought, but AI drives almost every aspect of your shopping experience. Amazon’s dominance in online retail is due in no small part to its mastery of artificial intelligence: the ability of machines to do work that was once the purview of humans alone.

“Amazon probably uses AI more extensively than any other company in the world today,” Pedro Domingos, a University of Washington professor and author of The Master Algorithm, told CNN Business. “Amazon is a company that wants to do everything, and in AI it has found a technology that is good for everything.”

Artificial intelligence promises to revolutionize everything from health care to farming, but we need not look to tomorrow to see its impact. The technology already underpins so many aspects of everyday life that we don’t give a second thought to: email spam filters and mobile check depositing apps, not to mention advanced automotive safety features like Cadillac’s Super Cruise and aircraft autopilot systems.

Inside an Amazon fulfillment center in Kent, Washington.

Delivery companies like FedEx and UPS rely upon artificial intelligence to plan routes and chat with customers. Tesla’s Autopilot semi-autonomous vehicle tech wouldn’t work without AI. And Netflix (NFLX) uses it to suggest shows you may like.

But no one uses AI quite like Amazon, where thousands of engineers work on it.

“It’s hard to imagine today’s Amazon without AI,” said Oren Etzioni, CEO of the Allen Institute for Artificial Intelligence. “An impressive, and growing, part of Amazon’s competitive edge is due to their sophisticated AI systems.”

Amazon recently invited CNN Business to its headquarters in Seattle to get a glimpse of the AI that brought those socks to my door. The company wouldn’t reveal all of its secrets, some of which are proprietary technology, but it shared enough to get an overview of the AI behind every purchase.

Smart shopping

The AI kicks in the moment I hit return after typing “Amazon.com” into my Web browser. Amazon’s homepage includes product recommendations that artificial intelligence curates given my history on the site – in this case dog treats and a Joni Mitchell album inspired by recent purchases for my golden retriever and my father.

An employee goes through a bin at the fulfillment center.

I don’t need chicken jerky tenders or a copy of The Hissing of Summer Laws, so I click the search bar. I get no further than typing “s” and Amazon’s AI is already offering suggestions. SD card. Spiderman PS4. Shower curtain. To hone this suggestion list, Amazon taps historical data from billions of searches, and the results evolve constantly to reflect how people who start a query with “s” typically complete it. Queries that have been most likely to lead to a sale top the list.

I choose “socks” from the dropdown menu, triggering several more AI systems compile the results. The algorithms list the options in an order designed to promote a quick, and satisfying, purchase. Products with the greatest number of sales and positive reviews tend to appear first.

“Every conceivable optimization opportunity has some form of machine learning applied to it,” said Srikanth Thirumalai, who leads search at Amazon. He’s worked at Amazon since 2005, and since day one has been centered on using artificial intelligence techniques to make products more discoverable, and shopping more pleasant. His very first project was identifying duplicate product pages.

Amazon sells a dizzying variety of socks. In fact, the query “men’s socks” returns more than 70,000 results. No one could possibly scroll through them all, especially while staring at a smartphone. It’s critical that Amazon shows customers what they truly want.

Amazon’s AI decides which socks to show me, and how high they should be placed on the page. It crunches sales data – Amazon won’t say how much – to determine which socks sell best. Those listings feature the orange “best seller” tag. (Sponsored items, in which merchants buy preferential placement on the results page, override the AI results.)

Crates zip through the fulfillment center on a conveyor belt.

When I click on some Puma socks that catch my eye, Amazon sends me to the product page. Here Amazon’s AI suggests other products I may be interested in, including Puma socks in a different color and Hanes boxers other customers have bought with their Puma socks.

Once I add the socks to my shopping car and click “buy,” I won’t think about them again until they land on my porch. But that’s when a whole new set of AI processes steps in.

Shipshape

First, Amazon uses AI to choose a fulfillment center to process my order. Several of these centers are within 140 miles of me, but Amazon may not use the one closest to my home in Washington, D.C. The company would not reveal how it selects a fulfillment center, but logistics experts said the choice probably reflects inventory (which is based, in part, on AI predictions on seasonal trends), shipping costs and how busy each center is.

In this case, Amazon’s AI did choose the center closest to me, in Baltimore, just 40 miles or so up the road. Before I bought them, my socks were sitting on a storage pod – which looks a bit like a four-sided bookcase about 6.5 feet tall. Once I clicked “buy” and placed my order, another AI system decided which warehouse robot would retrieve it from the pod and bring it to a human being. Proximity is only one factor the AI takes into account; the algorithms also considers how quickly a robot can reach an Amazon employee and how busy each employee is.

Robots are used in the Amazon fulfillment center.

Once dispatched, the robot slides under the storage pod, lifts it a couple inches, and scoots off to join the queue alongside an associate’s workstation. It might seem a whole lot easier, and cheaper, to have a staffer fetch my socks from the pod – but Amazon says bringing items to associates is far more efficient. AI lets Amazon maintain a balanced workload at each workstation while keeping the robots moving smoothly and quickly.

Until recently, an associate might reach into a bin to find nothing there. Artificial intelligence ensures my socks, or your whatever, are in the proper pod. Amazon calls its visual bin inspection, a technique it revealed for the first time to CNN Business.

Once an associate takes a product from a storage pod, a robot carts the pod to a holding area. Along the way, a camera snaps a photo of the pod and its contents. Computer vision software reviews the image; if anything looks amiss, the robot brings the pod to an inspection area, where a human checks the pod’s inventory and makes any needed corrections.

The visual bin inspection system at an Amazon fulfillment center.

An associate tossed my socks into a yellow crate, which then zipped through the warehouse on a conveyor belt to another workstation, where associates consolidate orders. If I’d selected the Nike tennis shoes Amazon’s AI had suggested, this is where someone would place them into a box with my socks. Once packaged, orders land on another conveyor belt and past a robot arm that slaps the mailing label on the parcel. AI then decides what to do next.

“Which door is it going out of the fulfillment center?” said Brad Porter, vice president of robotics at Amazon. “Which truck is it going to go out on?” Algorithms consider where the package is going, which other orders are going to the same general area and how quickly they must arrive at their destination. Artificial intelligence chooses the most efficient route for every order, no matter how small.

Amazon has started delivering some of its own packages, and it’s no surprise what makes this possible: Artificial intelligence creates the routes drivers take, and estimates delivery times, with the help of weather data, traffic patterns and several other proprietary data points Amazon won’t reveal. Rainy day? Your delivery may be slower. Heavy traffic? Drivers may be directed on an atypical route. And Amazon increasingly offers one-hour delivery times in some cities, which requires near perfect delivery routes.

Even after your package hits your doorstep, AI has one more job to do. The Amazon driver snaps a picture of your package and emails it to you so you know it arrived. The company trained an AI to recognize that it really is your package in the photo, and ensures the photo is framed in such a way to ensure you know exactly where to find your package.

Pretty smart. You could even call it intelligent.

As its hotly-contested and widely-publicized search for a second headquarters draws to a close, CNN Business is taking a top-to-bottom look at Amazon’s present — and what it means to our future in its series, The United States of Amazon.