How Amazon Is Trying To Get Rid Of Checkout Lines At Stores

At a recent Twenty One Pilots concert at Climate Pledge Arena in Seattle. These fans bought food and drinks under the watchful eye of dozens of AI enabled cameras, then walked out without stopping to pay. So the way you enter a just walk out store is by taking any sort of payment instrument, in this case a credit card, you simply tap it here on the entry gate.

Earlier that day, CNBC got a first ever on camera look at the mock store where Amazon developed the tech, which it calls Just Walk Out. You enter the store and now you're going to shop, just like you would at any other store. It's at 35 of Amazon's own brick and mortar stores across the US, UK, Australia and Canada, and now more than 200 third-party stores, too.

From concession stands at NFL stadiums like the Seahawk's Lumen Field to some 30 college campuses. When I'm done, I simply walk out the exit gate of the store and my virtual cart is charged to my credit card. That simple.

But earlier this year, Amazon pivoted away from using Just Walk Out in its own large stores, removing it from all Fresh grocery stores in the U. S and the two Whole Foods where it was being tested. They have clearly recognized that Just Walk Out is not going to be the saving grace of their physical retail efforts, and I think it's still to be determined as to whether or not there's enough interest, enough appetite among retailers for this technology, for Amazon to have a viable business.

Cnbc went to Seattle to see how Amazon is improving the tech with new generative AI tools, and to ask why it removed just walk out from its own stores while selling it to outsiders instead. Amazon is not alone in trying autonomous checkout start ups like AiFi and Grabango are trying something similar, but Amazon's soaring valuation and deep pockets give it a big advantage. It says it sold more than 25 million items where it's installed Just Walk Out in stadiums, arenas, conference centers, theme parks, airports, convenience stores, hospitals and college campuses, and that it's boosted sales by up to 112% in some locations.

The original project that would become Just Walk Out started in 2013. It was back when Jeff was still at the company. He was heavily involved early on.

John Jenkins has left Amazon since our interview, but he first started there in 2004, helping build the e-commerce site. We enabled people to shop for basically anything from their living rooms, and then we asked ourselves, well, like, how do you make the physical world equally as convenient? After some delays, it officially launched for public use at Amazon Go stores in 2018.

But if Amazon is to further disrupt the retail industry, the technology has to prove it works and customers will have to take to a technologically advanced retail experience. Early on, Amazon reportedly made plans for rapid expansion. Now a retail consultant, Jordan Berke, was leading Walmart's e-commerce in China at the time.

This was a quake moment. We all thought, wow, if Amazon can crack this, they will change. Probably the last remaining major friction point in retail, the checkout experience is always a downer.

The challenge is it's the hardest problem to solve. To use computer vision to remove checkout is a complete moonshot. When Amazon started opening fresh stores in 2020, it launched the tech in about 20 of them.

Then in April, it ditched the tech at all those US fresh stores. Grocery stores are incredibly complex, right? You're selling things loose produce, meat by the pound, floral arrangements, all kinds of stuff.

And so one of the reasons we wanted to go to those stores was to see, could we push this technology to its limits and make it work. And we did prove that we could. We've now sort of focused in on these smaller format stores where the technology tends to generate a little bit better ROI.

It works really well in things like small format places where there's like not a whole lot of inventory, and you have to deal with lots of bursts of people in a short period of time. So instead, at 50 US fresh stores and six Whole Foods, Amazon is relying on smart carts, which it also started selling to outside retailers this year. Dash carts identify items as they're placed inside and update a display with the total price in real time.

Just Walk Out, meanwhile, has shifted to become primarily a B2B service. Amazon started selling it to third parties in 2020 and says its popularity is accelerating. If you look at our earliest sort of third party customers, they would often open like one store the first year, and now we see customers coming to us that want to open seven stores the very first year.

Jenkins says Amazon will double the number of third party, Just Walk Out stores this year, with six new ones at Seattle's Lumen Field, seven at Northwest Stadium outside DC, one at M&T Bank Stadium in Baltimore, and new college locations. Now shoppers just need to get comfortable using it. One of the ways that it's making sure that there are accurate representations of what shoppers buy without cashiers around is by using LiDAR to do 3D scans.

So I'm going to go ahead and press play here. And this machine is getting a 3D scan of a store to make sure that all the items that are on the shelves are actually represented correctly. These are the blue boxes that represent the spaces where the products are.

The scan, which can also be done with an iPad, also assesses where to place cameras so they have the clearest view. The goal is to have the fewest number of cameras possible, so we optimize the camera placement so that we can get enough coverage on each fixture to see what is happening in the store. The cameras dangle from the ceiling and calibrate using QR codes on the floor, and they test what they see against other signals, like weight sensors on shelves.

The camera will probably have a pretty hard time telling how many of those I just took, right? But the weight sensor in that shelf, combined with the camera signal, can now get very, very accurate with what just happened on that shelf. In August, Amazon announced a new multi-modal AI system that improves accuracy by relying on transformer architecture.

The fundamental building blocks of generative AI. And then we have knowledge of the 3D map of the store and what products are where on what shelves. And we have catalog images of what those products look like.

We feed all of these inputs into our multi-modal AI model, and that model reasons over all of these inputs to figure out what actually went into the person's virtual cart and what we should add to their receipt. And in some stores, Amazon doesn't use cameras at all, relying on decades old technology, instead tagging each item with a cheap, unique radio frequency ID. Amazon is also using individual RFID tags so that it can track different individual pieces of clothing that might be different sizes, something that's hard for cameras to see.

RFID technology has traditionally been implemented for the benefit of the store owner. Right? To prevent theft.

We've taken a different approach. We've used RFID technology for the benefit of the shopper, right, to enable them to exit the store without having to wait in line. Shoppers are then sent a receipt, which Amazon told us typically takes tens of minutes.

And as a customer walks out of that store if they don't see their transaction immediately. It creates a natural uncertainty. Did they charge me the right amount?

Did they get the right items that I bought? And so the fact that often it takes four minutes, five minutes, sometimes up to 30 minutes before that receipt comes through, just shakes the confidence of the consumer on whether this is an experience I can trust. While nobody is watching live, Amazon does rely on people to label and review shopping data.

All of the stuff that's going on in the store gets ingested by a bunch of machine learning algorithms, and then receipts are produced. Now, it is true that we have some people that annotate videos. So any machine learning system, whether it's self-driving cars or just walk out stores or anything else, the way you start those systems up is you have humans label data.

But the massive data collection needed to make just walk out function brings up other concerns. You also need to expose, like what you're doing with the data and what data you're capturing and how long is it stored. Because all of those are reasons that we've lost trust in a lot of these institutions.

One way to get into some Just Walk Out stores is to sign in by scanning your palm, called Amazon One. At Red Rocks Amphitheater in Colorado in 2020. Concerns about what Amazon would do with that biometric data led to artist protests, and the venue canceled plans to use the tech at third party, just walk out stores.

Shoppers can swipe in with any credit card and don't need an Amazon account to shop. Still, Amazon has to convince retailers that they can trust one of their biggest competitors with handling valuable shopper data. And so, just like Netflix can deploy their entire sort of infrastructure and website on AWS, even though Amazon has Amazon Prime, a store owner who wants to use just walk out can have the same confidence that none of their data is ever going to leak over into Amazon's retail business from just walk out.

Amazon wouldn't disclose what it charges for stores to install the tech. They keep insisting that the number is down, the number is down, but they still don't share. Nobody shares the numbers of how much you know is a ballpark figure for what it costs, which suggests that it's still really expensive.

So how does the tech make money for stores? Lumen field here, where the Seahawks play in Seattle, they put just walk out technology into a store and they have 85% more people going through that store during the course of a game. To get a hot dog at halftime or to get a beverage at intermission.

That's an experience where speed is exciting. Getting back to your seat faster does make a difference. Amazon also says it reduces theft because of all the monitoring, and fewer staff are needed during peak traffic or times stores would otherwise be closed.

A just walk out store can operate unattended over the nighttime hours at a place like an airport. And so that's one of the reasons one of our partners, Hudson, has opened more than a dozen stores with us. Amazon says just walk out stores are still staffed.

Shelves need stocking alcohol. Purchases require an employee to manually check IDs, and shoppers need help getting used to the system. Sometimes when a store is new and a store owner will station someone by the gate and have them explain to people like, hey, here's how this store works and how you can interact with it.

At the exits, we saw some shoppers who were nervous about leaving without paying. It is perfectly valid to go like this. That's perfectly acceptable behavior.

And that's one of the cases where we know that you've just made a purchase, where that would have been theft in a traditional store. With a learning curve. Data concerns and large upfront investment.

The big question remains will just walk out ever hit wide scale adoption? There's almost a viral effect that will occur over time. It's just going to take a long time because you've got to cycle through everybody in America having this experience, this dies.

It will die because they didn't get adoption fast enough.