AI Agents Are Great, But...

can we talk nadn AI agents for a sec like are people taking this Tech selling them to businesses producing a positive return on investment for the businesses and then making some profit on the side or is it all just a bunch of hype so from what I can tell it's all just a bunch of hype I've sold several hundreds of thousands of dollars of AI automations over the course of just the last few years and I'm not going to bury the lead here from what I can tell the current iteration of nen AI agents is not at a point where you can realistically and reliably produce a return on investment for a business that you couldn't do easier and better through more conventional automation means conventional here just means the old left to right flow where you execute some trigger and then a bunch of actions occur immediately after now I'm not saying this because I hate AI agents or I think AI in general is a bad technology I run my entire life using artificial intelligence at this point it makes me all of my money it's all I talk about it's most of the content that I consume what I dislike is I dislike some of the claims that I'm seeing where this n runs my entire life it completely manages my business it does my laundry it pleases my wife right anytime you have some sort of technology or solution that supposedly does everything and cures everything you should be a little bit skeptical about whether or not it actually does what it's claimed to do those sort of thing is actually really common throughout history anytime a great technology is predicted and announced before it's actually fully developed you tend to have a big hype bubble where people overestimate the ability or capability of the current levels of Technology it's not the first time it's happened certainly not going to be the last time especially in our industry but I think if we're being honest with ourselves and if we wanted to produce the largest possible return on investment for the customer these n8na agents just ain't it dog they are not at the current level where you can reliably provide them as a solution to anybody and expect to make yourself a lot of money and that's ultimately all I care about okay so let me show you what I mean I have very simple nadn AI agent open here and what this is doing is it's getting a chat message after I get the chat message I am then communicating with the AI agent through an open AI chat model GPT 40 mini we're then storing the message history in the window buffer memory then on the right hand side here connect to this little tools Diamond I got a bunch of tools I could create an event in Google Calendar send a Gmail send an HTTP request to some weather API if I wanted to I could calculate stuff search up stuff on Wikipedia and then I could also access some Vector store that I put together in this case that just reference is the Bitcoin white paper Okay so we're seeing a lot of people say stuff like this end and agent runs my entire life the reason why I post I'm doing this right now and showing you guys what this looks like is just because like like why do people like this stuff cuz it looks really cool it has that wow factor when you you know put put a message in you're like hey how's it going you know it responds to you I guess sort of like a human would when you ask it to do something for you please book me a calendar event for tomorrow called Nick's birthday 2 p. m. when you ask it to do this stuff all right it actually goes down the road it creates the calendar event for you and then it like feeds a response back right like that stuff's pretty impressive from the from the bird's eye perspective so I guess what I'm trying to say is this can do stuff right we're at the current level of Technology we can actually go and it can do stuff but the fact that it can do stuff for you doesn't necessarily mean that this can do stuff for businesses to produce a positive outsized Roi all we're really seeing at this point is we're kind of seeing like a personal assistant and it's a nice personal assistant but you got to keep in mind that this isn't 100% reliable AI agents basically everybody here knows that is screwed around with these things they have a failure rate anywhere between maybe 1 to 5% per query um you know the better the model is the less likely the failure but keep in mind that since you're communicating with this flexible model it's not always going to get your category right so sometimes you'll say hey can you book me a calendar event and then maybe it'll do some HTTP request the weather API with some field called calendar right it'll hallucinate it'll invent some stuff for you so as somebody that sold several hundreds of thousands of dollars of these AI automations to businesses the number one thing that businesses tend to care about is they tend to care about ease of use and then they also tend to care about reliability a business has a very small window of available things that it does repeatedly in order to generate a return on investment for itself the more flexibility and Randomness and and variability and unreliability that you throw in the less valuable a solution like this will really look uh to a to a serious business owner that you know wants to produce some sort of uh outstanding Roi okay now contrast that with this flow this is an older School more traditional looking automation where we have three simple nodes a web hook that catches the request as it comes comes in then we are still using AI in the middle and we're also creating a calendar event over here on the right and I want you guys to see that this is the same sort of flow that people have been building for the better part of the last 20 years so there's nothing really new here but because we're using web Hooks and because we're using precisely formatted procedural events basically the reliability of a system like this is you know hundreds of times higher than that of a flexible AI system and I want you guys to see that we're still using AI in the system okay we still have an open AI model or node this node is pretty cool right it's actually doing something um as I'll show you guys in a second so typically the way that stuff like this works is I'm just going to test this we have some web hook URL that we'll paste in and you can add a parameter I'm going to say March 13 2025 at 1 p.

m. so instead of communicating with it in uh you know through some chat interface or whatever you might have an automation that takes the date of some trigger maybe it's I don't know your client books and onboarding call with you or something then you just grab the date and you can literally write it in plain text like natural language like I'm doing right here March 13th 2025 at1 PM okay when you put paste that in and I'm doing this through a get request which is why I'm getting that black screen but that doesn't really matter all too much when you paste that in you end up getting that event inside of naden with the date we're going to feed that into AI we're going to test that step that's going to convert it into our nice iso8601 format that's just a standard that a lot of calendar systems use and then finally we will test this too and we've now booked an event so why am i showing you these two systems um despite how much more boring this one looks uh businesses prefer this sort of stuff businesses prefer this sort of stuff because it works you know it's like Old Faithful it's the same reason that uh I don't know when like a new car model is made in the first year or something a lot of people that know cars typically don't like buy that model of vehicle why because they don't want to drive something that has a much higher probability of breaking down or whatever so businesses prefer like kind of the Old Faithful approach the vast majority of the time and in addition the benefit to a system like this is a system like this is substantially easier for a developer to set up it's also substantially more extensible and it's also substantially more maintainable okay this is basically a Lego block that you know works 99. 999999999% of the time it's like a pipe basically in some big offshore drilling rig pipeline thing you can tell I don't know my oil terminology but uh hopefully you guys get my point okay it's a part that you could swap out for something else the um AI agent sort of thing is it's not a defined part you know it's a squishy malleable thing and while that has a lot more potential over the course of the next few years than something like a rigid uh inflexible part you know right now it's just not really at the point where it can actually hold up the structure for us it can't really do anything of value so you can imagine hundreds if not thousands of different ways you could set up the same logic that I did there with that little get request with with the web hook with the open AI node in the Middle with the calendar booking but if you set them up they'll be 100 if not a thousand times more reliable they'll be substantially easier to interchange with other elements and stuff stuff like that and then the best part is you could always weave that flow into a future AI agent that actually has the capability to do this sort of thing more reliably in the first place now that you've seen what sort of the new school versus the old school way of doing things is and you know you sort of compared a little bit between the two I want you to ask yourself what intrinsic value does the first have that the second doesn't basically if we just be super realistic here the only real difference is the first had a little chat window so you got the perception that you were chatting with some sort of assistant or some agent right so that's sort of the benefit and and you know it is a benefit it is cool to some people uh you know some people work they do everything out of their phones for instance they don't want to have you know a form here and then some CRM there and some other system over there although you know if we're Frank if you're going to use an nna agent to update a CRM I guess the unspoken assumption is that you are going to be using that CRM at some time but I digress um you know like that is that is a benefit the fact that you have this one super cool simple screen you can host it on a website embed it on on your phone or whatever and that's fine for Less technical business owners but uh what is that coming at the cost of that's coming at the cost of reliability where the AI agent will sort of muck up your request maybe 1 to 5% of the time so you know one in 20 requests that you make just aren't going to work you might be under the assumption that they're going to work but they aren't how much money does it cost you when 5% of the time or one out of every 20 times or whatever the actual number is for crappy prompts it's going to be much higher for really good prompts it might be a little bit lower how much how much is that going to cost a business that relies on this a agent to do everything well it's going to cost it about 5% right like if you're using this to update a CRM if you're using this to send emails to people critical emails if you're doing this to I don't know add things to some sort of project board or tracker if you're doing this to automate some sort of cool task that maybe has to do with creating content you're essentially capable of doing 5% less there's also some potential issue where because you assume that some asset had been created or delivered um that might cause some runaway effect can actually impact your business more than that 5% but I'm just going to C it at 5% for now and that takes me to really this equation I'm going to put together the benefit to these AI agents is basically you get a chat box you get that little like cool factor which is nice as a business owner the cost though is you have I'm going to say around 5% unreliability so basically what you're doing is you're giving up I don't know man maybe about 5% of your freaking business for a little chat box that's cool any serious business owner isn't really going to give a crap about that they're going to want hold on to the 5% which might be you know $50,000 $500,000 a year uh whatever but like most things in life there's more there's another hidden cost that a lot of people don't talk about and that's that the prompt engineering of a lot of these built-in NAD AI agents um that's not you spell engineering The Prompt engineering is usually trash okay and it's trash um you know some of the it being trash is actually outside of your control so you can't fix it and that leads to some really funny situations where you'll send somebody an email using your naden AI agent assistant that you think is super smart or whatever and it'll just be a trash email there'll be tons of emojis in there there'll be weird confettis things will be formatted really oddly the tone of voice of the model is going to be weird like as a business owner you've lost a substantial amount of control over how you're perceived and a lot of the time this is in contact or communication with customers and it's like if you screw up the contact or communication with the customer you make them feel weird or something even even a little bit that can have massive Downstream effects on your ability to close a deal especially for one of these SBS not even talking about what happens at ENT priz businesses where that sort of perception might might be even more important so speaking of funny situations I just had it send an email to me Nicholas at gmail.

com supposedly wishing me happy birthday okay and it's responded with I've sent the birthday the birthday wishes to Nick and Nicholas arrive gmail.