Provalytics: AI, Ads & the Future ft. Jeff Greenfield

What do magic tricks and AI marketing have in common? A lot, according to Jeff Greenfield, CEO of Provalytics. In this conversation, we unpack the challenges of data privacy, the rise of Amazon and Walmart’s closed ecosystems, and why AI ads may be unavoidable. Jeff explains how his background as a magician and chiropractor shaped his approach to stress reduction and business problem-solving. It’s a fascinating, fast-paced look at how AI is reshaping advertising — and what that means for trust.

Transcript

0:17

welcome back everyone to another episode of the AI Rebels podcast as always I am your co host Spencer and I'm your other co host Jacob and we're very excited to have Jeff Greenfield on with us today who has a very unique background everything from CEO now of probabilistics to what I saw is this right Jeff that you were a magician is this yes this real OK yeah no it's real I wanted to make sure because that is awesome and I didn't want to be digging into that if that's not true so magician chiropractor is that also true that's also true yep my brother's a chiropractor so awesome I uh I kind of latched onto that so very diverse background and I think we're really excited to see how that has played a role in probabilistics and what you're working on now so yeah anyway thanks for coming on the show we're excited to have you here oh my pleasure I'm I'm excited to be here AI is uh everything these days you can't kind of turn away from it so it's it's making its way into every single sector every business and it's it's moving at light speed at this point which is yeah is awesome so true yeah we talk about it on here all the time how I don't know if there's ever been everything has been contingent on human minds innovating and I think AI is the first technology that is self improving like we can now turn AI on itself and say what can we do how can we get better and just use the technology to better itself it's really amazing and unique and terrifying at the same time better than us all reading self help books right yeah yeah as fun as that is yeah well Jeff tell us a little bit about your journey to where you are now with Provolytics what if how have you gotten to this spot in the AI space well it it it's kind of wild you know I I was in the I was a marketer after I was a chiropractor I was a chiropractor for a number of years I got injured in a car accident crushed a nerve in my arm so I couldn't take care of patients anymore and this is in the early to mid 90s so I kind of caught the early wave of the internet and ended up on the marketing side of things buying media for companies creating other solutions and one of my clients back in the 2005 2006 time frame had this issue where they were buying a lot of digital media and the numbers never added up meaning you know they would have 100 orders in a day when they add up what their partner said it would said that they had like 800 or 900 because everybody was taking credit for right and then they were also paying out commissions too that was the other problem that they were paying out multiple commissions so we had to create a solution to fix that so I built out what became C3 Metrics C3 Metrics one of the first enterprise multi touch attribution platforms and I scaled that company up and that company was all based on collecting 100% of the available data so that meant you put a tag on a website like G a 4 you collect all the clicks mm hmm any place that ads were running like on Yahoo or Google or on Facebook there would be a little tag there so you could collect all of the impression data and the system worked on the on the fact that you could collect all this data line it all up and and run machine learning on it and I scaled that company up and exited there in 2019 about a year before the pandemic thinking 12 years was enough I didn't know what I was gonna do next but I wasn't going to do measurement cause measurement didn't seem initially very sexy to me marketing is sexy you get to come up with big ideas but the reality is measurement is sexy because you're in the middle of all of these marketing plans you get to see what some of the biggest marketers in the world are doing which is pretty damn cool but during Covid we saw this like shift happen where the internet shifted to more a privacy centric world which meant the data that was used for measurement and also for targeting was now not gonna be available anymore people had to officially opt into things versus companies just collecting all this data and that created a problem in terms of how you measure because if you've been used to measuring for 20 years collecting everything and now you can only collect 20% of the data you can't make decisions based upon that so I saw an opportunity to come back in and be able to provide marketers the same type of output the same type of insights if you will telling them where to spend more where to spend less using less data and then allowing high level math statistical modeling and AI fill in those gaps and this is all around answering the famous question which is half the money I spend in marketing is wasted the only problem is I don't know which half and that's a problem right it's plagued marketers since the beginning of advertising and so now it's even more difficult because there's less data available but the reality is is AI helps kind of complete those gaps for us which is amazing I'm really interested to to hear um how do you measure the the efficacy of AI in filling in those gaps is it well in terms of looking at historical trends or yeah so in terms of looking at the efficacy of it Spencer it's really all about looking at how predictable is that model because what you're trying to do is is that we're not living in a deterministic world because we can't get all that data in fact we used to call it deterministic modeling because we had everything but the reality is is that you can't get inside somebody's head so nobody really knows what makes people act on something you can ask someone how did you hear about us and they say oh I saw an ad on YouTube but they may have forgotten that they saw 12 TV ads before they saw the ad on YouTube type of thing so but what happens is after the model is done and during that process you go back and you do what's called a validation on it and what I mean by that is you take typically 12 months worth of data and we take daily data it's very granular and we train the model for a month and we say essentially here's how many impressions there were for radio here's how many there were for TV here's how many there were for Facebook so we give it the total impressions at at very detailed and it's not just Facebook it's Facebook this campaign this ad set this creative had this many impressions this many clicks on this day we spent this much so we give it all of that data for every single day and we tell we tell it and this is how many orders came in we'll just use orders as an example and we do that for a month then we run the entire model through from that point forward five times and each time randomly selects days 20% of the days where after it's given those detailed impressions and holds back the orders and says okay model you have to predict what what is going to be the order count so you do it five times you end up holding back 100% of the data and now you can look at every single day for a year and you can see how many orders were there actually how many did the model actually predict and then from that you can pull out statistical numbers that are important to folks in this space mainly what's called an R square which tells you essentially how well the model predicts and then a MAPE which is kind of that error band if you will and for us when we have good data and when I say good data cause sometimes data comes over and someone says this is this is it and then we find out later on that there was something wrong somebody fat fingered something but when it's good data the models are typically R squares 1.9 or 1.93 and MAPE's that are in the single digits and so so what does that mean that means that to a marketer it says well this is something I can trust and maybe I should test it because most marketers don't don't have predictive models that they're using they're just using this kind of last click last one in wins type of of stuff so our models are always tested every time we get new data we reevaluate rerun the entire model because we want to know is the model holding up is it performing as well as it did before or is there something odd strange essentially that's going on yeah so you're building a model for each project each client is that the idea yeah it's it's a the the math is essentially the same that kind of all stays the same the the the platform is the same itself and it's built to create unique models based upon the data that comes in there so it's essentially agnostic you could say it's you know people say well there's a big difference between buying a car and buying um eye cream if you will you know people buy a car once every couple years people buy eye cream every couple of months the model doesn't care whether it's cream or auto yeah at because it's built from the perspective of saying what is it that marketers are buying well this is one of those things is that marketers invest dollars and most marketers today are confused they think that you're buying clicks cause clicks lead to sales and the reality is is that you're actually buying attention yeah and that attention leads to awareness and when awareness is built up that then leads to people walking into your store and that could be clicks or walking into a retail establishment and then making a purchase huh so what the I'm curious if you have any successful use cases or stories of clients that have built a model and what kind of results like the before and after picture essentially oh yeah yeah absolutely I mean this is the thing is that most marketers are very data driven meaning they follow the numbers even if the numbers that they're given are wrong so I can I can think of one client that when they came to us they're spending about uh$10 million a year in marketing so not not a for for a lot of marketers that's huge for others that's relatively small but 10 million is a large investment uh but 80% of it was spent based upon where the numbers were telling them to spend which was on Google search which is very down bottom of the funnel because yeah they were using Google Analytics to tell them what was working and what wasn't working and they found that Facebook wasn't working that well for them they had tried it and they weren't spending that much on it connected television their competitors were using they tried it but it didn't work for them because it didn't show up and they had tried other things too and none of it worked but paid search worked and they believed in it so we went in and we looked at not just what they had done recently but we went back an entire year and when we went back that entire year what we found is that that connected television actually worked really really well but it didn't translate into clicks cause there's nothing for you to click on and connect to the television they were just measuring based upon clicks same thing with Facebook that Facebook was working for them but they didn't actually see it because what was happening is that someone would be looking through their feed they would see an ad like a video they would scroll past it and then like a couple of days later they'd be on Instagram they see the video again now they'd watch it and then they would make a mental note to say hey that's pretty cool I'm I'm gonna get that the next time I run out of whatever it is that it was mm hmm and then two weeks later they're like oh my God yeah what was that thing called and they go to Google they type in the name cause they remembered the name they clicked on the Google ad and then they purchased it and so oh interesting so that's an example where Facebook built the awareness Instagram is really what closed them but they navigated to the sale through Google and in their results in GA four it showed that Google drove the value multiply that a couple thousand times at the end of the month if you're a smart marketer you're gonna cut your spend on Facebook and you're gonna spend more on Google and that's where we came in and that's exactly what the situation was so what we were able to do for them is we were able to show them that they had reached what we call a point of marginal return with Google it was oversaturated meaning they were the problem with Google is you can keep spending and they'll keep cashing those checks but yeah right your your your cost of acquisition will continue to rise your your return on ad spend will go down but Google will still take your money and what we were able to show them is just by moving a small percentage of that money over just 10% in the beginning back to Facebook they were able to increase their sales by about 20% and so that gave them enough power to be able to say well let's move a little bit more over and then after a couple months they were able to walk into finance and ask for more budget to now go back and do more connected television and so again it's all about half the money I'm spending is wasted I just don't know which half this really answers that question of which half and not only that but then it tells you how to redeploy those dollars yeah so that you can get a bigger bang for your buck wow so I'm I'm curious backtracking slightly you said that you use AI to fill in some of the you know some of the missing data essentially I'd be curious in hearing a little bit more about that is it a case are you using you know generative AI to create data that fits particular profiles um or is it you know some other some other method of no great great great question Spencer so what we did is when we built out the methodology in terms of what we use and what's fascinating is that we're big believers and this idea of kind of back to the future if you will because you know there's there's a lot of new stuff that's out there but it seems like the old stuff tends to work best one of the things as we talked about before because of Covid we lost all of these signals if you will we're kind of back to how things were before digital was around right where all you have in order to buy is I can go on Facebook now and I can buy a specific city maybe even like a zip code or an area like that I can buy a certain demographics but I can't go to the level of detail that they used to have like where you used to be able to bid on Ford F1 fifty leaseholders whose lease was gonna expire in three months that kind of stuff is that stuff is gone so what we did is is that we went back to use techniques that have been around for a while one which is very popular is something called Bayesian bayesian is a calculation that's actually been around since the 1700s and then we use a technique that was actually developed in the early 60s out of the university of Chicago called seemingly unrelated regressions and that allows us to solve all of these equations at once where we actually use AI in the platform is that when we built this out it was a four to five part process where data would go in it would complete the first process and then it would get evaluated by human and then there was a series of things that the human would have to do based upon what they saw and then they would have to start step two and so on and so forth so we started developing this in 2022 uh and we really started and then we built out our own kind of internal AI you know automation you you could call it AI you could call it intelligent automation if you will to carry through and do those steps that it used to be a human to do and what that allows us to do is to speed up the process itself so that there's no human intervention at all we check the data as it goes in and then it gets spit out at the other end nobody touches it wow the next step for us is we take the data it gets pushed into dashboards which is what folks in our our world like to look at but what people want and and you know our customers or marketers where it's just like any job you know things are going great and then your boss asks for something and you're like oh my god I don't I don't even know where to start with that and so what ends up happening is is that they end up reaching out to us to our team saying hey yeah I have a meeting in 35 minutes can you guys help me with this I need like one or two big insights to talk to my boss about oh this is great can you put it on a slide for me and I'm like this is a perfect job for a large language model a perfect job hundred percent we're now in the process of taking some beta datasets from our clients and training it with LLMs specifically only on their data and the access that we're gonna provide our clients based on the feedback that we've had is via text message that's what they like they wanna keep it out of Slack cause they want it to be an individualized thing so imagine you have a 24 7 uh real time AI analyst who knows your data inside and out who can make slides for you and give you insights yeah that's what our customers want so it seems to be that's a perfect fit for where the large language models are today if you will that's so that's super interesting yeah cause I've I've long had this I've been I'm a web developer by training and I I have a a passion for design well let me restate I have a I don't have a passion for doing design all the time um but I do have a passion for good design uh but that that aside from a web development perspective I've often thought that uh we're headed to a world where we're gonna end up with with a lot of generative UI right you know uh interfaces that are created on the fly for us um and this is a to me a really interesting realization of it in a you know in a rather oblique way but I've often thought that the first place we would see this happening is for analytics platforms um platforms with massive a raise of number at their disposal and then a steep learning curve to marshalling those numbers into something cool so that's that's a that's really fascinating and I'm super interested that they all said you know that that text messages how they wanted it delivered yeah so I'm I'm interested when you deliver those text messages is it just like a link to the slide that they generate or or what goes on there yeah that's that's the plan is to give them a link where they can go and pick that up many times for our clients you know the slide is kind of an afterthought it's really about like what's a bullet point or two that I can say about the numbers in the last month cause really it's about imagine you get in an elevator at work and you're riding up with the CEO and you're in there for like you know maybe if you're in New York City you may be in there for eight or nine minutes and you want to have something important to say and you can text and get an answer really fast I mean that's that's really what this is and that's the cool thing about having a via text is that no one else has to know about it it's it's only you for that one person but going back to what you said earlier Spencer about kind of the UI where where AI gets really fascinating from a marketing perspective and this is what's makes larger advertisers nervous so larger advertisers it's all about you know there's a saying I Learned many years ago it's more important to protect the brand than it is to promote the brand because that brand and I've spent a lot of money creating that brand making certain that you and I have agreement with that brand there's an emotional component to that brand yeah and so when I advertise you know everything has to go through legal it's got to be approved and everything like that with what's happening in Facebook and remember Facebook and meta was really designed for 90% of their business are small local advertisers your local florist your dentist stuff like that all you have to do is type in your name of your company and they will automatically generate ads for you which is really cool but the next phase of it is not only will they generate ads but Spencer you'll get one ad Jacob you'll get completely a different ad because yeah Spencer you like the color green and Jacob likes the color blue and Facebook knows this crazy or you'll have a different person in the background than Jacob Will and they'll be different messaging based upon where Facebook knows you are in your research journey if you will and that's really cool for local small businesses that is fantastic for big brands that is an absolute nightmare because for a big brand someone would take a screenshot and say what is this it would go up on on X and and there'd be a firestorm and people would get fired so but for small businesses it's it's phenomenal and that's that's kind of the next wave of this the problem though for brands even small brands is that you know 70% of the effectiveness of all ads is based upon the messaging and the creative and that emotional component of it and that's having someone else in control of that and individualizing it you don't get much insight back as a brand and you become very dependent on them and that's potentially one of the issues there yeah so orienting back to Provolytics here directly I'm really interested so you you mentioned that you know and you mentioned this on your website too I think that um we're headed into a world of wild gardens as far as the internet goes especially with the advent of AI agents proliferating on the open web um so I'm intrigued to hear are you guys able to like say say Jake and I are chatting in discord and Jake is like yo have you seen this toothpaste it's great I'm like oh no I haven't seen this toothpaste it does look great and then I go Google the toothpaste I'm interested to hear is there anything that you can do to kind of track chains like that or is that a case where it's like you can only cover so much it is a case of you can only cover so much you know there is that aspect of what you would call word of mouth if you will used to be back in the day before digital word of mouth was you know somebody would tell a friend if they had a good experience they'd tell three or four people each of them would tell three or four people and it would kind of explode like that and there was that aspect that would be like the organic growth of a brand you couldn't track it but it came from positive experiences if you will what's cool is that you can kind of see the after effect of that word of mouth online cause you can look at conversations and Reddit you can track brands who've had a lot of discord conversations reddit conversations and you can use things like Google Trends and you can see over time that the searches and the engagement around that brand goes up and and you can use things like that as kind of like a brand tracker if you will and there are companies out there that track brands and they what they'll do is every week they'll ask people have you heard about this brand what do you feel about this brand and you can use things like that as almost like a sentiment score if you will within the model to see how people feel about it and the reason you wanna do that is because what you don't wanna do is overvalue the impact of your advertising so right I'll give you a good example of that you know like for example right now there's a lot of companies out there that are mortgage companies that are advertising to get people to refinance their home and get a mortgage and then all of a sudden sometime the next couple of months rates are gonna drop and then and then people are gonna start going crazy mortgage applications are gonna go up through the nose now is that the result of advertising absolutely not there's an external event that occurred now we can pick that up in our models but it's also important that models are not mind readers so the more data we can give the model the better the problem is is that it's a catch 22 because information like interest rates and these numbers tend to come from you know the federal government and other sources different universities and so you know the data from last month may not be available for another two months yeah yeah so as so as a marketer you have to say to yourself do I want to wait three months to get an answer about what happened three months ago to help me make my decision this month or would I rather have something that is directionally correct faster and our philosophy is let's be less wrong than we were last month and one thing we can do about being less wrong is getting answers as quickly as possible because otherwise we're just gonna go with our gut and go with the GA four data which we know is at least we know probabilistics is less wrong than GA four definitely yeah right it's better yeah I mean it's a tricky thing I mean this is a common problem we're seeing is this idea of how do you have a live data how do we how can we integrate that into our models our platforms and it is a very tricky I mean it was not that long ago that like there was a cut off at what was the year what year did they cut it off with data I can't remember oh you were 21 with the LLMs yeah yeah it was like a it was like a it was it was like 2024 2023 something like that yeah and now it's going off and searching the web but here's what's interesting what we find with our data with our data and this is this is kind of and and maybe it's cause of my age and stuff like that but there seems to be there's an addiction with real time especially in marketing we've got marketers used to the idea that interesting that I'm advertising and I should be checking this data every hour and if things aren't going in the right direction I need to go in and change something and that trader mentality yeah that's that's actually a great a great analogy and marketing is not day trading it's definitely not it it's more it's more the Berkshire Hathaway method back back in the days of Mad Men they would plan out a campaign and they would launch it and then they would sit back and wait three to six months to see what would happen and I'm not suggesting that because with the tool sets we have today we can actually measure how long it takes for things to actually have an impact yeah we can measure it so we can see that you know for particular client their paid social is taking let's say 18 days on average to convert so what that means is is that uh we should probably wait 20 to 25 days to let everybody kind of go through a full cycle before we make a decision to cut a campaign or to or to double down on it type of thing so we have kind of these guardrails that let us know how often we should do things and even though we could update our models daily for clients we don't because we don't wanna encourage bad behavior we kind of we kind of encourage our folks to kind of sit on their hands I think there is a tendency for folks at least in my agency in in in my world to wanna keep themselves continuously occupied with the most updated bit of data because you know it seems like they're more up to date than anyone else and the reality is is that hey being a couple days behind isn't that bad sometimes you have a it's better to pan the camera back you get a broader perspective on what's actually going on when you're in the details of that of that minute to minute you can't spot trends you can't get the insights you need because you really need to in order to steer the ship you need that broader perspective got it so when you're importing let's say you're building a model for a company are you connecting to their live data for that or is it some sort of an export how does that process look as far as that's a great that's a great question so we have the capability to go let's say you know most people advertising on Google and most are advertising on meta and other places and there's these public APIs that are available and they could click a button give us access and we could automatically suck all this data down but and there's a big but most of the clients that we work with they have multiple partners or agencies that are doing their buying for them and so in order for our stuff to work there's like a couple of phases to it which is we provide insights but in order for those in order to to find that missing half we can find it for them but in order for them to actually redeploy it and spend smarter and better they actually have to act on it and the people that are acting on it in many cases are vendors that are working on their behalf advertising agencies yeah and so imagine this situation imagine you guys have your own ad agency you don't know about us the client hires us unbeknownst to you we we get access we go in and you guys are buying let's say Facebook meta and stuff like that so we go in there and we get all the data and then all of a sudden one day you're introduced to us and you're being told by us here's here's how you have to do things from now on you're gonna be like who are these people what what is going on here I thought yeah right great right yeah so in order for the client to have the big win we need to get everybody we need to build consensus with everyone across the board so what we found you know after 20 years or so almost of doing this is that being introduced in the beginning of when working with a client with their agency and becoming a partner with them and having them send us the data instead of us going in and grabbing it although it would be more convenient for them but sending us that data immediately puts us in communication and that's kind of the first stage because what's going to happen is is that they have a team of experts and you guys have a team of experts that you're new ad agency that know meta inside and out you guys understand the signals you guys know how to buy it and all of a sudden you know about a month or so from now we're gonna come with with new walking papers that say hey ignore what Facebook is telling you and follow this and you're gonna be like I'm an expert why would I do that and it's like well because meta doesn't know about the connected television meta doesn't know about all the billboards meta doesn't know about the direct mail meta doesn't know about the in store promotion that at several thousand targets around the US meta doesn't know about anything like that but politics does and so but we have to be in communication in order for us to build up some trust where you're willing to even test it does that make sense yeah yeah so for us we we're kind of old school that manual process plus the larger the client the more money they're spending we have some that are spending you know three to five hundred million dollars a year most of the data that they have is not available to click through an API some of their marketing data is but it's hidden behind firewalls we have to go through security checks and all that stuff and and a lot of the TV data that they're buying isn't available sometimes it's like a two or three week delay in terms of getting the data back that we need for the models got it so slight Tangent to that question so you know it's it's very popular for people to say you know the numbers don't lie but the reality is that we're very good at lying to ourselves using numbers um and I'm curious to hear does Provolytics do you wrap your numbers in a more opinionated frame so to speak of trying to you know point customers like hey like interpret the numbers this way or do you just promise customers like we will give you you know better more interesting raw data than meta analytics or Google Analytics well what we promise them is and you heard me say this before Spencer it'll be less wrong than what they're used to looking at but I think you hit on something incredibly important which is that you can take the same set of numbers and one person can look at it one way and the other person can look at it from another way it it I always talk about the car accident and there's people on each corner and the cop asked one person what did they see and then they go across the street and they hear something completely different and they have to interview everyone and kind of put together that complete picture and the reality is for our customers when they come to us they're usually seeking answers to specific questions that's usually what starts their search like why is it we can't get CTV to work and how do I get finance to to give us more money to pay for something that I can't prove is actually working and so so we start off with the numbers to answer the questions that they need to have answered that's where it starts at and then from there then we're able to kind of you know the numbers are the numbers but the insights that you pull from those numbers are really what matters to people cause raw data is just you know it's like people just like you know cause it's very easy you know I've seen reports from agencies before where it's an email with 35 bullet points of stuff and it's just like what am I supposed to do with this can you give me one sentence that tells me the most important thing cause I always tell the story that you may be putting together a presentation as an agency for your client and let's say it's 15 pages okay and it provides all the insights of everything that happens well now they have to present their numbers to the chief marketing officer and that chief marketing officer only wants it on one slide so now the CMO has one slide and the CMO has to give it to the CEO for a board presentation and so your 15 page deck goes down to one slide which then becomes a single bullet point for the board and so you always have to think in terms of what is the most important thing that people need to know from this most recent look at the data what's gonna change things the most for people and what is the perspective that they're looking at right now hmm yeah I I used to coach competitive debate um and there were a couple cases where there was a statistic that I told my students to use on both sides of the case that they wrote because you know it just matters how you spin it and then and then similarly to your point about like you know all of all of those bullet points being you know condensed down to a single bullet point sorry this is tangential but it was leading somewhere that I forgot now it's just tangential but anyways I like to call that like the outcomes versus the impact cause a lot of people conflate the two and they think that you know outcome and impact is the same thing and and it's not entirely true there's a lot of outcomes that are non impact excuse me non impactful um so it's interesting to hear about like the focus on answering specific questions I think that's really powerful and interesting approach and LLMs are perfect for that exactly even if you're not sure which one to pull from you can you can have a conversation with one of those that are out there like a chat GPT or others and it'll give you kind of like that distilled bullet point that the CMO yeah would give to the CEO and then you can look at it and say yeah that's not gonna help my client or no that's exactly what they're looking for and what I find where LLMs at least for me and my day to day life are useful is awesome writing partner awesome editor uh in terms of that and I encourage and I train my team how to use them on a regular basis because it's a tool just like anything else uh and it's silly not to utilize it but yeah you know really really good for summarization and pulling out those points and I think as well for people that are very junior in any industry finding those insights are very very difficult and just having a conversation with an LLM can help you gain and train yourself on how you would do it on your own as well too cause I don't think we wanna rely upon these always to pull those things out right right definitely not and plus yeah learning to do it on your own as well you know unlocks a higher level of LLM usage cause then it's if you know what a good answer looks like and you know your your field well enough you can start pushing the LLM to give you better answers ha ha yep that's 1,000% yeah without a doubt I've used I've used the LLMs to do some coding and languages that I don't know anything about but I do know enough about coding and development to know what's capable what can be done and I also knew what I wanted at the end so the more you know on your own the more you can extract from these awesome tools yeah totally agree it's I feel like I say this all the time but people think AI is this magic box and you just have to write a sentence you can say whatever you want to say and I'll give you the perfect answer and it's not not like that at all there's there's a lot of skill involved that's why people maybe it's starting to go away now because models are getting better but like prompt writing was this whole people were hiring expert prompt writers for a while because it really is there's a lot of best practices there's a lot of application of honestly managerial principles that we've tried to apply for a long time that you have to use with AI especially when you get into the world of agents and it's I wish I could just help people see that though the technology is novel the best practices and how to use it is not anyone can use it do what just use it in the way you would use other tools there's nothing complicated nothing crazy about it it just can unlock so much more capacity I the way I like to put it is is AI is like your your coworker Kyle like he's very good at his job but he's also really into energy healing so you better double check everything he says I will tell you guys one thing I sat in on a call about a month and a half ago with um there was probably about 20 of us and there were I think six or seven teenagers from 14 to 16 from all over the world and the purpose of the call was for us to ask questions about their AI usage and their usage of chat GPT interesting and these different engines and the things that I was fascinated by because we've talked about we started this conversation talking about how fast things are moving and how fast things are changing but to a 14 year old things aren't moving that fast yes it's like it's just it's just the technology so it's it's it's interesting that it you know when I go back and I remember Altavista av.com was the the top search engine and then Google came out and you know I could see how to someone who was like 20 years older my god what is what is happening here so to them it's just technology it's it's not moving that quickly right none of them said it feels like it's moving fast at all so it's just to us because it feels like our foundation is kind of it's cracking or it's broken or it's or it's moving or expanding but the reality is is that even though it feels like things are accelerating at a faster rate they're not to the younger people yeah yeah I hate to tell you both that you're old but you are old I know God dang it well I got it in find a new gig then podcasting is a young man's game alright Jeff I have a question that I have just been so curious what you would answer to and it's not I mean it could be related to probabilistics and everything but it's going back to your magician days and I'm curious what lessons you Learned as a magician that you now use both in AI and the business world well the the biggest lesson is that humans think in a linear fashion so when we think to solve problems and I wanna go from a to D we go a B C d magicians know this and we're able to exploit it because we know that even though you may say I'm not gonna go a B C d you're you're going to do that you just can't stop yourself it's the way the human mind works and so that that's that's a huge lesson both from a development standpoint of view and a technology standpoint of view is it things don't have to move in a linear fashion it definitely feels like all of us right now that it's not moving in a linear fashion right that that kind of acceleration aspect of things but from magic and you know the way I always viewed magic is that for most people who were viewing magic and were entertained they saw magic as a way to kind of think outside the box as a way you could say is like a stress reduction cause it takes you to a different place it expands your way of thinking and you know for some people it it gives them the belief that maybe there is still real magic in the world but more than anything else is that it puts you in a different place and kind of reduces your stress yeah when I was a chiropractor my job was to reduce stress on somebody's nervous system and now in the field that I'm in now in this world of marketing measurement where AI is a tool that we use really what we're doing is we're solving the biggest pain point that marketers have the biggest pain point that organizations have which is trying to get the numbers to line up because they don't so you could say that I'm really I've been in the stress reduction business my whole life that's kind of an expert the lesson that I kind of pull all the way through everything is is is that you know stress is not good for any of us so anything we can do to make our lives a little better and a little less stressful is is a good thing yeah I'd be interested one more slight technical question has it become more difficult to put these models together since the advent of llms or was the phase change really the privacy changes that happened just immediately post covid from our aspect it was the privacy changes because essentially you know there was kind of an understanding as part of the internet ecosystem the advertising ecosystem that all this stuff is free it's paid for by advertising and and in return we get all this data and then we can use that to hyper target we can use that to measure and then everything changed literally over the course of a year or two and then on top of that most people used to spend most of their time in the digital world on what we would call the open web you would go from website to website to website now these websites essentially are as we talked earlier these walled gardens their own ecosystems and it's still just a website but what there's a wall up that says hey you can't get any of this data it doesn't leave here so let let me give you an example if you're a large CPG brand selling detergent let's say you can go to Amazon and you can say I wanna I wanna I've I've got a list of my customers I wanna target my customers on Amazon and they say OK come here hook up and we'll do a data clean room you're gonna pass us your data it'll be anonymized and we'll find your customers on Amazon and you can spend you can buy Amazon ads to target them great oh wow this is working great we saw a lift in sales I'd like to take that same list now and go over to Walmart with it and they're like oh no so now I have to go to Walmart and do the same thing with Walmart but Walmart data is not gonna talk to Amazon data which is not gonna talk to Disney data so yeah it's like it's almost like Amazon and Walmart have their own internet it's their own universe which is great for them bad for the advertiser that's the problem I was gonna ask yeah so there's there's this feels like there's this balance that has to happen between rent seeking behavior and advertising right I've never personally been bothered by advertising that much cause I just kind of you know I just ignore it I usually I'm very good at ignoring ads um anyways but it's interesting to see there's been kind of a backlash against advertising as a funding model but in reality like I don't know how you have the open web without there was a there's a there's a a Open AI member of Open AI who is popular on Twitter room and he posted the other day something about you know along these similar lines and it got me thinking hard about you know where do LLM providers go and I feel like inevitably there's going to be ads injected and I'd be interested to hear your both of your perspectives on on that prospect cause on one hand I don't like the idea of it because I like you know the idea of my my chat remaining pure and unsullied but on the other I sort of get like you know there's there's there's a necessity of funding here and you think back to public commons in the past that were commercialized right newspapers etcetera like they were all running ads and that's that's how they got their funding anyways long way of saying I'd be interested to hear both of your perspectives on you know yeah prospect of ads intruding into something you can know them and the also the the impact of you know walled gardens versus open spaces governed by yeah fueled by ads I was gonna say off of a point you actually said earlier on Jeff where you mentioned and Spencer you mentioned too this idea of generative UI and then like meta building ads and unique ads based on the individual and honestly it makes it's kind of scary at when I start thinking about the way an LLM could advertise because if it's if we're not careful it could very quickly turn into it's just blending the advertisement into its answers to you yeah yeah and it becomes like you're not even aware you're not aware like a newspaper right it's the classified you go to that page you're seeing ads you know their ads but it becomes a very fine line to walk where who's gonna be checking the ads that chat GPT is giving me if it's not even there's no delineation it's just its answer and I base my decision on that it can get scary it's very similar to Google yeah Google started with two ads on the right hand side and then they put one up top and now depending upon the search it can be the entire first page if you will so yeah intellectual advertising has been a part of the growth of the digital advertising ecosystem but going back Spencer to what you were asking about in terms of is advertising going to be here we don't have to go back that far we just go back to streaming yes anytime you have something a product where your cost to deliver that product increases exponentially is more people utilize it you need to have a monetization platform behind it that's based upon usage and subscriptions didn't do it the streaming companies you can't can't do that you can't do a a free model you have to have some sort of subscription right subscriptions they were still losing money but advertising works because the more people that watch the more minutes are available so streaming up until five years ago was the most expensive undertaking for companies they were spending billions hundreds of millions of dollars to buy content and there was a war to buy just about everything if you had a series that had that had run once before someone was gonna buy it because they needed to fill their catalogs up but now we've found something even more expensive to run these large language models and so right the only the only thing that will pay for it is advertising and if you go back over 10 years worth of of investor quarterly calls with Netflix and Reed Hastings they talked about how there will never be advertising on Netflix and now there's advertising and it's growing and it's doing really well for them yeah that kind of open the door for all streaming to be able to do the same thing we've already seen in some of the earlier code that there's integrations for it we have also seen you know taking this contextual aspect to another level a lot of people are utilizing and I think what they're doing is they're seeing what are people using this for one of the things that they're seeing people are using this for is what's the best air filter for me in my situation and it goes and searches the web and finds the best ones and and tells the person this is why this is the best one for you would you like to buy it click here and they won't even go to shopify there's gonna be a complete integration in there which is going to upend other business models on the web and stuff like that which is going to completely it's gonna feel like things are moving very quickly but to today's 11 or 12 year old it's just gonna be business as usual for them so you do it yeah but that's that's gonna be the way they monetize the way they're gonna monetize is have a direct link to commerce which makes it very powerful which makes it almost when you think about Amazon Amazon has advertising and commerce all built in so it's it's kind of its own complete engine when you when you start to think about it Google has advertising but there's no commerce you have to click to go someplace else Facebook has started integrating through Instagram it'll it'll bring up a picture of it TikTok has a TikTok shops which are directly integrated chat GPT is gonna go direct as well too but it'll it'll be part of that same window that'll be there so that'll give them that same end to end that Amazon has which makes it incredibly incredibly powerful but from a consumer standpoint of view fewer clicks makes my job easier I I I trust this this this this model it hasn't hasn't pointed me in the wrong direction yet so I trust its recommendation so and in return chat GPT or whichever model it is gets a gets a commission as like an affiliate yeah right right and probably also from that click if you will also gets paid for that click too so double dipping just the same way that Amazon does so more power to them I think that's the direction that things are going but that's the only way that they will ever pay for all the power requirements and the chip requirements for this you're right that it scales that's the only that's one of the few revenue streams that can scale with usage that's right yeah no it's it's something that I've been thinking about a lot lately cause I my knee jerk reaction is disgust I don't like the idea right but then I think about it more and I'm like well you know my revealed preference kind of says differently it's not like I it's not like ads on every other website have driven me away from the internet yeah right I spend too much time there right hahaha yeah that's the point and so it's it's an interesting tension to me between you know stated and revealed preferences and I I and I wondered how that's gonna go when it and I and I wonder too which company is gonna be first to do it is it gonna be Google they already have you know a massive advertising engine is it gonna be meta I wouldn't be surprised Zuck loves to monetize they'll get so much hate and then everyone will do it yeah exactly it's gonna be it's gonna be fascinating to watch yeah yeah yeah it's gonna be very interesting like you said moving fast for us not for everybody but for us moving fast that's right that's right and also remember it's not always the first company that wins right um yeah if you if you go back in the history of Microsoft Microsoft was very slow to the internet they sat back and they waited they jumped into this one a lot earlier but it's gonna be interesting I think we're gonna see a lot of kind of late bloomers that are gonna come on board and then also you know the real question is is that these initial entries here you know the perplexity and yeah and the open AI's they may power other companies that kind of layer on top of it so right you could look at something like an open AI as to be like an AWS if you will for AI yeah that'll be interesting it's gonna be interesting to see where this where the business models come out five years from now cause it's definitely gonna change yeah it will well Jeff this has been an excellent conversation really fascinating yeah very fast just it's just a totally different little niche little view on this this whole AI revolution that's happened yet it's been really good yet Jeff if people want to follow you well actually first we always like to ask at the end for a quick piece of advice for followers who may not be technical who are scared who are not super well versed in AI what advice would you give them to stay even though it's moving fast for many to stay current to figure it out I I I would say you know I mean this is kind of what I've used for myself not just with AI but with digital and the internet things have always been moving always been changing you need to you know dedicate time each week to challenging yourself to learn new things yeah I think that's really the key you know and and you know don't feel like you're behind because you're you're you're you're always gonna be behind there's always gonna be people that are there's always people that are out there right now that are innovating that are thinking ahead that are coming up with the next the latest and the greatest so you're you're not gonna be them necessarily maybe you will be but the most important thing is to you know give yourself an hour or two each week maybe it's the weekends yeah where you say hey I'm gonna figure out how to use these agents that open AI has cause I read something about it and just play around with it and it doesn't mean that you're gonna start using it but at least you become familiar with it yeah something you at least have a conversation about so every week I try to teach myself something different cause any advice I give right now if this airs a week and a half from now it's gonna be outdated at that point but the main thing is totally what will never be outdated is to challenge yourself and always look for new stuff yeah that's awesome thanks Jeff my pleasure if people wanna follow you follow everything you're you're working on what are the best channels best platforms for that our best platform would probably be LinkedIn find me on LinkedIn if they wanna learn more about provalytics you can go to provalytics.com or you can go to get Prova which is G E t P R O v a dot com Prova is uh means proof in Italian so get proof cool that's what it's all about I love it love that awesome well thanks Jeff we'll definitely stay in contact we're excited to see where you go in the coming months awesome thank you yeah thank you