#130 Productizing Decisional AI

Subscribe to get the latest

on Wed Mar 22 2023 17:00:00 GMT-0700 (Pacific Daylight Time)

with Darren W Pulsipher, Matthew Pulsipher,

In this episode Darren interviews his son Matthew Pulsipher about productizing decisional AI. Matthew has recently modernized and product development pipeline to include decisional AI in his product development.

Keywords

#ai #decisionalai #aiproductization #people #process

Listen Here

In this episode, Darren Pulsipher, Chief Solution Architect of Public Sector at Intel, interviews his son Matthew Pulsipher, a Product Manager, about the productization of Decisional AI. Matthew explains that generative AI is built on general datasets and is good for general knowledge questions but lacks predictability and determinism, making it difficult to automate processes. Decisional AI, on the other hand, is simpler in scope but more focused in context, allowing it to make data-driven decisions based on specific company needs. Matthew shares his experience integrating Decisional AI into products and highlights the importance of context in AI.

Type of AI

There are different types of AI, and each has a unique ability to help organizations automate processes, make business decisions and augment human work. Decisional AI is primarily used for making decisions and is based on models generated from previous data. Predictive AI, on the other hand, generates predicted values based on custom models and data sets. Training models is critical to the deployment and implementation of AI solutions. The key is to identify a real problem that is operationally relevant and achievable within a reasonable timeframe.

AI to streamline processes

Artificial intelligence (AI) can be used to streamline decision-making processes in businesses. It is important to scope the AI’s capabilities to a specific set of options, so as not to overwhelm the system and make the decision-making process more efficient. AI is best suited for processes that are repeated, involve data-driven decisions, and require subjective human reviews. For example, a financial institution can use AI to validate driver’s licenses using extracted data. Continuous training through user feedback to the AI can improve its decision-making abilities and eventually replace the need for human review altogether. Ancestry.com’s indexing project as an example of how reinforced learning can reduce the need for human involvement over time.

Human in the Loop

When building a machine learning backend, it’s essential to keep the user’s needs in mind. The goal is to streamline their current process and provide an aid to help them do their job more effectively. To achieve this, it’s crucial to interview and observe users in their current environment to understand their behavior, identify inefficiencies, and document any inferences they make that may not be documented. By doing so, you can curate the data to pull out inferences before sending them to the model, which will result in more accurate results based on replicating human behavior. It’s important to remember that the inferences are often more critical than the raw data, and by understanding the user’s behavior and needs, you can design a better AI product.

Another key factor in deploying AI is the human factor. It’s about establishing the right context when implementing automation to avoid the fear of job loss. How to deal with potential AI/Human problems, such as stakeholders circumventing the AI put in place for them. One solution is to design the interface to make it clear when a user has reviewed a specific data point and provide overrides where needed. Additionally, asking stakeholders about their reasons for bypassing the system can help improve the model and prevent gaming of the API. Ultimately, AI and humans can work together to achieve better results.

AI can take care of easy tasks if it’s well-trained, and humans can learn to leverage AI better over time. Building collaborative interfaces that make AI a team member rather than a cold algorithm, allowing for more natural interactions that can help it learn better. AI will become essential in any job dealing with stakeholders who process human data due to the variability involved.

Podcast Transcript

Hello, this is Darren

Pulsipher, chief solution,architect of public sector at Intel.

And welcome to Embracing

Digital Transformation,where we investigate effective change,leveraging people processand technology.

On today's episode Productizing Decisional AIwith special guest Matthew Pulsipher.

Matthew, welcome to the show.

Hey, Darren

It's weird You can't call me Darren.

I'm your dad.

Fair enough. Yeah.

No, no.

This the first time I've interviewedone of my kids on the show.

And it's apropos right nowbecause of what's going on inartificial intelligence.

So, Matthew,give us a little bit of your background,where you're coming fromand why we have you on the show today.

So I'm a product managerand I've worked withseveral companiesjust modernizing their product line.

And in my most recent role,one of the things I've worked onvery particularlywas integrating Decisional A.I.into the product.

And I found that was a veryintriguing, interesting experience,and it showed where a lot of the promisesand a lot of the exact tacticsthat you have to have when doing so.

You know, it's really interestingbecause not a lot of companieshave actually productizedany kind of eithertheir internal stuffor a chat bot here or something like this.

So this is a new field, right?

Oh, absolutely.

And the thing is that it's very commonthat everyone just kind of assumes A.I.is chat CBTor just other generative AI systems.

Right?

And in business, there's definitely a fitfor those systems and they become moreand more sophisticated every day.

But what matters the most in

AI is context.

And the issue with generative

AI is, is that you have limitedcontext of requestbecause you were limited to the prompt.

Everything beyond that contentor that promptis coming from its general dataset.

Okay.

But before we get into that, whatlet's let's simplify this a little bitbecause there's lots of different

AI techniques out thereand you kind of talk to me a little bitabout three different types of A.I.,

So let's let's kind of let's set thatground first.

What are the the main three types of of

AI that we see out there today?

Sure.

Well, you've got generative, which iswhere most of the hype is right now,and that's what Chat GPT is and othertexture writers image generators.

These generate text or imagesor contentand they're built on general datasets.

So it's just kind of takinga slice of whateverit was trainedon, which is just about everything.

For each query that you're sending to it,you need to give it the contextper query,and that can be kind of limited,which makes it good for general knowledgequestionsand difficult, complex individual tasks.

But it doesn't make it very goodfor making focuseddecisions based on your company.

Okay, so generative

AI generalized right IT general knowledgetrained on lots and lots of datafrom lots of different spaceswhen you ask it questions it.

It's like talking to someonethat reads encyclopedias.

Right.

And it can be a little bitunpredictable, too, right?

Because you don't alwaysget a standard response back.

Oh, well, how so?

Well, there's never a 100% chanceyou can ask it very specific questionsand offer a format.

And there's a very good chanceit will come back in that format.

But you never have 100% guaranteebecause there's alwaysa little bit of randomness to it.

Otherwise it would always produce the sameanswers for every single question.

So that's why if I ask it the same thingtwice, I don't get the same response.

I get a little bit different responseeach time.

Right. Okay.

All right.

So there's no absolute nesswith generative AI.

Very good AI, very good tool.

I play around with chat GPT all the time.

I use it for my podcastto find out information.

It's good at summarizing informationfor me and pulling in different formatsthat I need, but that lack of predictable,deterministicresponse makes it very hardto automate processes using it.

Okay, is this where the next step is?

Which is decision way AI or. Yeah.

Okay, so explain decisional AI then.

So the difference between generativeand decisional, a generative

AI is based on general knowledgeand it generates text or contentbased on that general knowledge.

With Decisional A.I., it is a lot simplerin its scope, but it's a lot morefocused in its context.

So Decisional, the AI chooses the bestoption given a set of data based on modelsgenerated from your previous dataand trained using your own company's data.

So this is this helps me focusand target decisionsbased off of previous decisions

I have made in my in my companyor in my industry.

Right?

If I'm sharing models in industry models,then that would be the case.

And you can take those modelsand train them according to your employeesor departments behavior.

Okay.

And Decisional, I used primarilyfor making decisions.

Decision only makes decisions. Right?

And those decisions could be usedfor balletic,validationand verification of uploads from a user.

It could be used to presentthe best product to a customerbased on their activities, on your siteand other information you know about them.

And it's based on a lot of contextfrom your company's data.

So it's the opposite of Jeremy Vinethat Jeremy knows a lotabout everythingand it's never really deterministic,but Decisional

AI knows a lot about your specific caseand it will return deterministic results.

Okay.

And I want that determinismwhen I'm dealing with business processand things like that.

So I would use decision when I do decidewhether a product is good,good enough to ship,or whether a customer that I have is worthextending a line of credit to or a vendoris giving me the best price in timeso I can use decision making for businessprocess and business process automation.

That's what I'm hearing is that right?

Correct.

It's very good for automation becauseit picks between options A, B, C, or D,

Okay.

All right.

So the last one hereand where we want to get to, right?

Eventually predictive.

I want it to tell mewhere the stock market's going,why SBB Bank actually failed.

And I would have gotten my money outbefore it failed.

This is predictive AI Right.

This is where I really wantto get to, right?

So predictableis almost a cross point between the two,but it's a lot more similar to Decisional.

So it's based on the custommodels and datasets that you have,but it generates a predicted valuebased on those other numbersand data points that you have.

And that predictive valueis based on the previous performanceof thatyou've worked in decision.

We are not predictive by decision AI.

So I don't, I just go grab this modeland it solves all my problemsfor decision making.

I mean, you can do it, all right?

So I just grab it, right?

I just

I just go to Walmart, buy my decision.

Okay?

I for industrial for my manufacturingby my decision,

I plug it in and I'm ready to go.

Is that how it works?

I mean, if you're dealing with a modelthat is working with cost softwareand it's using establishedpre-trained things, it can be that simple.

But in practice, yeah, I mean,actually there are marketplaces of modelsthat you can importthat take standardized datathat are pre-trainedto like a certain standard, right?

So you can start using those immediatelyand then train it further using your usersdata. Okay.

So I don't have to start from scratch.

Not all.

And, and it's not like I'm going and justbuying something that plugs in directly.

There's a little bit of work involved.

There's some integration work.

There are some SAS vendors out therethat offer Decisional is kind of a serviceand you can set up your containerand make those API callsand you're just feeding your data into itand getting those decisions backthen, sending corrections if needed.

And what kind of get intoif you were to start from scratch,what that would look like.

All right, so let's start.

How do I get started?

How do I decide whether I can use decision

AI or not?

Walk me down the process.

Sure.

Well, the first thing that you need to dois identify the problem that youwant to solveand once you've done that,you can determine whether it's a good fitfor A.I., it's a good fit for A.I.if it's not too large.

Otherwise, you're going to spend monthsand months trying to accommodatea very complex set of datawith not necessarily accurate model.

You want to pare it down to the pointwhere it is something that's achievablewithin about six months.

If you want to be successful right now.

And part of that, too, is pickingsomething that's operationally relevant.

If you're deciding to solve a problemthat doesn't really matterto your operations or your business users,you're doing it to say that you're using

AI and that's all you're going to achieve.

You need to solve a real problemthat helps the business.

So you need to scope your probleminstead of saying,

I want it to do a whole bunch of things,scope it to I want it to helpmake decisions on this specific thing.

Is that what I'm hearing?

Right?

You want to find somethingthat is has reasonable boundsso that you can limit your optionsbecause you're trying to have a decisional

I for a limited set of options, Right.

Let's say five options at mostit could be more, could be less.

But you don't want to make it toodifficult on yourself because that needsto be an answer to the query,whatever that decisional option is.

And we'll get into that in a minute.

We talk about influencersand decision options.

Another thing that makes problemsa good fit for A isif it's a repeated process,is this something that your business usersare doing every day?

Is it busywork?

Well, I mean, so. Whoa.

But decisions have to be made here.

So this is not just like an RPA,a robotic process automation, where

I'm just, oh, they, they go and do this,then they do this, then they do this.

There'sdecisions involved in this automationas well, right?

It's notit's well, there are a lotthere are a lot of thingsthat happen in businessesthat are repeated processesbut still require subjective human review.

Gotcha.

And the example I want to usethe other thing is that's data driven,but we can kind of talk about this exampleand how it applies all these things.

Let's talk about a financial institutionthat needs to validate driver'slicenses using extracted data.

So I have a driver's licenseand I can run it through OCRand I can take those data points from it.

Right?

I need to determinewhether these data pointsare logically consistent and each statehas slightly different rules.

And what I want is, is this good?

Is this definitely bad,or does it probably need human review?

This way I can speed up and clearout the human queue of workand still have aamount of risk management involved.

All right.

So this this helps with making decisions,which means I can take the humanout of out of the loop where humanslooking at a driver's license saying, yes,that looks good against this pictureof a California driver's licensecompared to a Georgia driver's license.

I know what they're supposed to look like.

I can check it.

That can happen automatically.

But then you can have onethat says it needs to be thrown outand then another onethat says it needs human review.

Do you eventually see canthe decisional models get betterso I can throw out the human review?

Absolutely.

And that's the intent ofif you've built the product correctly,what you've built is an interfacethat streamlines the manual processthat they're already doing,because a lot of timespeople will establishmanual processes for these thingsthat there's no system to do it for them.

They could be doing it in Excel sheet,they could befollowing their own rules on it.

So what you want to dois figure out what those rules are,and then you want to come upwith data points around those rulesand then use those data pointsto train the A.I..

Then you feed it back into an interfacethat lets them do the work.

And as they're doing the work,you can do a couple of things.

You can give them a predeterminationthat upon human review you can validateyes or no on.

So you're kind of pre filling stufffor them, which helps productivity.

Or you can skip the queue entirelybecause you're confident that it's good,which speeds up the user experienceand increases productivityor you canjust say needs review.

The A.I.doesn't knowthere is enough confidence scoreand then when the human reviews it,they will submit their response.

And that response goes back to the modeland trains the model continuously.

So you know what this reminds me of?

It reminds me of workthat ancestry.com com was doingwith indexing handwrittendocuments from the 16 1700scensus documents and birthcertificateand birth records and counties.

They had legions of people that wouldgo on and say, I think this is what it is.

And I noticed over the yearsbecause we did indexing, it wasit was kind of fun, right,to look at these old documents.

And I noticed something really coolas the years went on and doing it,the decision, it would say,we think it's this is that yes or no?

And I go, Yes, guess what? Indexing.

You don't need asmany people anymore because the

AI has gotten really smart, right?

So this reinforced learning is part ofit needs to be part of your productoffering that you're working inin your offering, it sounds like.

Yeah.

And to make the most effective productthat uses this kind of machinelearning back end,you're going to build something thateven if you didn't have the machinelearning back end builtin, would still speed up the processthat your users are currentlygoing through by streamlining itand keeping them focused on what matters.

Oh, I like what you said thereabout keeping them focused on what mattersbecauseyou don't want the

AI to get in the way right?

Right.

So it's supposed to just be like an aidto help them do their job more effectivelyand in better, you'd want to interviewand get a betterunderstanding of what the users are doingnow in their current process.

Okay. Andthe best thing to do is just sitand watch them after you've readthe documentation that drives the process.

Sometimes they've documentation,sometimes they don't,but sit and watch themand take notes of what they're doing.

Are there things they're looking at?

So going back to the driver's license,are they looking at the issue datesin relation to the expiration date?

Are they looking at certain otherindicators of the driver's license?

Maybe if it's Georgiaand it has a gold border, it's invalidor something along those lines.

You just need to figure outwhich inferencesthey're getting from the dataon the driver's license.

And this is very important.

And we'll talk a little bit about howthe inferencesare more important than the raw data.

And if you can write codeto pull those out beforesending them to the model,you're going to get much more accurateresults based on tryingto replicate the human behavior.

So if you can identify areaswhere you want to do the inference.

So it's that interviewwhen when you talkedabout the interview, it reminded meof my first job out of college.

I don't know if I've ever told youthe story.

Oh, no.

So I my first company I worked withwas ASG Technologies Medical Imaging.

My job was to help come upwith the user interface for a radio,a digital radiography station.

Now, this wasvery, very cool stuff.

Sorry, this was very, very cool stuffbecause backthen people put films on light boards,so the first thing they hadme do was go and watchradiologists in these light boardreading rooms for three days.

And I just took notes on what they did,how they workedand it was fascinating because I learnedfrom the waythat they interacted with the film,what we could do inin our product to make itso it was used to how they workedand and I saw the inefficienciesin what they did and the tricksthey used to get around things.

Sounds like we want to use the sametechnique when we're developing A.I.internal products and also external,it sounds like, right?

Because usually I assumethat when you did this, you had some kindof a guide that they were trained on thatyou also read before? No.

Okay.

No, I sat there with seasoned whiteguys that were reading film for 20years, and I sat there withnew doctors.

Right. That have beenthat were still residents.

They were six months into this.

So it was fascinatingwatching what they did and how differentbetween the two, right?

Yeah, absolutely. Absolutely.

Because that's one of the things you'llfind is seasonedpros will come up with shortcuts.

And the shortcuts are oftentimesinferencesthat may not be documented orthey may not have been trained in school,but these guys have seen enough recordsthat they know thatif this is the case on a record,then we can skip this other stuffbecause we know that this is a problem.

As an example. So how do youhow do you do?

I mean, you literally just sit thereand watch people take a recorded session,have them talk through whatthey're doing and why,or you could do a silentinterview.

It really just dependson which stakeholder you're working with.

And if they're nervous,have them talk through it.

Otherwise they won't feel like you'rejust watching themand judging their performance.

Yeah, yeah, that's just me.

I think that's if you're just tryingto understand what they do and why.

I think that's why they put someoneso junior.

I mean, fresh out of college,sitting in a dark room, they were like,

Oh, he's just a piano over there.

He's not.

You're not about it'sthis is a performance review.

It's not understand what you're doing.

Yeah.

Because that's going to bea fear of a lot of people.

Have you had to dealwith any fear of people saying, well, if,if they're coming in here to helpautomate the process with an ally,is that replacing my job?

I mean, do people feel that way?

I think to a degree they do, but it's allabout establishing the right contextbecause they're really trying to dois help themwork on their workmore effectively and faster.

Right.

So another thing I've seen isif you aren't the kind of personthat interacts with the stakeholders thebest, have the UI designer do the studythinkthey have a lot of experience in the area?

So getting a good UI designer.

So this is what I'm hearing,a UI designer or productmanager, that'sthat's good at it, taking copious notes.

And so you have these stakeholdersthat you're working with.

Have you run into any problems where,where people are circumventing the APIthat you put into place?

Oh, absolutely.

Yeah. So how do you deal with that?

Well, to a degree, you do have to be ableto trust the stakeholderand you need to have overrideswhere neededbecause ultimately that departmentcan make their own decisions and the airis meant to help them,not interfere with them.

But the other thing that I've found that'shelpful isif they're trying to, let's saythat they want to mark something.

No, but we don't want to train the

AI to reject those in the futurebecause it was a weird exception.

It doesn't really fitin the standard rules.

I asked themthe reason they're marking it.

No, in your interfaceand then under behind the scenes,choose not to train the AI with it.

Oh, interestinginterest. So.

So when they mark something.

No. I ask them why?

Because it could bethat the rules are wrong, right?

Or it could be.

This is an exceptionthat only happens once every million.

Or there's some circumstancewhere it's not.

They need to, let's say it's driver'slicensevalidation and independent of the imagethat was sent in,we found out this driver'slicense is bogus.

They still need a way to do their work.

But if you don't give them a wayto do their workthat fits in the confines of the system,they will find ways to justmake it happen.

And you don't want thatto harm the models.

So if you were to ask them, well,why are you doing this?

You don't even need to factorthat into the model.

And you could include thatas a data point,but you could also just use as a datapoint as to whether you send it or not.

Interesting.

And I actually I actually like thatsometimes we don't we don't think aboutthose sorts of things.

What about stakeholdersgaming the, the API?

Um, unfortunately I do think that there isa little bit of that going onor it can happen.

Definitely.

What you can do to prevent that isyou want to design the interfaceto make it very clearwhen it's a user that revieweda specific data point orand either review the specific data pointand make it clear to themit's just it's there to help you.

It's there to protect it.

And if it's not deadon, you can correct it.

Now, gaming it.

You just have to make surethat your model is aligned with whatthe user is doingout of it.

No. No. Okay.

No, that makes sense.

I, I just thought because APIsare susceptible to to human interaction.

Right.

You could get a group of peopleto help deaths right.

That are or piracyflagging stuff so they say no.

So they say noall the time which they can skip over.

So somehow you've got to be ableto capture those sorts of thingsin the back end.

It sounds like.

And allies just look at the reasonsand try and think of them ahead of timeand make sure your interface accommodatesfor that and also allows them to feelheard if they need to bypassthat.

Again, go back to the reason dropdown.

If there's a reason dropdown there,we are accommodatingthe fact that we know that they're goingto need to bypass this sometimes,especially if I wasn't confident.

Right, Because it

Yeah, I wasn't confident.

It's an edge caseand eventually what ends up happeningis, is the model gets better.

Your team is really only going to workon edge cases and stuff.

That is not as clear.

So their work will get harder,but they'll be doing less of it,which is exactly where humans are useful.

Well, yeah.

You sound like my interview with chat

CBT humans will be useful for.

I'm just giving you a hard timethat. Yeah.

So it's it's interestingbecause the way I kind of see this goingthen is the AI'swill take care of all the easy stuffultimately if you're if you've got a welltrained model.

Yes. Gotcha.

And I also thinkthat humans are trained as wellhow to leverage the AI betteras as they continually go through it.

They're going to learnhow the AI is going to be predictiveand they're going to know the areasto only focus on those areasmoving forward.

It's in our it's in our nature toto learn.

So it's an interesting combination wherewe're learning together, the AI and thestakeholderare learning how to do this together.

That's that's actually fascinating.

And you want to build an interfacethat, you know, the AI is a team mate.

It's helping youand it needs to be collaborative.

The kinds of thingsthat you would put intoa collaborative interface where multiplepeople need to work on a single fileare the things that you would wantto integrate into an AI interface as well,because it appears another teammember to the user.

Oh, that's interesting.

I never thought of it that way.

I always thought of it as this coldheartedcomputer algorithm,but I like what you're saying there.

If I include it into the team, it'spart of my team,then I can interact with itmore naturallyand it can learn better from thatnatural interaction just in termsof how it appears in the tool.

Of courseyou can't have chat.

GPT participate in Slack channels oranything along those lines, but not yet.

Just give it some time.

Oh, so if you could stackdecisional generative,

I mean you could just tolerate APIs. So.

So where do you see this movingforward, Matthew?

I mean,do you see this as a blossoming industryfor product managers that got to learnhow to add a leverage into their products?

Or is this a special niche?

AI What what do you see?

I think it's going to become essentialfor any kind of jobwhere you're dealing with stakeholdersthat do human dataprocessing of any kind.

And why just human data processingjust because it's so much variability?

Well, if right now we've already kind ofhad the digital revolutionto a degree, right?

Yeah, Yeah.

If you already have it in dataand you can do hard line logicto solve the problem,you've probably alreadysolved it or should solve itusing just hard line logic.

But where decisional is arethe most usefulare cases where you need to have it.

Normally right nowa human review something manuallybecause it doesn't quite add up.

Gotcha.

Okay.

So that's that'sthat's the the key point there.

All right, great.

Matthew,any any last words for my audience today?

Well, I reallyappreciate you having me on the show.

And I am looking for my next opportunityto transform a business using A.I.. So.

All right. So there you go. There's this.

There's his pitch.

Go hire this kid, please.

I don't want him moving backin with me, butthank you for listeningto Embracing Digital Transformation today.

If you enjoyed our podcast,give it five stars on your favoritepodcasting site or YouTube channel,you can find out more informationabout embracing digital transformationand embracingdigital.org

Until nexttime, go out and do something wonderful.