AI agents demystified

Garima Dosi (Moderator):

Good morning, good afternoon, one and all. Welcome back again after a long break of about two months. So, it was, I think two months back in October 2024 when we had discussed, RAG with LLMs under our learning series, Roots to Fruits Decoding AI.

Just for a quick recap, as you can also see on screen, the concept of fruits to Roots to Fruits is where we learn a topic similar to how a tree grows, establishing its roots, and then growing into the trunk, interesting in variety of branches, leaves, and finally bearing fruits. So, for Decoding AI, also, we had started with the roots of LLMs, which was LLM itself. We went into the trunk of how to directly interact with LLMs through prompt engineering of prompts, and then interview branches, of workflow interactions, and also RAG, which was our last topic. And in the midst of all of this, we had also discussed an early fruit of LLMs, right at the top there, which was how to get to value from Dark Data.

And so, this New Year, we will start with the leaves of the tree of learning, which are AI agents. So, agents, agentic workflows, agentic this, agentic that. So, all of these are buzzword these days. And so, if I had to simply define agents, I would say they are human-like automations or intelligent automations of processes or human activities done using LLM or Gen AI, simply just to put it that way. And in the webinar today, we will see how are such automations technically possible, and more importantly, how do these agents empower businesses.

So, to take us all into the complex world of AI agents, we have our AI Solutions Engineer, Gil Fernandez with us, and he will help demystify AI agents with this knowledge and experience about this interesting topic. But, before we actually begin some routine announcements, that the resources of this webinar would be made available to you after the session, so please feel free to immerse your brains totally into demystifying AI agents with Gil. Thank you.

Gil Fernandes (Speaker):

Thank you very much, Garima. I hope I’m audible. So, this is today’s agenda. We are going to recap, the previous webinars very briefly. Then we are going to really get a solid understanding about agents. After that, we’ll talk a little bit about the aspect of agents and self-improvement. And finally, we’ll talk about the upcoming webinars and, ideas that are going to be discussed in the next webinars related to agents. After that, if time permits, we’ll have a Q&A session. Thank you.

Garima Dosi (Moderator):

Gil, just sorry to interrupt. Are you sharing something on the screen?

Gil Fernandes (Speaker):

Oh, yes, sorry. Yeah. Thank you very much for reminding me.

So, first we’ll start with insights, from the previous webinar, but before we go there, we’ll have an audience poll. Can we have the poll? Thank you very much. So, this poll is about which AI agents you are using. So, are you using search agents, agents, which I use to search knowledge or related to search engines, are you using database agents, for example, or personal email agents that actively go through your emails and perform actions for you. So, which type of agents are you already using? Please let us know. And I’ll continue here.

This is just a recap from the previous webinars. We talked a lot about Large Language Models, LLMs, and we mentioned that they are really great at some tasks, and they are not so great for in, in other contexts, let’s say. So, they are great for Natural Language Processing tasks. This means summarisation, for example, or keyword extraction. These are just two examples, or in context learning, that’s another example. They are also great to convert unstructured textual data into structured data, which you can store in a relational database, for example. And because they are so good at structuring data, they can be used in the context of Dark Data. Like Garima already mentioned. We have a webinar on Dark Data, and I really suggest that you actually have a look at it. And, like I said, they’re also very good at learning from context. And this allows us to create products like Retrieval Augmented Generation based products. And, and this was also the topic from our previous webinar.

Now, Large Language Models have been in the past limited, in reasoning tasks, even though they improved quite a lot. Now, example of such models that have improved immensely in this area are, for example, ChatGPT o1 and, I think hopefully o3. They have typically also knowledge cutoffs. So, they know about the world until a certain date. And after that, their knowledge is not there because this knowledge was actually in their training data. So, they know until a specific date. Then, also they are, were typically in the past not very good at long-term planning, even though this has now improved a lot with some recent developments, like the ones they already mentioned before from ChatGPT. And they also has a problem that actually they don’t mention sources very often. It’s not known very often from where the knowledge is actually coming. And this is a big problem, like, for example, Sora from such models. Then, they tend to have hallucinations to make up, you know, text or to make up, facts, that actually are not correct. And they also have the problem of refusals very often. So, these models are trained within ethical boundaries, and if they falsely detect that you are, going outside of these boundaries, they might refuse to give you some information that should be actually okay to give to you. And especially, if you look at Large Language Models, and if you, for example, just download the simple, Large Language Models using tools like Ollama or something like that, you will notice that they actually cannot execute tasks on their own. They cannot, for example, open your calendar, Google calendar and look into it. They can’t do these kind of things. And so this is where agents, I mean, so agents extend the functionality of Large Language Models. And so that’s why we want today to look at agents.

But, before we actually go to the main topic here, I would like to see the results of the poll, so to see which types of agents, are already in use amongst our audience. And it seems that search agents are relatively popular and that most people actually don’t have experience with agents yet. And that is actually kind of expected. So, you might actually be using agents without noticing it, so that sometimes there is software that silently embed some functionality, which actually is agent driven. So, you might actually be using agents without noticing it sometimes, database agents, so someone is using it. And also personal email agents are very surprised that actually four people are actually using it. But it seems that the most common type of agent that is actually being used are search based agents. So, perhaps someone is here using Perplexity AI, which actually acts a little bit like a search agent. Yeah. But anyway, thanks a lot. So, I’ll just close the poll for now and I’ll continue.

This is now the main question. What are agents? Let’s try to understand. Actually Garima gave a great explanation, about agents and, so, but, I’ll start again and try to explain them again in very, very simplistic, easy to understand terms. The agent, the AI agent acts as an intermediary. So, like, an expert who is, between a user and an expert system, an intermediary, for example, in the world outside would be a real estate agent, for example, is, an example of an intermediary, but also, another example would be the insurance broker. Basically an expert in terms of claiming insurances in terms of, finding the best offerings in the market for you. For example, this is what insurance brokers typically do, they’re also agents. Another example in the world outside for an agent would be financial agents, that actually invest your money, and then they, will obviously receive some commission from you. But these are real world intermediaries. But now the AI agent is also an intermediary, like these examples. The only thing is that this agent is AI driven. So, there is a user, then there is the agent intermediary. In this case, it’s an AI agent, basically, its engine is actually not a human engine, but it’s an AI engine. And then there is an expert system with which, the agent intermediary is actually acting. This is perhaps the most, basic explanation of what an agent is. It is an intermediary, an AI driven intermediary.

Now let’s try to understand, the components, the central components, the basic components of an agent. An agent has a brain and it has tools like the normal intermediary out there, like the real estate agent. It has some kind of reasoning capabilities, and it also has some tools. It has some tools, like it has some knowledge about, its, problem domain and, can use these tools to navigate through this, knowledge domain. So, the AI agent is also like that. It has a brain, and this brain typically is the Large Language Model, or, it could also be, MLLM. So, a Multimodal Large Language Model. So that access the brain as the part of the system, that reasons. And then it has tools. And these tools are specific to the problem domain. For example, our insurance broker will have access to all insurance companies to their database and product offerings, right? And it’ll also have a tool that allows to compare multiple prices, you know, of, insurance, offerings. So the main components are the brain, and the tools.

Now we are going to dive into the brain. So, what is actually the brain of an agent? Like I said, the brain has two parts. It has a reasoning part that does this reasoning. So, that is the intelligence, and it also has a memory. So, it has reasoning abilities. So, that means, it is able to discern, decide, and then also act on these decisions. And it also has a memory. So, it has like some imprints that help it to reason better. Now I’m diving from the brain, I’m going to dive now specifically into the memory to see what is in this memory.

So, there are, in AI agents, there are four types of memory. There is the interaction history, which we call chat history. It might not be actually chat history. I would prefer interaction history in this case because, if you had a verbal, like audio-based interaction with the agent, which is possible these days also, then you have to have this kind of, interaction history in audio format. So, that’s one type of memory. And this is extremely useful to know what was the previous context, what is the, the context of the previous conversations, because they have a big influence on the outcomes of an interaction. Then, normally an agent has to have, its kind of the domain knowledge. And this, the main knowledge is the external knowledge base. So, this is like the, the knowledge really that defines the specialty of the agent. So that’s the knowledge about the specific problem domain. So, if we go back to the insurance broker, the insurance broker has a lot of knowledge about insurance policies, how to make claims. It knows all the processes, it knows all the workflows, for example, and this is exactly this, external knowledge base, which is the domain knowledge base. Basically. Then, agents might have optionally, knowledge about the user. So, they have knowledge about the user, and, they know the profile of the user. They know, the background, the preferences of the user. They know about, his job, perhaps they know about his LinkedIn profile. And this is also extremely important because agents should be able to adapt to the conversation partner or the interaction partner in this case, and they should be able to adapt their behaviour to them. And then finally, the agents should have some mechanism for remembering the, what happened with this specific, users. And they should be also able, and not only with this specific user, with all users, so they have, they should have some kind of internal caching of previous interactions so that they can, react quicker on, repeated interactions. So, this I call internal caching. So, for me, these are all the types of memory that an agent can actually have, so that the agent is really powerful. So, chat history, the domain knowledge, the user profile, knowing about the user and internal caching knowledge about previous interactions that can be used to make future interactions actually faster.

Now, very often what I have seen is people selling, AI based workflows as agents. And, sometimes I think, oh, that’s not quite correct, that is actually a misuse of the term agent, because the term agent sounds actually cooler than actually I have here an AI workflow, and I’ll try to explain the difference between AI driven workflow and the agent. And so, the main difference is that the agent is, capable of dynamically establishing a path in the workflow, whereas the workflow path is actually predefined and it’s static. This is quite theoretical, but, let me give you an example. For example, if you have a child and you want the child to go to the supermarket to buy some, vegetables, to buy some fruits, yeah, let’s say, then you tell the child, you give the child actual a workflow instruction. And this means, you go out of the door, this, type of workflow that you would give to the child would look like this. You tell the child, you go out of the door, you turn right, then you turn left, then you walk, 200 meters, and then you find the supermarket on your left, you go into the supermarket, and then you go to the groceries, and then you buy some groceries there. And then you come back home using the same path. So, basically what you’re actually giving to the child is actually a workflow. And you could do the same thing with an AI, actually tell the AI exactly which steps to do and, and let the AI actually execute it. Now, the agents are non-deterministic, and the agents, will act in a different way. Now, in this case, imagine that you are interacting with an adult and you want the adult to go out and actually buy groceries at the best possible price. Then you actually, you, you’ll have to interact with this adult in a slightly different way. You’ll have to tell the adult, buy me groceries at the best possible price. So, now the, the adult will go out and will try to go to multiple supermarkets, and it has the knowledge on where to find the supermarkets, and then we’ll go to multiple supermarkets and analyse them, compare all the prices, and then finally go to the best offering and bring it back to you. So, this is more like, this is a purchasing agent or like, so a supermarket agent or something like that. So, in this case, you actually don’t know what the, the person is going to do because the person will, will perhaps go to this or that supermarket, and you, you don’t even know the order in which the person is going, to go to these supermarkets and what kind of calculations the a this person is going to do. This is a proper agent. This is really the difference. So, in the first case that I explained, it’s deterministic. In the second case, you don’t know the execution path.

So, these are some examples of, this is not an agent here, this is a workflow. And this is just a workflow that builds a knowledge graph, and it goes step by step. In the first step, it reads files, then it chunks the files. So, means it cuts the, the files in specific parts, then it extracts the graph, then it does some augmentation of the graph. So, basically it finds communities there, then it summarises the knowledge of these communities in some text, and then it stores it somewhere. So, like you see, it’s just a recipe. You follow one step, another step, another step. You know exactly what this workflow will do. This is not an agent.

Now this, what you’ll see here is actually an example of an agent. So, the first step is always there is some user interaction. A query is, requested. And then, the first step is always the Large Language Model, receives this query. Then it decides, which is the next step. And this might be, a multitude of steps. In this case, this is a SQL agent, a SQL or a database agent. And, in the first step, it’ll probably, get to the table information about the database. It’ll try to learn some knowledge about the database, if the query is related to the database. If the query is not related to the database, it probably will give you an error probably. But anyway, the first step is probably get the table information. Then it replies, actually this arrow is not correct. It should be tool replies goes to ask LLM what to do. This should go the other way around. So, this is not quite correct. And then, or it gives an error when it executes the tool, but it always goes back to the Large Language Model. So, and then the Large Language Model in the second step will probably, generate some SQL and, check if the, the query is okay. So, it use the second tool that you see here. Then again, the tool would reply, go to the Large Language Model then, or it would give an error. It’ll say, no, no, no, your SQL query is actually not good. Generate it again, and it would generate it again. Check it again. So, what happens here is, to be, succinct is just, that you actually don’t know, which will be the execution path. So, the big difference between a workflow and, proper AI agent is that the agent will take its own decisions. It’ll determine by itself the execution path inside of, a workflow. And they will also learn from errors and, they can adjust in the middle of the execution. They can adjust the execution path also.

I wanted to talk a little bit about the relationship between Retrieval Augmented Generation systems and agents, because RAG systems are typically workflows. Now, I’ve seen in the past some hybrids between agents and the Retrieval Augmented Generation systems, and I’m just going to mention them briefly.

One is this Multi-Hop RAG, which is a system that, you know, like all RAG systems, they first query a database, extract some context, and then send this context typically, or merge this content together with the initial question. And then, put it back to the Large Language Model. Then the Large Language Model looks at it, and based on the context and the question, it then, produces synthesises a reply. Now, this is the simple RAG, but, Multi-Hop RAG, actually interacts with the knowledge base multiple times and accumulates knowledge multiple times and does it an unspecified number of times until it has accumulated enough knowledge, it reaches a certain threshold and then can give the response. So, as you can see, you have already this kind of looping and this kind of unpredictability here. So, you don’t know how many times it’ll actually, go to the knowledge base, how many queries it actually does in Multi-Hop RAG. and so that’s the beauty of it. So, in a certain sense, it is, a hybrid between our workflow because you know what it is going to do, but you don’t know how many times it is going to do that. So, Multi-Hop RAG is like a bit of a hybrid between an agent and a workflow. It’s not quite there to be an agent.

And then there are products like STORM that is, these are systems that are used to create essays based on specific topics they have inside of them already. Two little entities like, little type of, little agents. One is the Wikipedia writer, another one is the expert, and they interact with each other a number of times. And so, this means in the case of, STORM, it’s a Retrieval Augmented Generation system with some agentic elements inside of it. So, yes, sometimes you might find out there some, hybrids between workflows and agents, but I just wanted to point out that very often many, AI products out there, they’re sold as AI agent, but they’re only AI workflows because they have a fully deterministic path, even though they can handle errors. But handling errors is just, not really, this quality of really being non-deterministic in the path execution.

Anyway, here are some agent examples. This is our next topic. The first example is the SQL agent. It’s just an intermediary between a user who speaks in natural language and a database, a normal database, enterprise grade databases. It can be anything, can be Oracle, can be Postgres, can be whatever, but it could also be something like, also a NoSQL database would also apply like MongoDB or something like that. But anyway, it’s a very simple principle. Again, that’s the idea of the intermediary. Knows about databases, understands natural language, can convert natural language into the language of the database.

Now, how would you build such an agent? we have built such an agent and we’re going to demo such an agent. So, normally the ingredients that you actually need is the Large Language Model. I use ChatGPT in this case. And then, I created a couple of tools, one for table listing, another for, retrieving the table information. So, details about the specific, table. Then there is a query verifier and a query executor. So it has some tools and it has, a brain, which is the Large Language Model. Now, another example would be a SQL agent, but instead of just communicating with one database, it would communicate with multiple databases. Now, this can be very, very useful for enterprises that you have some kind of chat bot or agent, that you can ask about, questions about, the enterprise data. Now this is something very useful, to have because, the entry barrier to accessing data that in a direct and, chat-based kind of manner, the barrier is very low. You just ask questions. For example, you could ask questions, which item has had the most sales in the last week? And the agent would then generate all the queries and give you a properly formatted answer to this question. Now, this is a very powerful system, and that’s, system that might be useful. Actually, I think this might be useful for most companies out there.

Another example, a real estate agent, for example, this is just an agent that’s like this intermediary instead of being a human intermediary, is just an agent that can interact with, in this case, I call it trust interface with the technical interface that gives you property information.

Another example, this is not another example. This is the same example in how you would build it because I tried to build this type of agent. This is how you actually build it. You need to build three tools and you need to connect, to the Large Language Model. Obviously you need a prompt that, is suitable for this type of agent. And, it has three tools. One is a search tool, which is perhaps the most important one, but this search tool doesn’t understand, the location that is spoken. So, you need to have a translator between the spoken translate, spoken location and the spoken property type, and the codes that are used in the search tool. So, that’s why we have two other tools is location search and property type mapper, because these are just translators between the spoken language and the technical language. So these are the basically ingredients of a real estate AI agent.

And finally, we have also another example, which is the program of thought, which is like a code monkey, AI type of code monkey where you actually interact with it. It generates some code that executes the code and then sends you the results in nicely formatted way, in non-technical way. So, basically, this is like, instead of having a, a software developer or a code monkey here that receives some questions and generate some codes, executes the code and so on, you have actually the Large Language Models actually doing this.

This is just, a detailed example of how you actually, create the program of thought agent. So, typically, in essence you just ask the Large Language Model a question. Then if it, the question is suitable for code generation, it generates code, executes the code, collects the output, then the Large Language Models again checks the, if the output is valid. If there was not an execution error, there was an execution error, it tries again to generate the code a couple of times. and if it keeps on going like this for too long, actually it generates an error. So, all the agents also don’t have unlimited patience, let’s say. So, if all interactions are, producing errors and if party says stop, that’s enough for me. I’ll give you an error and goodbye. And then so it can also deal with errors. Obviously all error, all agents should be able to deal with errors. I’m just going to demo this kind of little agents here.

Let me stop sharing and let me share a little demo where you can see three of these agents, which we created. Now I just need to find where it is. I hope you can see.

Let me go to the homepage. So, I have here three agents. So, one is the property agent, real estate agent, basically, and the other one is the coding monkey coder agent I call, and then the SQL Query agent. Let me give you just some, examples of the property agent. The property agent, is able to answer questions like this one, can you find properties in Aberdeen under £ 2 million? For example. And, what the agent actually does internally, it actually, like I said, it produces thoughts and you can see the thoughts on the right side, where it actually is thinking himself. We can see the thoughts there. To find properties in Aberdeen under £ 2 million, I first need to identify the location in Aberdeen. I will perform. So it actually is saying I’m going to perform a location search. And then, it actually found the location for the location code for Aberdeen, and now it, tries to find the property identifier for houses, and it found it, and finally it found, some properties successfully. Very good.

This is a live demo, by the way. I’m not just. Sorry. I lost something here. Let back to here. And as you can see, you should be able, hopefully it’s, showing it properly. It shows some properties, and then you can look at the properties in detail. And there, there they are the properties. So, you can see some property. I’m just not, I’m not trying to advertise properties for anyone here. So, then there is also the coding agent. And the coding agent can answer questions, like, can you calculate some matrix operations here. Like for example, the inverse of some matrix here, actually, it’s very fast because I think it has some caching, this agent, so it already knew the result for this question and just trying to, yeah, so sometimes, because the agent that I created, they have internal caching, so if you ask twice the same question, they think, ah, I know already the answer. They give you the answer very, very fast, and that’s what happened. So, it actually inverted this matrix anyway, so they’ll know about mass that generate some code. You can see the code that is generated there.

And finally, there is the SQL query agent, and this SQL query agent can answer your questions about, the content of a database. This is a movie database. It is a hypothetical database. There are no real actresses or actors in here. And, for example, there is a movie called, yeah, I could first of all just list the tales in the database, but I already know that there is a movie called, Academy Dinosaur in there. And I’m just going to create a question about this Academy Dinosaur movie, which is a non-existing movie, actually and I’m going to ask you the specific, Robert Dukakis, this Dukakis, in this movie, acting in this movie. And so actually what it is actually doing internally generating SQL statements, and you can see it generates quite a good number of statements, and it is trying to figure out if Robert Dukakis is acting in this movie. And no, Robert Dukakis is not acting in this movie. And Penelope Guiness is, for example, whoever that is. Anyway, these are my three simple agents that we created as a demo here. Let’s go back to the presentation.

So, let me share again. hold on. I need to get to the sharing window again and find it again. So, that was the demo. Go back to it. so hopefully this is working.

There are agents, so you can then expand on these simple agents and for example, create agents that instead of having simple tools, they have agents as tools. And like this, you can build actually very, very complicated, recursive structures. Basically, you can build, agents that actually interact with other agents, and you can build like a big tree, of complex interactions with them. So, that’s just an idea.

Now the, which is the main technique, how that you actually can use for building agents. This technique is called ReAct. It’s actually Reason and Act, and it has, three steps, but Reason and Act is just a foundational workflow pattern, used in all agents. So, all agents use this, so believe me, agents are actually simpler. The people come up with incredibly complex, use cases for agents, but they are based on this pattern. This pattern allows to perform multi-step reasoning with error correction on top of it. And it also allows agents to access, information which are not accessible to the central intelligence unit, Large Language Models or the MLLM, by the way.

So, the main ingredients are actually three verbs. It’s a think, act and observe, think is actually about creating context, about thinking about the problem. but it is also expressing that in a sentence or in sentences. Then the second, step is act. Act is actually choose which tool to perform, a specific, action. And the third, verb is actually the observation. Observation is actually executing the tool and retrieving the results. So, who actually is responsible for the, the thinking? The Large Language Model, who is responsible for the acting is the Large Language Model. It’s choosing the tool. And who is responsible for the observation is actually the specific tool, the main tool, basically.

Here’s just an interaction example, where you see that there is a question by the user first step. Second step is reasoning about the question, and the reasoning looks like this. I should look up the best size for AI related news to provide accurate and up to date URLs. Just thinks about it then tells what it’s gonna do. I will perform a search using this tool. It already actually mentions the tool. Then in the action, actually it decides I’m going to call SerpAPI tool, a specific tool, whatever that tool is. And, with specific arguments, with specific, parameters, arguments, basically. And then, the tool executes and retrieves some search results. And these search results that, for example, some webpages like you see here. And then with all this information, with this extended context, the agent decides I can already reply to the question, and we’ll then synthesise an answer, which, and it says, here are some of the best sites for a right, AI related news and blah, blah, blah.

So, I’m just going to have here some typical ReAct based prompts. Normally this prompts instruct the model to use the ReAct pattern. So, the most important parts here in this, example here is this prompt, where it says, to do this, you will interleave thought tool, name and tool arcs and receive an, a resulting observation. So, this is the instruction that enforces the ReAct pattern.

So, here I have a second example where it, you see this internal prompt where it says you run in a loop of thought to pause observation. Again, it’s instruction, the model to use the ReAct pattern. So, every prompt that, so all the, the agents have an underlying prompt and they have all this kind of, react pattern in them. So, I’ve tried out multiple agent, multiple prompts, and these two prompts actually work very well in, in the agents that I’m using.

Now just talking about the idea of self improving agents. So, like I said, they have this pattern of being able to deal with errors and shooting different paths to circumvent these errors. So, this is one of these kind of ideas, which I already explained.

The second thing agents, like you saw before when I was doing the demo, actually agents should be able to cache so they know about previous interactions, and if you, they detect that an interaction is already kind of, has already been tried and there is no, change, possible change in the, in, in the query, then actually they can, instead of executing the whole agent, it can bypass all of that and just give you the answer straight away. And that happened to me actually in the demo. So, caching is also, an example of an agent that can improve itself.

And then there are also techniques where you actually gather the output of the agent and then use this output to tune the prompts. Now, there are specific, advocates of this kind of technique, and this technique is about really, creating a data set, based on interactions over time, and then creating some metrics like an evaluator that can actually evaluate the quality of specific interactions. And then based on the quality of interactions, it can actually optimize the prompt. So, this is, something very, very interesting.

There is a framework that specializes on auto prompt tuning based on the interaction. So, it means that over time the agent can actually improve its own prompt its way of interacting with the Large Language Model. So, and that the GSPY is declarative, self-improving patents, very interesting framework, and it attempts exactly that. So, the idea of creating agents that over time they’ll learn with interaction, but you need to obviously, have a specific setup to, to make this actually possible.

Now, after talking so much about agents, I would like to ask you, in which areas do you think you can use agents? So, agents can be using product search, for example, in retail. They can also be used in product recommendation also in retail or gathering information like in consulting businesses or, also in other types of legal, businesses like document search, for example. Just let me know where you actually think you can use, in which areas you actually think you can use agents. And so let me continue here.

I’ve been using some agent toolkits, here they are. Specifically I have used DSPY and LangGraph. DSPY because it allows you really to build agents that can improve themselves over time. AutoGen I use more in the context of, creating. So AutoGen I use more in the context for creating actually, teams of agents that interact with each other as well as CrewAI. So, nowadays there are more and more agent toolkits out there, but also more products that allow you to easily build agents.

I would also like to go to this question. What is next? Because I just am scratching the surface, but I, today I was scratching the basics, let’s say. So, the next big topics, I, I think in the industry, but also in these webinars are agents swarms and simulations, like what Garima has already mentioned, and voice and video agents. So, agents that not necessarily actually, perform, simply, chat operations, but they can also listen to your voice and they can also synthesise the answer.

But I would like now to, see what you have, what are your actual plans for using agents? And it seems that we have here quite, as I can see, actually it’s very well distributed, the plans for using agents as I see it. So, I see that there is perhaps the most popular one was, database search and document search. So, basically it’s about knowledge. How do you say about information management? So, as I see, so most people are thinking about these areas of information and knowledge management. What, what else is there? But retail is also quite a popular in terms of, as I see it, as well as consulting in any case. It’s interesting. It seems that the information and knowledge management are, is actually perhaps the preferred usage of agents, but however, it’s a bit, hard to just analyse this because everything is so even here. Thanks a lot.

I’m just wanting to, to give some credits, to ChatGPT, DSPY and all these other frameworks that I’ve been using and also mentioning the most important paper, in my opinion, the most important paper in this whole agent kind of thing and foundational paper is this paper about ReAct, Reason and Acting. It’s called actually ‘Synergizing Reasoning and Acting in Large Language Models’. And here is the arxiv link. So, for anyone who actually wants to look at it and see, it’s actually a kind of a genius kind of thing because based on this paper, you actually can build, huge businesses, actually huge ecosystems on top of these type of ideas. Anyway, thank you very much. I think I’ll stop here. I’ve spoken enough already. Now I would like to have the Q&A session. Thank you.

Garima Dosi (Moderator):

Great Gil. A good discussion on agents there. We had, some questions, related to the limitations of agents. Like, do agents inherit the limitations of LLMs because they use LLMs, at their core, or what are the other limitations of agents?

Gil Fernandes (Speaker):

That’s definitely, sure. Because Large Language Models can, hallucinate, agents can also hallucinate. However, I would, rather suggest that they hallucinate less because they are actually using tools that can counteract those hallucinations. So, that’s, also, the same thing is also happening there with the Retrieval Augmented Generation systems. So they inherit some of the weaknesses of Large Language Models. They do definitely, but these weaknesses are not so visible or they are not, so how do, how do you say they don’t have such a big impact, but now how do you counteract these, weaknesses? So, the main weakness I see there is, hallucination. There is, other weaknesses of Large Language Models. We mentioned them is knowledge cutoffs. Now the knowledge cutoff can actually be mitigated with tools. So, I can actually build into my agent a tool that retrieves up to date knowledge. So in a sense, there are some weaknesses of Large Language Models that can be mitigated better and some which are more difficult to mitigate with agents. Agents are also great to expand the reasoning capabilities of Large Language Models. So, there they are also very effective. And, I think the main problem that remains is really the problem of hallucination, of still hallucinating, even though you actually have there some tools that can actually counteract this hallucination. This problem is still there. And, the way to actually mitigate this, which I think this main problem is actually to build really good fact checking tools that actually go against, the model’s output and actually evaluate it. So, I would say create fact checking tools and also create also some kind of tools that can, inspect the results of the agent so that you can, fine tune the prompt of this agent to counteract the problem of the main problem of hallucinations. So, I hope this, this makes some sense what I’m saying here.

Garima Dosi (Moderator):

Yes, Definitely. There was another questions by a question while you were demoing, the agents. And, the question was that the response of an agent, is it better to send it as stream output?

Gil Fernandes (Speaker):

What I was doing there, so the agent has a thought process, like we said, because it is using the ReAct pattern. So, it’s always reasoning, deciding the next tool to use, then calling the tool, then getting some, observation from the tool. It’s, and what I was trying to do there in the demo was actually to stream out the, the single steps of the execution. I was not doing, the same for all agents, but the idea was actually to give you an idea of how the agent is actually thinking. Yes, I think it’s much better to stream the thinking process of the agent so that the end user does not see only the response, but also understand has the possibility of understanding what the agent actually has thought, what was the reasoning process behind it? Because imagine if there was a flaw in this reasoning process and you just get the answer, the end answer, then actually you cannot debug the thinking of the agent if you don’t have the thinking process there. So, yes, I would say yes, you should actually stream out, to the end user the thinking process of the agent so that the user can actually debug and understand better the final answer. Actually, it’s a very good idea to do that.

Garima Dosi (Moderator):

And then the last question is around, the cost of, maintaining, deploying, creating. So, creating, deploying, executing, and also maintaining agents. What is the cost involved in this?

Gil Fernandes (Speaker):

So, the cost are linked to the usage of the Large Language Models mainly, and also to, to the cost of the tools. Now, for example, these days, for example, if you’re using search tools like Tavily or, the one SerpAPI, these tools are also costly because they use internally Google and these APIs, they cost you some money. Now, the cost of the tools, kind of depends. If for example, if you build in your company your own tools, obviously they will be free for you, right? That’s obvious, right? But, one question remains, the other cost is the cost related to the Large Language Model. So, the cost of the agent is actually the cost of the tools plus the cost of the Large Language Models. So, you have to kind of, have this in mind, right? So, there is also the cost of the Large Language Models. Obviously, in my tools I’m using ChatGPT for something and I’m paying for it, right? So, and the agent is more expensive than the, how do you say this simple interaction with the Large Language Model because actually interacting multiple times with the Large Language Model. So, it means agents are always, in terms of interactions with the Large Language Model, more intensive. And so the biggest costs that I think you’re going to have are the cost of usage of the Large Language Model. Now you can again, mitigate those costs by, in housing the Large Language Model. So, instead of using ChatGPT, you could actually set up some server with something like Ollama or something like that where you actually use Llama 3 or something like that. And then it is cheaper. Obviously you need to run a big beefy machine to run this, Large Language Model too. So you have to see, definitely agents are more expensive than single interaction agents, have two, sources of cost. One is the tooling cost and another one is the Large Language Models cost, and you can mitigate the Large Language Model by in housing the usage of the, Large Language Model, basically.

Garima Dosi (Moderator):

Yeah. I think we’ve got, so many questions for today and, thank you Gil for, explaining agents and the concept so well with examples. The demo is great and, so, please do contact Gil, with any queries or feedback, on this session. His contact details are on the screen. you can email him at techtalk@onepointltd.com. And also, he’s available on LinkedIn.

The upcoming webinars. So, the upcoming webinars would all be focused on the new leads of agents, mostly around agents, so deeper dive into certain areas and topics of agents. So, while you wait for the next webinar, you can also go through the existing ones, the link for which would be, shared post this webinar and is also displayed on the screen. So, I think that’s it from us, for today. And bye for now and have a good day ahead. Thank you so much.