Autonomous Hierarchical Agent System using CrewAI

Hello Techies👋! Hope you all are doing amazing stuff. I'm back with an amazing concept in GenAI, I'm sure you'd love it "Agents in GenAI". You can now make your custom Automation for your use case using Agents and Generative AI technology. Sounds Interesting right!!.

There's a lot hype going on in the genAI market regarding this topic these days. So, Today in this article I'm going to practically show you how to make use of CrewAI framework, prompt-engineering, OpenAI API and build your own newsletters daily without you searching and browsing through the internet everytime, directly using GenAI Agents.🤩

This Article is beginner friendly and considers the fundamental knowledge of prompt-engineering or OpenAI technology. You can follow my previous article regarding advance prompt engineering techniques to learn more about it.

Note: Bonus!! at the end I'm going to discuss the Daily Auto Newsletter generator using crewAI, serper API and OpenAI gpt-4.

Excited!! Without any further ado, Let's dig into the topic

Let's first see our Agenda!

What are Agents? How they help in building an Automation
Multi Agent: The GenAI secrete weapon for enterprise success.
Industry Use-cases implementing Multi-Agent Systems.
Challenges with Agent Systems.
CrewAI v/s Autogen: Framworks to build GenAI multi-Agent systems
Fundamental Flow of building multi-agent system using CrewAI Framework
Sample Project: Built an Autonomous newsletter Creator Multi-Agent system using CrewAI, OpenAI gpt-4 and Serper-API.

What are Autonomous Agents?

Autonomous agents offer a new approach to building generative AI models. Autonomous agents are software programs that can act independently to achieve a goal. In the context of generative AI, autonomous agents can be used to generate content without the need for human intervention.

Autonomous agents can overcome the limitations of prompt engineering in several ways. First, autonomous agents can learn to generate content without the need for human-provided prompts. This allows them to be more creative and flexible in their output. Second, autonomous agents can be given goals that are more general than specific prompts. This allows them to generate different types of content that are still relevant to the goal. Thirdly, these agents can make decisions on their own without any human intervention and can achieve the goal with a set of provided tools.

They can use the right prompts themselves. Autonomous agents can learn to identify the right prompts for a given task, without the need for human intervention. This allows them to be more efficient and effective in their output.
They can validate the output. Autonomous agents can evaluate the output of their generation process and make adjustments as needed. This ensures that the output is of high quality and meets the user's requirements.
They can apply multiple iterations. Autonomous agents can generate content iteratively, making improvements each time. This allows them to create high-quality content that is tailored to the user's needs.
They can learn from their mistakes and improve over time. Autonomous agents can learn from their mistakes and improve their performance over time. This allows them to become more intelligent and capable of generating high-quality content without human assistance.

Multi Agents: The GenAI secrete weapon for enterprise success

Multi-agent systems can assume complex organisational roles through actors (synthesized through agents) to automate collaboration for solving complex tasks. Such capability can produce outstanding business results and drive optimised business outcomes.

Think of multi-agent frameworks as being able to harness the power of multiple generative AI models, plugins or agents and tools, where collaborative software entities, assigned with different roles to different models are combined cohesively to build a more intelligent and assertive system.

The image below, shows a way to visualise how multi-agent systems can be combined, being flexible to perform simple to more complex tasks. We can see agents that interact with end users, LLMs, code generation and code interpreters, safeguards, and the use of internal or external tools in the enterprise.

Multi-agent framework, managing the interaction between multiple agents and tools

A recent paper from Microsoft, Pennsylvania State University and the University of Washington (https://arxiv.org/pdf/2308.08155.pdf), is one example of a multi-agent system. The paper states that by combining multiple agents and focusing on the tasks they are best for, we end up having better outcomes.

It’s important to highlight, that this doesn’t have to be just chaining of LLMs, multiple types of existing or new AI/ML models can be aggregated to address specific goals and tasks, and integration with third-party tools that sometimes can perform extremely well for specific problems.

Here are a few key features of multi-agent systems:

· Create new types of intelligence that can perform more complex tasks, with greater results.

· Use of LLMs and/or tools to create intelligent agents to communicate and collaborate.

· Generate improved inputs for intelligent agents, which can help them to perform the tasks more effectively.

· Create a collaboration mechanism between agents that results in increased efficiency and better outcomes for the asks made by humans or other applications.

Looking at the diagram below, we can see diverse applications that can be built using multi-agent systems.

Example of applications of Multi-agents systems, and how the agents can collaborate for augmented outcomes

Math solving, where the agents are combined to address questions related to math, where LLMs are usually not good for, however using multiple agents we can indeed perform accurate calculations, still providing a very compiling answer.

Multi-agent coding, where agents are being used to generate user stories, code, tests or even graphical design for software/applications, while a simple agent would not be able to perform all mentioned roles, or not able to be specific for the different roles associated with coding. You could see an analogy in the real world, where a developer doing it all, gathering requirements, doing the design, coding and then testing, would not have the best results.

Conversations interactions, where multiple interactions between user and machines are possible, having agents acting as mediators for conflicts, expediting trivial and non-complex or critical judicial processes that could drive to the agreement between parts without the need for complex processes taking a long time to get up to results. Traditional chatbots are very dialogue flow oriented while conversational AI will provide a much better experience to the end user besides being able to cover more questions or asks.

Business process automation, being supported by corporate operational systems, agents will be used to interact with humans or applications in other to achieve a high level of automation without such enhanced customer experience and overcome the challenges imposed by legacy bots.

Online decision-making, where documents, webpages, third-party systems, or even abroad knowledge bases are used to enrich decision-making inside the organization. This is an example where web interaction tasks.

Retrieval-augmented Generation is known as a technique that improves the quality of text generated by LLMs by incorporating information from corporate knowledge sources. The text is captured before being submitted with the original prompt to LLMs that will originate the final output, where information from inside corporations will still be kept safe and not exposed outside of the context of the task being performed.

What are then the benefits that we could expect out of Multi-agent systems and are the first applications at the enterprises?

Here are some advantages:

· Increased flexibility and scalability: by easily adapting to changing business needs and can scale to meet the demands of even the largest enterprises.

· Improved efficiency and productivity: automating many tasks that are currently performed by humans on distinct roles, freeing up employees to focus on more strategic work.

· Enhanced decision-making: Agents can make decisions based on local information, leading to a faster and more efficient decision-making process. This is crucial in dynamic business environments where quick responses are necessary.

· Improved customer service: by creating personalized customer experiences and responding to customer inquiries more quickly and efficiently considering the different roles that might be required.

· Improved planning: by creating the entire set of artefacts required for the planning of specific goals, along with an optimized plan of resources.

· Complex problem-solving multi-agent systems excel in solving complex problems that involve a large amount of data and variables. In an enterprise, this could mean optimizing supply chains, predicting market trends, or managing large-scale projects.

Industry Use-cases implementing Multi-Agent Systems

Supply chain management: Where multi-agents can be used to optimize the flow of goods and materials through a supply chain, to predict demand, predict and assign orders, identify bottlenecks, launch marketing campaigns, and analyze the results to apply improvements along the sales process.
Create marketing campaigns for products: where multiple roles and collaborations of humans would be required, and long meetings, for those goods not sold or about to expire, others could be targeted for campaigns that can be planned and managed, all having human security and safety in mind but also the importance of minimizing waste and minimize loses.
Enhanced Customer Experience: Personalization, agents can analyze customer data and behavior to provide personalized experiences, whether it is in customer support interactions, product recommendations or marketing campaigns. This individualized touch can significantly enhance customer satisfaction and loyalty.

Challenges with Agent Systems

While multi-agent systems hold great promise as a secret weapon for enterprise success, there are indeed challenges and concerns that need to be carefully considered and addressed. There are also some challenges and concerns associated with the use of multi-agents because as they can be used to perform more effectively and faster, they could be used to create harmful content, such as fake news, propaganda, sophisticated scamming and cyberattacks which have generated a lot of discussion and investment in ethical AI. Of course, looking at the positive side, they can also be used to detect and automatically prevent or minimize the impact of attacks whilst executing a neutralization strategy.

Addressing these challenges requires a combination of technical expertise, rigorous testing, clear policies and regulations and ongoing monitoring. As the field of multi-agent systems continues to evolve, so will the solutions to these challenges. Enterprises investing in this technology must remain vigilant, adaptable, and committed to ethical and responsible AI practices to unlock the full potential of multi-agent systems for their success.

CrewAI v/s Autogen: Frameworks to build GenAI multi-Agent systems

GitHub stars rating evolution of AutoGen and CrewAI

CrewAI is built on top of LangChain and allows one to orchestrate multiple agents working on a user-defined task.

Same as AutoGen , CrewAI is an open-source and uses the concept of agents with different roles, but on top of that, CrewAI allows agents to delegate work to each other.

For code execution Autogen creates python code and run that code inside a Docker Container, while CrewAI generates a report regarding the code and using expected_output one can save the code and run anywhere as it wants without security and other concerns.

AutoGen and CrewAI they have proven the ability to deliver some interesting and useful examples quickly. While CrewAI is younger than AutoGen, it would be cool to see benchmarks and evals from both frameworks to make it easier for developers to make the right decision when deciding.

I heard from some developers that they chose CrewAI because they were already familiar with LangChain, and others argued that AutoGen is more customizable. However, when discussing with developers, most said that they don’t see a big difference between CrewAI and AutoGen as they accomplish similar tasks.

In my personal opinion and experience Autogen is more customisable like langchain and CrewAI is more featured and easy to use like llama-index

Fundamental Flow of building multi-agent system using CrewAI Framework

CrewAI is built on top of LangChain and allows one to orchestrate multiple agents working on a user-defined task.

Key Features

Role-Based Agent Design: Customize agents with specific roles, goals, and tools.
Autonomous Inter-Agent Delegation: Agents can autonomously delegate tasks and inquire amongst themselves, enhancing problem-solving efficiency.
Flexible Task Management: Define tasks with customizable tools and assign them to agents dynamically.
Processes Driven: Currently only supports sequential task execution and hierarchical processes, but more complex processes like consensual and autonomous are being worked on.
Save output as file: Save the output of individual tasks as a file, so you can use it later.
Parse output as Pydantic or Json: Parse the output of individual tasks as a Pydantic model or as a Json if you want to.
Works with Open Source Models: Run your crew using Open AI or open source models refer to the Connect crewAI to LLMs page for details on configuring your agents' connections to models, even ones running locally!

Follow below steps to create your agent system using crewAI. It's simple!!

Create An Agent
Define and create a Task
Define a Crew Process
Crew Kickoff to get your crew to work!

Flow to start your coding with crewAI right away:

import os
from crewai import Agent, Task, Crew, Process
from crewai_tools import SerperDevTool

os.environ["OPENAI_API_KEY"] = "YOUR_API_KEY"
os.environ["SERPER_API_KEY"] = "Your Key" # serper.dev API key

# You can choose to use a local model through Ollama for example. See https://docs.crewai.com/how-to/LLM-Connections/ for more information.

# os.environ["OPENAI_API_BASE"] = 'http://localhost:11434/v1'
# os.environ["OPENAI_MODEL_NAME"] ='openhermes'  # Adjust based on available model
# os.environ["OPENAI_API_KEY"] ='sk-111111111111111111111111111111111111111111111111'

search_tool = SerperDevTool()

# Define your agents with roles and goals
researcher = Agent(
  role='Senior Research Analyst',
  goal='Uncover cutting-edge developments in AI and data science',
  backstory="""You work at a leading tech think tank.
  Your expertise lies in identifying emerging trends.
  You have a knack for dissecting complex data and presenting actionable insights.""",
  verbose=True,
  allow_delegation=False,
  tools=[search_tool]
  # You can pass an optional llm attribute specifying what mode you wanna use.
  # It can be a local model through Ollama / LM Studio or a remote
  # model like OpenAI, Mistral, Antrophic or others (https://docs.crewai.com/how-to/LLM-Connections/)
  #
  # import os
  # os.environ['OPENAI_MODEL_NAME'] = 'gpt-3.5-turbo'
  #
  # OR
  #
  # from langchain_openai import ChatOpenAI
  # llm=ChatOpenAI(model_name="gpt-3.5", temperature=0.7)
)
writer = Agent(
  role='Tech Content Strategist',
  goal='Craft compelling content on tech advancements',
  backstory="""You are a renowned Content Strategist, known for your insightful and engaging articles.
  You transform complex concepts into compelling narratives.""",
  verbose=True,
  allow_delegation=True
)

# Create tasks for your agents
task1 = Task(
  description="""Conduct a comprehensive analysis of the latest advancements in AI in 2024.
  Identify key trends, breakthrough technologies, and potential industry impacts.""",
  expected_output="Full analysis report in bullet points",
  agent=researcher
)

task2 = Task(
  description="""Using the insights provided, develop an engaging blog
  post that highlights the most significant AI advancements.
  Your post should be informative yet accessible, catering to a tech-savvy audience.
  Make it sound cool, avoid complex words so it doesn't sound like AI.""",
  expected_output="Full blog post of at least 4 paragraphs",
  agent=writer
)

# Instantiate your crew with a sequential process
crew = Crew(
  agents=[researcher, writer],
  tasks=[task1, task2],
  verbose=2, # You can set it to 1 or 2 to different logging levels
)

# Get your crew to work!
result = crew.kickoff()

print("######################")
print(result)

The Code is available here, So before going into the further discussion you can visit and check the code to directly start working: Autonomous Newsletter Creator Multi-Agent system.

It's better to start splitting and segregating the code as per the flow into multiple modules and scripts. Like Declare Agents into agents.py, Task under tasks.py, Tools under different .py files of tools etc. SDE hack:)

I've structured my code in the same way. let's discuss each section in-detail and what it does.

Note: in the .env.example file define the OpenAI_API_KEY and SERPER_API_KEY, put your own credentials over there and use pyproject.toml to install the dependencies to your system for running the application.

Create Agents

Agents are basically like the task doors, they have some particular roles and they perform the tasks defined to them.

For the Automated Newsletter I've create 4 agents i.e.
news_fetcher_agent will fetch the data from the internet and ensure to fetch the latest/upto the date information from the internet.

news_analyser_agent: News_analyser will summarise the news fetched by news_fetcher_agents and curate those to write down to the .md file.

newsletter_compiler_agent: this agent will ensure attention to detail and it changes the look and structure of the md file so as to look like a newsletter.

editor_agent: So as to maintain the TopAI news in the market, ensures the user-engagement and Latest News in the feed of newsletter.

Create Tasks

Now let's define tasks to each agents. For the tasks, crewAI just added some new features like async operation for the agents, expected_output we can define to the agents, callbacks are also added so, we can run that agents multiple time to improve the response or output. Process we can now define Hierarchical ie. agents can now perform more complex tasks fasterrr.

In the tasks.py created 3 tasks i.e. to fetch_news from tool, to analyse the news and to compile the news, editor is just the reviewer of the end process/newsletter So, not assigned any task to editor agent.

Create Tools

As our task requires some tool that will fetch the data from the Internet or browser and then gives us the data to create and compile the newsletter. hence used a serper API to browse through the internet and fetches the latest trending AI data.

under the tools folder created a search_tools.py where searching code through google using serper-API resides.

Create Output tool

define any output tool or file to trigger or write your output to In this case final output should be an .md file looks like a newsletter that we can publish on to Linkedin or any blogs platform. We can trigger the blogs platform or linkedin to schedule the newsletters using our above workflow, but for now we as storing that into a output file, file_io.py contains the output file code.

Orchaestrate the Agent flow

Finally in the main.py defined the code orchaestration using crew and main thing here as per new crewAI launch defined the process as hierarchical, by which we can run multiple agents at the same time.

Finally we kickoff the Crew to complete it's journey from fetching the news from Internet to Write the Newsletter into a file.

Hurray!! Now You can built your company with agents will act as your employees to do your end-to end work😄. Seems Interesting Right!!

I'm planning to make more such complex use-cases on GenAI agents Automation and overall GenAI applications from RAG to fine-tuning etc.

Stay tuned and follow our newsletter to get daily updates and projects in the industry every week!! Connect with me on linkedin, github, kaggle.

Let's Learn and grow together:) Stay Healthy stay Happy✨. Happy Learning!!

Future of AI Automation: Built Autonomous Agents using CrewAI