workflow-builder/README.md

# Agent Workflow Builder

**AI workflow builder that turns plain English into complex automation pipelines with drag-and-drop magic ✨**

An open-source, visual workflow builder for creating AI-powered agent workflows, similar to N8N or Zapier, built with React Flow and modern web technologies. Create sophisticated automation workflows with AI models, web scraping, and data processing capabilities.

**🚀 Build AI workflows without code: describe, drag, drop, execute. Like Zapier but smarter!**

## Data Flow Architecture

```mermaid
sequenceDiagram
    participant U as User
    participant UI as React UI
    participant S as Zustand Store
    participant E as Execution Engine
    participant P as Processors
    participant A as AI Services
    participant D as Database/APIs

    U->>UI: Create/Edit Workflow
    UI->>S: Update Workflow State
    S->>UI: Re-render Components

    U->>UI: Execute Workflow
    UI->>S: Trigger Execution
    S->>E: Start Execution Engine

    E->>E: Create Execution Plan
    E->>E: Validate Workflow
    E->>P: Execute Node Processors

    P->>A: Call AI Services
    A-->>P: Return Results
    P->>D: Call External APIs
    D-->>P: Return Data

    P-->>E: Node Results
    E->>S: Update Execution State
    S->>UI: Update UI with Results
    UI-->>U: Show Execution Results
```


## 🚀 Features

### 🎨 Visual Workflow Editor

_Turn ideas into workflows with zero coding_

- **🎯 Drag & Drop Interface**: Intuitive node-based workflow creation
- **⚡ Real-time Connection**: Connect nodes with visual edges
- **📚 Node Palette**: Easy access to all available node types
- **📱 Responsive Design**: Works on desktop and tablet devices
- **🔗 Smart Variable Substitution**: Use `{{nodeId.output}}` or `{{nodelabel.output}}` to pass data between nodes

### 🤖 AI-Powered Node Types

_Pre-built intelligence for every automation need_

- **🌐 Web Scraping**: Extract data from websites using Firecrawl AI
- **🤖 LLM Task**: General language model operations (DeepSeek, OpenAI)
- **🧠 Embedding Generator**: Create vector embeddings for semantic search
- **🔍 Similarity Search**: Find similar content in vector stores
- **📋 Structured Output**: Parse and structure data with JSON schemas
- **📥 Data Input**: Workflow entry points (text, JSON, CSV, URL, PDF)
- **📤 Data Output**: Workflow exit points with multiple formats

### 🧠 AI Copilot Assistant

_Your AI assistant that reads your mind and builds workflows_

- **🗣️ Natural Language Processing**: Describe workflows in plain English
- **✨ Intelligent Workflow Generation**: AI creates workflows from descriptions
- **💡 Contextual Suggestions**: Get smart recommendations based on your workflow
- **🔍 Validation & Optimization**: Automatic workflow validation and improvement tips
- **🎯 Mixed Intent Analysis**: Handle complex, multi-step workflow requirements

### 📁 Workflow Management

_Organize, share, and scale your automations_

- **💾 Save & Load**: Persistent workflow storage in browser
- **📤 Import/Export**: Share workflows as JSON files
- **📊 Version Control**: Track workflow changes with timestamps
- **📚 Workflow Library**: Manage multiple workflows with search and filtering
- **🎯 Demo Workflows**: Pre-built example workflows to get started

### ⚡ Execution & Testing

_See your workflows come to life in real-time_

- **🚀 Real-time Execution**: Step-by-step workflow processing with live updates
- **👁️ Visual Feedback**: Live status indicators for each node (idle, running, success, error)
- **🧪 Test Interface**: Run workflows with sample data and custom inputs
- **📋 Execution Logs**: Detailed execution history and debugging information
- **🛠️ Error Handling**: Clear error messages and recovery suggestions
- **📊 Performance Monitoring**: Track execution times and resource usage

### 🔧 Advanced Features

_Enterprise-grade capabilities for complex automations_

- **⚙️ Configurable Parameters**: Customize each node's behavior and settings
- **🌊 Data Flow Visualization**: See data passing between nodes in real-time
- **📊 Execution Results**: View and export workflow outputs in multiple formats
- **🧩 Modular Architecture**: Easy to extend with new node types
- **📄 PDF Processing**: Upload and process PDF documents with text extraction
- **🔌 API Integration**: Seamless integration with DeepSeek, OpenAI, and Firecrawl APIs

## 🛠️ Technology Stack

_Built with modern, battle-tested technologies_

- **⚛️ Frontend**: React 18 + TypeScript
- **🎨 Visual Editor**: React Flow v11
- **💅 Styling**: Tailwind CSS with custom animations
- **🗃️ State Management**: Zustand
- **🎯 Icons**: Lucide React
- **🔔 Notifications**: React Hot Toast
- **🤖 AI Integration**:
  - DeepSeek API (primary LLM)
  - OpenAI API (alternative LLM)
  - Firecrawl AI (web scraping)
- **📄 PDF Processing**: Browser-based PDF.js
- **🔨 Build Tool**: Create React App

## 🚀 Quick Start

_Get up and running in under 5 minutes_

1. **📥 Clone the repository**

   ```bash
   git clone https://github.com/Nikhil-Doye/workflow-builder.git
   cd workflow-builder
   ```

2. **📦 Install dependencies**

   ```bash
   npm install
   ```

3. **🔑 Configure API Keys** _(Optional but recommended)_

   - The app will prompt you to configure API keys on first launch
   - You can also configure them later through the settings menu
   - Required APIs:
     - **DeepSeek API**: For LLM tasks and AI processing
     - **Firecrawl API**: For web scraping functionality
     - **OpenAI API**: Alternative LLM provider

4. **▶️ Start the development server**

   ```bash
   npm start
   ```

5. **🌐 Open your browser**
   Navigate to `http://localhost:3000`

## 🎯 How to Use

### Method 1: AI Copilot (Recommended) 🤖

1. **Start the Application**

   - Open the app in your browser
   - Click "New Workflow" to create your first workflow

2. **Use AI Copilot**

   - Click the "AI Copilot" button in the toolbar
   - Describe your workflow in natural language (e.g., "scrape a website and analyze the content")
   - Let the AI generate the workflow structure for you
   - Review and modify the generated workflow as needed

3. **Test and Execute**
   - Use the Testing Panel to provide sample input data
   - Click "Run Test" to execute the workflow
   - View results in real-time as each node processes

### Method 2: Manual Workflow Creation 🎨

1. **Add Nodes**

   - Click "Show Node Palette" to reveal available node types
   - Drag nodes from the palette onto the canvas
   - Connect nodes by dragging from output handles to input handles

2. **Configure Nodes**

   - Click on any node to open its configuration panel
   - Set parameters like URLs, prompts, model settings, etc.
   - Use variable substitution: `{{nodeId.output}}` or `{{nodelabel.output}}` to pass data between nodes

3. **Test Your Workflow**

   - Use the Testing Panel to provide sample input data
   - Click "Run Test" to execute the workflow
   - View results in real-time as each node processes

4. **Save and Export**
   - Click "Save" to persist your workflow
   - Use "Export" to download your workflow as a JSON file
   - Share workflows by importing JSON files

### 💡 Example Workflows

#### 🌐 Web Scraping + AI Analysis

_Extract data from websites and analyze with AI_

1. **Data Input** → **Web Scraping** → **LLM Task** → **Data Output**
2. Configure the Web Scraping node with a URL
3. Set up the LLM Task with a prompt like "Analyze this content: {{scraper.output}}"
4. Test with a sample URL

#### 📄 Document Processing Pipeline

_Process PDFs and extract structured data_

1. **Data Input (PDF)** → **LLM Task** → **Structured Output** → **Data Output**
2. Upload a PDF document
3. Extract and process the text with AI
4. Structure the output according to a JSON schema

#### 🔍 Embedding & Search Workflow

_Find similar content using vector search_

1. **Data Input** → **Embedding Generator** → **Similarity Search** → **Data Output**
2. Create embeddings from input text
3. Search for similar content in a vector store
4. Return the most relevant results

## 🏗️ Architecture

_Scalable, modular design for enterprise use_

### 📁 Project Structure

```
src/
├── components/           # React components
│   ├── nodes/           # Node type components
│   │   ├── BaseNode.tsx
│   │   ├── DataInputNode.tsx
│   │   ├── DataOutputNode.tsx
│   │   ├── WebScrapingNode.tsx
│   │   ├── LLMNode.tsx
│   │   ├── EmbeddingNode.tsx
│   │   ├── SimilaritySearchNode.tsx
│   │   └── StructuredOutputNode.tsx
│   ├── WorkflowEditor.tsx
│   ├── WorkflowList.tsx
│   ├── WorkflowToolbar.tsx
│   ├── NodeConfiguration.tsx
│   ├── ExecutionPanel.tsx
│   ├── TestingPanel.tsx
│   ├── CopilotPanel.tsx
│   ├── OnboardingModal.tsx
│   └── OpenAIConfig.tsx
├── services/            # External service integrations
│   ├── openaiService.ts
│   ├── firecrawlService.ts
│   ├── copilotService.ts
│   └── pdfService.ts
├── store/               # State management
│   └── workflowStore.ts
├── types/               # TypeScript definitions
│   └── index.ts
├── utils/               # Utility functions
│   ├── workflowSerialization.ts
│   ├── workflowValidator.ts
│   ├── workflowGenerator.ts
│   ├── variableSubstitution.ts
│   └── patternMatchers.ts
├── examples/            # Example workflows and documentation
└── App.tsx             # Main application
```

### 🔧 Key Components

- **🎨 WorkflowEditor**: Main visual editor using React Flow with drag-and-drop functionality
- **🤖 CopilotPanel**: AI-powered workflow generation and assistance
- **⚙️ NodeConfiguration**: Modal for configuring node parameters and settings
- **⚡ ExecutionPanel**: Real-time execution monitoring with live status updates
- **🧪 TestingPanel**: Test interface with sample data and custom inputs
- **📚 WorkflowList**: Workflow management with search, filtering, and CRUD operations
- **🗃️ workflowStore**: Centralized state management with Zustand for workflow data
- **🔌 Services**: Modular service layer for AI APIs, web scraping, and PDF processing

## 🔧 Configuration

_Set up your AI integrations in minutes_

### 🔑 API Keys Setup

The application requires API keys for full functionality:

1. **DeepSeek API Key** (Primary LLM)

   - Get your key from [DeepSeek Platform](https://platform.deepseek.com/api_keys)
   - Used for LLM tasks and AI processing
   - Format: `sk-...`

2. **Firecrawl API Key** (Web Scraping)

   - Get your key from [Firecrawl](https://firecrawl.dev)
   - Used for web scraping functionality
   - Format: `fc-...`

3. **OpenAI API Key** (Alternative LLM)
   - Get your key from [OpenAI Platform](https://platform.openai.com/api-keys)
   - Used as fallback for LLM tasks
   - Format: `sk-...`

### Node Types and Parameters

Each node type supports specific configuration options:

#### Variable Substitution with Node Labels

The workflow builder now supports intuitive variable substitution using node labels:

**Traditional Method (Node IDs):**

```
{{node-1234567890.output}}
```

**New Method (Node Labels):**

```
{{Website URL.output}}
{{Web Scraper.output}}
{{Content Analyzer.output}}
```

**Benefits:**

- **More Intuitive**: Use descriptive names instead of cryptic IDs
- **Easier to Read**: Workflow configurations are self-documenting
- **Better UX**: No need to remember or look up node IDs
- **Backward Compatible**: Both methods work together

**Requirements:**

- Each node must have a unique label within the workflow
- Labels are case-sensitive
- Empty labels are not allowed
- The system validates label uniqueness automatically

#### Web Scraping Node (Firecrawl AI)

- `url`: Target website URL (supports variable substitution)
- `formats`: Output formats (markdown, html, text, summary, links, images)
- `onlyMainContent`: Extract only main content (boolean)
- `includeTags`: CSS selectors to include (comma-separated)
- `excludeTags`: CSS selectors to exclude (comma-separated)
- `maxLength`: Maximum content length to extract
- `waitFor`: Wait time for dynamic content (milliseconds)
- `timeout`: Request timeout (milliseconds)

#### LLM Task Node

- `prompt`: The prompt to send to the language model (supports variables)
- `model`: AI model selection (deepseek-chat, deepseek-reasoner)
- `temperature`: Response creativity (0-2, default: 0.7)
- `maxTokens`: Maximum response length (default: 1000)

#### Data Input Node

- `dataType`: Input data type (text, json, csv, url, pdf)
- `defaultValue`: Default input value or sample data

#### Data Output Node

- `format`: Output format (json, text, csv)
- `filename`: Output filename for exports

#### Embedding Generator

- `model`: Embedding model selection (text-embedding-ada-002, text-embedding-3-small)
- `dimensions`: Vector dimensions (default: 1536)

#### Similarity Search

- `vectorStore`: Target vector database (pinecone, weaviate, chroma)
- `topK`: Number of similar results to return (default: 5)
- `threshold`: Similarity threshold (0-1, default: 0.8)

#### Structured Output

- `schema`: JSON schema for structured output
- `model`: AI model for structured generation

## 🚀 Deployment

_Deploy to your favorite platform_

### 📦 Build for Production

```bash
npm run build
```

The build artifacts will be stored in the `build/` directory.

### Deploy to Vercel

```bash
npm install -g vercel
vercel --prod
```

### Deploy to Netlify

```bash
npm run build
# Upload the 'build' folder to Netlify
```

### Deploy to GitHub Pages

```bash
npm run build
# Push the build folder to a gh-pages branch
```

### Environment Variables

For production deployment, you may want to set up environment variables:

- `REACT_APP_DEEPSEEK_API_KEY`: DeepSeek API key
- `REACT_APP_FIRECRAWL_API_KEY`: Firecrawl API key
- `REACT_APP_OPENAI_API_KEY`: OpenAI API key

Note: API keys are currently stored in browser localStorage for security.

## 🤝 Contributing

_Help us make automation accessible to everyone_

We welcome contributions! Here's how you can help:

1. **🍴 Fork the repository**
2. **🌿 Create a feature branch**: `git checkout -b feature/amazing-feature`
3. **💾 Commit your changes**: `git commit -m 'Add amazing feature'`
4. **📤 Push to the branch**: `git push origin feature/amazing-feature`
5. **🔀 Open a Pull Request**

### 📋 Development Guidelines

- ✅ Follow TypeScript best practices
- 🎨 Use Tailwind CSS for styling
- 📝 Write meaningful commit messages
- 🧪 Add tests for new features
- 📚 Update documentation as needed

## 🐛 Troubleshooting

_Quick fixes for common issues_

### 🔧 Common Issues

**🔑 API Key Issues**

- Ensure API keys are properly configured in the settings
- Check that API keys have the correct format (sk-... for AI, fc-... for Firecrawl)
- Verify API keys have sufficient credits/permissions

**🔗 Nodes not connecting?**

- Ensure you're dragging from output handles (bottom) to input handles (top)
- Check that nodes are properly positioned and not overlapping
- Verify the workflow has a clear data flow path

**⚡ Workflow not executing?**

- Verify all required node parameters are configured
- Check the execution panel for detailed error messages
- Ensure all nodes are properly connected in sequence
- Check that variable substitutions are correct (e.g., `{{nodeId.output}}` or `{{nodelabel.output}}`)

**🌐 Web Scraping Issues**

- Verify the target URL is accessible and not blocked
- Check Firecrawl API key and credits
- Try different output formats or content selectors
- Ensure the website allows scraping (check robots.txt)

**🤖 LLM Task Issues**

- Verify AI API key is valid and has credits
- Check that prompts are properly formatted
- Ensure variable substitutions in prompts are correct
- Try different models or adjust temperature settings

**📤 Import/Export not working?**

- Ensure you're using valid JSON format
- Check browser console for error messages
- Verify the workflow structure is complete

**📄 PDF Processing Issues**

- Ensure the PDF file is not password-protected
- Check file size limits (large files may timeout)
- Verify the PDF contains extractable text (not just images)

### 🆘 Getting Help

- 🔍 Check the **Execution Panel** for detailed error messages and logs
- 🧪 Use the **Testing Panel** to debug with sample data
- 🤖 Review the **AI Copilot** suggestions for workflow improvements
- 🖥️ Check browser console for technical error details

## 📄 License

This project is licensed under the MIT License - see the [LICENSE](LICENSE) file for details.

## 🙏 Acknowledgments

_Built with amazing open-source tools_

- [React Flow](https://reactflow.dev/) for the visual editor
- [Tailwind CSS](https://tailwindcss.com/) for styling
- [Lucide React](https://lucide.dev/) for icons
- [Zustand](https://github.com/pmndrs/zustand) for state management
- [DeepSeek](https://www.deepseek.com/) for AI language models
- [Firecrawl](https://firecrawl.dev/) for web scraping capabilities
- [OpenAI](https://openai.com/) for alternative AI models

## 📚 Additional Resources

_Everything you need to master workflow automation_

- 📖 [Getting Started Guide](GETTING_STARTED.md) - Detailed setup and first workflow tutorial
- 🔗 [Variable Substitution Examples](src/examples/variableSubstitutionExample.md) - Learn how to pass data between nodes
- 🎯 [Demo Workflows](public/demo-workflows.json) - Pre-built example workflows

## 📞 Support

_We're here to help_

- 🐛 **Issues**: [GitHub Issues](https://github.com/your-username/agent-workflow-builder/issues)
- 💬 **Discussions**: [GitHub Discussions](https://github.com/your-username/agent-workflow-builder/discussions)
- 📚 **Documentation**: Check the Getting Started guide and examples

## 🔮 Roadmap

_What's coming next_

- [ ] 🤖 More AI model integrations (Claude, Gemini)
- [ ] 📋 Advanced workflow templates
- [ ] 👥 Team collaboration features
- [ ] ⏰ Workflow scheduling and automation
- [ ] 🛠️ Custom node development SDK
- [ ] 📊 Workflow analytics and monitoring
- [ ] 🗄️ Database integrations
- [ ] 🔌 API endpoint creation

---

**Built with ❤️ for the AI community**

_Empowering developers to create sophisticated AI workflows without the complexity._