Webhooks

Webhooks provide real-time notifications about workflow execution progress, eliminating the need to continuously poll the status endpoint. When you provide webhook URLs in your workflow execution request, Storyteller will send HTTP POST requests to those URLs at key stages of the workflow process.

Webhook Types

There are four types of webhooks you can configure:

Webhook Type	Trigger	Purpose
Processing	When workflow execution begins	Confirm workflow has started
Progress	During workflow milestones	Track intermediate progress
Processed	When workflow completes successfully	Handle successful completion
Failed	When workflow fails	Handle errors and failures

Webhook Configuration

Configure webhooks when executing workflows by including the webhook URLs in your request:

{
  "workflows": ["add-media"],
  "mediaUrls": ["https://example.com/video.mp4"],
  "metadata": {
    "https://example.com/video.mp4": {
      "Title": "My Video"
    }
  },
  "workflowProcessingWebhookUrl": "https://your-app.com/webhooks/processing",
  "workflowProgressWebhookUrl": "https://your-app.com/webhooks/progress",
  "workflowProcessedWebhookUrl": "https://your-app.com/webhooks/processed",
  "workflowFailedWebhookUrl": "https://your-app.com/webhooks/failed"
}

Webhook URL Requirements

• HTTPS Required: All webhook URLs must use HTTPS
• Publicly Accessible: URLs must be accessible from the internet
• Response Time: Endpoints should respond within 30 seconds
• Status Codes: Return 2xx status codes to acknowledge receipt
• No Authentication: Webhooks don't include authentication headers

Webhook Payload Structure

All webhooks send a JSON payload with the following structure:

{
  "correlationId": "12345678-1234-1234-1234-123456789012",
  "webhookType": "processed",
  "timestamp": "2025-01-15T10:30:00Z",
  "workflows": [
    {
      "code": "add-media",
      "status": "Finished",
      "message": "Workflow successfully completed",
      "timestamp": "2025-01-15T10:30:00Z",
      "steps": [
        {
          "name": "Create Media Asset",
          "status": "Finished",
          "message": "Media asset created successfully",
          "timestamp": "2025-01-15T10:29:45Z",
          "locations": [
            "https://yourtenant.usestoryteller.com/assets/12345/edit"
          ]
        }
      ]
    }
  ]
}

Payload Fields

Field	Type	Description
correlationId	string	Unique identifier for the workflow execution
webhookType	string	Type of webhook: processing, progress, processed, failed
timestamp	string	ISO timestamp when the webhook was sent
workflows	array	Array of workflow status objects

Workflow Object

Field	Type	Description
code	string	Workflow identifier
status	string	Current status: Running, Finished, Failed
message	string	Human-readable status message
timestamp	string	ISO timestamp of last status update
steps	array	Array of workflow step details

Step Object

Field	Type	Description
name	string	Step name
status	string	Step status: Running, Finished, Failed
message	string	Step status message
timestamp	string	Step timestamp
locations	array	URLs to created resources (optional)

Webhook Implementation Examples

JavaScript

```javascript const express = require('express'); const app = express();

// Middleware to parse JSON bodies app.use(express.json());

// Processing webhook - workflow started app.post('/webhooks/processing', (req, res) => { const { correlationId, workflows } = req.body;

console.log(🚀 Workflow started: ${correlationId}); workflows.forEach(workflow => { console.log(${workflow.code}: ${workflow.status}); });

// Acknowledge receipt res.status(200).json({ received: true }); });

// Progress webhook - workflow milestones app.post('/webhooks/progress', (req, res) => { const { correlationId, workflows } = req.body;

console.log(⏳ Workflow progress: ${correlationId}); workflows.forEach(workflow => { workflow.steps.forEach(step => { console.log(${step.name}: ${step.status}); }); });

res.status(200).json({ received: true }); });

// Success webhook - workflow completed app.post('/webhooks/processed', (req, res) => { const { correlationId, workflows } = req.body;

console.log(✅ Workflow completed: ${correlationId}); workflows.forEach(workflow => { workflow.steps.forEach(step => { if (step.locations) { console.log(🔗 Created: ${step.locations.join(', ')}); } }); });

// Process successful completion handleWorkflowSuccess(correlationId, workflows);

res.status(200).json({ received: true }); });

// Failure webhook - workflow failed app.post('/webhooks/failed', (req, res) => { const { correlationId, workflows } = req.body;

console.log(❌ Workflow failed: ${correlationId}); workflows.forEach(workflow => { console.log(${workflow.code}: ${workflow.message}); workflow.steps.forEach(step => { if (step.status === 'Failed') { console.log(Failed step: ${step.name} - ${step.message}); } }); });

// Process failure handleWorkflowFailure(correlationId, workflows);

res.status(200).json({ received: true }); });

async function handleWorkflowSuccess(correlationId, workflows) { // Your success handling logic // - Update database records // - Send notifications to users // - Trigger dependent processes console.log(Processing success for ${correlationId}); }

async function handleWorkflowFailure(correlationId, workflows) { // Your failure handling logic // - Log errors // - Retry if appropriate // - Notify administrators console.log(Processing failure for ${correlationId}); }

app.listen(3000, () => { console.log('Webhook server listening on port 3000'); });

=== "Python"

    ```python
    from flask import Flask, request, jsonify
    import logging

    app = Flask(__name__)
    logging.basicConfig(level=logging.INFO)

    @app.route('/webhooks/processing', methods=['POST'])
    def processing_webhook():
        data = request.json
        correlation_id = data['correlationId']
        workflows = data['workflows']

        app.logger.info(f'🚀 Workflow started: {correlation_id}')
        for workflow in workflows:
            app.logger.info(f'  {workflow["code"]}: {workflow["status"]}')

        return jsonify({'received': True}), 200

    @app.route('/webhooks/progress', methods=['POST'])
    def progress_webhook():
        data = request.json
        correlation_id = data['correlationId']
        workflows = data['workflows']

        app.logger.info(f'⏳ Workflow progress: {correlation_id}')
        for workflow in workflows:
            for step in workflow['steps']:
                app.logger.info(f'  {step["name"]}: {step["status"]}')

        return jsonify({'received': True}), 200

    @app.route('/webhooks/processed', methods=['POST'])
    def processed_webhook():
        data = request.json
        correlation_id = data['correlationId']
        workflows = data['workflows']

        app.logger.info(f'✅ Workflow completed: {correlation_id}')
        for workflow in workflows:
            for step in workflow['steps']:
                if 'locations' in step:
                    app.logger.info(f'🔗 Created: {", ".join(step["locations"])}')

        # Process successful completion
        handle_workflow_success(correlation_id, workflows)

        return jsonify({'received': True}), 200

    @app.route('/webhooks/failed', methods=['POST'])
    def failed_webhook():
        data = request.json
        correlation_id = data['correlationId']
        workflows = data['workflows']

        app.logger.error(f'❌ Workflow failed: {correlation_id}')
        for workflow in workflows:
            app.logger.error(f'  {workflow["code"]}: {workflow["message"]}')
            for step in workflow['steps']:
                if step['status'] == 'Failed':
                    app.logger.error(f'  Failed step: {step["name"]} - {step["message"]}')

        # Process failure
        handle_workflow_failure(correlation_id, workflows)

        return jsonify({'received': True}), 200

    def handle_workflow_success(correlation_id, workflows):
        """Handle successful workflow completion"""
        # Your success handling logic
        # - Update database records
        # - Send notifications to users
        # - Trigger dependent processes
        app.logger.info(f'Processing success for {correlation_id}')

    def handle_workflow_failure(correlation_id, workflows):
        """Handle workflow failure"""
        # Your failure handling logic
        # - Log errors
        # - Retry if appropriate
        # - Notify administrators
        app.logger.info(f'Processing failure for {correlation_id}')

    if __name__ == '__main__':
        app.run(host='0.0.0.0', port=5000, debug=True)

C#

```csharp using Microsoft.AspNetCore.Mvc; using Microsoft.Extensions.Logging; using System.Text.Json;

[ApiController] [Route("webhooks")] public class WebhooksController : ControllerBase { private readonly ILogger _logger;

public WebhooksController(ILogger<WebhooksController> logger)
{
    _logger = logger;
}

[HttpPost("processing")]
public IActionResult ProcessingWebhook([FromBody] WebhookPayload payload)
{
    _logger.LogInformation("🚀 Workflow started: {CorrelationId}", payload.CorrelationId);

    foreach (var workflow in payload.Workflows)
    {
        _logger.LogInformation("  {Code}: {Status}", workflow.Code, workflow.Status);
    }

    return Ok(new { received = true });
}

[HttpPost("progress")]
public IActionResult ProgressWebhook([FromBody] WebhookPayload payload)
{
    _logger.LogInformation("⏳ Workflow progress: {CorrelationId}", payload.CorrelationId);

    foreach (var workflow in payload.Workflows)
    {
        foreach (var step in workflow.Steps)
        {
            _logger.LogInformation("  {Name}: {Status}", step.Name, step.Status);
        }
    }

    return Ok(new { received = true });
}

[HttpPost("processed")]
public async Task<IActionResult> ProcessedWebhook([FromBody] WebhookPayload payload)
{
    _logger.LogInformation("✅ Workflow completed: {CorrelationId}", payload.CorrelationId);

    foreach (var workflow in payload.Workflows)
    {
        foreach (var step in workflow.Steps)
        {
            if (step.Locations?.Any() == true)
            {
                _logger.LogInformation("🔗 Created: {Locations}", string.Join(", ", step.Locations));
            }
        }
    }

    // Process successful completion
    await HandleWorkflowSuccessAsync(payload.CorrelationId, payload.Workflows);

    return Ok(new { received = true });
}

[HttpPost("failed")]
public async Task<IActionResult> FailedWebhook([FromBody] WebhookPayload payload)
{
    _logger.LogError("❌ Workflow failed: {CorrelationId}", payload.CorrelationId);

    foreach (var workflow in payload.Workflows)
    {
        _logger.LogError("  {Code}: {Message}", workflow.Code, workflow.Message);

        foreach (var step in workflow.Steps.Where(s => s.Status == "Failed"))
        {
            _logger.LogError("  Failed step: {Name} - {Message}", step.Name, step.Message);
        }
    }

    // Process failure
    await HandleWorkflowFailureAsync(payload.CorrelationId, payload.Workflows);

    return Ok(new { received = true });
}

private async Task HandleWorkflowSuccessAsync(string correlationId, WorkflowStatus[] workflows)
{
    // Your success handling logic
    // - Update database records
    // - Send notifications to users
    // - Trigger dependent processes
    _logger.LogInformation("Processing success for {CorrelationId}", correlationId);
}

private async Task HandleWorkflowFailureAsync(string correlationId, WorkflowStatus[] workflows)
{
    // Your failure handling logic
    // - Log errors
    // - Retry if appropriate
    // - Notify administrators
    _logger.LogInformation("Processing failure for {CorrelationId}", correlationId);
}

}

public class WebhookPayload { public string CorrelationId { get; set; } public string WebhookType { get; set; } public DateTime Timestamp { get; set; } public WorkflowStatus[] Workflows { get; set; } }

public class WorkflowStatus { public string Code { get; set; } public string Status { get; set; } public string Message { get; set; } public DateTime Timestamp { get; set; } public WorkflowStep[] Steps { get; set; } }

public class WorkflowStep { public string Name { get; set; } public string Status { get; set; } public string Message { get; set; } public DateTime Timestamp { get; set; } public string[] Locations { get; set; } }

```

Webhook Security

Verify Webhook Source

While webhooks don't include authentication headers, you can verify they come from Storyteller by:

1. IP Address Filtering: Allow requests only from Storyteller's IP ranges
2. HTTPS Only: Always use HTTPS endpoints
3. Correlation ID Validation: Verify the correlation ID matches your expected values

// IP address filtering middleware (example IPs)
const STORYTELLER_IPS = ['52.1.2.3', '52.4.5.6']; // Replace with actual IPs

app.use('/webhooks', (req, res, next) => {
  const clientIP = req.headers['x-forwarded-for'] || req.connection.remoteAddress;

  if (!STORYTELLER_IPS.includes(clientIP)) {
    return res.status(403).json({ error: 'Forbidden' });
  }

  next();
});

Correlation ID Tracking

// Track expected correlation IDs
const pendingWorkflows = new Set();

// When starting a workflow
async function executeWorkflow(payload) {
  const result = await fetch('/api/workflows/execute', {
    method: 'POST',
    body: JSON.stringify(payload)
  });

  if (result.ok) {
    const { correlationId } = await result.json();
    pendingWorkflows.add(correlationId);
    return correlationId;
  }
}

// In webhook handlers
app.post('/webhooks/processed', (req, res) => {
  const { correlationId } = req.body;

  if (!pendingWorkflows.has(correlationId)) {
    console.warn(`Unexpected correlation ID: ${correlationId}`);
    return res.status(400).json({ error: 'Unknown correlation ID' });
  }

  // Remove from pending when complete
  pendingWorkflows.delete(correlationId);

  // Process webhook...
  res.status(200).json({ received: true });
});

Error Handling & Retries

Webhook Delivery Behavior

• Retry Policy: Storyteller retries failed webhook deliveries up to 3 times
• Retry Intervals: 5 seconds, 30 seconds, 5 minutes
• Failure Criteria: Non-2xx status codes or connection timeouts
• Final Failure: After 3 failed attempts, webhook delivery stops

Handle Delivery Failures

// Idempotent webhook handling
const processedWebhooks = new Set();

app.post('/webhooks/processed', (req, res) => {
  const { correlationId, timestamp } = req.body;
  const webhookId = `${correlationId}-${timestamp}`;

  // Prevent duplicate processing
  if (processedWebhooks.has(webhookId)) {
    console.log(`Duplicate webhook ignored: ${webhookId}`);
    return res.status(200).json({ received: true, duplicate: true });
  }

  try {
    // Process webhook
    handleWorkflowSuccess(req.body);

    // Mark as processed
    processedWebhooks.add(webhookId);

    res.status(200).json({ received: true });
  } catch (error) {
    console.error('Webhook processing error:', error);
    res.status(500).json({ error: 'Processing failed' });
  }
});

Testing Webhooks

Local Development with ngrok

For local testing, use ngrok to expose your local server:

# Install ngrok
npm install -g ngrok

# Start your local server
node webhook-server.js

# In another terminal, expose port 3000
ngrok http 3000

Use the ngrok HTTPS URL in your webhook configuration:

{
  "workflowProcessedWebhookUrl": "https://abc123.ngrok.io/webhooks/processed"
}

Mock Webhook Testing

// Test webhook handler with mock data
const mockWebhookPayload = {
  correlationId: "test-12345",
  webhookType: "processed",
  timestamp: new Date().toISOString(),
  workflows: [
    {
      code: "add-media",
      status: "Finished",
      message: "Test workflow completed",
      timestamp: new Date().toISOString(),
      steps: [
        {
          name: "Create Media Asset",
          status: "Finished",
          message: "Test step completed",
          timestamp: new Date().toISOString(),
          locations: ["https://test.storyteller.com/asset/123"]
        }
      ]
    }
  ]
};

// Test your webhook handler
app.post('/test-webhook', (req, res) => {
  // Simulate webhook call
  fetch('http://localhost:3000/webhooks/processed', {
    method: 'POST',
    headers: { 'Content-Type': 'application/json' },
    body: JSON.stringify(mockWebhookPayload)
  })
  .then(response => response.json())
  .then(data => {
    res.json({ test: 'completed', result: data });
  })
  .catch(error => {
    res.status(500).json({ test: 'failed', error: error.message });
  });
});

Best Practices

1. Always Use HTTPS - Webhook URLs must be HTTPS in production
2. Respond Quickly - Return 2xx status codes within 30 seconds
3. Handle Duplicates - Implement idempotent processing
4. Log Everything - Comprehensive logging for debugging
5. Validate Payloads - Verify correlation IDs match your expectations
6. Graceful Degradation - Fall back to polling if webhooks fail
7. Monitor Performance - Track webhook delivery success rates
8. Secure Endpoints - Use IP filtering and correlation ID validation

Troubleshooting

Common Issues

Webhook Not Receiving Calls

• Check URL accessibility: Test with curl or browser
• Verify HTTPS: Ensure webhook URLs use HTTPS
• Check firewall settings: Allow inbound HTTPS traffic
• Validate URL format: Ensure proper URL structure

Webhook Timing Out

• Reduce processing time: Move heavy processing to background jobs
• Return early: Send 200 response immediately, process asynchronously
• Increase server capacity: Scale webhook handling infrastructure

Missing Webhooks

• Check server logs: Look for error messages or crashed processes
• Verify uptime: Ensure webhook server was running
• Review retry logs: Check if retries were exhausted

Related Documentation

• Workflow Execution - How to configure webhooks
• Workflow Status - Alternative polling approach
• Troubleshooting - Common webhook issues

Support

For webhook-related issues:

• Email: [email protected]
• Subject: Include "Webhook" in the subject line
• Information to provide:
• Webhook URLs being used
• Correlation IDs experiencing issues
• Server logs showing webhook calls
• Expected vs actual behavior