API Feed Monitor with Discord Webhook Integration

A robust, asynchronous Python service that monitors live activity feeds and sends structured notifications to Discord webhooks. It continuously polls an external API, processes various feed events, and forwards them to your configured Discord channels.

This is an improved version of the original implementation by PomfQ, with enhanced stability, better error handling, cleaner code structure, and more resilient state management.

---

Features

Modern Asynchronous Architecture

• Built entirely with asyncio and aiohttp for efficient non-blocking operations
• Context managers for proper resource cleanup
• Single session reuse for better performance
• Type hints throughout for better code safety

Supported Feed Types

The monitor recognizes and processes seven distinct event categories:

Feed Type	Description
status_updated	User status posts
wall_post	User-to-user wall messages
game_published	Newly published games
marketplace_buy	Avatar/shop purchases
badge_earned	Badge acquisitions
username_updated	Username changes
POST	General posts

Each event generates a clean, structured Discord embed with relevant metadata, timestamps, user info, and properly escaped text.

Intelligent Webhook Delivery

• Automatic retries with exponential backoff for failed requests
• Smart rate limit handling (respects Discord's 429 responses)
• Failed webhook queue with periodic retry attempts
• Configurable concurrency limits to prevent overwhelming endpoints
• Optional delays between webhooks for rate limiting

Robust State Management

• Persistent tracking of processing position:
  • last_id - the most recently checked feed ID
  • last_valid_id - the last ID that returned actual data
• Atomic file writes using temporary files (prevents corruption)
• Thread-safe state updates with async locks
• Automatic state recovery on startup

Smart Recovery System

• Detects extended sequences of missing or invalid feeds
• Automatically jumps forward from the last known valid position
• Prevents getting stuck on sparse or unstable feeds
• Configurable error thresholds and jump sizes

Clean Data Processing

• Markdown-safe text escaping for Discord
• Smart image URL normalization and cleaning
• Filters out default/placeholder avatars
• Proper timestamp formatting
• Safe JSON parsing with fallbacks

Built-in Health Monitoring

A lightweight HTTP server runs on http://localhost:8080 with two endpoints:

• / - Returns JSON status response
• /health - Returns JSON status response

Perfect for container orchestration, uptime monitoring, or load balancers.

Comprehensive Logging

• Structured logging with timestamps and log levels
• Detailed information about API calls, webhook deliveries, and errors
• Configurable log levels (DEBUG, INFO, WARNING, ERROR, CRITICAL)
• Exception tracebacks for debugging

---

Requirements

Python 3.8 or higher is required.

Install dependencies:

pip install aiohttp aiofiles

Note: beautifulsoup4 is no longer required in the improved version.

---

Configuration

All settings are managed through a config.json file in the same directory as your script.

Configuration Options

Key	Type	Description
SERVER	string	Base API URL (no trailing slash)
STATE_FILE	string	Path to state file for tracking progress
FALLBACK_START_ID	integer	Starting ID if no state file exists
API_POLL_INTERVAL	float	Seconds to wait between feed checks
WEBHOOK_DELAY	float	Seconds to wait before sending each webhook
API_TIMEOUT	float	Timeout in seconds for API/webhook requests
LOG_LEVEL	string	Logging level: DEBUG, INFO, WARNING, ERROR, CRITICAL
CONCURRENT_WEBHOOKS	integer	Maximum simultaneous webhook deliveries
WEBHOOKS	object	Map of feed types to Discord webhook URLs

Example Configuration

{
  "SERVER": "https://www.kogama.com",
  "STATE_FILE": "state.json",
  "FALLBACK_START_ID": 32262350,
  "API_POLL_INTERVAL": 1.0,
  "WEBHOOK_DELAY": 0.6,
  "API_TIMEOUT": 10.0,
  "LOG_LEVEL": "INFO",
  "CONCURRENT_WEBHOOKS": 6,
  "WEBHOOKS": {
    "status_updated": "https://discord.com/api/webhooks/YOUR_WEBHOOK_ID/YOUR_TOKEN",
    "wall_post": "https://discord.com/api/webhooks/YOUR_WEBHOOK_ID/YOUR_TOKEN",
    "game_published": "https://discord.com/api/webhooks/YOUR_WEBHOOK_ID/YOUR_TOKEN",
    "marketplace_buy": "https://discord.com/api/webhooks/YOUR_WEBHOOK_ID/YOUR_TOKEN",
    "badge_earned": "https://discord.com/api/webhooks/YOUR_WEBHOOK_ID/YOUR_TOKEN",
    "username_updated": "https://discord.com/api/webhooks/YOUR_WEBHOOK_ID/YOUR_TOKEN",
    "POST": "https://discord.com/api/webhooks/YOUR_WEBHOOK_ID/YOUR_TOKEN"
  }
}

Note: To disable notifications for a specific feed type, simply remove it from the WEBHOOKS object.

---

Getting Started

Installation

1. Clone or download the repository

   git clone https://github.com/yourusername/api-feed-monitor.git
   cd api-feed-monitor

2. Install dependencies

   pip install aiohttp aiofiles

3. Create your configuration file

   • Copy the example configuration above
   • Replace webhook URLs with your Discord webhook URLs
   • Adjust settings as needed

4. Run the monitor

   python api_monitor.py

The monitor will start processing feeds from the configured starting ID and automatically save its progress to the state file.

---

Finding the Latest Feed ID

To start monitoring from the most recent activity, you'll need to find the current feed ID. Here's how:

Using Browser DevTools

1. Open your profile and prepare to create a new post
2. Open Developer Tools (F12) and go to the Network tab
3. Filter by /feed/ in the search bar
4. Create a post on your profile
5. Look for the new request that appears in the Network tab
6. Click on it and open the Response tab
7. Find the ID in the JSON response:

{
  "data": {
    "id": 32220804,
    "feed_type": "status_updated",
    ...
  }
}

8. Use this ID as your FALLBACK_START_ID in config.json, or manually set it in state.json:

{
  "last_id": 32220804,
  "last_valid_id": 32220804
}

Visual Reference

Here's what this looks like in DevTools:

In this example, the feed ID is 32220804.

---

How It Works

Monitoring Loop

1. Fetch the next feed ID from the API
2. Process the feed if data is returned:
   • Parse feed type and extract relevant information
   • Create a formatted Discord embed
   • Send to the appropriate webhook
3. Update state tracking:
   • Increment to next ID
   • Save progress to state file
4. Recover if feeds are missing:
   • Track consecutive errors
   • Jump forward after threshold is reached
5. Retry any failed webhooks periodically
6. Repeat after configured delay

State Management

The monitor maintains two important values:

• last_id: The current position in the feed sequence
• last_valid_id: The last position that had actual data

If the monitor encounters many empty responses, it will intelligently jump forward from the last valid position to catch up with active feeds.

Recovery Strategy

When the monitor hits a long streak of empty feeds (default: 130 consecutive errors), it will:

1. Calculate how far to jump based on the error count
2. Jump forward from the last valid ID
3. Reset the error counter
4. Continue monitoring from the new position

This prevents the monitor from getting stuck when there are large gaps in the feed sequence.

---

Advanced Configuration

Adjusting Performance

For high-volume feeds:

• Decrease API_POLL_INTERVAL (minimum: 0.5)
• Increase CONCURRENT_WEBHOOKS (up to 10)
• Reduce WEBHOOK_DELAY (minimum: 0.3)

For rate-limited scenarios:

• Increase WEBHOOK_DELAY (1.0 or higher)
• Decrease CONCURRENT_WEBHOOKS (3-4)
• Increase API_POLL_INTERVAL

Logging Levels

• DEBUG: Extremely verbose, shows every operation
• INFO: Normal operation, shows processed feeds
• WARNING: Important events and recoveries
• ERROR: Failed operations that will be retried
• CRITICAL: Fatal errors that stop the monitor

Custom Recovery Behavior

You can modify these constants in the code:

@dataclass
class Config:
    # ... other fields ...
    max_consecutive_errors: int = 130  # Trigger recovery after this many errors
    retry_max_attempts: int = 3        # Retry failed webhooks this many times
    recovery_jump_size: int = 100      # Jump this far if no valid ID exists

---

Monitoring and Maintenance

Health Checks

The built-in health server makes it easy to monitor the service:

curl http://localhost:8080/health

Response:

{
  "status": "ok",
  "timestamp": "2026-01-12T10:30:00.000000"
}

Viewing Logs

The monitor logs all operations to stdout. To save logs to a file:

python api_monitor.py > monitor.log 2>&1

Or use a logging service like systemd, Docker logs, or a log aggregator.

State File

The state.json file tracks progress:

{
  "last_id": 32262450,
  "last_valid_id": 32262448
}

You can manually edit this file to:

• Jump to a specific feed ID
• Reset to the beginning
• Resume from a known good position

---

Troubleshooting

Monitor keeps skipping feeds

• Check your API_POLL_INTERVAL is not too aggressive
• Verify the API is responding correctly
• Check logs for rate limiting messages

Webhooks are failing

• Verify Discord webhook URLs are correct
• Check if Discord is rate limiting you
• Increase WEBHOOK_DELAY to space out requests
• Look for 429 responses in logs

State file is corrupted

Delete state.json and restart. The monitor will begin from FALLBACK_START_ID.

High CPU usage

• Increase API_POLL_INTERVAL to reduce polling frequency
• Check for infinite error loops in logs
• Verify the health server isn't being hammered

Recovery triggers too often

• Increase max_consecutive_errors in the Config class
• Check if the feed API is having issues
• Verify your starting ID is in an active range

---

Running as a Service

Using systemd (Linux)

Create /etc/systemd/system/feed-monitor.service:

[Unit]
Description=API Feed Monitor
After=network.target

[Service]
Type=simple
User=your-user
WorkingDirectory=/path/to/monitor
ExecStart=/usr/bin/python3 api_monitor.py
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl enable feed-monitor
sudo systemctl start feed-monitor
sudo systemctl status feed-monitor

Using Docker

Create a Dockerfile:

FROM python:3.11-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY api_monitor.py .
COPY config.json .

CMD ["python", "api_monitor.py"]

Build and run:

docker build -t feed-monitor .
docker run -d --name feed-monitor \
  -v $(pwd)/state.json:/app/state.json \
  -p 8080:8080 \
  feed-monitor

---

Credits

This project builds on the original work by PomfQ. The improvements include:

• Modern Python patterns (dataclasses, enums, type hints)
• Better error handling and recovery logic
• Atomic state file operations
• Context managers for resource safety
• Enhanced logging and monitoring
• Performance optimizations
• Cleaner, more maintainable code structure

---

License

This project is provided as-is for educational and monitoring purposes. Check the original repository for licensing information.

---

Disclaimer

This tool is provided strictly for legitimate monitoring, moderation, and research purposes. Users are solely responsible for ensuring their use complies with:

• Applicable laws and regulations
• Platform terms of service
• Privacy and data protection rules
• Ethical standards

The authors assume no responsibility for misuse, including but not limited to harassment, stalking, unauthorized tracking, or privacy violations. Always respect others' privacy and use automation responsibly.