External API Monitoring#

This section explains how to monitor external API endpoints by counting the number of requests, measuring request duration, and tracking various other metrics. SystemGuard uses Prometheus to display metrics in its UI. There are two types of metrics displayed:

  • Historical Metrics: Show Counter and Gauge metrics. These include real-time and historical data.

  • Summary Metrics: Show Summary metrics with real-time updates.

Counting Requests#

To count the number of requests made to specific API endpoints, you can use the Counter class from the prometheus_client library. The following example demonstrates how to count requests to the homepage:

from prometheus_client import Counter

homepage_request_count = Counter('homepage_request_count', 'Count of requests to homepage')

@api.route('/')
def homepage():
    homepage_request_count.inc()
    return 'Hello, World!'

Measuring Request Duration#

To measure the time duration of requests to specific API endpoints, you can use the Summary class from the prometheus_client library. The following example demonstrates how to measure the request duration for the homepage:

from prometheus_client import Summary

homepage_request_duration = Summary('homepage_request_duration_seconds', 'Time spent processing request to homepage')

@api.route('/')
@homepage_request_duration.time()
def homepage():
    return 'Hello, World!'

Combining Request Counting and Duration Measurement#

You can combine both request counting and duration measurement for a comprehensive monitoring solution. The following example demonstrates how to count requests and measure their duration for the homepage:

from prometheus_client import Counter, Summary

homepage_request_count = Counter('homepage_request_count', 'Count of requests to homepage')
homepage_request_duration = Summary('homepage_request_duration_seconds', 'Time spent processing request to homepage')

@api.route('/')
@homepage_request_duration.time()
def homepage():
    homepage_request_count.inc()
    return 'Hello, World!'

Additional Metrics#

In addition to counting requests and measuring request duration, you can monitor other important metrics such as response size, error count, active requests, request latency, and random gauge values.

Size of Responses in Bytes#

To track the size of responses in bytes, you can use the Summary class. The following example demonstrates how to measure the size of responses:

from prometheus_client import Summary

response_size = Summary('response_size_bytes', 'Size of responses in bytes')

@api.route('/')
def homepage():
    response = 'Hello, World!'
    response_size.observe(len(response))
    return response

Total Number of Errors#

To count the total number of errors, you can use the Counter class. The following example demonstrates how to count errors:

from prometheus_client import Counter

error_count = Counter('error_count', 'Total number of errors')

@api.route('/error')
def error_endpoint():
    try:
        # Simulate an error
        raise ValueError('An error occurred')
    except ValueError:
        error_count.inc()
        return 'Error', 500

Active Requests Gauge#

To track the number of active requests, you can use the Gauge class. The following example demonstrates how to monitor active requests:

from prometheus_client import Gauge

active_requests = Gauge('active_requests', 'Number of active requests')

@api.before_request
def before_request():
    active_requests.inc()

@api.after_request
def after_request(response):
    active_requests.dec()
    return response

Histogram for Request Latency#

To measure request latency in seconds, you can use the Histogram class. The following example demonstrates how to track request latency:

from prometheus_client import Histogram

request_latency = Histogram('request_latency_seconds', 'Request latency in seconds')

@api.route('/')
@request_latency.time()
def homepage():
    return 'Hello, World!'

Random Gauge Value#

To track a random gauge value, you can use the Gauge class. The following example demonstrates how to set a random gauge value:

from prometheus_client import Gauge
import random

random_gauge = Gauge('random_gauge', 'A random gauge value')

@api.route('/random')
def random_value():
    value = random.random()
    random_gauge.set(value)
    return f'Random value: {value}'

Firewall Configuration#

Make sure the exporter port (e.g., 5001) is accessible by updating your firewall rules. For example, with UFW:

sudo ufw allow 5001/tcp

Configure Prometheus to Scrape the Exporter Target#

After running your Prometheus exporter (e.g., at http://localhost:5001/metrics), you need to configure SystemGuard to add this target so that Prometheus can begin scraping metrics.

Add a Target in SystemGuard#

  1. Navigate to the /system/targets page in the SystemGuard UI.

  2. Click on “Add Target”.

  3. Fill in the following fields:

    • Job Name: Enter a descriptive job name (e.g., my_function_exporter).

    • Target URL: Enter the full URL to your metrics endpoint (e.g., http://localhost:5001).

    • Scrape Interval: e.g., 15s (how frequently Prometheus should scrape this target).

    • Username: admin Optional, if your exporter requires authentication.

    • Password: •••••••• Optional, if your exporter requires authentication.

  4. Click the “Add Target” button to save the target.

  5. Scroll down and click on the “Update Prometheus” button to apply changes and restart Prometheus with the new configuration.

Add a Chart Configuration in SystemGuard#

To visualize your metrics in the SystemGuard UI:

  1. Navigate to /chart_configurations.

  2. Click to Add a new chart.

  3. Fill in the following fields:

    • Metric Name: my_function_total_calls, my_function_temperature_celsius, etc.

    • Title: Descriptive name for your chart

    • X-Axis Label: e.g., Time

    • Y-Axis Label: e.g., Temperature (°C), Call Count

Viewing Metrics#

  • Historical Metrics: Available at /system/historical_system_metrics

  • Summary Metrics: Available at /system/summary_metrics

You can now see both real-time and historical metrics from your system directly in the SystemGuard dashboard.