15 KiB
Service monitoring
A core requirement of microservice governance is service observability. As a shepherd of microservices, it is not easy to keep track of the health status of various services. Many solutions have emerged in this field in the cloud-native era. This component abstracts telemetry and monitoring, the important pillars of observability, to allow users to quickly integrate with existing infrastructure while avoiding vendor lock-in.
Install
Install components via Composer
composer require hyperf/metric
The hyperf/metric component has Prometheus dependencies installed by default. If you want to use StatsD or InfluxDB, you also need to execute the following commands to install the corresponding dependencies:
# StatsD required dependencies
composer require domnikl/statsd
# InfluxDB required dependencies
composer require influxdb/influxdb-php
Add component configuration
If the file does not exist, execute the following command to add the config/autoload/metric.php
configuration file:
php bin/hyperf.php vendor:publish hyperf/metric
use
Configuration
options
default
: The value corresponding to default
in the configuration file is the driver name used. The specific configuration of the driver is defined under metric
, using the same driver as key
.
'default' => env('METRIC_DRIVER', 'prometheus'),
use_standalone_process
: Whether to usestandalone monitoring process
. It is recommended to enable. Metric collection and reporting will be handled in theWorker process
after shutdown.
'use_standalone_process' => env('TELEMETRY_USE_STANDALONE_PROCESS', true),
enable_default_metric
: Whether to count default metrics. Default metrics include memory usage, system CPU load, and Swoole Server and Swoole Coroutine metrics.
'enable_default_metric' => env('TELEMETRY_ENABLE_DEFAULT_TELEMETRY', true),
default_metric_interval
: The default metric push interval, in seconds, the same below.
'default_metric_interval' => env('DEFAULT_METRIC_INTERVAL', 5),
Configuring Prometheus
When using Prometheus, add the specific configuration of Prometheus to the metric
item in the configuration file.
use Hyperf\Metric\Adapter\Prometheus\Constants;
return [
'default' => env('METRIC_DRIVER', 'prometheus'),
'use_standalone_process' => env('TELEMETRY_USE_STANDALONE_PROCESS', true),
'enable_default_metric' => env('TELEMETRY_ENABLE_DEFAULT_TELEMETRY', true),
'default_metric_interval' => env('DEFAULT_METRIC_INTERVAL', 5),
'metric' => [
'prometheus' => [
'driver' => Hyperf\Metric\Adapter\Prometheus\MetricFactory::class,
'mode' => Constants::SCRAPE_MODE,
'namespace' => env('APP_NAME', 'skeleton'),
'scrape_host' => env('PROMETHEUS_SCRAPE_HOST', '0.0.0.0'),
'scrape_port' => env('PROMETHEUS_SCRAPE_PORT', '9502'),
'scrape_path' => env('PROMETHEUS_SCRAPE_PATH', '/metrics'),
'push_host' => env('PROMETHEUS_PUSH_HOST', '0.0.0.0'),
'push_port' => env('PROMETHEUS_PUSH_PORT', '9091'),
'push_interval' => env('PROMETHEUS_PUSH_INTERVAL', 5),
],
],
];
Prometheus has two working modes, crawl mode and push mode (via Prometheus Pushgateway ), which can be supported by this component.
When using the crawl mode (Prometheus official recommendation), you need to set:
'mode' => Constants::SCRAPE_MODE
And configure the crawling address scrape_host
, the crawling port scrape_port
, and the crawling path scrape_path
. Prometheus can pull all metrics in the form of HTTP access under the corresponding configuration.
Note: In crawl mode, standalone process must be enabled, ie
use_standalone_process = true
.
When using push mode, you need to set:
'mode' => Constants::PUSH_MODE
And configure the push address push_host
, push port push_port
, push interval push_interval
. Push mode is only recommended for offline tasks.
Because of the differences in basic settings, the above modes may not meet the needs. This component also supports custom mode. In the custom mode, the component is only responsible for the collection of indicators, and the specific reporting needs to be handled by the user.
'mode' => Constants::CUSTOM_MODE
For example, you may want to report metrics through custom routes, or store metrics in Redis, and other independent services are responsible for centralized reporting of metrics, etc. The [custom escalation](#custom escalation) section contains corresponding examples.
Configure StatsD
When using StatsD, add the specific configuration of StatsD to the metric
item in the configuration file.
return [
'default' => env('METRIC_DRIVER', 'statd'),
'use_standalone_process' => env('TELEMETRY_USE_STANDALONE_PROCESS', true),
'enable_default_metric' => env('TELEMETRY_ENABLE_DEFAULT_TELEMETRY', true),
'metric' => [
'statsd' => [
'driver' => Hyperf\Metric\Adapter\StatsD\MetricFactory::class,
'namespace' => env('APP_NAME', 'skeleton'),
'udp_host' => env('STATSD_UDP_HOST', '127.0.0.1'),
'udp_port' => env('STATSD_UDP_PORT', '8125'),
'enable_batch' => env('STATSD_ENABLE_BATCH', true),
'push_interval' => env('STATSD_PUSH_INTERVAL', 5),
'sample_rate' => env('STATSD_SAMPLE_RATE', 1.0),
],
],
];
StatsD currently only supports UDP mode, you need to configure UDP address udp_host
, UDP port udp_port
, whether to batch push enable_batch
(reduce the number of requests), batch push interval push_interval
and sample rate sample_rate
.
Configuring InfluxDB
When using InfluxDB, add the specific configuration of InfluxDB to the metric
item in the configuration file.
return [
'default' => env('METRIC_DRIVER', 'influxdb'),
'use_standalone_process' => env('TELEMETRY_USE_STANDALONE_PROCESS', true),
'enable_default_metric' => env('TELEMETRY_ENABLE_DEFAULT_TELEMETRY', true),
'metric' => [
'influxdb' => [
'driver' => Hyperf\Metric\Adapter\InfluxDB\MetricFactory::class,
'namespace' => env('APP_NAME', 'skeleton'),
'host' => env('INFLUXDB_HOST', '127.0.0.1'),
'port' => env('INFLUXDB_PORT', '8086'),
'username' => env('INFLUXDB_USERNAME', ''),
'password' => env('INFLUXDB_PASSWORD', ''),
'dbname' => env('INFLUXDB_DBNAME', true),
'push_interval' => env('INFLUXDB_PUSH_INTERVAL', 5),
],
],
];
InfluxDB uses the default HTTP mode, you need to configure the address host
, UDP port port
, username username
, password password
, dbname
data table and batch push interval push_interval
.
Basic abstraction
The telemetry component abstracts three commonly used data types to ensure decoupling of concrete implementations.
The three types are:
Counter (Counter): An indicator used to describe one-way increments. Such as HTTP request count.
interface CounterInterface
{
public function with(string ...$labelValues): self;
public function add(int $delta);
}
Gauge: An indicator used to describe an increase or decrease over time. Such as the number of available connections in the connection pool.
interface GaugeInterface
{
public function with(string ...$labelValues): self;
public function set(float $value);
public function add(float $delta);
}
- Histogram: used to describe the statistical distribution produced by continuous observation of an event, usually expressed as percentiles or buckets. Such as HTTP request delay.
interface HistogramInterface
{
public function with(string ...$labelValues): self;
public function put(float $sample);
}
Configure middleware
After configuring the driver, you only need to configure the middleware to enable the request Histogram statistics function.
Open the config/autoload/middlewares.php
file, the example is to enable middleware in http
Server.
<?php
declare(strict_types=1);
return [
'http' => [
\Hyperf\Metric\Middleware\MetricMiddleware::class,
],
];
The statistics dimension in this middleware includes
request_status
,request_path
,request_method
. If yourrequest_path
is too large, it is recommended to rewrite this middleware to remove therequest_path
dimension, otherwise the high cardinality will cause memory overflow.
Custom use
Telemetry via HTTP middleware is just the tip of the iceberg of what this component can do. You can inject the Hyperf\Metric\Contract\MetricFactoryInterface
class to telemetry business data yourself. For example: the number of orders created, the number of clicks on ads, etc.
<?php
declare(strict_types=1);
namespace App\Controller;
use App\Model\Order;
use Hyperf\Di\Annotation\Inject;
use Hyperf\Metric\Contract\MetricFactoryInterface;
class IndexController extends AbstractController
{
/**
* @var MetricFactoryInterface
*/
#[Inject]
private $metricFactory;
public function create(Order $order)
{
$counter = $this->metricFactory->makeCounter('order_created', ['order_type']);
$counter->with($order->type)->add(1);
// order logic...
}
}
MetricFactoryInterface
contains the following factory methods to generate the corresponding three basic statistic types.
public function makeCounter($name, $labelNames): CounterInterface;
public function makeGauge($name, $labelNames): GaugeInterface;
public function makeHistogram($name, $labelNames): HistogramInterface;
The above example is the generated metrics within the scope of the statistical request. Sometimes the indicators we need to count are for the complete life cycle, such as counting the length of asynchronous queues or the number of items in stock. In this scenario, you can listen to the MetricFactoryReady
event.
<?php
declare(strict_types=1);
namespace App\Listener;
use Hyperf\Event\Contract\ListenerInterface;
use Hyperf\Metric\Event\MetricFactoryReady;
use Psr\Container\ContainerInterface;
use Redis;
class OnMetricFactoryReady implements ListenerInterface
{
protected ContainerInterface $container;
public function __construct(ContainerInterface $container)
{
$this->container = $container;
}
public function listen(): array
{
return [
MetricFactoryReady::class,
];
}
public function process(object $event)
{
$redis = $this->container->get(Redis::class);
$gauge = $event
->factory
->makeGauge('queue_length', ['driver'])
->with('redis');
while (true) {
$length = $redis->llen('queue');
$gauge->set($length);
sleep(1);
}
}
}
In terms of engineering, it is not suitable to query the queue length directly from Redis. The queue length should be obtained through the
info()
method under theDriverInterface
interface of the queue driver. Just a simple demonstration here. You can find a complete example in thesrc/Listener
folder of the component's source code.
Notes
You can use #[Counter(name="stat_name_here")]
and #[Histogram(name="stat_name_here")]
to count the invocation and running time of the aspect.
For the use of annotations, please refer to the Annotation Chapter.
Custom Histogram Bucket
This section only applies to Prometheus drivers
When you are using Prometheus's Histogram, sometimes there is a need for a custom Bucket. Before starting the service, you can inject the dependency into the Registry and register the Histogram by yourself, and set the required Bucket . When you use it later, MetricFactory
will call you to register the Histogram of the same name. An example is as follows:
<?php
namespace App\Listener;
use Hyperf\Config\Annotation\Value;
use Hyperf\Event\Contract\ListenerInterface;
use Hyperf\Framework\Event\BeforeMainServerStart;
use Prometheus\CollectorRegistry;
class OnMainServerStart implements ListenerInterface
{
protected $registry;
public function __construct(CollectorRegistry $registry)
{
$this->registry = $registry;
}
public function listen(): array
{
return [
BeforeMainServerStart::class,
];
}
public function process(object $event)
{
$this->registry->registerHistogram(
config("metric.metric.prometheus.namespace"),
'test',
'help_message',
['labelName'],
[0.1, 1, 2, 3.5]
);
}
}
After that, when you use $metricFactory->makeHistogram('test')
, the returned Histogram is your pre-registered Histogram.
Custom report
This section only applies to Prometheus drivers
After setting the component's Promethues driver working mode to the custom mode ( Constants::CUSTOM_MODE
), you can freely handle indicator reporting. In this section, we show how to store metrics in Redis, then add a new HTTP route to the Worker that returns Prometheus-rendered metrics.
Storing metrics with Redis
The storage medium for metrics is defined by the Prometheus\Storage\Adapter
interface. Memory storage is used by default. We can change to Redis storage in config/autoload/dependencies.php
.
<?php
return [
Prometheus\Storage\Adapter::class => Hyperf\Metric\Adapter\Prometheus\RedisStorageFactory::class,
];
Add /metrics route to Worker
Add Prometheus routes in config/routes.php.
Note that if you want to get metrics under Workers, you need to handle the state sharing between Workers yourself. One way is to store the state in Redis as described above.
<?php
use Hyperf\HttpServer\Router\Router;
Router::get('/metrics', function(){
$registry = Hyperf\Utils\ApplicationContext::getContainer()->get(Prometheus\CollectorRegistry::class);
$renderer = new Prometheus\RenderTextFormat();
return $renderer->render($registry->getMetricFamilySamples());
});
Create console in Grafana
This section only applies to Prometheus drivers
If you have default metrics enabled, Hyperf/Metric
prepares a Grafana console for you out of the box. Download the console json file, import it into Grafana and use it.
Precautions
- To use this component to collect metrics in a
hyperf/command
custom command, you need to add the command line parameter:--enable-event-dispatcher
when starting the command.