ETCD on Pigsty v5.0

E-R Model of Etcd Cluster

Mon, 01 Jan 0001 00:00:00 +0000

The ETCD module organizes ETCD in production as clusters—logical entities composed of a group of ETCD instances associated through the Raft consensus protocol.

Each cluster is an autonomous distributed key-value storage unit consisting of at least one ETCD instance, exposing service capabilities through client ports.

There are three core entities in Pigsty’s ETCD module:

Cluster: An autonomous ETCD service unit serving as the top-level namespace for other entities.
Instance: A single ETCD server process running on a node, participating in Raft consensus.
Node: A hardware resource abstraction running Linux + Systemd environment, implicitly declared.

Compared to PostgreSQL clusters, the ETCD cluster model is simpler, without Services or complex Role distinctions. All ETCD instances are functionally equivalent, electing a Leader through the Raft protocol while others become Followers. During scale-out intermediate states, non-voting Learner instance members are also allowed.

Module: ETCD

Mon, 01 Jan 0001 00:00:00 +0000

ETCD is a distributed, reliable key-value store for critical system config data.

Pigsty uses etcd as DCS (Distributed Config Store), critical for PostgreSQL HA and automatic failover.

The ETCD module depends on NODE module and is required by PGSQL module. Install NODE module to manage nodes before installing ETCD.

Deploy ETCD cluster before any PGSQL cluster—patroni and vip-manager for PG HA rely on etcd for HA and L2 VIP binding to primary.

flowchart LR
    subgraph PGSQL [PGSQL]
        patroni[Patroni]
        vip[VIP Manager]
    end

    subgraph ETCD [ETCD]
        etcd[DCS Service]
    end

    subgraph NODE [NODE]
        node[Software Repo]
    end

    PGSQL -->|depends| ETCD -->|depends| NODE

    style PGSQL fill:#3E668F,stroke:#2d4a66,color:#fff
    style ETCD fill:#5B9CD5,stroke:#4178a8,color:#fff
    style NODE fill:#FCDB72,stroke:#d4b85e,color:#333

    style patroni fill:#2d4a66,stroke:#1e3347,color:#fff
    style vip fill:#2d4a66,stroke:#1e3347,color:#fff
    style etcd fill:#4178a8,stroke:#2d5a7a,color:#fff
    style node fill:#d4b85e,stroke:#b89a4a,color:#333

One etcd cluster per Pigsty deployment serves multiple PG clusters.

Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Before deployment, define etcd cluster in config inventory. Typical choices:

One Node: No HA, suitable for dev, test, demo, or standalone deployments using external S3 backup for PITR
Three Nodes: Basic HA, tolerates 1 node failure, suitable for small-medium prod
Five Nodes: Better HA, tolerates 2 node failures, suitable for large prod

An even-member Etcd cluster is technically valid, but it does not tolerate more failures than an odd-member cluster with one fewer member, while increasing deployment and quorum cost. Production therefore usually uses one, three, or five members; clusters larger than five are uncommon.

Administration

Mon, 01 Jan 0001 00:00:00 +0000

Common etcd admin SOPs:

Create Cluster: Initialize an etcd cluster
Destroy Cluster: Destroy an etcd cluster
CLI Environment: Configure etcd client to access server cluster
RBAC Authentication: Use etcd RBAC auth
Reload Config: Update etcd server member list for clients
Add Member: Add new member to existing etcd cluster
Remove Member: Remove member from etcd cluster
Utility Scripts: Simplify ops with bin/etcd-add and bin/etcd-rm

For more, refer to FAQ: ETCD.

Create Cluster

Define etcd cluster in config inventory:

Playbook

Mon, 01 Jan 0001 00:00:00 +0000

The ETCD module provides two core playbooks: etcd.yml for installing and configuring etcd clusters, and etcd-rm.yml for removing etcd clusters or members.

Architecture Change: Pigsty v3.6+

Since Pigsty v3.6, the etcd.yml playbook focuses on cluster installation and member addition. All removal operations have been moved to the dedicated etcd-rm.yml playbook using the etcd_remove role.

`etcd.yml`

Playbook source: etcd.yml

This playbook installs and configures an etcd cluster on the hardcoded etcd group, then launches the etcd service.

Monitoring

Mon, 01 Jan 0001 00:00:00 +0000

Dashboards

ETCD module provides one monitoring dashboard: Etcd Overview.

ETCD Overview Dashboard

ETCD Overview: Overview of ETCD cluster

Dashboard provides key ETCD status info. Notable: ETCD Aliveness—shows overall etcd cluster service status.

Red bands = instance downtime; blue-gray below = cluster unavailable.

Alert Rules

Pigsty provides five preset alert rules for etcd, defined in files/victoria/rules/etcd.yml:

Metrics

Mon, 01 Jan 0001 00:00:00 +0000

This snapshot records 177 monitoring metric families for the ETCD module. The metrics present at runtime vary with package version, enabled collectors, and target state.

Metric Name	Type	Labels	Description
etcd:ins:backend_commit_rt_p99_5m	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd:ins:disk_fsync_rt_p99_5m	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd:ins:network_peer_rt_p99_1m	Unknown	`cls`, `To`, `ins`, `instance`, `job`, `ip`	N/A
etcd_cluster_version	gauge	`cls`, `cluster_version`, `ins`, `instance`, `job`, `ip`	Running version. 1 = ‘cluster_version’ label with current version
etcd_debugging_auth_revision	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Current auth store revision.
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_disk_backend_commit_rebalance_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_disk_backend_commit_spill_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_debugging_disk_backend_commit_spill_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_disk_backend_commit_spill_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_disk_backend_commit_write_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_debugging_disk_backend_commit_write_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_disk_backend_commit_write_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_lease_granted_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Total granted leases.
etcd_debugging_lease_renewed_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Renewed leases seen by leader.
etcd_debugging_lease_revoked_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Revoked leases.
etcd_debugging_lease_ttl_total_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_debugging_lease_ttl_total_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_lease_ttl_total_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_mvcc_compact_revision	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Last compaction revision in store.
etcd_debugging_mvcc_current_revision	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Current store revision.
etcd_debugging_mvcc_db_compaction_keys_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	DB keys compacted.
etcd_debugging_mvcc_db_compaction_last	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Last db compaction unix time. Resets to 0 on start.
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_mvcc_db_compaction_pause_duration_milliseconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_mvcc_db_compaction_total_duration_milliseconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_mvcc_events_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Events sent by this member.
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_mvcc_index_compaction_pause_duration_milliseconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_mvcc_keys_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Total keys.
etcd_debugging_mvcc_pending_events_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Pending events to send.
etcd_debugging_mvcc_range_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Ranges seen by this member.
etcd_debugging_mvcc_slow_watcher_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Unsynced slow watchers.
etcd_debugging_mvcc_total_put_size_in_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Total put kv size seen by this member.
etcd_debugging_mvcc_watch_stream_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Watch streams.
etcd_debugging_mvcc_watcher_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Watchers.
etcd_debugging_server_lease_expired_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Expired leases.
etcd_debugging_snap_save_marshalling_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_debugging_snap_save_marshalling_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_snap_save_marshalling_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_snap_save_total_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_debugging_snap_save_total_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_snap_save_total_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_debugging_store_expires_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Expired keys.
etcd_debugging_store_reads_total	counter	`cls`, `action`, `ins`, `instance`, `job`, `ip`	Reads (get/getRecursive) to this member.
etcd_debugging_store_watch_requests_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Incoming watch requests (new/reestablished).
etcd_debugging_store_watchers	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Active watchers.
etcd_debugging_store_writes_total	counter	`cls`, `action`, `ins`, `instance`, `job`, `ip`	Writes (set/compareAndDelete) to this member.
etcd_disk_backend_commit_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_disk_backend_commit_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_disk_backend_commit_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_disk_backend_defrag_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_disk_backend_defrag_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_disk_backend_defrag_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_disk_backend_snapshot_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_disk_backend_snapshot_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_disk_backend_snapshot_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_disk_defrag_inflight	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Defrag active. 1 = active, 0 = not.
etcd_disk_wal_fsync_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_disk_wal_fsync_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_disk_wal_fsync_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_disk_wal_write_bytes_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	WAL bytes written.
etcd_grpc_proxy_cache_hits_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Cache hits.
etcd_grpc_proxy_cache_keys_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Keys/ranges cached.
etcd_grpc_proxy_cache_misses_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Cache misses.
etcd_grpc_proxy_events_coalescing_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Events coalescing.
etcd_grpc_proxy_watchers_coalescing_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Current watchers coalescing.
etcd_mvcc_db_open_read_transactions	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Open read transactions.
etcd_mvcc_db_total_size_in_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	DB physical bytes allocated.
etcd_mvcc_db_total_size_in_use_in_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	DB logical bytes in use.
etcd_mvcc_delete_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Deletes seen by this member.
etcd_mvcc_hash_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_mvcc_hash_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_mvcc_hash_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_mvcc_hash_rev_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_mvcc_hash_rev_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_mvcc_hash_rev_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_mvcc_put_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Puts seen by this member.
etcd_mvcc_range_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Ranges seen by this member.
etcd_mvcc_txn_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Txns seen by this member.
etcd_network_active_peers	gauge	`cls`, `ins`, `Local`, `instance`, `job`, `ip`, `Remote`	Active peer connections.
etcd_network_client_grpc_received_bytes_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	gRPC client bytes received.
etcd_network_client_grpc_sent_bytes_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	gRPC client bytes sent.
etcd_network_peer_received_bytes_total	counter	`cls`, `ins`, `instance`, `job`, `ip`, `From`	Peer bytes received.
etcd_network_peer_round_trip_time_seconds_bucket	Unknown	`cls`, `To`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_network_peer_round_trip_time_seconds_count	Unknown	`cls`, `To`, `ins`, `instance`, `job`, `ip`	N/A
etcd_network_peer_round_trip_time_seconds_sum	Unknown	`cls`, `To`, `ins`, `instance`, `job`, `ip`	N/A
etcd_network_peer_sent_bytes_total	counter	`cls`, `To`, `ins`, `instance`, `job`, `ip`	Peer bytes sent.
etcd_server_apply_duration_seconds_bucket	Unknown	`cls`, `version`, `ins`, `instance`, `job`, `le`, `success`, `ip`, `op`	N/A
etcd_server_apply_duration_seconds_count	Unknown	`cls`, `version`, `ins`, `instance`, `job`, `success`, `ip`, `op`	N/A
etcd_server_apply_duration_seconds_sum	Unknown	`cls`, `version`, `ins`, `instance`, `job`, `success`, `ip`, `op`	N/A
etcd_server_client_requests_total	counter	`client_api_version`, `cls`, `ins`, `instance`, `type`, `job`, `ip`	Client requests per version.
etcd_server_go_version	gauge	`cls`, `ins`, `instance`, `job`, `server_go_version`, `ip`	Go version running. 1 = ‘server_go_version’ label with current version.
etcd_server_has_leader	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Leader exists. 1 = exists, 0 = not.
etcd_server_health_failures	counter	`cls`, `ins`, `instance`, `job`, `ip`	Failed health checks.
etcd_server_health_success	counter	`cls`, `ins`, `instance`, `job`, `ip`	Successful health checks.
etcd_server_heartbeat_send_failures_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Leader heartbeat send failures (likely overloaded from slow disk).
etcd_server_id	gauge	`cls`, `ins`, `instance`, `job`, `server_id`, `ip`	Server/member ID (hex). 1 = ‘server_id’ label with current ID.
etcd_server_is_leader	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Member is leader. 1 if is, 0 otherwise.
etcd_server_is_learner	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Member is learner. 1 if is, 0 otherwise.
etcd_server_leader_changes_seen_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Leader changes seen.
etcd_server_learner_promote_successes	counter	`cls`, `ins`, `instance`, `job`, `ip`	Successful learner promotions while this member is leader.
etcd_server_proposals_applied_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Consensus proposals applied.
etcd_server_proposals_committed_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Consensus proposals committed.
etcd_server_proposals_failed_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Failed proposals seen.
etcd_server_proposals_pending	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Pending proposals to commit.
etcd_server_quota_backend_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Backend storage quota bytes.
etcd_server_read_indexes_failed_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Failed read indexes seen.
etcd_server_slow_apply_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Slow apply requests (likely overloaded from slow disk).
etcd_server_slow_read_indexes_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Pending read indexes not in sync with leader or timed out read index requests.
etcd_server_snapshot_apply_in_progress_total	gauge	`cls`, `ins`, `instance`, `job`, `ip`	1 if server applying incoming snapshot. 0 if none.
etcd_server_version	gauge	`cls`, `server_version`, `ins`, `instance`, `job`, `ip`	Version running. 1 = ‘server_version’ label with current version.
etcd_snap_db_fsync_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_snap_db_fsync_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_snap_db_fsync_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_snap_db_save_total_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_snap_db_save_total_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_snap_db_save_total_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_snap_fsync_duration_seconds_bucket	Unknown	`cls`, `ins`, `instance`, `job`, `le`, `ip`	N/A
etcd_snap_fsync_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_snap_fsync_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
etcd_up	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
go_gc_duration_seconds	summary	`cls`, `ins`, `instance`, `job`, `quantile`, `ip`	GC pause duration summary.
go_gc_duration_seconds_count	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
go_gc_duration_seconds_sum	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
go_goroutines	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Goroutines.
go_info	gauge	`cls`, `version`, `ins`, `instance`, `job`, `ip`	Go environment info.
go_memstats_alloc_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Bytes allocated and in use.
go_memstats_alloc_bytes_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Bytes allocated, even if freed.
go_memstats_buck_hash_sys_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Bytes used by profiling bucket hash table.
go_memstats_frees_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Frees.
go_memstats_gc_cpu_fraction	gauge	`cls`, `ins`, `instance`, `job`, `ip`	GC CPU fraction since program started.
go_memstats_gc_sys_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Bytes used for GC system metadata.
go_memstats_heap_alloc_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Heap bytes allocated and in use.
go_memstats_heap_idle_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Heap bytes waiting to be used.
go_memstats_heap_inuse_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Heap bytes in use.
go_memstats_heap_objects	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Allocated objects.
go_memstats_heap_released_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Heap bytes released to OS.
go_memstats_heap_sys_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Heap bytes obtained from system.
go_memstats_last_gc_time_seconds	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Seconds since 1970 of last GC.
go_memstats_lookups_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Pointer lookups.
go_memstats_mallocs_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	Mallocs.
go_memstats_mcache_inuse_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Bytes in use by mcache structures.
go_memstats_mcache_sys_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Bytes used for mcache structures from system.
go_memstats_mspan_inuse_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Bytes in use by mspan structures.
go_memstats_mspan_sys_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Bytes used for mspan structures from system.
go_memstats_next_gc_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Heap bytes when next GC will take place.
go_memstats_other_sys_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Bytes used for other system allocations.
go_memstats_stack_inuse_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Bytes in use by stack allocator.
go_memstats_stack_sys_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Bytes obtained from system for stack allocator.
go_memstats_sys_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Bytes obtained from system.
go_threads	gauge	`cls`, `ins`, `instance`, `job`, `ip`	OS threads created.
grpc_server_handled_total	counter	`cls`, `ins`, `instance`, `job`, `grpc_code`, `grpc_method`, `grpc_type`, `ip`, `grpc_service`	RPCs completed on server.
grpc_server_msg_received_total	counter	`cls`, `ins`, `instance`, `job`, `grpc_type`, `grpc_method`, `ip`, `grpc_service`	RPC stream messages received on server.
grpc_server_msg_sent_total	counter	`cls`, `ins`, `instance`, `job`, `grpc_type`, `grpc_method`, `ip`, `grpc_service`	gRPC stream messages sent on server.
grpc_server_started_total	counter	`cls`, `ins`, `instance`, `job`, `grpc_type`, `grpc_method`, `ip`, `grpc_service`	RPCs started on server.
os_fd_limit	gauge	`cls`, `ins`, `instance`, `job`, `ip`	FD limit.
os_fd_used	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Used FDs.
process_cpu_seconds_total	counter	`cls`, `ins`, `instance`, `job`, `ip`	User + system CPU seconds.
process_max_fds	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Max FDs.
process_open_fds	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Open FDs.
process_resident_memory_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Resident memory bytes.
process_start_time_seconds	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Start time (unix epoch seconds).
process_virtual_memory_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Virtual memory bytes.
process_virtual_memory_max_bytes	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Max virtual memory bytes.
promhttp_metric_handler_requests_in_flight	gauge	`cls`, `ins`, `instance`, `job`, `ip`	Current scrapes.
promhttp_metric_handler_requests_total	counter	`cls`, `ins`, `instance`, `job`, `ip`, `code`	Scrapes by HTTP status code.
scrape_duration_seconds	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
scrape_samples_post_metric_relabeling	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
scrape_samples_scraped	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
scrape_series_added	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A
up	Unknown	`cls`, `ins`, `instance`, `job`, `ip`	N/A