Google Cloud Platform (GCP) PubSub metrics
Collect pubsub metrics from Google Cloud Platform (GCP) with Elastic Agent
Version |
2.33.1 (View all) |
Compatible Kibana version(s) |
8.12.0 or higher |
Supported Serverless project types |
Security Observability |
Subscription level |
Basic |
Metrics
The pubsub
dataset fetches metrics from PubSub in Google Cloud Platform. It contains all metrics exported from the GCP PubSub Monitoring API.
You can specify a single region to fetch metrics like us-central1
. Be aware that GCP PubSub does not use zones so us-central1-a
will return nothing. If no region is specified, it will return metrics from all buckets.
Sample Event
An example event for pubsub
looks as following:
{
"@timestamp": "2017-10-12T08:05:34.853Z",
"cloud": {
"account": {
"id": "elastic-obs-integrations-dev",
"name": "elastic-obs-integrations-dev"
},
"instance": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"machine": {
"type": "e2-medium"
},
"provider": "gcp",
"availability_zone": "us-central1-c",
"region": "us-central1"
},
"event": {
"dataset": "gcp.pubsub",
"duration": 115000,
"module": "gcp"
},
"gcp": {
"pubsub": {
"subscription": {
"backlog": {
"bytes": 0
}
}
},
"labels": {
"user": {
"goog-gke-node": ""
}
}
},
"host": {
"id": "4751091017865185079",
"name": "gke-cluster-1-default-pool-6617a8aa-5clh"
},
"metricset": {
"name": "pubsub",
"period": 10000
},
"service": {
"type": "gcp"
}
}
Exported fields
Exported fields
Field | Description | Type | Metric Type |
---|---|---|---|
@timestamp | Event timestamp. | date | |
agent.id | Unique identifier of this agent (if one exists). Example: For Beats this would be beat.id. | keyword | |
cloud | Fields related to the cloud or infrastructure the events are coming from. | group | |
cloud.account.id | The cloud account or organization id used to identify different entities in a multi-tenant environment. Examples: AWS account id, Google Cloud ORG Id, or other unique identifier. | keyword | |
cloud.account.name | The cloud account name or alias used to identify different entities in a multi-tenant environment. Examples: AWS account name, Google Cloud ORG display name. | keyword | |
cloud.availability_zone | Availability zone in which this host, resource, or service is located. | keyword | |
cloud.image.id | Image ID for the cloud instance. | keyword | |
cloud.instance.id | Instance ID of the host machine. | keyword | |
cloud.instance.name | Instance name of the host machine. | keyword | |
cloud.machine.type | Machine type of the host machine. | keyword | |
cloud.project.id | Name of the project in Google Cloud. | keyword | |
cloud.provider | Name of the cloud provider. Example values are aws, azure, gcp, or digitalocean. | keyword | |
cloud.region | Region in which this host, resource, or service is located. | keyword | |
container.id | Unique container id. | keyword | |
container.image.name | Name of the image the container was built on. | keyword | |
container.labels | Image labels. | object | |
container.name | Container name. | keyword | |
data_stream.dataset | Data stream dataset. | constant_keyword | |
data_stream.namespace | Data stream namespace. | constant_keyword | |
data_stream.type | Data stream type. | constant_keyword | |
ecs.version | ECS version this event conforms to. ecs.version is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword | |
error | These fields can represent errors of any kind. Use them for errors that happen while fetching events or in cases where the event itself contains an error. | group | |
error.message | Error message. | match_only_text | |
event.dataset | Event dataset | constant_keyword | |
event.module | Event module | constant_keyword | |
gcp.labels.metadata.* | object | ||
gcp.labels.metrics.* | object | ||
gcp.labels.resource.* | object | ||
gcp.labels.system.* | object | ||
gcp.labels.user.* | object | ||
gcp.labels_fingerprint | Hashed value of the labels field. | keyword | |
gcp.metrics.*.*.*.* | Metrics that returned from Google Cloud API query. | object | |
gcp.pubsub.snapshot.backlog.bytes | Total byte size of the messages retained in a snapshot. | long | gauge |
gcp.pubsub.snapshot.backlog_bytes_by_region.bytes | Total byte size of the messages retained in a snapshot, broken down by Cloud region. | long | gauge |
gcp.pubsub.snapshot.config_updates.count | Delta of the cumulative count of configuration changes, grouped by operation type and result. | long | gauge |
gcp.pubsub.snapshot.num_messages.value | Number of messages retained in a snapshot. | long | gauge |
gcp.pubsub.snapshot.num_messages_by_region.value | Number of messages retained in a snapshot, broken down by Cloud region. | long | gauge |
gcp.pubsub.snapshot.oldest_message_age.sec | Age (in seconds) of the oldest message retained in a snapshot. | long | gauge |
gcp.pubsub.snapshot.oldest_message_age_by_region.sec | Age (in seconds) of the oldest message retained in a snapshot, broken down by Cloud region. | long | gauge |
gcp.pubsub.subscription.ack_latencies.value | Distribution of ack latencies in milliseconds. The ack latency is the time between when Cloud Pub/Sub sends a message to a subscriber client and when Cloud Pub/Sub receives an Acknowledge request for that message. | object | |
gcp.pubsub.subscription.ack_message.count | Delta of the cumulative count of messages acknowledged by Acknowledge requests, grouped by delivery type. | long | gauge |
gcp.pubsub.subscription.backlog.bytes | Total byte size of the unacknowledged messages (a.k.a. backlog messages) in a subscription. | long | gauge |
gcp.pubsub.subscription.byte_cost.bytes | Delta of the cumulative cost of operations, measured in bytes. This is used to measure quota utilization. | long | gauge |
gcp.pubsub.subscription.config_updates.count | Delta of the cumulative count of configuration changes for each subscription, grouped by operation type and result. | long | gauge |
gcp.pubsub.subscription.dead_letter_message.count | Delta of the cumulative count of messages published to dead letter topic, grouped by result. | long | gauge |
gcp.pubsub.subscription.mod_ack_deadline_message.count | Delta of the cumulative count of messages whose deadline was updated by ModifyAckDeadline requests, grouped by delivery type. | long | gauge |
gcp.pubsub.subscription.mod_ack_deadline_message_operation.count | Delta of the cumulative count of ModifyAckDeadline message operations, grouped by result. | long | gauge |
gcp.pubsub.subscription.mod_ack_deadline_request.count | Delta of the cumulative count of ModifyAckDeadline requests, grouped by result. | long | gauge |
gcp.pubsub.subscription.num_outstanding_messages.value | Number of messages delivered to a subscription's push endpoint, but not yet acknowledged. | long | gauge |
gcp.pubsub.subscription.num_undelivered_messages.value | Number of unacknowledged messages (a.k.a. backlog messages) in a subscription. | long | gauge |
gcp.pubsub.subscription.oldest_retained_acked_message_age.sec | Age (in seconds) of the oldest acknowledged message retained in a subscription. | long | gauge |
gcp.pubsub.subscription.oldest_retained_acked_message_age_by_region.value | Age (in seconds) of the oldest acknowledged message retained in a subscription, broken down by Cloud region. | long | gauge |
gcp.pubsub.subscription.oldest_unacked_message_age.sec | Age (in seconds) of the oldest unacknowledged message (a.k.a. backlog message) in a subscription. | long | gauge |
gcp.pubsub.subscription.oldest_unacked_message_age_by_region.value | Age (in seconds) of the oldest unacknowledged message in a subscription, broken down by Cloud region. | long | gauge |
gcp.pubsub.subscription.pull_ack_message_operation.count | Delta of the cumulative count of acknowledge message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count. | long | gauge |
gcp.pubsub.subscription.pull_ack_request.count | Delta of the cumulative count of acknowledge requests, grouped by result. | long | gauge |
gcp.pubsub.subscription.pull_message_operation.count | Delta of the cumulative count of pull message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count. | long | gauge |
gcp.pubsub.subscription.pull_request.count | Delta of the cumulative count of pull requests, grouped by result. | long | gauge |
gcp.pubsub.subscription.push_request.count | Delta of the cumulative count of push attempts, grouped by result. Unlike pulls, the push server implementation does not batch user messages. So each request only contains one user message. The push server retries on errors, so a given user message can appear multiple times. | long | gauge |
gcp.pubsub.subscription.push_request_latencies.value | Distribution of push request latencies (in microseconds), grouped by result. | object | |
gcp.pubsub.subscription.retained_acked.bytes | Total byte size of the acknowledged messages retained in a subscription. | long | gauge |
gcp.pubsub.subscription.retained_acked_bytes_by_region.bytes | Total byte size of the acknowledged messages retained in a subscription, broken down by Cloud region. | long | gauge |
gcp.pubsub.subscription.seek_request.count | Delta of the cumulative count of seek attempts, grouped by result. | long | gauge |
gcp.pubsub.subscription.sent_message.count | Delta of the cumulative count of messages sent by Cloud Pub/Sub to subscriber clients, grouped by delivery type. | long | gauge |
gcp.pubsub.subscription.streaming_pull_ack_message_operation.count | Delta of the cumulative count of StreamingPull acknowledge message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count. | long | gauge |
gcp.pubsub.subscription.streaming_pull_ack_request.count | Delta of the cumulative count of streaming pull requests with non-empty acknowledge ids, grouped by result. | long | gauge |
gcp.pubsub.subscription.streaming_pull_message_operation.count | Delta of the cumulative count of streaming pull message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric <code>subscription/mod_ack_deadline_message_operation_count | long | gauge |
gcp.pubsub.subscription.streaming_pull_mod_ack_deadline_message_operation.count | Delta of the cumulative count of StreamingPull ModifyAckDeadline operations, grouped by result. | long | gauge |
gcp.pubsub.subscription.streaming_pull_mod_ack_deadline_request.count | Delta of the cumulative count of streaming pull requests with non-empty ModifyAckDeadline fields, grouped by result. | long | gauge |
gcp.pubsub.subscription.streaming_pull_response.count | Delta of the cumulative count of streaming pull responses, grouped by result. | long | gauge |
gcp.pubsub.subscription.unacked_bytes_by_region.bytes | Total byte size of the unacknowledged messages in a subscription, broken down by Cloud region. | long | gauge |
gcp.pubsub.topic.byte_cost.bytes | Delta of the cost of operations, measured in bytes. This is used to measure utilization for quotas. | long | gauge |
gcp.pubsub.topic.config_updates.count | Delta of the cumulative count of configuration changes, grouped by operation type and result. | long | gauge |
gcp.pubsub.topic.message_sizes.bytes | Distribution of publish message sizes (in bytes) | object | |
gcp.pubsub.topic.oldest_retained_acked_message_age_by_region.value | Age (in seconds) of the oldest acknowledged message retained in a topic, broken down by Cloud region. | long | gauge |
gcp.pubsub.topic.oldest_unacked_message_age_by_region.value | Age (in seconds) of the oldest unacknowledged message in a topic, broken down by Cloud region. | long | gauge |
gcp.pubsub.topic.retained_acked_bytes_by_region.bytes | Total byte size of the acknowledged messages retained in a topic, broken down by Cloud region. | long | gauge |
gcp.pubsub.topic.send_message_operation.count | Delta of the cumulative count of publish message operations, grouped by result. For a definition of message operations, see Cloud Pub/Sub metric subscription/mod_ack_deadline_message_operation_count. | long | gauge |
gcp.pubsub.topic.send_request.count | Delta of the cumulative count of publish requests, grouped by result. | long | gauge |
gcp.pubsub.topic.streaming_pull_response.count | Delta of the cumulative count of streaming pull responses, grouped by result. | long | gauge |
gcp.pubsub.topic.unacked_bytes_by_region.bytes | Total byte size of the unacknowledged messages in a topic, broken down by Cloud region. | long | gauge |
host.architecture | Operating system architecture. | keyword | |
host.containerized | If the host is a container. | boolean | |
host.domain | Name of the domain of which the host is a member. For example, on Windows this could be the host's Active Directory domain or NetBIOS domain name. For Linux this could be the domain of the host's LDAP provider. | keyword | |
host.hostname | Hostname of the host. It normally contains what the hostname command returns on the host machine. | keyword | |
host.id | Unique host id. As hostname is not always unique, use values that are meaningful in your environment. Example: The current usage of beat.name . | keyword | |
host.ip | Host ip addresses. | ip | |
host.mac | Host mac addresses. | keyword | |
host.name | Name of the host. It can contain what hostname returns on Unix systems, the fully qualified domain name, or a name specified by the user. The sender decides which value to use. | keyword | |
host.os.build | OS build information. | keyword | |
host.os.codename | OS codename, if any. | keyword | |
host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword | |
host.os.kernel | Operating system kernel version as a raw string. | keyword | |
host.os.name | Operating system name, without the version. | keyword | |
host.os.name.text | Multi-field of host.os.name . | text | |
host.os.platform | Operating system platform (such centos, ubuntu, windows). | keyword | |
host.os.version | Operating system version as a raw string. | keyword | |
host.type | Type of host. For Cloud providers this can be the machine type like t2.medium . If vm, this could be the container, for example, or other information meaningful in your environment. | keyword | |
service.type | The type of the service data is collected from. The type can be used to group and correlate logs and metrics from one service type. Example: If logs or metrics are collected from Elasticsearch, service.type would be elasticsearch . | keyword |
Changelog
Version | Details | Kibana version(s) |
---|---|---|
2.33.1 | Enhancement View pull request | 8.12.0 or higher |
2.33.0 | Enhancement View pull request | 8.12.0 or higher |
2.32.1 | Enhancement View pull request | 8.12.0 or higher |
2.32.0 | Enhancement View pull request | 8.12.0 or higher |
2.31.2 | Bug fix View pull request | 8.7.1 or higher |
2.31.1 | Enhancement View pull request | 8.7.1 or higher |
2.31.0 | Enhancement View pull request | 8.7.1 or higher |
2.30.1 | Bug fix View pull request | 8.7.1 or higher |
2.30.0 | Enhancement View pull request | 8.7.1 or higher |
2.29.1 | Bug fix View pull request | 8.7.1 or higher |
2.29.0 | Bug fix View pull request | 8.7.1 or higher |
2.28.5 | Enhancement View pull request | 8.7.1 or higher |
2.28.4 | Enhancement View pull request | 8.7.1 or higher |
2.28.3 | Enhancement View pull request | 8.7.1 or higher |
2.28.2 | Enhancement View pull request | 8.7.1 or higher |
2.28.1 | Enhancement View pull request | 8.7.1 or higher |
2.28.0 | Enhancement View pull request | 8.7.1 or higher |
2.27.0 | Enhancement View pull request | 8.7.1 or higher |
2.26.0 | Bug fix View pull request | 8.7.1 or higher |
2.25.1 | Bug fix View pull request | 8.7.1 or higher |
2.25.0 | Enhancement View pull request | 8.7.1 or higher |
2.24.0 | Enhancement View pull request | 8.7.1 or higher |
2.23.0 | Enhancement View pull request | 8.7.1 or higher |
2.22.1 | Enhancement View pull request | 8.6.0 or higher |
2.22.0 | Enhancement View pull request | 8.6.0 or higher |
2.21.0 | Enhancement View pull request | 8.6.0 or higher |
2.20.1 | Bug fix View pull request | 8.6.0 or higher |
2.20.0 | Enhancement View pull request | 8.6.0 or higher |
2.19.1 | Enhancement View pull request | 8.6.0 or higher |
2.19.0 | Enhancement View pull request | 8.6.0 or higher |
2.18.0 | Enhancement View pull request | 8.6.0 or higher |
2.17.2 | Bug fix View pull request | 8.6.0 or higher |
2.17.1 | Enhancement View pull request | 8.6.0 or higher |
2.17.0 | Enhancement View pull request | 8.6.0 or higher |
2.16.2 | Bug fix View pull request | 8.5.0 or higher |
2.16.1 | Bug fix View pull request | 8.5.0 or higher |
2.16.0 | Enhancement View pull request | 8.5.0 or higher |
2.15.2 | Enhancement View pull request | 8.5.0 or higher |
2.15.1 | Enhancement View pull request | 8.5.0 or higher |
2.15.0 | Enhancement View pull request | 8.5.0 or higher |
2.14.0 | Enhancement View pull request | 8.3.0 or higher |
2.13.0 | Enhancement View pull request | 8.3.0 or higher |
2.12.1 | Bug fix View pull request | 7.17.6 or higher |
2.12.0 | Enhancement View pull request | 7.17.6 or higher |
2.11.12 | Bug fix View pull request | 7.17.6 or higher |
2.11.11 | Bug fix View pull request | 7.17.6 or higher |
2.11.10 | Enhancement View pull request | 7.17.6 or higher |
2.11.10-beta.6 | Enhancement View pull request | — |
2.11.10-beta.5 | Enhancement View pull request | — |
2.11.10-beta.4 | Enhancement View pull request | — |
2.11.10-beta.3 | Enhancement View pull request | — |
2.11.10-beta.2 | Enhancement View pull request | — |
2.11.10-beta.1 | Enhancement View pull request | — |
2.11.9 | Bug fix View pull request | 7.17.6 or higher |
2.11.8 | Enhancement View pull request | 7.17.6 or higher |
2.11.7 | Enhancement View pull request | 7.17.6 or higher |
2.11.6 | Enhancement View pull request | 7.17.6 or higher |
2.11.5 | Enhancement View pull request | 7.17.6 or higher |
2.11.4 | Enhancement View pull request | 7.17.6 or higher |
2.11.3 | Enhancement View pull request | 7.17.6 or higher |
2.11.2 | Enhancement View pull request | 7.17.6 or higher |
2.11.1 | Enhancement View pull request | 7.17.6 or higher |
2.11.0 | Enhancement View pull request | 7.17.6 or higher |
2.10.0 | Enhancement View pull request | 7.17.6 or higher |
2.9.0 | Enhancement View pull request | 7.17.6 or higher |
2.8.0 | Enhancement View pull request | 7.17.6 or higher |
2.7.0 | Enhancement View pull request | 7.17.6 or higher |
2.6.0 | Enhancement View pull request | 7.17.6 or higher |
2.5.0 | Enhancement View pull request | 7.17.6 or higher |
2.4.0 | Enhancement View pull request | 7.17.6 or higher |
2.3.0 | Enhancement View pull request | 7.17.6 or higher |
2.2.1 | Enhancement View pull request | 7.17.6 or higher |
2.2.0 | Enhancement View pull request | 7.17.6 or higher |
2.1.0 | Enhancement View pull request | 7.17.6 or higher |
2.0.0 | Breaking change View pull request This change involves Users with GCP integration enabled will need to input values for these
variables again when upgrading the policies to this version.
| 8.3.0 or higher |
1.10.0 | Enhancement View pull request | 7.17.0 or higher |
1.9.2 | Bug fix View pull request | 7.17.0 or higher |
1.9.1 | Enhancement View pull request | 7.17.0 or higher |
1.9.0 | Enhancement View pull request | 7.17.0 or higher |
1.8.0 | Enhancement View pull request | 7.17.0 or higher |
1.7.0 | Enhancement View pull request | — |
1.6.1 | Enhancement View pull request | 7.16.3 or higher |
1.6.0 | Enhancement View pull request | — |
1.5.1 | Enhancement View pull request | 7.16.3 or higher |
1.5.0 | Enhancement View pull request | 7.16.3 or higher |
1.4.2 | Bug fix View pull request | 7.16.3 or higher |
1.4.1 | Bug fix View pull request | 7.16.3 or higher |
1.4.0 | Enhancement View pull request | — |
1.3.1 | Bug fix View pull request | 7.15.0 or higher |
1.3.0 | Enhancement View pull request | 7.15.0 or higher |
1.2.2 | Bug fix View pull request | 7.15.0 or higher |
1.2.1 | Bug fix View pull request | — |
1.2.0 | Enhancement View pull request | 7.15.0 or higher |
1.1.2 | Enhancement View pull request | 7.15.0 or higher |
1.1.1 | Bug fix View pull request | — |
1.1.0 | Enhancement View pull request | 7.15.0 or higher |
1.0.0 | Enhancement View pull request | — |
0.3.3 | Enhancement View pull request | — |
0.3.2 | Enhancement View pull request | — |
0.3.1 | Enhancement View pull request | — |
0.3.0 | Enhancement View pull request | — |
0.2.0 | Enhancement View pull request | — |
0.1.0 | Enhancement View pull request | — |
0.0.2 | Enhancement View pull request | — |
0.0.1 | Enhancement View pull request | — |