Log names, buckets and scopes

Photo by Tim Mossholder on Unsplash

“So, what is in my name to you?” (A. Pushkin)

This post is a part of the How Cloud Logging works series.

There are many things about Cloud Logging that users aren’t aware of until they meet an unexpected behavior or need to implement a specific use case. This post about three such things: Log names, Log scopes and Log buckets.

Log buckets

Let’s start from the last one ‒ Log buckets. Simply put, all logs (actually they are log entries a.k.a. LogEntry objects) are stored in storage abstractions called buckets. These buckets are called Log buckets to distinguish from Storage buckets that often are references as “just” buckets. By default, each project (but also folder and organization) has two buckets: _Required and _Default. _Required bucket is used to store audit logs. You cannot write to this bucket or to modify it. So, all logs that are written by resources in the project and applications that are deployed into the project are written to the _Default bucket. However, you can modify this behavior by defining routing rules a.k.a. sinks. Unless you want to export ingested logs somewhere outside Cloud Logging (e.g. BigQuery) or to stream them to Pub/Sub, you have one out of two options:

1. Ingest logs to another Log bucket hosted in your project.
2. Route your logs to a different project where you can ingest them into _Default bucket of that project or to custom Log bucket.

Plain and simple. There is no hierarchy in Log buckets. All logs written to a bucket are searchable using Logs Explorer UI or gcloud CLI. They are uniquely identified using three fields of the log record: timestamp, logName and insertId. We will talk about logName later. Using different buckets you can control who can access logs and define where the log data is physically stored and also how long Cloud Logging keeps it.

DO NOT use Log buckets as long term storage for your logs. Export your logs to Storage bucket (as JSON strings) or to BigQuery (as structured records).

However, you might not see logs that you redirected to non-default buckets, especially if these buckets are hosted in a different project. When it happens you may need to fine tune the scope of your query.

Scopes and Views

Scopes define the visibility of logs for your query and are a part of the access control solution. In Logs Explorer you can refine a scope by selecting a “scope by project” or a “scope by view”:

For every log bucket, Cloud Logging automatically creates the _AllLogs view, which shows all logs stored in that bucket. Cloud Logging also creates a view for the _Default bucket called _Default. The _Default view shows all logs except Data Access audit logs. You can create additional views for a Log bucket if you need to narrow the scope of the visible logs in this bucket. You will need to grant users permissions to view logs in the custom views. The automatically created views do not require explicit permissions.

By the default, you query logs in the project scope. But how logs are ingested into a particular project in the first place?

Log Names

In Cloud Logging each log entry is uniquely identified by a triplet that includes a log name, a timestamp in RFC3339 UTC and an insert ID. The purpose of the last one is to handle use cases when more than one log entry share the same log name and timestamp. You can find the detailed description in the official documentation of the LogEntry structure that describes all fields of the log entry. It defines the log name (logName) as a string that follows one of the following patterns:

"projects/[PROJECT_ID]/logs/[LOG_ID]"
"organizations/[ORGANIZATION_ID]/logs/[LOG_ID]"
"billingAccounts/[BILLING_ACCOUNT_ID]/logs/[LOG_ID]"
"folders/[FOLDER_ID]/logs/[LOG_ID]"

For most applications only one pattern matters:

"projects/[PROJECT_ID]/logs/[LOG_ID]"

In this pattern the PROJECT_ID holds a unique project ID. And LOG_ID is a URL encoded string that uniquely identifies a log within this project. Currently LOG_ID must be less than 512 characters long. When the log name is malformed the Logging API will return a response with code 400 that looks something like the following:

{
  "error": {
    "code": 400,
    "message": "Name \"invalid_log_name\" is missing the parent component. Expected the form projects/[PROJECT_ID]/logs/[ID]",
    "status": "INVALID_ARGUMENT"
  }
}

Now, this pattern is used by Cloud Logging to determine to which project the log entry belongs when you run a query scoped by a project and which project quota will be used when Logging API entries.write is called. There are ways to call the write API in a way that the log name contains one project ID while the call consumes quota of another.

As I explained, when you write logs to Cloud Logging they will be ingested into the _Default Log bucket of the project which is identified using the logName field of the log entry. However, if this project has defined Sinks (a.k.a. routing rules) the log entry can be redirected to a different destination. One of them is another Log bucket possibly hosted in a different project. For example, you can define a sink in a project with the project ID ”experiment-project” that redirects all logs ingested with LOG_ID “take-me-away” to be stored in the default Log bucket of another project with the ID “another-place-for-logs”:

gcloud logging sinks create demo-sink \
 "logging.googleapis.com/projects/another-place-for-logs/locations/_Default" \
 --log-filter='LOG_ID("take-me-away")' \
 --project="experiment-project"

Now, if you try to write a log to the project “experiment-project” with this log ID:

gcloud logging write "take-me-away" "this log will be redirected" \
 --project="experiment-project"

you will be able to see the log entry both in the logs of the project “experiment-project” using the scope “Scope by Project”. And also in the logs of the project “another-place-for-logs”. BUT, if you query for it in Logs Explorer for that project, you will not see it. It is because the default scope is “Scope by Project”. Changing it to “Scope by storage” and selecting “_Default” Log bucket with the “_AllLogs” view will make the trick. Checking the field logName of this log entry you will see that its value is

projects/experiment-project/logs/take-me-away

Using REST API you can compose the log name explicitly. You can try this using GUI.

You also can use one of the Logging client libraries. Depending on the language you will have to define the log name directly (e.g. Go) or to provide a project ID and a log ID separately (e.g. Java).

When you ingest logs using logging agents (e.g. Ops agent) by writing your log entries to standard output, both the project and log IDs are determined by the agent.

Summary

You generally can control the destination (storage location) of you log entries at time you write your logs to Cloud Logging. However, you might not find your logs where you expected them to find. Do not worry. They are there. Just follow these simple steps:

• Check the scope and views of your query
• Verify that the project you expect should host your log entries does not have Log sinks that redirect your written logs elsewhere
• Ensure that when you write you logs, you indeed send them to that project by reviewing the code to determine which project id is used or, if you write to standard console, the logging agent documentation.